OpenLLMetry is all you need

Key Takeaways

hey everyone I'm near the CEO of Trace Loop and today I'm going to talk to you a bit about open elemetry which is a nice open source project we' built you can probably hear from the name open elemetry that it originates in another project called open Telemetry and in case you're not familiar with open Telemetry I'm going to spend the next couple of minutes explaining to you what open Telemetry is so you know we're not talking about gen yet we're not talking about llms just plain simple cloud observability open Telemetry is an open source project it's maintained by the cncf it's one of the largest projects out there after kubernetes uh that is maintained by the cncf and it standardizes a way to do cloud observability in your you know Cloud environment currently it's supported by every major avability platform from Spang data dog din Trace new grafana honeycom and many many others so if you're using open Telemetry you can use it in conjunction with any of these platforms pretty easily but what is open Telemetry exactly open Telemetry is a protocol first and foremost that H standardizes the way to do logging metrics and traces H in your uh Cloud application logging I think I don't need to explain to you what is what is logging exactly H because if you've ever written some python you know script and you you've written print then then you've done some logging so logging is an arbitrary event you can send uh anytime you want H in the life cycle lifetime of your application and it just emitted as is and you can view it later on possibly with some metata metrics on the other hand is completely different metric is something you want to see on an aggregate level you want to see how it behaves across days or across users or you know whatever you want and when we're thinking about metrics in the traditional Cloud world we are probably talking about CPU us usage a memory usage or um latency and if you want to you know talk a bit about gen if you're thinking about which metrics we want to see when we are building a gen based applications it's probably things like token usage latency error rate and so on lastly open Tel defines tracing actually this was the first thing that was defined uh with open Telemetry but H it's the I would say the least trivial one so tracing is basically tracking of a multi-step process so again thinking about you know Cloud environments you have microservices they're talking to each other and you want to see some process that spans across multiple microservices you want to use a trace for that and then you can see you know how a certain request coming from the user is processed across these uh microservices H specifically for Gen I think tracing is actually pretty common because we have we are using you know a lot of multi mstep processes whether it's chains or workflows or even you know agents are multi-step processes that me that interact and run tools so yeah logging metrics and traces this is what open Telemetry as a protocol defined but it doesn't stop there because you know what can you do with a protocol so open Telemetry is also an ecosystem and it contains h sdks instrumentations lectors sdks are the way you can you know manually send out these logs metrics and traces from your application it open Telemetry currently has 11 different languages uh supported in 11 different languages of sdks from python typescript go C++ and many others you know every language you're probably using has an open Telemetry SDK instrumentations are a way to do this automatically so remember if you're using an SDK and you you want to send out logs metric and traces you need to do it manually you need to actually send out a log or a Met a trace or a span or a metric but instrumentations can do it automatically and we going we're going to talk about it in a bit and lastly collectors allow you to do some processing to your observability data before you send it out to whatever a observability platform you're using so what are instrumentations instrumentations are a way as I said to automatically get some observability data some visibility into some part of your application let's say you have you're using a an SQL Server then you can use an instrumentation for that SQL Server Like postgress and get some uh logs metrics and traces automatically uh the way these instrumentations work is that they monkey patch the client library that you're using within your application and then emit all the data that is that that you want to probably want to see in your uh in your observability platform everything happens on the application side and they're designed H magnificant you know engineering level so that the latency impact is almost negligible and you get a you know a nice view of everything that's happening in your system without doing anything collectors are a self- Deployable components you can deploy in your own you know Cloud environment on kues or whatever you want and can provide you some pre-processing before you send data out to whatever you know platform you're using for example if you want to filter out some data that is not important for you or you want to obscure pii or obscure sensitive data and hide it you can use the collector to do it and and those you know ready made components that you can just deploy and they're completely open source have a lot of these uh features just built in and lastly if you also want to send this out send the you know the observability data out to multiple providers you can also do it with the collector I think at this point you're probably asking me hey near you you talked a lot about open Tel but we are in a gen conference what what does that has to do with with Gen this is where open elemetry comes in we took this Amazing Project called open Telemetry and extended it to support a lot of you know gen uh Frameworks a foundation models and Vector databases that some of you know you can see some of the logos that of the instrumentations we built here so we extendedly to support all of these you know amazing products and because we relying on open Telemetry you can then get observability in whatever platform you're using you want you want to see you know traces within data dog just use open Elementary you want to see them in Sentry you want to see them in grafana Tempo or in Dino Trace just you know just use open Elementry configure it correctly and that's it you get logs metrics and traces automatically in your favorite platform it's kind of nice um we've worked we've worked a lot on on on building instrumentations with our community and now we have more than 40 different providers so we're talking about Foundation models like open AI entropic coher Gemini bedrock and many others H we're also talking about Vex databases like Pine con chroma and many others and we're also we also have a support for Frameworks like L chain Lama index crew Ai and hstack so you have you know the instrumentations that automatically emit logs metrix and traces and then just connect it to whatever platform you want and you get it out of the box and remember because these are instrumentations it's it's kind of done automatically so uh it's it's pretty cool it's like a magic just just to give you an example what what is what an instrumentation for let's say pinec con would look like so uh if the the pine con instrumentation will contain you know ways to see the queries going out to Pine con see indexing happening within Pine con and also in and also ability to investigate vectors return from Pine con so you want to see you know the data the distances the vector distances that Pine returned or H scores latencies all of these are available in the standard open Elementary format so open El it's it's a great you know way to to connect uh llm based applications to whatever platform you're currently using and because it's a standard protocol you you're never tied to a specific platform you can easily switch between platforms is just a matter of you know a configuration change H because all of the platforms that support open Telemetry H supported with the same with the exact same format H that's it if you have any questions uh I'm available I will be in the conference I'm available in the conference or otherwise at near at tr.com thank you very much