Building & Evaluating RAG Pipelines

Key Takeaways

Hello everyone and thank you for joining today's session. My name is Ree and I'll be your moderator today. We're going to kick off today's session in just a few minutes. We're just waiting so everyone has a chance to join. If you haven't already, make sure that you register for this session. You can do so by scanning the QR code that's on screen. You can also find a link to do so in the chat and the video description as well. Also pinned to the chat are the resources for the session. So, if you're looking for lots of content on rag, uh then please do check out the resources document. We also have the slides we're going to be covering today as well as a link to pre-order our guests book. So, please do check out the resources in uh linked in the chat and the video description. Also, uh we do have our uh flagship virtual conference coming up in two days on Thursday. So, if you haven't already, make sure that you register for radar. I will be putting a link in there very shortly. Actually, I'll put it in there right now for you. Uh but yeah, we've got speakers from uh yeah, many different uh companies. Pine Cone, Snowflake, uh Mastercard, uh Llama Index, H&M Group, the list goes on. So, please do check that out and sign up if you haven't done so already. If you have any questions throughout the session today, please let us know in the chat. We're going to be covering your questions for the last 10 minutes of the session. So, please do stick around for that. And yeah, I think that's everything from me. I'll repeat these messages one more time for any people that are joining just now. So, if you've just joined us, welcome uh and thank you for joining today's session. My name is Ree and I'll be your moderator today. Uh if you haven't registered already for this session, please do. You can do that uh using the QR code that's on screen now. You can also find the link that's pinned in the chat and also in the video description. Uh you'll also find a link to the resources for today's session. Uh that will include the slides we're going to be covering today, a link to pre-order the book of our guest Abby and also uh everything rag that we've got going on on data cam including courses, blogs and tutorials. So please do check out the resources document if you are interested. Also uh we put on these sessions for free. So please do uh help us out and uh reach more people like yourself. Subscribe to the channel and also give the video a like if you haven't done so already. will be taking attendance in the form of likes on this video. So yeah, good to see you all in here. Please do let us know where you're joining from in the chat. It's always good to know uh yeah, what a diverse audience we've got. So yeah, I think with that we I'll hand over to Richie and we'll get started. So yeah, Richie, please take it away. Hi there, data scamps and data champs. This is Richie. Uh I got to say I do appreciate you all being here. I know we had a few uh little uh changes of dates going on. So, I'm glad you all made it. Uh and please do let us know where you're joining from in the chat. Always nice to hear uh where in the world you're coming from. Now, uh today we're talking about uh retrieval augmented generation or rag. So, this just um is fast becoming a foundational technique for anytime you're working with AI. And one fact particularly uh in a lot of enterprise AI use cases the real challenge is not about like uh crazy uh difficult reasoning. It's really about looking up the right information and just uh getting uh that uh those details that about your business or about other um facts into the query so the the AI doesn't hallucinate. That means uh making use of rag is incredibly important uh in in a business context. So uh whether you are a data scientist or an AI engineer uh in order to uh make use of rag uh you need to understand uh things about the architecture you need to understand best practices about building uh building into AI applications and so uh that's what we're going to learn about today uh the core concepts van rags of what it is and what its role is in modern AI systems. Our guest today is Abby Arian. So, hi Abby. Uh Abby is the founder of the AI assisted content creation platform Abide AI. Uh she's got a decade of experience as a machine learning scientist and engineer across e-commerce, insurance, and media. And she's also the author of the forthcoming book LLMOPS about managing large language models in production. So, an incredibly important topic and uh yeah, I think you can pre-order the book already with the link in the chat. So, yeah. Yeah, welcome Abby. Please uh take it away. Tell us all about Rag. Thank you so much, Richie. Hello everybody. Um well, let me check if my slides are already available. Fantastic. They are on my screen now. Okay. So, welcome everybody to the talk. Uh today we're going to be talking about debugging and monitoring LMS in production. Um before I go into the talk just something I'll mention which is a you can keep asking questions throughout the talk as well. We'll take your questions towards the end of the talk though. Now let's go to today's session. Uh about me I think Richie has already talked about this so we don't need to go too much into the detail of who I am. Um but I'm one of you pretty much. Um the agenda for today's um talk uh we'll cover three things today. The first thing is basically why is it hard to be able to debug and why is it important to be able to monitor our LLM pipelines understanding what are the challenges over there what are the techniques that we can apply and I've seen too many people doing very random things and just jumping from one tool to another. So we'll talk about the systematic way to look at this thing and the third thing which is thinking about the tooling and the feedback loops and we'll discuss what are the tooling that I recommend. Obviously do not take my word for granted. Uh if you're already using something stick to that one but hopefully I'll just make some very generalistic recommendations in this part. Understanding the challenges. Um so the first part is really understanding why evaluating elements is hard. So you know there there's I think this is a topic we've all talked about a lot but it's essentially the scale of the problem is massive because again we're trained on massive amounts of data. The and even if you start evaluating there are far too many things to be able to evaluate which is the informativeness. You're looking for effectuality. You're looking for fluency and coherence. you're looking for safety and bias, grounding, efficiency and whatn not. So this is the first thing even if you figure out what do you want to evaluate the second part is really defining what makes something good or how can we determine if this is good enough for this particular set of problem or maybe not for this another application. So that becomes another problematic thing with evaluating large language models which is the task specific success that needs to be defined uh accuracy in factuality efficiency safety and fairness. Again how do you measure good across all of these things that's the hard part. So as of now there are couple of ways to be able to measure or evaluate large language models. You know you can look at the language features. You can look at the task dependence metrics specifically. You can define some task independent metrics as well which is the independent ones would be toxicity fairness bias. And I know you're probably thinking why don't we use some evaluation library like LLM harness um or you know the Helm one by Stanford and obviously those are very good collections but again I think it's very important to be able to understand what how many grants are you covering with those ones in terms of the library use either but always try to have couple of metrics across all of these evaluation criteria. ideas. So what are the ones that are important? I would say human evaluation hardest to do a few very few people are actually doing it very well. When it comes to the frameworks, that's probably the easiest place you want to start at. Then the yellow one is where you need to put a little bit effort, but again, it's very doable, which is defining the quantitative metrics, which is the blue score, the rogue scores, and the accuracy scores. The LLM specific metrics, which would be the glue, super glue, and helisphere metrics. And you know, defining the custom metrics, which could be for you uh could be let's say similarity score. So the mover score and stuff um or you could use the looking at the number of nwords as well which are being en grants that are being used as well. So those are some evaluation methods to be able to evaluate large language models. But with all of these ones there are some limitations. Human evaluation obviously is limited. It's expensive. It's time consuming. It's very subjective. the existing benchmarks that there are a lot of these benchmarks if you're looking at like let's say hell swag they haven't been updated in a while so most of the benchmarks are a one is they're suspectable to reverse engineering which I think has been talked about enough in the community with most of the models sort of reverse engineering um the sort of performance using the benchmarks and the second thing is basically not being updated in real time essentially and the Third thing is the LLM using large language models as an evaluator which I've seen plenty of people do but although it's powerful but again I think it's very biased opique as well as resource intensive. Most of the applications I've seen are using small and large language models to be able to evaluate the bigger large language models. Fantastic approach but explanability and interpretability are still hard. Now this is just about you know the large language models itself. But now let's go one step further. Let's talk about let's think about even if you manage to do all of this. The next part where it gets even more complicated is when we start thinking of pipelines really uh which is when we talk about observability in pipeline there are a lot of things that are unpredictable and not just the models itself although LLMs are basically unlike the traditional models which is their outputs are shaped not just by input data but also by prompts system messages retrieval context and even temperature settings. That entire thing makes observability a multi-layered challenge. You need tooling to be able to capture every piece of context influencing the modeling, not just the model itself, everything that's designed around it. Otherwise, when you're debugging a large language model pipeline, basically your debugging looks like looking for basically like a black cat in a dark room, which is something I've seen people doing, which is they isolate just the model and they forget about all of these things, which is the prompts, the system messages, the retrieval context and temperature settings, everything that just builds up the pipeline itself. Now the reason why it's harder is the first thing is the same model input can yield different outputs. Um so debugging requires full visibility into the prompt model behavior as well as the context. The second thing is the hallucinations, latency spikes and ungrounded outputs they can all happen. You can't fix what you can't see. So it's very important to have some sort of visibility into the system itself. The third thing is basically errors can impact trust, cause brand damage or even legal risk which is the leaking of the PIIs which you've seen with plenty of cases like Sony, Samsung and whatnot. The other part is basically like LLMs need specialized monitoring for prompts completions as well as reasoning change not just accuracy itself. So you need explanability, you need traceability and observability to be able to stay compliant and earn trust. Most of the people are doing only one of those things. And I think there are plenty of people who are just implementing large language model pipelines, even end to end pipelines without doing any of this part really. It's it's just you're building up the code and then building a rest API around it and maybe just add langu combined with it. But that's that's pretty much what I've seen most people in the industry do. Our discussion here is we need to think a little bit more than this which is we need to think about what are the failure modes that we can see in LLM pipelines. Not all the failure modes are made equal. So hallucination, you will have some sort of false claims or misinformation when you're talking about prompt regression. That's also a AI system failure that can happen. So what you need to do is basically categorize your failure modes into these four categories. There are more categories that you can create, but these are the four main categories across which you can detect what's essentially going wrong. Most of these failures aren't detectable with traditional ML metrics, for example, like accuracy or loss. They can't detect any of these things. A hallucination might still be grammatically correct and fluent, but still be factually wrong. Similarly, a slight increase in latency might be fine in dev, but it can cause timeouts in production. So, you need observability that is focused on output quality, user experience as well as system safety. Now, this diagram that you're looking at is the foundation of everything that we're going to cover today. Each box here, whether that's the prompt builder or whether that's rag based, represents a spot where something can go wrong or for us where something can essentially be improved. For example, poll retrieval leads to hallucination and broken prompt templates can lead to confusing responses. Our observability strategy needs to watch over every link in this chain that you're looking at which is across the user interface across the orchestration layer the pre-processing layer across the retriever across the LLM inference post-processing as well as the user facing or the feedback capture. There are different kinds of risk that again these things can create you'll have access to these slides later to be able to read most of this stuff. Now let's talk about monitoring and debugging techniques. What how can we solve some of these challenges essentially? So think of monitoring as happening on three levels. What goes in basically the prompts, what goes inside the box which is the inference and what comes out which is essentially the responses. If you watch only one of these then you'll miss out the full story. The best setups usually monitor each layer separately and correlate them through logging and tracing. So what you're looking for is basically drift monitoring and business KPI monitoring. Most of the people are doing only one of those things which is either they're doing business KPI monitoring which is looking at conversions engagement time to resolution for enterprise apps or looking at drift monitoring which is the changes in the query type or the prompt or shifts in the retrieval quality for like rag applications specifically. How can you how can you actually solve it? Um, one of the ways to be able to solve this problem is using structured logging and tracing for large language models. Without structured logging, debugging large language models is extremely extremely hard. You need full visibility into what prompt was used, what retrieval context was injected, what model was called, and how it exactly responded. Then tracing adds another layer which helps you be able to spot the bottlenecks in these complex chains essentially. Um so for it can be very different based on like your implementation strategy can vary based on if it's an isolated event versus it's a full session. For different kinds of complexity you would have different types of logging as well. Um so for an isolated event you would do some sort of structured logging where you're logging in the individual request in key value format itself then now if you have an application and if you're thinking about okay now I've already implemented something for just like individual uh events itself now you're thinking about how can I make sure that this stays consistent across the entire user session. So what you're thinking about here is basically how do you standardize uh the log schemas across devices. How do you include the version info which is the model prompt retriever as well as the chain. How do you sanitize logs for PII before storing? Um these are these are some things that obviously you need to think about. So three things that you need to do u depending on if you're implementing for an isolated event or full session. The first thing is implement some sort of structured logging which is use JSON or key value formats. You need to log a couple of things. You need to log the prompt. You need to log the system message, the model parameters, user input, response, latency, and the token count. Now, there are plenty of tools that allow you to do this. I'm not going to mention a name. I think you guys can drop a name in the chat itself. I'm pretty sure most of you are using one or another tool to be able to do this. U most of the monitoring tools um that do exist in the market allow you to be able to do this. So there's no custom writing any of these things. What you can do is now the next step from this point is to be able to enable the query across all of these logs which is now show me all completions with more than thousand tokens or with greater than or less than 60% helpfulness let's say that's how you can do structured logging tracing the second part uh when you're thinking about tracing what you want to do is to be able to visualize the the execution across components so these components is what we're looking at. So from the retriever to the prompt builder to the large language model to the parser. So you want to be able to see the entire pipeline end to end. Then implement distributed tracing that helps you pinpoint where the latency or errors originate. This is ideal not for every single type of large language model application and this is the reason I'm basically mentioning this separately which is structured logging works for almost all LLM application tracing is ideal if you're using chains or if you're using rag pipelines essentially then the next step is the third thing that you can also do is this session context logging which is log full user session including the history the feedback and the reruns And then you can group the logs by session ID or by the conversation ID itself. That's something which most of the companies are already doing right now. I don't know if you guys have implemented something like this. But this is a very simplistic uh implementation. Amazon uh AWS allows you to be able to do it as well within your cloud services whichever one you're using. Now the next part is um once you've done this three-part setup depending on your application itself whether you did the structured logging or whether you did the system whether you implemented tracing and the next part as well which is whether you did session context logging. The next part is now how to do anomaly detection in production. Now this is the hard thing. There are couple of ways to do this. The first method is to set up some threshold based alerts. The second method is to implement some sort of statistical anomaly detection. The third method is to look for drift detection. And the fourth one is feedback signal analysis. You can use one or combination or all of these techniques as well to be able to now do the anomaly detection. And the reason for this is large language models don't throw exceptions when they hallucinate. Most of the times you would see them failing silently which is the reason why anomaly detection is important. Think of it like a smoke detector. You want catch every fire early but you want to be able to catch the most before it spreads. Right? So when it comes to the tools, the usual tools will allow you to be able to do this which is the Promethus and Grafana which allow you to do be able to collect the metrics and show real-time dashboards or use ML floor use lang. Here on your screen you're seeing an example of um a visualization in graphana and the way that they show the metrics. So let's say we're setting up threshold based alerts. Now what to set? Ideally what you want to set here is some sort of hard threshold on the runtime metric. So for example the if my response time is greater than 2 seconds then alert ops. U if my token count is greater than 12 1,024 then you want to be able to truncate or reject the response. Um now what tool can you do use just to do this threshold based alert setup? You can simply use permitis and graphana for this which will allow you to be able to auto notify the slack teams and you know also create a log incident in the issue tracker itself. The other method which we discussed earlier was the statistical anomaly detection. Here we can use any of the metrics the conventional quantitative metrics which is like you could let's say look at the average latency you could look at the mean standard deviation then zcore you can use rolling stats across all of the metrics essentially so for example whenever there's a spike in the response time zcore which is greater than three or you know whenever we have an output average length which is suddenly dropping Then you want to be able to slack or auto notify within Slack or Teams whatever you're using and highlight the impacted trace cluster in Langmith. Graphana allows you to be able to do most of this which is you can set up Graphana with any sort of alert manager and tie up your APIs very quickly within your Slack with in your Lansmith as well. Um now the next one drift detection. So how do we do drift detection or how how how do we think about the implementation here? What you're looking at is basically thinking about the embedding histograms. So one of the ways I've personally implemented this is using fice and lchain hooks. what you want to do is to trigger retriever uh retriever refreshes and retraining jobs every single time something goes wrong. So for example, if you have an input drift uh which is you know the the queries are starting to get low quality or if you have retrieval drift let's say which is the embedding similarity now that drops then you want to be able to start a retraining job to make sure that now this doesn't get included or you want to be able to trigger the refresh essentially. So that's that's how you do um a drift detection essentially. Now another one is the feedback signal analysis. The feedback is the customer feedback essentially which is whenever there's a sudden drop in thumbs up or thumbs down ratios. Um can again use ML flow for this thing or can use lang if you're already using lang for this. What are the signals that you're looking for here? um sudden drop in thumbs down, thumbs up ratio or you know a sudden increase in the thumbs down ratios or search back in um or any any surge in fall back completions which is you know your model has started to create very generic or default responses that's that's when you know or that's something that you want to monitor for. The action here should be to tag the model version to link to the degraded outputs and correlate with the deployment logs and create an error message itself. That's how you're thinking about feedback signal analysis. So as of now we've looked at four different methods to do anomaly detection. I think it's pretty obvious now as to why I was saying which is think about large language models not just from like the conventional ML perspective but from more like fraud detection kind of application because these applications especially in production look very similar. Your operational workflow here should be something like if you're using Prometheus then collect the runtime as well as the system metrics. If you're using graphana, most of the times you would be using a combination. If you're using graphana, think about setting up visual as well as threshold alerts. The threshold alerts that we saw here in this part which is on the response time as well as the token. Um then on MLflow which allows you to be able to track the version impact on metrics. Uh this is the part where we discussed this essentially uh which is the model input dist distribution anytime we're detecting the drift essentially and lang langsmith allows you to be able to do trace level flagging. So all of this most of these methods would allow you to do some sort of combination which is the structured logging as well as trace level flagging. LSmith is one of those tools that allows you to be able to do a some trace level flagging automatically, but you can also set up your own custom flagging over there and to be able to debug a lot of these issues that we talked about. Now let's go to the next step which is now you understand your failure points, you understand how to be able to capture those failure points. The next natural thing is how to think about tooling and feedback loops. The first thing is thinking about metrics that matter. Uh there are four essential metrics that always matter with large language model applications in pipelines. Um you're looking at performance, you're looking at drift hallucination and if the if we are still following some sort of ethical and safety metrics. So any sort of library or any framework that you're using for doing evaluations, ideally you want to have a combination of some metrics along all of these four categories. So define your performance metrics. It could be uh the latency, it could be the token usage, thut it could be the timeouts. um define your custom metrics when it comes to drift which is I think we already discussed a lot of ways of thinking about drift which you can again set up in promise twist as well as graphana or even in langmmith um that's possible hallucination and output quality here again we can use lang to be able to do this which is um to be able to c allow tracing whenever something goes wrong and then to be able to capture the exact model and the exact output where things started to go wrong which is here you're thinking the factual accuracy the consistency across reruns the response structure correctness and the ethical and safety metrics that we're looking for are the toxicity levels bice detection PII leakage detection and safety classification scores this is something where we need to define the SLAs and SLI based on these metrics um you can use again the automated eval frameworks like lang or you can define your own custom pipelines as well. I don't really have a preference here. That's totally up to you. Um ideally at every point in time you want to be able to store all of these metrics with the version context which is it should always include your log should always include the model the prompt and the retriever version if you're using a rack application. Not all the met metrics matter equally. Focus on metrics that reflect user experience and downstream business impact. For rag application, hallucination often goes back to the retrieval quality, not just the model behavior itself. For chat bots, let's say the latency and toxicity detection might be your top concern. So there's no oneall solution at any point in time for different large language model applications. Now basically different I think this comes down to different LLM architectures requiring different observability strategies which is the reason it's very hard to be able to show one demo and say like okay this is how you implement this for every single application. No it doesn't work that way. For example, a chatbot would need real-time sentiment in user feedback hoops whereas a rag would need retrieval quality application. The enterprise apps they would need rock solid audit logs and compliance. So knowing your architecture and tuning your observability accordingly is super important. When it comes to rag applications itself, there are couple of things that you want to think about which is you're looking at the retrieval relevance. Again, depending on if this is your only application, if this is the only application, I would say implement everything end to end which is document the logs, do drift detection, do failure tracing, do retrieval relevance as well at every point in time. Um the tools for doing this is basically using lang and vector store logs which most of the times I personally use fice for this. Fice integrates really well with lang to be able to store the custom embeddings and draw the histograms that we were looking at in one of the earlier slides. Um and then again if you want to set up some sort of alert management on top of this then you can use open telemetry or anything similar uh to be able to set up your alerts here. When it comes to chatbots you're doing session based tracing which is tracing the user flow. Again sessions don't make much of a difference for every single application for rag application as well as for chatbots. The entire session is what you need to have in perspective at all times. It's not just a one-off response. So you always want to look for the you always want to track the user flow. Look for turnbyturn behavior. What are the escalation metrics you you're looking at which is how often do the users retry or escalate to human agents. One really good example of this being misimplemented in a couple of companies is what they're saying. they don't allow you an option to be able to um now escalate the problem to a human and they just only have one option on your screen which is basically close the ticket now my thing was resolved so ideally I hope most people don't implement it that way but u a good way to look at this is basically looking at um how many times was something escalated to human agents if it says zero then you really have a problem something was is implemented in your system. Um, so a lot of telecommunication companies to be able to keep their scores or to obviously be able to advertise themselves better and be like, okay, we have zero queries on the hold out. No, that's not really true. You just don't like allow people to be able to escalate them. Um, toxicity and safety filters at response as well as the prompt level. and for feedbackdriven implement improvement loops. So what you're looking at is basically doing implementing languid then some structured logging and some feedback dashboards. Feedback dashboards is the thumbs up and thumbs down thing that we were talking about earlier. So you need a combination of four these all of these four things and all these four things always do flow into each other. Now depending on let's say you have an enterprise AI application you want to be able to define the hard targets this is something that we saw in the earlier slides which is the response time under 1 second or 2 second 99% of time so first part is really defining your SLAs and SLI uh the data governance logging ethical compliance checks and auditability again the requirements these may be very extensive for a consumer application or for a basic rack application that is being used inhouse itself. Um so think very carefully on what architecture you're following and what observability strategies do you need for something like that. And the tools here are um Promethus and Graphfana do allow you to be able to do some sort of visual uh evaluations um as well as to set up the alerts and then to automatically send them those alerts into your Slack system, implement MLflow or ZenML or whichever model tracing tool data as well as model tracing tools that you want to be able to implement and some internal logging frameworks especially for like complian checks itself. Most of the large enterprise companies do already have their internal locking frameworks. Some in general best practices. Um, so I know although it may seem like I'm trying to probably suggest everybody to start using Langmith, totally not the intention here, but it's a tool that does a lot of things really well. um especially like most of the dashboarding tools that I've seen. Um so there's weave by weights and biases. Weave does um traceability really well but the structured logging a lot of other logging systems it fails. It doesn't really integrate with a lot of other systems really well. So for that reason I always I've personally use LSmith primarily for that reason. um or so but you're feel free to use any other to be able to trace the full LLM pipeline execution. Most of the tools especially like some of the earlier tools only are able to trace like the large language model performance alone and not not really like a lot of stuff on the side that we were talking about the system prompts the context side of things. Always store the logs with your metadata which is the model version, the prompt version and retriever version. If you're using ML flow then most of the times or even with most of the observability tools these are automatically do get locked. Include the user feedback hoop hooks for all user facing LLM systems tag and version all experiments. Again, this is one of those basic advice that everybody has been giving since 2018 and things are still broken. Um, for MLOps, everybody was saying this, which is tag and version all your experiments. Every single time you're changing up the models, you're changing up the prompts, you're changing up anything in the context logic. Um, but again, the same advice stays in 2025 as well, eight years later. Finally, the feedback loops that we were talking about earlier. Again the there are three types of feedback loops. Uh the explicit feedback, the implicit feedback and the system feedback. Most of the people are tracking only one kind of feedback which is the let's say the user thumbs up or the star rating or the comments whichever that you're looking at. Um but you also want to be able to look for the implicit feedback as well. And this is where observability gets interesting. If you're looking at really building a strong oper, look at the retries, the reformulations, and the click-through rates as well. Track where users abandon or escalate the task. And the system feedback is again the eval scores that the quantitative scores that you're essentially looking at. And the logs should be flagged for anomalies as well as for hallucination. And before we go here, make sure that the PII is always handled. There are a couple of tools that allow you to be able to handle it right before it even reaches your model. So, for example, Arthur.ai has a fantastic system. Guardra.ai. Again, fantastic to be able to ensure your PI is handled before those logs get stored for training because once they do, then it's very hard to be able to deal with that. How do you retrain the model? Uh once you have all of this so reward the consistency as well as fuality and updates aggregate at you session for every single user level for context at every single time. Um trace the feedback by model as well as the prompt version itself because now all of this has to go back into the model itself. So I hope that answers everything. Um so that's that's my talk here. Um, these are the things I wanted. All right. Uh, that was very cool stuff. Uh, I learned a lot there. Um, brilliant. So, uh, we've got plenty of time for questions from the audience. So, uh, yeah, for anyone who has any questions for Abby, please do ask them. In the meantime, I've got a few questions for you. So, um, I guess to begin with, just I'm always curious on uh for like, uh, good use cases, good ideas to get started with. So, do you have any ideas for just sort of simple things you can put in production? Just practice monitoring like what's a what's a really simple example? Um, simple just sorry go just simple things to build and then monitor like what's a good first project to try out a good first project is usually some sort of like rag application itself. Um, so one of the things that I built personally was to give it a lot of my Google Docs for my book because I'm writing a book. So I saved a lot of papers that I was reading constantly and then to be able to query things very quickly out of those ones and although that's something where you can set up your observability but observability al always makes sense when you have a dynamic system once you take it to the next step and once you create an API around it that's where observability start to make sense. So if you're looking to build something which is end to end pipeline, implement a simple chatbot application, ask your 10 friends to use it and that's where you can implement most of this stuff and start implementing uh integrating language. Okay. Yeah. So certainly once you got changing data or you've got um other users then that's when chaos starts to happen and that's when the observability becomes much more important. Okay. Um, so I guess uh the biggest sort of mistake I think a lot of people make is like okay I've built something now going to put it in production and it's like oh maybe we should start thinking about observability now what are we going to track and that's obviously like a really terrible process because once things are going live it's too late. So can you talk me through like what's a sensible development process like where do you start building in like uh deciding on observability um parameters and things like that? So a good step and one of the reasons people say once it's all once it's deployed it's already too late but while you're doing training itself ideally you're looking at the metrics that we discussed in the first part of the talk which are the blue scores the rogue scores and all of that stuff that part has to be done be during the training part of the process itself. Once that part is implemented, then once the deployment kicks in, that's where you're starting to think about observability really and that's where you start implementing your lang or graphana or prometh with whichever systems that you already rely on. Okay. So it seems like it's pretty early on then that you need to start worrying about all this stuff. Um yeah. Okay. And so you you mentioned in the first run itself which is when you're doing the training start implementing at least the basic metrics for large language model evaluation observability. So I think here's here's the difference. There are two words observability and evaluation. Evaluation is limited to large language models. Observability always comes in when we are talking about a pipeline entire pipeline. when you're implementing or deploying a model itself, you still need to do evaluations. So implement evaluation techniques and evaluation metrics over there and the next step is when you go into the ops once you've already done the deployment. Okay. So um evaluating the large language model outputs that's going to happen pretty soon and then the next thing once you start building up like a whole system a whole pipeline then you need to worry about the broader observed earth problems. All right. Uh so I liked your framework with the four different categories. So it was what uh performance drift hallucinations ethics was it if I remember right so you got these four things talk me through like um are there any situations where some of these are more important than others like when do you care about uh particular metrics um so performance metrics makes perfect sense when you're using building up an application like child GPT so for example like um okay my the speed of the responses makes tons of difference or you know I'm tied in terms of the cost itself. So when you have massive amount of users and you know that your application is already being used in multiple multiple location you really care about the performance metrics and the ethical safety metrics again are super important. I don't want to say that those are not important but they can take a they can be depprioritized sometimes over the performance metrics. But when it comes to healthcare applications or the financial applications, that's where you're really thinking about the ethical and safety metrics first. Um when we're looking at uh for example legal application, then the hallucination and the output quality matters way more than the performance metrics essentially. Or anytime we're building something which is like a rag application the hallucination and the output quality would make way more sense than let's say the ethical and the safety metrics because most of the rag applications are built to be used inhouse most of the times or the data and query drift those again make sense when you have an application running for like massive points in time and when you know it's being used for like multiple in multiple domains. So for example, people with charge GPT they are trying to ask multiple types of questions. Now this isn't really the case with very focused uh like let's say Chevronly deployed their system. People are not going to come ask about cars. People are only going to come ask about cars and not something else. So that's where basically the data and drift uh query drift doesn't m make too much of difference if that makes sense. Now okay that does make a lot of sense. It seems like some of these are then in opposition. So, I was thinking about uh like whenever you do a Google search, I mean quite often it'll put some sort of AI summary there and because it's search, you're trying to get your answers as soon as possible. It's got to generate something really quickly and so they use whatever like a cheap lightweight model that sometimes makes stuff up. So, there's a trade-off then between hallucinations and ethics versus latency. So, they care about the performance metrics. Most of the big browser applications and most of the user applications in the space which is like let's say the top 10 companies meta if they build up an application they would care about the performance metrics more than the ethics for and I mean if it's ex throwing a few companies under the bus they don't care about ethics at all hopefully there's like some like responsible AI team somewhere in these companies that where someone does care and I I really hope So um okay uh so uh the other thing is you mentioned a lot of technologies today uh lang came up a few times but there were quite a few different things. So which are the most important uh tools do you think uh people should be learning about if you want to be an AI engineer and uh is there a particular order to learn them in? So I think the most important thing especially now um and I must say this because we're living in a bubble right now where a lot of companies are getting funded the valuations are overhyped and we'll see a lot of companies failing as well which is they're not not being able to make revenues um so stick with the tools that are built by wellunded companies and that's the reason I mentioned Langsmith because they have like a really good history of being funded really Well, um, Llama index happens to be one of those ones that happens to be well funded. They again I think are their biggest customer is MongoDB which brings them a lot of customers as well. Um, so that's that's the reason I mentioned Langsmith is the one which is a little bit more reliable as compared to others because I don't know if other tools will still be here two years from now. Um but when you're thinking about [Music] implementations, think really about what metrics are you looking for? What are you thinking about with the fundamentals really matter more than the tool? If you have the fundamentals right, then you can use a rule-based system and do your own evaluations as well and do a lot of these visualizations manually. You don't really need a tool. um you can build your own dashboarding solution and that's essentially what we did at one of the companies I was working with. Okay. Uh so yeah uh you mentioned Lindex and we talked about lang to like the lang chain ecosystem. I guess uh I was I just feel like um hastack from deep set that's like kind of the third framework. So I guess learn one of those three and probably you're going to be okay picking at least one of those ecosystems. Um okay nice. Uh we're going to go to some audience questions in a moment. Uh before that I just want to say for those of you who have not yet registered for the radar virtual conference on Thursday uh please do so. Uh I think uh re will pop another link in the in the chat shortly but yeah uh there we go. Uh please do come for please do come for that. It's going to be a very exciting event. Uh all right so uh we got some questions. Uh oh both from Krishna. Krishna is doing all the questions. So Krishna asks are human evaluations still uh considered the gold standard for evaluating these things or uh even though it's timeconuming. So yeah uh you can have like LLMs acting as a judge for other LLM output but humans are sometimes still used when do you choose one or the other? Uh are they still considered gold standard? I think um it's going down a little bit as the evaluation frameworks and libraries are coming up which they are able to do a lot of stuff manually but you know they're they've not really reached the human level yet. So it's going to take a bit of time, probably a year or two, and we'll still have some cases, especially like with anomaly detection, you still need humans to look at if that was essentially an anomaly or whether that was flagged incorrectly and what was the reason for that flagging by the model itself. So that's that's where we are moving in the future, which is although humans will still evaluate and they will still be considered the gold standard, but it will not be human evaluations the way that they've been happening for the last two years, which is humans looking at every single input output. Humans will just be looking at the alerts flagged by the system itself and we'll have really good anomaly detection systems in place. Um, yeah. So, I guess the ideal situation is you want as much as possible to be done by the ALM because that scales cheaply, but sometimes it just doesn't work. Oh, actually, you know what? I was reading recently about uh the first is the first benchmark for deep research tools and the way they built it. They start off just having an LLM judging everything and it was too sick of it was like it was like all the stuff that was created was getting high marks. So they had to have humans write reports and then get the LM to compare to the human written report. So you got the automous judge but with comparing against human data as a gold standard. So yeah humans are still needed at least for the time being. That's one of the that's one of the big mistakes that people make which is usually use one of those techniques. Um so just use if you're using any one technique to be able to do evaluations on the model itself you're always going to fail. So have a combination of some engram metrics that you're always evaluating some sort of similarity based metrics. So whether that's your mover score, your similarity score and then using a large language model. Usually one of the three systems would actually always flag it. Okay. Yeah. So really just like try multiple systems because they're going to be perform in different ways and that's probably going to get you the best. I mean have three different pipelines. It's not very resource consuming. Okay. Um that that's uh good advice. All right. Uh we got one more question from Krishna. So Krishna says um I've been playing around with MLflow regarding model versioning and logging. Is that good for beginners? So I mean uh yeah talk me through uh what what are the choices MLflow or what? Yeah. I mean that is more like when you're doing the training itself. So fantastic for beginners especially in the training stage but not for the observability not for the end to end pipeline. So I don't think you can even you should never deploy model without implementing MLflow. So I agree with that part which is yes you have to play around with MLflow a little bit for that reason but don't just end there. uh if you've deployed a production level application then implement some at least one uh tracing tool over there and you can choose your tracing tool. Okay. Uh all right. So that seems pretty useful. So when you want to check like the accuracy of um models and if you're changing having different versions of models you need to keep checking over and over again. MLflow simplifies that. Uh you mentioned like um the you want some some tracing tools. Just uh remind everyone what are your favorite tracing tools then? So I guess uh Honeyhive is a really good one. Um Viv does really well for code checks whenever you have you're building an application like cursor. Uh they do really good job every single time your code fails or even if you're building chains with like uh agent workflows we works really well over there and otherwise just use lang is fantastic for like rag applications as well as chatbot applications. Okay. uh a good uh choice of options there then it seems like uh this space is is pretty well covered. Uh all right I've I've got one more question for you. So you talked a bit about one more I'll mention uh if you're using large language model based evaluation which is you're using one large language model to evaluate another there's a tool called log 10.io that does that fantastically well. Okay. Uh cool. Uh all right so last question from me. Um you mentioned anomaly detection and I quite like this because it's like a sort of a favorite tool of data scientists. So it's good to see this is also being used by uh AI engineers. What do you need to know about uh anomaly detection uh in order to u