LLM in a Loop: Automate feedback with evals

Key Takeaways

Hey everyone, I'm Shaw. This is the fourth video in a larger series on AI agents. Here I'm going to talk about running LLMs in a loop. I'll discuss why we might want to do this and then walk through a concrete example with Python code. So if you've ever used an LLM system like ChatBT or Claude, then you've probably experienced the power of trial and error. For example, if I wanted ChatGBT to rewrite my resume for me, I might simply just get my resume, write this prompt, can you rewrite my resume? And then pass it into Chat GBT. But usually what happens is that the first rewrite that Chat GBT does isn't always so good. It might be a little bit better, but it's probably got something wrong with it. For example, like maybe it doesn't use very strong action verbs for each of the bullet points under my work experience. So, what I might do is just respond back to Chad GBT and say, "Can you use stronger action verbs for each bullet point and then it'll return a slightly better version of the resume." Still, this isn't like 100% there. Maybe it's a little bit too long. So, I'll say something like, "Please compress to one page. Limit bullet points to two to three max." So, I pass this back to Chad GBT. And then it comes back with an even better version of the resume. And then this might be pretty close to the final output I'm looking for, but maybe I just want it in markdown. So I'll just say, "Hey, can you format this in markdown?" Then we'll have the final version of the resume that can help get me hired. So this simple example is representative of a larger pattern that comes up when working with LLMs. Namely, a lot of times it doesn't get it perfectly right on the first try. It takes some trial and error and refinement to get the LLM's response to be exactly what you're looking for. This basic idea of improving an LLM's response through this iterative back and forth process is the key motivation for running an LLM in a loop. But rather than having a human in the loop who is manually interacting with the model, the idea of LLM in a loop, as I'll discuss it here, is automatically providing the feedback to an LLM to improve its outputs. And so there are countless ways that this can look in practice, but just a few examples are maybe you have a coding agent that is tasked with completing GitHub issues. So maybe the way this will look is we'll have the agent write some code. Then we'll see if the code produced by the agent passes unit tests and actually runs. And if the code doesn't run or it fails some of the unit tests, we can gather that feedback and feed it back to the model and have it try again. And then we can just continue doing this process until the code runs successfully and all the unit tests pass. Or maybe we have an argument writer. So, we have an LLM that is making some case in a debate and then we have a different LLM which is taking the opposite position that responds to the original one and then they just go back and forth until one of them wins the argument. So, this could be a way of generating compelling arguments. Another version of this is maybe you want a LLM to come up with video ideas for you, but a lot of times the video ideas that directly come from an LLM aren't so good. So maybe you'll have an LLM judge that is aligned on user preferences. So let's say this idea generator comes up with five different ideas. And then this judge LLM picks out the best three ideas and passes that back to the generator who generates a few more ideas and that gets passed to the judge again who again picks the top three. And then this gets passed back to the idea generator. And this loop continues until you have a refined list of ideas. Or maybe you have a LLM that can write X posts or Twitter posts for you. So basically, you'll have it write a post and then you'll pass it feedback of different engagement metrics, maybe the like to impression ratio or the comment to impression ratio, some real world signal that gives the LLM feedback on its responses. And so across all these cases, just like how giving Chad GPT feedback on rewriting my resume led to improved responses, giving an LLM feedback across all these different use cases will lead to better and better outputs. This is important for building AI agents because this allows them to one do more and more sophisticated tasks without a human in the loop and two it allows the agent to get feedback from its environment which is a central feature of agency. A critical ingredient of getting an LLM in a loop to work well are the evaluations that we bake into the system. Here I'm going to focus on three types of evals we can use when generating this automated feedback in an LLM loop. So the first type are rulebased ones, the second are LLM based ones and the third are real world data. A critical point in picking your evaluations for an LLM loop is ensuring that the eval are highly correlated with your desired outcome. While this is still a bit more art than science, in the upcoming slides, I'll give a handful of examples to hopefully give you a better idea of what might be the best choice for your specific use case. So starting with type one, rule-based evals are tests and metrics that can be built using simple code. This includes binary tests. For example, if the code generated by your coding agent raised an error, that can be a pass fail metric. Or maybe you can check the formatting of the LLM's output using some kind of linting tool to ensure that it follows a desired format. You can check if a particular string is present in the model's output. You can check if the correct answer exists in the model's output and countless others. Another flavor of rule-based evals are continuous or at least non-binary metrics. So this can be things like the length of the output, the faithfulness. So this could be for like a summarization task. You can see how faithful the model's output is to the source documents. You can use traditional NLP metrics like the rogue score or blue score to compute the similarity between the output and a reference text. You can compute things like a readability score to see what reading level the model outputs. All these eval are going to be very use case specific, but these are just a handful to hopefully get the gears turning. Another important point here is that even if you're using continuous metrics like length, faithfulness, so basically anything that's not binary, you'll still need to translate it into a binary eval so you can define your stopping criteria for your LLM loop. And so this is just going to come down to defining a threshold for each of your continuous metrics. So, while the great thing about rule-based deval is that they're simple to implement and interpret, it is fundamentally limited in that there are just some things that can't be built with code. You can't assess the empathy of a response with code or accurately classify the sentiment of a response. For these more sophisticated tests that you might want to run, you can turn to type two, which are LLM based eval. These give us a more flexible way to assess a model's output using judge LLMs. So the great thing about using an LLM as a judge is that one, it can give you a pass fail metric or evaluation. So you can have like pass fail, does the model's output meet some criteria that you define. Another version of this is A or B. So rather than having this global assessment of is the output good in this global sense, you can also have it evaluate outputs in a relative sense. Basically taking an output and then the previous output and asking the LLM judge to assess whether it improved. The other thing we can get from an LLM judge is not just this past fail score, but also additional guidance on how the original LLM can improve its response. While this might be the first thing you might want to reach for because it gives you a simple way to automate the feedback loop, there is a big challenge with getting LLM judges to work well, which is aligning this judge LLM with the desired result you're looking for. And so in my experience, getting an LLM judge to work well can be a whole undertaking in of itself, but sometimes it might be the only viable way to do certain types of evaluations. The final type of eval I'll talk about are real world data. This consists of giving an LLM realorld feedback. Some examples of this are the click-through rate on a website. So let's say you're having an LLM write you copy for a landing page every single day. you can gather the click-through rate for that day and then you can take that click-through rate, pass it back to the LLM, have it try to rewrite the copy to improve the click-through rate, and continue this loop every single day. Another version of this is maybe you're having an LLM write you sales outreach script or maybe some ad copy, then you can use the number of people who actually bought your product or service as the feedback signal to the model. Finally, something like user satisfaction. So basically, if you have a customer support bot, you can have thumbs up, thumbs down as the signal that you pass back to the LLM. The power of using real world data in this feedback loop is that a lot of times these measures are ultimately the thing that we care about. When it comes to sales, we care about conversions. When it comes to customer support, we care about user satisfaction. These are the metrics that we're trying to optimize for or they're at least highly correlated to the metric we're trying to optimize for. But of course, the limitations here is that, you know, maybe the data integrations may not be available or maybe you have low data volume. Maybe you're not getting enough impressions on your landing page to pass CTR back to the model. Or maybe your sales cycle isn't on the order of days, but on the order of months. So it would take a really long time to know if a outreach script is actually effective. Now that we have a basic idea of what an LLM in a loop is, why we might want to do it, and different evaluations we can use in these feedback loops, let's look at a concrete example here. I'm going to build a system to refine my Upwork profile using this LLM in a loop idea. For those who don't know, Upwork is a freelancing website. So you can go on there as a freelancer, apply to jobs. You can also have a profile and clients can reach out to you about potential gigs. So it's just a marketplace for people to get freelance work. While I don't freelance anymore, I did this while I was in grad school. And looking back at my profile, it's actually pretty funny that anyone hired me at how bad the text is on my profile. So let's see if GBT can rewrite this profile for me to make it more compelling and converting. The process will be like this. I'll take my profile, give it to GPT40, and then I'll run a suite of different tests, which we'll talk about later. And then based on these test results, we'll give GBT feedback and ask it to rewrite the profile. And then we'll just continue in this loop until all the tests are passed or we hit a maximum number of iterations. We'll start with some imports. So really not much to import here. Basically just OpenAI's API. I do create a set of custom functions which we can talk about later. And then is just how I'm importing my OpenAI API key from av file. The first thing I'm going to do is write the instructions for GPD40. This was historically called the system message or the developer message. But now OpenAI has changed their API and now this is called instructions. To define the instructions, I have these two text files. One is called instructions.txt. The other is called example.txt. So I'll show these text files in a second here, but first I'll talk about how I generated them. to come up with instructions. I went on Upwork and then I searched for freelancers that have made over a million dollars freelancing AI or data science services. So I found five profiles that met this criteria and then I pasted those into chatbt and I asked it to find the different commonalities and best practices from those profiles and then synthesize a set of instructions for writing a profile based on those best practices. That's how I generated these instructions. And then additionally, I had ChadBT use the instructions and those five examples to write a synthetic example. These are both available on the GitHub repository linked down here. And just to show it, here's the repo. And then if we go to this context subdirectory and then we can click on instructions, we'll see this markdown file with the best practices. This was all generated by Chad GBT. So it's saying Upwork profile generator instructions. Follow these instructions to create high-converting Upwork profile that positions you as a premium expert in your field. Define your niche and ideal client. Draft your headline, hook with pain points, show your value, break down your services, add credibility and social proof, background and tools, pre-qualify, call to action, and then final tips. Then we can go look at the example. This is an example based on those top five freelancer profiles I found and the synthesized instructions that I just showed. Here we have unlock AI powered automation without the overhead of a full dev team. You know AI has potential but between jargon hype and tools that only half deliver, it's hard to know where to start. Then it has all these pain points. It's positioning the freelancer and then it's saying who I help, what I build, how I work, why clients work with me, so on and so forth. So this is just going to be a helpful example in guiding the model. If you want to use any of these, it's freely available on the GitHub. And then I'll just call out this read context is a custom function. It's really simple. It just reads a text file from that context directory. And so we imported instructions and we imported examples. Then I just used this prompt template to create the final instructions. So I'm overwriting the instructions variable here. Start with the instructions. I give an example. And then I add this important guideline. The next step is crafting the initial prompt to pass to GPT. For that, I'm going to read in this background.txt file, which is basically my current Upwork profile. Looking at that now, we can see that it doesn't follow any of those best practices we saw in the instructions. The worst thing I did is that it's all about me me. hates about Shaw and how qualified he is, how eccentric he is, how educated he is, how experienced he is, and the things that he does, but it has nothing to do with the customer and their problems, which is ultimately what matters. So, I'm importing the context. I'm also going to define the customer. This will also help GPG write the copy in a compelling way. So here I put founders and CXOs of small to mediumsiz businesses seeking guidance with AI use cases. And so I imported these two and then I will construct this prompt. Here I have Upwork profile rewrite task below a freelancer's background. Your task is to rewrite it based on high converting Upwork profile best practices. Their ideal customer avatar is so I'll insert the customer here and then I'll paste in my current profile here. This is going to be the initial input to GPT to kick off this whole rewriting loop. The next step is to define the eval. This is going to be the test suite that we're going to evaluate model outputs on to see if it's any good or not. Here I define four different evals. And again, I used chatbt to analyze those top five freelancer profiles to brainstorm different evaluations. And so these are the ones I came up with. They're all rule-based evals. They don't involve an LLM and they don't require any real world data. These are just things we can implement with code. The first is checking the word count. All the profiles were between 300 and 800 words. Another one is a heruristic for checking if the profile is client focused. So the way I do this is basically count the number of times you or your is mentioned in the profile. The threshold here is five or more. Another evaluation is to check for social proof. And so the heristic I use here is whether a dollar sign appears in the text. This would indicate you're talking about some past impact where you generated some dollar figure in value for a client or the profile includes a quotation from a past client. And then finally, I compute the readability of the profile and ensure that it's no greater than a 12th grade reading level. Originally, I tried eighth grade, but this was never passing. And I think part of the reason is that AI is a technical field, and they're just keywords that will never pass a middle school reading level. So, I just gave up and went for 12th grade. So, I've implemented all these functions in Python, and we can take a quick look at that. So, the first eval is the word count. It's really simple. It just counts the number of words, and then it will be true if it's within a lower bound and upper bound. And then it'll be false otherwise. This is checking if it's client focus. So basically it's counting the number of times you or yours mentioned and then it's checking if it's above or equal to the threshold. Then we have the social proof eval. So this doing a regx search to see if you have a dollar amount with some number and then million or thousand at the end of it or it's checking if there are quotations in the text. And looking at this I feel like this is redundant. It's doing the same thing. That's what you get for having AI write your code for you. And then passing if either of these conditions are true. Finally, doing the readability eval. First, I'm going to clean up the text to remove any emojis or markdown. And then I'll run this flesh concaid grade from this text stat library. This will just return the grade level. And so here I put nine. Oh, I guess I went with a nth grade reading level instead of 12th grade. Finally, I'll combine all the evals into this run all eval function. And this will check for everything and return the results in a dictionary. Now that we've defined all the evals, we can run this whole thing in a loop. The code is pretty straightforward for running these LLM loops. Here I'm just going to initialize this all past flag. Initialize it as false. I'll initialize the counter as zero and then I'll define max number of iterations. Basically, as long as all the tests are not passed and the counter has not exceeded the max number of iterations, we're just going to continue going in this loop. So, the first thing that happens is that the profile is rewritten. This takes in the instructions that we defined earlier and the prompt we defined earlier. And then client is just the connection to the OpenAI API. This will rewrite the profile and then I have this function to save the profile text as a TXT file. Once we have that, we can run all the evaluations. So this will return a dictionary with the test results. Then once we have the results, we can generate a new prompt. This is an entirely rule-based function that'll generate a new prompt based on the test results, which we can take a peek at. And then at the end, we'll update the all passed. If all the tests are passed, we'll return true. Otherwise, we'll return false. And then we'll update the counter. And then we'll just keep going in this loop. These are what the results look like after the first iteration. and it actually does a pretty good job. So, word count, client focus, and social proof. All these tests pass, but readability does not pass. So, it tries again. This time, the word count actually gets worse. Presumably, what happened is that it was trying to shorten it for readability, but then the profile got too short. It went under 300 words, so that's why this false. And then it kind of figured out, okay, that didn't work. Let's try something else. It got the word count back, but still readability failed. And then again, readability failed. But then on the very last iteration, right before it got killed, all four evaluations passed. If you're curious, you can check out all five iterations in the GitHub repo. There's this folder called profiles. So you can see how the profile text evolved over the iterations. But we can just look at the final result. So this is what it looked like before. This is the original text. And then this is the final text. And already this is easier to skim. So, a client who's rushing through and looking at a dozen different profiles is going to be able to easily skim this. And then maybe their eyes are going to see not sure how to use AI, overwhelmed with data, no clear insight, stuck with manual tasks that waste time. And they're going to say yes, that's exactly me. Then it says I can help. So that's perfect. And then what I offer, how I work. Honestly, that's pretty good. I think I might, even though I don't freelance anymore, I might just update the profiles to not embarrass myself with this profile. But I do need to make a few changes because this what clients say and proven success is completely made up. It's hallucinated. And the reason is that I didn't give it enough context to be able to write these things. But if I paste in some testimonials or different impact metrics into my original text, it'll probably be able to do it for me. And then the last thing I want to call out is this generate eval report. This is the function that rewrites the prompt and gives feedback to GPT on how to update the profile text. So this function is entirely rule-based. No LLM or machine learning is used under the hood. It's just preceded responses that pop up if certain tests pass or fail. So it's kind of hard to see what's going on here. So I had Chad GBT write it out. So basically it's this pre-anned thing where it has this preamble upwork profile evaluation and rewrite task. Here it'll paste the most recent version of the profiles text and then it'll summarize the evaluation results here. There's this little table for each of the evals and then finally it'll have rewrite instructions since just word count and social proof failed. It'll have these pre-anned instructions on how to improve those. And then I'll have this final text here. Although running an LLM in a loop gives us a new path to getting better and better responses and performance from AI agents, it does come with limitations. And the main one is so-called reward hacking. What this means is we have our LLM, it generates some kind of output which is then evaluated against some tests whether it's rulebased, LLM based or just real world data. those results are passed back to the LLM and this loop continues which hopefully makes the response better and better. So running an LLM in a loop in this way will incentivize the model to maximize its performance according to whatever evals you define and basically ignoring everything else. While this can be a very powerful force, it also opens up the door to reward hacking, which is when your model maximizes the positive feedback it's getting from your eval suite without necessarily improving the desired outcome that you're looking for. A concrete example of this could be a Twitter agent. Let's say you have a model that writes posts for you on Twitter or X. It'll write the post. It'll get fedback engagement metrics. It'll write more posts and get more engagement metrics. And then it'll continue in this loop. And its goal is to get greater and greater engagement metrics. What'll happen in this loop is that the tweets will become more and more engaging, but also they become more and more toxic. This is just the reality of any optimization process. Your system is just going to optimize for the metrics you're optimizing for. it's not going to consider anything else. So unless you explicitly have a penalty for toxicity in this feedback loop, the Twitter agent is going to ignore toxicity. It'll just be dead focused on improving engagement. This is an example from reference number one where they talk more about this reward hacking phenomenon specifically at test time. So we're not training any model parameters here. We're just feeding the output of the LM back to itself to improve it. The moral of the story is you need to be careful what you optimize for because computer programs, they're going to maximize the incentives presented to them. While this can be a superpower, if the incentives aren't properly set up, this can lead to very bad outcomes. So, I hope this video was helpful to you. If you have any questions, please let me know in the comments section below. And as always, thank you so much for your time and thanks for watching.