How to Create a Self Healing Code Agent

Key Takeaways

Hi, this is Katherine from Lang. One of the most common use cases we see with LMS is code generation. But here's the question. How do you improve the accuracy and performance of the code generated by your code agent? One answer is static analysis. Checking the code for errors or for inconsistencies without executing it. This is where tools like Pyate or My Pi comes in. So now you could actually build that kind of check into your agent loop. This is what we call a reflection step. A point in the loop where the agent looks back at what it just generated and validates or improves it before continuing. We'll show you an example of how you can create such a reflection step in your architecture, evaluating its own output before returning a result. Instead of building evaluation logic from scratch, we'll show you how you can leverage open evals, an open- source package that provides you with out- of-the-box tools for things like type checking and sandbox evaluation. Open evals includes a number of helpful out ofthe-box evaluators. This includes ones form as a judge, rack, and code evaluation. Today, we're going to focus specifically on the code evaluator section. Open evals gives you several pre-built evaluators for generated code such as type-checking generated code, sandbox type checking and execution evaluators, and finally using Elm as a judge to evaluate code quality. Since LM outputs often contain both code and non-code text, open evaluators also include flexible code extraction utilities. You can either use an LLM to extract relevant snippets or simply extract markdown code blocks directly. Now let's deep dive into the several techniques that we can take to validate code. First, you can type check generated code with Pyite and my PI. These are lightweight approaches to type check Python code, but they won't install or check for missing dependencies. For TypeScript, there's a similar evaluator for type checking. Second, you can use LM as a judge to prompt LM to provide feedback on your code. LM as a judge evaluator here defaults to no code extraction strategy, but it gives the option to enable extracting code, which could be helpful to reduce noise or distraction, especially when using smaller models. Finally, open evals integrates with E2B to run some code evaluators in isolated sandboxes. Static analysis is a useful tool for caching mistakes in generated code. Open evals lets you perform this fully locally. Sometimes generated code can contain and misuse third party libraries. So rather than installing or executing these arbitrary dependencies locally, you can use the sandbox environment to effectively perform type checking. There are two key sandbox evaluators. First is the sandbox typeing evaluators. One example is the sandbox pyate. This evaluator parses out the required dependencies, installs them in a sandbox, and runs pyate. It returns any type-checking errors found for TypeScript. Similarly, there's a type-checking sandbox evaluator. The second type of sandbox evaluator is the sandbox execution evaluator. This evaluator goes a step beyond installing dependencies, actually running the code inside the sandbox and checking for runtime errors. Now let's take a look at the example we saw earlier to see how it's applied in action. Now back to the example we saw in the beginning. The base agent has a simple React architecture implementation where it works off of a text file embedded in the system prompt and it has tools to fetch further information to improve the correctness of generated code. Mini chat link chain includes a reflection set which verifies the generated code through the sandbox typeing evaluator in the loop. Let's take a look at how it's constructed. So this is our reflection node. In our reflection step, we first create a sandbox pyite co-evaluator where the evaluator extracts generated code from the agent's output through markdown and pushes it up to an E2B sandbox to run Pyrite over it. We obtained a result from running the evaluator. And now there are two situations where we will want to directly output the response to the end user. The first scenario is if the code generated does not have any type-checking issues. That is within the result score field it outputs true. So it wouldn't need any code regeneration. The second scenario is if the original agent response did not include any code at all. We can check for this by checking the result metadata to see if code extraction step in the evaluator field. In this case, we also want to directly output our agents response. So now transitioning that logic into code in the case that our agent's response contains code and finds false evaluator results. We want to feed the error message back to the agent to regenerate code. We do so by attaching a user message to the LM giving context on the errors found and directions to regenerate code. With this information in the prompt, the agent then has to inform context to regenerate and modify based on the error. Let's try this out. I can feed an example input to our agent asking to create a swarm style agent. The query is routed to our agent node which generates its respond while calling a set of tools. Once the response is finished generating, this is passed to a reflection node to create the results. Looks like the reflection node has caught some errors which is now prompting the agent to regenerate the code. Now our agent regenerates the response which now passes the evaluator test and their outputs our final answer. We can also inspect the trace and lang to take a detailed look at the run behavior. So first we can see that the base agent called a get langraph docs content tool. This fetches relevant content from relevant documents and then it calls the LLM to generate a response. The response is then passed to the reflection step where the evaluator is run. Taking a closer look at the evaluator results. It passed through a score of false and based on the error message several required arguments seems to be missing. So this includes model name, timeout and stop. So this is then passed on as a part of the human message. If we scroll down where we included the error found and asking the agent to try to fix it. So now having the context of the error message, the agent generates another round of outputs. The generated output is then passed to the reflection step. And during the second time of running the evaluator, we can see that no errors were found returning true. And from there, the final response is passed on to the customer. Adding this reflection step within our architecture. This helps catch some of the type checking mistakes that are being made by the LM and this can ensure better code quality for our agent. Thank you for watching.