Summarization Middleware (Python)

Key Takeaways

Hey folks, it's Sydney from Linkchain and I'm super excited to be back with another episode in our Python middleware series. This time we're going to be covering our summarization middleware. So context engineering is all the rage these days. But what does that actually mean? Context engineering is giving your model, which powers your agent, the right information and tools at the right time so that it can execute a given task. One of the most important tools that you can use to optimize your context for an agent is summarization. In particular, we're seeing agents run for longer and longer durations of time, which means that there is very long conversation histories with important information. But in order for your agent to perform well at the next step, it needs to have the right bits of information from the full conversation history. That's where summarization comes into play. With summarization, you can help your agent focus on the right information. One real world example of this that you might see in your day-to-day is that Claude Code autocompacts your conversation history when it gets too long. So, if you're asking Claude Code to help you with a pretty involved refactor, a couple minutes in, it might autocompact. You can do the same thing in just a couple lines of code with Langchain's new summarization middleware. All right, so we're looking at Lingchain's new middleware documentation where we can see the docs for the built-in summarization middleware. It's got a pretty simple interface here. So when you're creating an agent with this primitive, you generally specify model and tools and then you can pass in the summarization middleware with a couple of knobs that you turn. So first is the model that you use for summarization and then we have a context size that you trigger with that can either be specified in tokens, messages or proportion of the available context window. And then we also have a retention policy that's the context size that you want to keep. Let's jump over to Langchain's new API documentation to do a little bit more of a deep dive. So again, we can see those types here. I mentioned the context size. We also have things that you can customize like the summary prompt, whether or not you want to trim the context before it goes into the summary model. Um, and then also a detailed token counter. You can also see all of our other pre-built middlewares here and then some utilities for building custom middlewares. Let's jump into some code. So for this example, we are going to build an agent that can retrieve information from Wikipedia. We're using this nice retriever tool from Langchain community which we wrap in a fetch Wikipedia data tool. Then we use a custom summary prompt for our summarization middleware. Additionally, we configure a couple of those things like the model we want to use for summarization and then the fact that we want to trigger summarization when 70% of our context window size has been used and then our retention policy is that we're keeping 30%. You might be wondering, how does Langchain know what 70% of the context window size is for the GPT3.5 turbo model, for example? Well, we just released a new model profiles package that has information about model capabilities that we use under the hood to intuitively make decisions about agent behavior. So, that contains information like tool calling capacity, structured output features, context window size, and much more. Let's test this out in the live debugger. So, for this demo, I'm going to fetch information on each of the founding fathers and their birthdays. I'm using this example because I anticipate that there is tons of content on the Wikipedia pages for people like George Washington, John Adams, Thomas Jefferson, etc. So, let's kick this first query off. We can see a successful tool call to the fetch Wikipedia data tool with George Washington here and then a very verbose tool result and then that summary message from the model. It looks like our before model summarization tooling is not yet triggered. Let's kick this off for John Adams next. And then finally Thomas Jefferson. Great. So we can see that the summarization middleware was kicked off. Here we see a summary was generated. And then we use the fetch Wikipedia data tool to get information on Thomas Jefferson. And here's that final response. All right. So let's look at the trace view for these as well. So we see turn one which was George Washington. Turn two which was John Adams. and then turn three which was Thomas Jefferson. And then we see the summarization middleware was kicked off. Here we can see our custom prompt with the messages to summarize and then the output of that summary which is then fed into the final model request with that Thomas Jefferson tool call as well. Thanks for joining me for a quick demo of Lingchain's new summarization middleware, which is particularly helpful if you are diving into the art of context engineering in order to optimize agent behavior. See you in the next one.