Context Engineering – Managing Intelligence at Scale

Key Takeaways

So, thanks to everyone who has joined. This is session number three on the kind of course on building agentic applications and going into some of the details and the underpinnings of these agents. So, one of the we're on session three now. My name is Kweku Ackom. I'm the director of AI solutions at Sambanova. We're a full stack AI companies specialized on specialized inference. And we have our own chips and we basically have super efficient and super fast inference on our kind of RDU chip. It's an equivalent to the GPU. And so today I'm going to be talking about context engineering, what that means and how it comes how to kind of manage it in production as well. So, I'm going to share my screen. If there's any questions, please ask. And you can also follow along using the using the link here. So, if you if you want to follow along, there is a repo here. So, I think it's already been shared. But if you want to go follow along to the code that I'll be going through in this session, you can do at this GitHub here. Again, if there's any questions, I will try to answer them in the session. I've also got one of my colleagues here who will also jump in and try to answer some as well. [snorts] So, again, before I kick off, just kind of give you a quick overview again who we are. So, this is the Sambanova cloud and again, we specialize in super fast inference. So, if you want to log in to the cloud or sign up or kind of follow along, you'll need to get a Sambanova API key. So, just kind of go here and you'll be able to log in and then get your API key here. And once you do, you'll be able to go on to our playground and start using our models. So, again, one of the things we're very very good at is the fast inference. So, you can see here the kind of the tokens per second compared to the GPU. Cool. So, now that we've kind of got all that over with, we'll be talking about context engineering and managing intelligence at scale. So, let's have a look and we're going to talk about um a few things today. So, kind of take you through some of the theory and then we're going to walk through some of the practical examples using our Jupiter Labs. Okay. So, the first thing that we're going to look at here uh is the recap and today's agenda. So, if I look at the agenda at the moment, you can see that we are going to talk about the context collapse problem, what happens when um what happens when uh context collapses and why context engineering has become such a pivotal part of agentic applications. We're going to in our previous demos, we've been building um deep agents from scratch in like the the the kind of lowest level primitives in LangGraph. Today, we're actually going to move to the extract abstraction, which is the deep agents library, which is open source and we'll be using to kind of um highlight or bring to life some of these theoretical things. Again, there's lots of SDKs that do this. This is an open source one from um LangChain that we're going to use, but you know, um other other harnesses allow you to do similar things as well. We'll go through middleware. We'll go through some of the patterns that you see in production um and then we're going through some of the context types as well. So, what you can hear see here is kind of like when when um when agents lose context or when they get into really long context, it it drastically reduces the effectiveness of the agent. So, you can you can start to see these things and this comes from one of the papers that was on context collapse. You can see that accuracy starts to go down. You can see latency um goes down as well because obviously you you are managing your context. You don't have this huge, you know, huge context that's growing um and then you you you are able to kind of better manage your context window of the of you the model that you're using to give it the most relevant context at the time. So, in a nutshell, what context engineering is is making sure that your agent or agents, your workflow has exactly the right context at the right time and nothing more. And this is super important because as we go forward with agents, you can see the fact that you've got context coming from everywhere. And I'll talk a little bit about that in in in the sessions that we'll go through. So, if you think about a modern agent, you know, we talked about it in sessions one and two. You build your agent, you give it some tools. When it goes out into the world, it has so many so many pieces of context coming in. It's got the the context from the runtime. It's got context from, you know, API calls. It's got tool calling context. And we're going to go through how that gets lost, how it grows, and how to manage it as well. So, you've got two different problems that cause agent failures. You've got token limits and the degradation of attention. So, the token limits are, you know, are hard cut off, right? So, you know, in a lot of open source models, we've kind of got around, you know, this kind of 128 to 200k tokens that you're seeing on some of the frontier open source models. And for the frontier closed source models, you're kind of moving up towards the million context. So, Opus has got a million, I think. And Gemini's the same as well. So, you've kind of got this context window that's fixed. You have a conversation history that keeps going up, right? So, as you're talking to the agent, you're you're doing conversations, you're doing tool calls. Those are still living somewhere, so your context window very quickly gets filled up. And then you kind of, you know, once it's full, in general, the oldest the oldest message start to get dropped. And then the agent starts to lose access to early instructions and context, right? So, it starts to forget unless it's actually managed. So, if you just run context in a in a vanilla way, this is what will happen in most agent frameworks. Then, you've got this whole thing of attention degradation. What you've got here is this kind of lost in the middle. So, the model attends less to mid context information, even when it sits within the context context uniform, the attention is not uniform, right? So, so basically, models are strong base they strongly attend to the beginning and the end of the context. So, stuff in the middle gets reduced, and this has been proved as well. So, basically, and then the key thing here is that critical details buried in long context are effectively invisible to some models. And again, models have been getting better at that, but it's still somewhere that models struggle a little bit. So, what you're trying to do in the context window and context engineering is making sure that your agent has access to the right things at the right time. And that usually means being very, very, very disciplined about what goes into your agent's context window and what doesn't. So, you know, moving things to pointers and moving things so that the agent can reference them. And we'll go through this in in in example as well. So, as I said, the key thing here is kind of talking about these five context types, right? So, you've got first cut time kind of context you got is the input context. So, these are things like system prompts, memory, skills, static, anything that's loaded before the user says anything. So, some of this, you know, these a lot of this is repetitive, and and a lot of this is is is a is is starting to drive a lot of the input to agents. This is why, you know, the the skills and the the the memory and the prompts are so important because they go in at every every step. Then, you have a run time context. This is everything that, you know, the user metadata, you know, things like keys, connections, things that propagate automatically to the sub agents. Then, we've got kind of this this compression, right? So, you can offload large outputs to files. You can summarize old messages. So, this is again a a a a type of context because we're now seeing that the compression that goes into context is something where we take everything that's been happening before and we we pass that back to the back to the agent instead of trying to forget things that are happening earlier, we try and summarize them as well. And then the next thing that we're going to talk about specifically in this session is isolation. So, you can also have isolated context types. So, this is this is where you pass context to your sub-agents. So, it means that you keep the context window of your main agent clear and you offload the large context to task-specific sub-agents. And we'll we'll do a little bit of that as well. And then finally, kind of the the last context type is this long-term memory. This is very much memory that you manage from the application layer. So, say in a system in a in a store somewhere, depending on what you're using, this could be Postgres, Redis, or somewhere that you can store this data. And this is this survives across conversations. So, this could be anything like, you know, lessons learned, any particular memories and things like that that you want to store in the longer term. So, one of the things that you want to ask when you're debugging your agent is which of which of these context types is broken? Is it my input? Is it my runtime? My compression? My isolation? Or my long-term memory? So, this this tells you exactly where to look. And we'll build kind of solutions for types three and four because this is where you're you I'm sure you've seen if you've used a modern system that they actually compress your conversation. So, if you go into Claude, you use Claude code, or any other open source, you see that they automatically run comp context compression and they automatically use isolation as well. So, they send context to sub-agents there as well. So, what we're also going to talk about in this session is how we're moving from what I would say is the um moving from coding things from scratch. So, in the first two sessions I was very much keen to get people to code things by hand so that they understood what's going on within your deep agent, right? So, you understand how to these these primitives that are going into deep agents like, you know, um tools uh to do list, memory, and things like that. Um planning, we we we built those primitives by hand, but now we're actually going to use the abstraction. Um and one of the abstractions there's there's more um libraries that do this, but we're going to concentrate on nine graphs um kind of deep agents. And you can see the abstractions for the for the code that we've been writing so that we can really focus on the context part of things as well. So, nothing really here changes. You just You'll just see slightly cleaner code. We will still go through some examples for writing specific middleware tools um that will manage context, but we're also going to use some of the built-in tools that come with uh lots of platforms now as well. So, first thing I want to make sure is I'm going to make sure if there's um any questions from any folks. Um do do do do do do do do do do do do do do do do do do Okay, so we we didn't get any questions. Okay, so what we've got here is we have a new abstraction here, which is called create deep agent. So, we pass in the model, we pass in the prompt, and then we pass in the tools, and more importantly, we pass in things like middleware and sub-agent. So, this is kind of, you know, we're going to move a little bit into how we use middleware and sub-agents because managing context, um middleware and sub-agents are really good way of doing that. Um and we'll I'll talk to you about how this works um in a in a little while. Okay. So, the next thing that we're actually going to do is we're actually going to The easiest way to kind of talk about context um compression action and talk about context engineering is to actually start to go through a an example. So, we're going to go through an example, and we're going to kind of go through this search and offload. So, in in lots of one of the easiest ways to think about context engineering is thinking about an agent that has access to a web tool. So, every time you go and search the web, you've got tons of results that are coming in, right? And, you know, you can go and search the web and you might get a, you know, it might get 10 search results and each of those 10 search results might be a full web page worth of a XML. Now, that's going to be lots and lots of tokens. So, what we're going to see here is a real-life version of managing that context. Without managing that context, you are feeding all of those results into the agent's context window every time it goes through to do a turn. So, what we're going to show is what are the other potential ways that we can handle this or we can work with it as well. So, I'm going to I'm going to switch over now to the to the um to the notebook and we're going to start going through some some things there as well. Cool. So, I'm just going to go into the Q&A and answer a couple of questions as well. So, so, questions from in the multi-agent systems, what should be the context holding window? So, the context holding window, I'm trying to understand the question here. So, managing your context in multi-agent systems very much depends on your your strategy here, right? So, you can hold context in a few places, like I said. You can hold it within the within the the agent's context window. You can offload it to something like cat to to something like Redis. So, where something like Redis would come in is where you wanted to offload some of these things. So, if you wanted to offload documents or past conversation history and things like that, memories, to a to a cache store, you can. And we'll talk a little bit about that as well. But, but yeah, I hope that answered your question. Okay, so we're I'm in I'm in the notebook now and again, we're going to we're going to use LangSmith to kind of do some of our tracing. And and you can see here that I've got a I've got a benchmark task, right? So, you can see the benchmark task is here. We're going to research the current global AI landscape. We're going to cover the EU AI Act, the US executive orders, um, China's AI governance approach and then we're going to save the files, right? So, what I've actually done here is I've actually told I've I've created a specific task that we're going to get the the agent to run. And again, you can see the five contact types that I talked about and more importantly, when it's set. So, if you look think about the input context, when do you set that? You set it at run time. So, every time like, you know, it's so it's static per run. So, these these skills can be updated automatically. So, you there there are ways that you can kind of update this during the system prompt, you know, based on feedback of the traces, but usually it's fairly static. Then, you've kind of got the run time context, you know, this this can change based on the tools that are called, the user that's calling it, things like that. And then, you've got things like the compression. This is, you know, when it's set automatically when limits are reached. And then, isolation to sub agent happens when a task is delegated. And then, I think this is what the question was talking about, longer term memories. Longer term memories are assisted by some sort of cache or file store and these persist across conversations if if that's what the, um, application builder has allowed as well. So, what we're actually going to do is we're going to build this kind of con- custom context offloading tool. So, before we kind of move on to the abstractions, I wanted to really go through and dive into what this means. So, what we're going to build is a pattern. We're going to build a search tool that actually saves the results to a file and resu- returns only a brief summary. So, what happens here is the the model never actually sees the raw search dumps in its messages. So, we are manually managing the context. So, this is a very, you know, a custom example because it it's showing you how if you built a tool, let's say a tool that goes and gets email or Jira, or even goes and gets the news, this is a way that you can actually manage what goes into the into the context window of the model. So, again, the tool itself isn't isn't too complex. I'll I'll run through the code fairly quickly. So, I I I decorate it with the app tool decorator to make sure that it is a tool object, and then I call it search and offload. And what I'm doing here is I'm invoking the query from the user. I'm getting the I'm getting the list of um of web search results, then I'm going through each item, um cleaning up some URLs, getting some telemetry, and building some custom search tools as well. So, again, nothing nothing too crazy here. Um I'm you'll you're going to see um essentially the the summaries, and then the the files saved as well. So, what we're going to do here is we're going to we're going to actually build a deep agent with our custom search tool plugged in. So, here again, we're using LangChain Sabanover. You can follow along. We're using the deep agent abstraction. This is the one that we've been building by hand. And then we're going to actually, you know, build a build a workflow that uses the to-dos. As you can see here, I'm saying for each research topic, immediately call search and offload, um and then offload it there. So, read the file, review your files, uh save findings, and then use write file to write the final report. So, you can see here we're being fairly prescriptive. So, again, if you can see what we're doing here, you can see that we are calling the model, we're calling our to-do list middleware, calling some tools, and then feeding back into the model. So, what we're going to do now is we're going to run run this on our benchmark task, and we're actually going to see the See the results come through now. So, I'm going to While that's happening, I'm just going to move to the Q&A very quickly. Um I don't think I've got too many there, so I'm just going to move to the chat and make sure I'm answering all the questions there as well. Okay, cool. So, as as this runs, basically what What we're going to see here is that we We're getting We're getting the model to go out and run a a ton of search queries, right? And then we're coming back, and we are passing them through our custom-made tool. And our custom-made tool is actually going to do summarization on each of the results that comes back, and it's basically and going to give us a summarized version of this. And what we What we're actually going to be able to see is that you can You can start to see the difference that it makes in managing the context and doing summary. So, you can see here, we've got 26 total messages, 10 tool calls, five to-dos, and 12 follows. So, what we're going to do is what we're going to actually going to show you exactly what happened and what we compressed. So, here you go, right? So, these These are the queries that these that the agent did, right? So, we have 1 2 3 4 5 6 7 8 9 10 11 12 13. So, what you can see here is Let me just make this a little bit bigger. So, you can see here You can see the raw search characters. You can see what was saved to file. And you can see the in-context summary. And you can see the compression ratio of what it's done. >> [clears throat] >> Now, as you can see, if you were managing, you know, thousands of SQL tables or maybe thousands of Salesforce queries, you can start to see how even a 250K context window could quickly become bloated, especially with messages and conversation history and things like that as well. So, it's You can see how this can really really really add up. So, then you can see here. So, the in-context messages was this, and you can see how much I've offloaded. So I have part of context management and context engineering is offloading the context. So in order to make sure that your your agency is the right thing at the right time, a good idea is to offload the context, give a pointer to the file and we'll go through the way that deep agents does this on its own with its own tool and essentially [snorts] give like a summary of the file or just a pointer to the file. So here we're actually giving a summary. So you can see the the purple is what the original thing was in terms of in terms of number of characters and you can see what I'm actually sending through. So I'm actually sending through kind of 400 to 500 character summary versus the you know, the 3000 to 3500 full full text file. Now that makes a huge difference in terms of what I can actually put into the context window and as if you can imagine if I put all of these into the context window, it would get super bloated and if you can imagine there could be five to six of the tool calls that have their own context that need to be managed as well. So this is just an example of what the difference context compression can make in in in real life. So again, you can kind of see the three results that are saved. You can see the sources. You can actually see you know, we've got a file here EU act research the first 400 characters. Now this is super important. It's very It's a very good practice now to offload big files to basically a file system. Be that virtual or something that you've implemented through like a a store and basically point to that information and get it later when the when the when the model needs it as opposed to keep it in its memory. So I'm just going to stop there and just make sure that there aren't any questions. If not, I will carry on. So what we what we what I will kind of want to make sure that people get from that is the fact that the way that you can manage your context in real life is through these kind of um pieces of code that do something to the context, move it somewhere, um move it back into the into the um agent's memory, compress it, summarize it, you know, save it somewhere else for that for the for the model to come back to later. What then you're doing is that you're keeping the context window clean. So, you're passing through, you know, in summary of those documents, maybe 1,000 characters as opposed to like 40,000, which would massively blow your context. And the way that is done, and what we'll come to in part two of this demo, once I kind of go through some of the um theory behind this, is the how. So, the how this happens in agents is basically a layer that we call middleware or hooks. So, essentially, what you've got in in um in LangChain and LangGraph is middleware, and you've got this in things like ADK, um which is the the Google uh platform or framework, and then you've got the Microsoft uh kit as well. And middleware allows you to actually hook in into kind of the pre- and post-model hooks. So, what does that look like? So, if I if I kind of go back to this to this um to this diagram here, where hooks kind of go here is before and after. So, you can here you can see here is an example of middleware. We have a patch tool calls middleware. So, this runs something before the model makes a tool call. And then you've got this kind of to-do list middleware after model. And this basically says, "Well, after you've done something, go and check your to-do list and make sure that you are on track." So, this is where you actually inject things to manage your context. So, you can manage your context before the agent does the call, or you can manage your context after it does the call. So, again, this middleware is the implementation layer that you can use to actually get this get this stuff done. And I'll go through some of the some of the the theory behind that as well. Cool. So, before I continue, let me just make sure that I haven't got anything in the chat. I can see that I've got a couple of Q&As. So, so there's two questions here. So, how do how do you assess the quality of your context compression? I think that's a really good I think that's a really good questions. So, I think the best way to analyze your context compression, if I if I go back to if I go back to this, is to kind of run run your own e-vals on it, right? So, if you can see here, you know, what is the what is the quality your objective quality in terms of the the the task that your agent is doing. So, if it's doing like a you know, a web a web search, then it's going to some source systems and returning an answer to the user, you can actually analyze the quality of that answer. Does it degrade? Does it get better with the same with with the summarization of the context? That's one way of doing it. Basically, analyze it in the whole end-to-end flow. The second is to actually, you know, take out the context layer in itself and say, basically, you know, come up with some sort of algorithm or benchmark in terms of does how well does this summary capture the information or point to the information that's been captured? And that can be kind of your own internal benchmarks or, you know, how well the agent is able to carry on its task with that context compression in there as well. So, I think those are both good examples, but the best is honestly to test it on the on the task that you're trying to do. Does it If if I compress the if I compress the context, does my agent stay on track? Does it kind of vary or does it kind of carry on as if it had full context? So, that's a really good idea as well. Then, the next question is how do I route the right kind of context in more complex applications? I think this is a really good question. I think this is very much um there's a couple of ways you can think about this. Uh so you've kind of got as I said you've got these five types, right? So you you need to think about where you want to route them. So things like skills and memory and things like that are things that are going to be loaded in all the time. So if you're thinking about if you've got a ton of context in say um in in SQL or you've got it in maybe Word documents, the thing that you kind of want to ask is what is the next action of my agent, right? So in a lot of system prompts they will actually say you have pointers to files, you know, you can if if you need to, you know, perform a SQL uh query, you know, look at this skill file. Also basically you can you can you can guide it within the system prompt, right? So you can say, "Hey, when you've got this kind of query could maybe go and find the context from here." You'll have statics files that you can look at. So guide it through the system prompt. The second is a bit more I guess in code and middleware cuz that's like the prompting, right? So if you want to have like a belt and braces approach, you can basically say something like, you know, if if a query if it's a web query, then you in your middleware you can force your agent to go and look at the um look at the files that contain those. So that's like a belt and braces where you're not you're not relying on the agent to do the right thing per se, but you're actually handling it in middleware and injecting it automatically based on the kind of query. So that's a bit more I guess deterministic, um but it does give you a bit less flexibility. Uh the last way that you can kind of do this and you're starting to see this in more complex agentic systems and we do this as well is that [snorts] if what we what when me and my team build tools, if we have say a a tool that is going to go and get stuff from Jira. What we'll do is we'll say, if you call that tool, then we inject more context. So, it's kind of like based on the run time. So, we basically say based on the user's query, if it looks like you're about to call said tool, then that tool then injects some extra context that can basically give the the model some information at run time to say, "Hey, now you're using the Jira tool. Actually, Jira like you know, large Jira are dumped here and you can go and get information. Here's a pointer to the issue and things like that." So, those are some of the some of the potential ways that you can you can root and assess the quality of your context. I hope that answers the question. So, so does the compression allow multimodal type? So, do you see the use case for clinical research applications for drugs? So, so there's a couple of things here. So, multimodal, I think you can compress. So, anything anything that has data, you know, you can compress. The thing is what how easy and it there's there's plenty of ways you can do kind of like lossy lossy compression for different modalities. The question becomes when you compress it, where should you put it? So, I think something like a multimodal type, maybe you might want to say I think that would be a good use case for pointing to it. So, for instance, if it was load of images, you might want to save those images securely somewhere and then say to the model, "Hey, you can if you want to see said images, you can go to this location. This is a pointer to them and bring them into your context when you see the need for." So, again, I think there is definitely a use case for those kind of applications that being built and you'd have to kind of think about how to manage your context if you if you do deal with a lot of you know, audio and things like that and images. And my kind of suggestion there would be to create a pointer to something so that you can actually get to the to the real information elsewhere when the agent needs it. Cool. Awesome. So, I'm going to move on to some of the additional bits that I was talking about here. So, what we've talked about here is kind of context compression in action. So, what we're actually now going to talk about is the the kind of how. So, we talked about, you know, a a place where we'd want to add observability, maybe guardrails, context enrichment without changing the agent code. So, how do you do it, right? So, this concept of middleware is is exactly where you do that. And the key key kind of principle of middleware is that you want to intercept the agent loop. So, you want to go you want to get into the agent loop and you want to put pre and post hooks. So, you can see here, without middleware, you've just got this kind of LLM tool LLM. So, the LLM calls a tool, kind of classic react, gets the tool results, thinks, "Hey, have I answered the question?" Goes and maybe calls another tool, carries on. Now, the problem that you've got with this, as we just mentioned, is that your context bloats really quickly. And again, we used that web search result as a way for you to see that in action. So, what middleware does is it says, "Hey, let's actually inject something before the LLM calls the tool or the agent calls the tool." And then, additionally, we can put something after it does that. And additionally then as well, you could also put something at the stop as well. So, after, you know, the LLM does its final thing, you could also put something in here as well. So, this this that wouldn't be middleware, that would be kind of just a hook. But, there are very there's a few places where you can inject it. The kind of pre and post tool is very good and very common because if you think about it, a lot of your context is coming from your external tools, right? So, you know, even with your skills files and things like that, even though they are massive, if you are runtime and you're making tool calls, you're going to get a lot of context from those calls. So, that's where you want to manage them. So, here what you've kind of got is you've got this kind of uh middleware object in in the deep agents and LangChain. So, this allows you to kind of use custom and pre-built ones. So, you can you know, the the the the common middleware patterns of all context engineering patterns that we're seeing is summarization, compression, maybe you know, redacting PII and things like that. Um dynamic prompts. Um so, there's lots of use cases you can use for middleware. Um and one of them that we're seeing for instance is injecting lessons learned. So, if you want to if you want your agent to be more dynamic in itself improvement, a way to do that potentially is going to get the information from the trace, putting that into the prompt, asking the LLM to analyze it, basically analyze its own performance and things like that. Um so, this is this is uh a kind of slide on how it happens in practice and we'll go over some of that code um as well. So, the next thing where I'm going to talk about and we're going to kind of go through in the examples is talking about context isolation um via sub-agents. So, again, one of the one of the cleanest patterns that you're seeing and you'll see in lots of programs that you use whether or not it's obvious is a different matter is that the best kind of harnesses or agentic uh products are managing context via um sub-agents. So, what does this mean in practice? So, you have a most you know, in most kind of uh workflows now, you kind of have the supervisor agent that's the planner. That usually has a higher performance reasoning model that's a bit cleverer and that is the agent that you want to keep its context as clean as possible. So, that supervisor agent should really be thinking about, you know, um task description, how that task is progressing, what the user wants and things like that. It shouldn't be, you know, loading web searches and going out to SQL databases or going to get images or things like that. That should be left to the sub agent who can manage its own context and then feed back the key outputs to the to the supervisor agent. Now, of course, if you've used Codex or Claude code or open code, you can actually see that in how they run. So, when you open something in say open code and you and you ask a task, you can see it actually delegate it to sub agents. So, beyond the under the hood, that's what it's doing. It's taking your query, it's decomposing it, and then it's immediately spawning sub agents to actually go and do the work. And you'll see this in Claude code. You can see it's got like a plan agent and then you see it's got like a research agent. And these are these are, you know, dedicated sub agents that go off and work async while the supervisor agent is looking after the whole workflow. So, again, this is a super super powerful concept and I would really I would really advise people to kind of when they think about context isolation in your task to really think about using sub agents. It it manages the context in a cleaner way as possible without you having to do too much you know, coding. And you it's flexible enough that you can actually do you can add your own information into a sub agent. You can change exactly what goes back and forth. So, it does give you enough flexibility to if you want to get deeper into what the sub agent is doing. So, here we've got an example with that we're going to go through in code as well. So, as I said, the the problem is that you've got these noisy results. It pollutes the context and it gets distracted by these details. Now, sub agents sub agents they're isolated, you know, they can return a compressed summary and they have access to the virtual system or the actual source system if you actually deployed one. And then you can have your supervisor stay focused on the orchestration, which again is the more complex your tasks are, this becomes super super key. So, here in this example, what we've got is you've got this kind of supervisor agent, and we've got three different research agents. So, you it it basically decomposes the task and give it gives it to these agents. And what they do is they go off with their own context, you know, they have their own context windows, right? Because these these agents themselves have an LLM, that LLM has, you know, 200k or so, whatever. So, we've got a new a fresh context window for each of these sub agents to manage, and then they do their task and return the the the main the main thing to the main agent there as well. So, you can see an example in code here that we'll walk through. Um so, what we're going to actually do now is we're actually going to go through a demo where we actually use middleware, and we use sub agent delegation. Again, this is one of the most powerful ways to manage context at runtime, because this this can grow with the complexity of your task. So, we have several sub agents that we have running in some of our demos, and I can I can try and show you one, and you can see exactly what's happening to these to these sub agents. They're able they're able to kind of go and and run these run these complex queries by and manage the context window as well. So, let me just get this, and we can go to here. Okay, so I can see that I've got a few Q&As, so I'm just going to pop in here very quickly. Okay. Okay, perfect. I think I've I think I've answered most of those. Right, so >> [snorts] >> I'm going to go here and go into the middleware section. So, so as I said, middle middleware wraps the agent loop. So, what you can see here is we're going to make some custom middleware. And again, that's very easy. We're using deep agents, but again, other other frameworks will have the same kind of primitives. And so, this is the middleware primitive. So, what I'm going to do, I'm just going to add a kind of a pretty naughty observability middleware. All it's going to do is it's going to log a money model to call, track token usage, and basically count the invocation of tools. Now, again, this this is done by many of observability tools, so you don't need to do this yourself, but you can imagine that if this was, you know, your own your own data that was coming in, you you could count whatever you wanted. You could basically build middleware to to manage context in any way that you want. You can do regex and things like that. So, it really is up to you what what you do in this middleware layer. So, what I've done here is I've created this middleware that basically counts some tokens. So, you can see before the model it says it runs before the model, it captures the contact size, so you can see that. You can see it returns calls, messages, tokens, and whether we're over budget. And then I've got an after model, i.e. the post hook, that then goes and runs after every model call, counts the tool invocations as well. So, what we're going to do is we're going to basically reset our middleware telemetry here, so you can see that we're we're we're creating a new deep agent, and we're running it with custom middleware. After I do that, I'm actually just going to create a um a plot, so you can actually see what's happening when it runs the when it runs the um deep agent with middleware. And what you'll actually see is that the the token count doesn't jump as it normally would with all of these messages. You can see that it's managed because we are using this kind of um middleware to kind of track it there as well. So, you can see how this works in in practice. So, let me just dip back into the Q&A. Okay. So, is the compacting part what LLMs like Claude do post-task when they auto-compact? Yes, it's exactly that. So, the compact the the compaction part, when you see Claude code do the auto-compacting, that is exactly the the the process that we're going through there. It's context management via mid via a hook. And so, um I believe most most kind of I can't tell you the exact kind of percentages, but most LLM's or most frameworks will now kind of auto compact when they see that you're getting to within 85 to 90% of the context window and they want to compress it. So, it means that you're not running out of context and they will do that automatically. So, that's the kind of thing where middleware comes in. So, you can imagine what that middleware is doing under the hood. It's basically saying, "When the length of this conversation with the user, with, you know, with tools and stuff comes to 85 to 90% of the context window of the model, then auto run the context compression software to compress it." And then it will have, of course, their own compaction algorithm that runs as well. But it's it's exactly what's happening in in Claude code when they auto compact, for sure. So, single agent with large context and made tools versus multi virtual model of context. Yes, for sure. So, so, is there a threshold or context cooling? So, yeah, this is this is this is I think this is very dependent, right? So, I think what you're finding is that you don't want to context engineering is really making sure that you're managing the context. So, the model sees exactly what it needs to see and nothing more and then at the right time. So, it's kind of this less is more approach. That's why you see compaction. That's why you see summarization. That's why you see these kind of plan.md documents. It's because it's saying that a a a model or an agentic system can always go to a file. It can always say, "Hey, you know, you've got this plan here. Go and read it if you need to." Where it gets harder for it is where it has to carry that in memory. So, and generally, what you're starting to see is that if you've got a, you know, a single agent with a ton of tools, it's it's usually better to go to that pattern where you have this, you know, single agent that then delegates to sub agents, right? Who have their own um context window and their own tools. That's leading to a much better quality, and that's where you're seeing a lot of these harnesses land on. That's what Crew Code does. That's what Open Code does. That's what Deep Agent does. So, they usually, you know, you're you're getting that is the kind of design pattern that people are using now. So, your next question is there threshold or context size? I think that kind of 85%, you know, if you're if you're getting to kind of that that if you're getting to 85% full in your in your context size just by the tools that you're giving to your agent and a normal conversate in a normal kind of turn, then you probably want to reduce it, right? And I think that will always depend on your use case, but I definitely think that people want to think of this less is more approach with context management. You really want to make sure that you are very protective, right? Of the context that goes into your model, um and not kind of too liberal with it. And I think that's what a lot of the designers, Crew Code, and people are doing now. They're being much more stringent, I would say, with what they're putting into the into the run time context of the of the model. Okay. So, here what we've got is it's kind of run the custom middleware, and you can see kind of exactly what's happened here, right? So, you can see the you can see the cool the tool call breakdown. You can This is basically every time it intercepted something, um and it did that. Now, you can see the number of tokens that it got here as well. And what I'm doing here is again, I've This is me kind of set setting a token budget. So, you can imagine if you're and the another reason to do this, by the way, there's it's not just latency and forgetting. There's a huge thing, cost, right? So, if you're if you're running this in your a business and you're at the CFO, you really would do want to manage your tokens, right? Like, you don't want to be, you know, spending tokens, you know, when you don't need to, when you can use file storage and things like that, which are much cheaper. Um you you want to make sure that you're managing your tokens effectively. And of course, you that allows more people to use stuff, but it also means that you're not wasting tokens, right? So, you can see here that we intercepted nine model tool calls, we offloaded nine files, and the context grew by only 999 tokens, right? That's the key thing. So, we did nine searches, and as you can see, those searches all have around two to three K in context, but we were able to basically run the whole framework and only grow our context by 999. That's the kind of power of middleware as well. So, what I'm going to now show you is, okay, this is how you can do what we've just done, but using a pre-built middleware. So, again, you should understand how to build middleware by yourself, but you should also know when, hey, this this framework that I'm using actually has something very close to the summarize to the middleware that I need, so maybe I can just use that. So, what we're going to use here is the summarization middleware that comes out of deep agents. So, again, we're going to do exactly the same thing. We're going to basically you're going to what you're going to see is that we're going to run the the deep agents with the summarization middleware. So, as you can see here, this is basically coming from the um from the actual framework. So, you can see here from middleware summarization import this, and then you can see I'm importing the state back end, and then all I'm doing here is instead of passing through this custom tool, is I'm just simply invoking the deep agent with the middleware, which is the summarization middleware that we've that we've added here as well. So, if that is run, yes, it has. So, you can see here, the the the So, it the model didn't need to compact here, but you can actually see exactly what is happening here, right? So, we're using it with the summarization middleware. It's doing the exact same thing. It's doing it automatically. So, you can see here, so you can see files in state 28, final context 97. So, you can see files 96. So, again, this one, you know, even though the the context is still big, it it as you grow this, you can see for sure how this would how this would this would scale. And again, you can see that we haven't had to call the compacting of context, but we're actually going to do that into the next step as well. So, the next thing that I want to talk about is automatic compression. But before I do that, I'm just going to kind of go into my Q&A's and try and answer a few of them as well. How is this How is this different from invoking Claude compact before start of every task? So, you can so So, Claude compact is is is very similar to what's happening here. And again, you've got two options about so about when to invoke context management, right? So, you can invoke it yourself. So, you can have it invoked at any time kind of through those backslash commands that happen in Claude or through any other thing you depending on what framework you're using. So, it's it's very similar to that. What I'm kind of taking you into is what's going on behind the scenes when it's actually running it and how it's running it as well. So, in a pre and post hook, you can basically run it automatically. But what you find with middleware is that most middleware is invoked automatically for you at some sort of threshold one or you can allow the user to do it at the beginning of every task so that they can do it themselves as well. So, it's very similar to answer your question. Cool. So, what we're going to do now is we're going to do like automatic compression. So, let's just kind of run this code. So, if you can see here, we're going to do this kind of tool fetch huge doc, right? So, we're going to get this huge doc. We're going to basically call an agent that will actually trigger this. So, we know that we are we are forcing we are forcing our deep agent to actually trigger its contact compression software. So, you could actually see here that I'm going to basically pass in this huge doc. Um I'm going to pass in this system prompt, and I'm going to pass in the model. So, I'm going to inspect what um what the files that were auto-created. So, what deep agent will do is it's going to auto-create, very much like Claude does, it will auto-create a file that it will then point to and offload to and tell the model, "Hey, I've done this, but now um basically you can look to this file if you want to see it." So, here, here you go. Right. So, you can see the size of the tool call is massive, right? 129,000 characters. Now, what you can see here is what's actually happened. So, I've replaced it to a pointer of the file. So, what the So, you can actually kind of go into into this. So, you can see here, without you having to actually write your own compression, you've basically got that for free. And a lot of frameworks are going to start building in some of these very useful context engineering primitives into their middleware, so you won't have to code them. But, it's good to understand how they work, so in case that you and your company want to extend it, you want to build some middleware that's custom only to you, you can. And you can see here, you can you know, the tool you can see exactly what the LLM was fed in its context. It said, "Tool result too large, therefore the result of this tool call identifier was saved at this path." So, now the model knows that, "Hey, if if the follow-on question or your follow-on reasoning means that you need to relook at this tool result, this is where you can find it." And again, this is where a lot of context engineering is moving to, especially with these huge complex systems is, "Hey, here is a pointer to said file or said or said tool result or said JSON, you can find it here if you want to. Here is the file as well." So, then that makes life very, very simple. And again, the key thing is you keep your contacts clean. So, what we're actually going to do now is we're going to talk about the last um section, which is sub-agent delegation. So, what we're going to do here is we're going to create a supervisor. Here's the supervisor prompt and a research sub-agent. So, the research sub-agent says, you do focus research, you offload where you can, you know, return a two to three-sentence summary. The supervisor basically writes to do's, um delegate the task to to whoever the sub-agent is, which is in this case the researcher, and then, you know, keep your your contact stays clean, sub-agents handle the noise, right? So, this is very much, you know, the the sub- the the supervisor and sub-agent pattern which I was talking about, which is how a lot of complex tools are now being managed. So, [snorts] you basically design your system, so you have some very common, you know, sub-agent types, so maybe a researcher, a planner, or an executor. And then, you they get called at any point in your flow where you need some deeper information, you want to keep the the uh coord- the supervisor focused on coordinating and making sure that the kind of horizon of a complex task is all that it's focusing on, and it's handing off specific tasks to the sub-agents as well. So, this is again, this is super key, and it's quite easy to do in terms of uh managing this with a uh with a package like Deep Agent or any of the other ones um that are being used for agent development. Okay, so the originals provide the originals preserved on a file system when the agent or on the age on Yes, yes, so the originals are preserved. Now, this is to be clear, this is a virtual file system, right? So, this is where the cool things A lot of agent frameworks, they will start you off with a virtual file system. So, this is a virtual file system and will persist to the end of the session. However, what you most people do is that when you build these virtual file systems, it's very easy to sub them in for an actual file system. So, like a Redis or Postgres or something like a MongoDB that will actually securely store this data for a user for a thread and isolate it. And so, when the agent actually wants to revisit them, we give them, you know, the tools so that it's they're they're able to go to that um store and get the information. So, yes, it is preserved on the system either virtually um per session or if you want to persist it further in production, for instance, you would have a an actual um proper store. So, the next question is, after offloading context to a file pointer and read it when necessary, can So, yes, so can the context be literally cleaned up? Yes. So, you can actually you can make it so that it's, you know, you can you can again through middleware again, you can basically clean up your um tool calls or your the those files. You can basically maybe go through and have a middleware that cleans context and things like that. It would be very similar to kind of context compression but would go through your through your um through your data store and say, "Hey, we don't think these tool calls are relevant anymore. Let's just get rid of them." Um So, if so, would that really help in context management with regard to cost? So, again, re if if all that the in the context management, once we offload the context, right? It's costing us less in terms of tokens, right? So, if that context lives on say a file system and I'm not referring to it um every time I do an agent call, then the that's the main cost saving, right? So, that it's offloading it, that's the cost saving. Now, of course, if I then remove that context off my file system, then of course, I, you know, I I guess I save some space, but the real money saving is saving the token. So, it's offloading the context to either a virtual file system or like a a store. That's the saving because it's not in my context. I'm not, you know, resending the same code token cost. I'm not paying for those input tokens. So, when you say delete them from your file store, if you wanted to, you don't get that much of a saving. But, of course, it's cleaner. You don't have your, you know, file system ex- you know, expanding massively. And again, this if you're in production with, say, 20,000 users, then that cost would, you know, soar, right? If you have, you know, users saving all of their web searches, suddenly your your space is going to soar. So, you probably want to manage it then, and that's where I think cost would start com- coming into it. So, to perform every action could be right read process. We have to give write a tool and give this to uh them. So, I don't I'm not sure I understand that question. So, the the LLM already has access to a file system, and that's part of the deep agents. If you were here in the previous sessions, you can basically have some primitives, and they are able to be able to be run using using that process. So, so yeah, that's you do the LLM does have access to that, just to be clear. Okay. So, I see that we're getting close to time here, so I'm going to go on. So, as you can see here, you can see exactly what's happened, right? You can see that we've had 12,991, and we've actually offloaded 153,000 tokens. So, this is a huge deal, right? So, you can see here, these are the files that were collected, and you can see exactly in the workflow what happened. It called the to-dos, and then what you'll see here is it you can see it's actually delegating each of these to the sub agent. So, it actually each of these tasks go to a sub agent. So, this is super super powerful, right? So, you can see that it's gone and it's saved its information, and you can see that it's done a search and offload, and you can see the tool output. It's calling that and things like that. So, this is again, if there's one graph you should take, it's basically what happens when you start offloading your context to your sub agents. It is a huge save, and I think it's super clean, and it allows your your main agent to stay focused. Um and so, that's kind of what I wanted to cover in the in the in the um Jupiter notebook. Uh so, you can again, feel free to to use this on your own and go through those as well. So, again, these are some of the production context patterns that you're seeing. People have mentioned it, auto compaction, um Manas has a KB cache, you've got code-based reindexing, and you've got kind of like LangChain's kind of you know, what to compress and isolating heavy uh processing in sub agents, which is what we've used today. Um and again, the key takeaways that I kind of want that context engineering is probably the number one lever in how well complex agentic systems are running. Ones that do it excellently are really diligent with context are the ones that are winning in really large production context uh um uh deployments even. So, we talked about the context types, we talked about um compression can basically prevent token limit failures, and you know, basically um attention degradation as well. Middleware's how we deploy these kind of um context management tools, and it can add observe observability and guardrails. And then, lastly, that sub sub agent delegation is a context strategy, and it's probably one of the most effective in terms of really cleanly isolating. So, there's managing the context at the supervisor level, at the agent level, but in terms of strategically across your agents, making sure that your your sub agents have the that that context is really key. Um so, that's kind of what I wanted to cover. So, I know I've got a few questions. So, I'm just going to stop sharing and kind of get back to some of those uh questions as well. So, da da da da da So, let me see. Is using skills instead of having all the MCP tools. Yes, that is that is actually a context management strategy. So, using skills and saying to the agent, "Hey, only use these skills when you when you feel the need to." is a strategy because then you can actually make sure that you Yeah, you're not loading all those tools as well. So, MCP, it will load all the tools and that's why I think it's a very heavy strategy and people do That's why skills have really taken off because they do allow you to manage your context in a in a much cleaner way there as well. Um so, let me go to the Q&A and I'll try and answer a few of those questions. Um so, so, will compressing cause um any [clears throat] important information to be lost? Of course, I think that is definitely That is the That is one of the key skills is making sure that your compression algorithm is as good as possible. So, I think making sure that you lose as little uh important information in compression is the game. So, if you can win that game, your agents will be doing really well. Um is it possible but to answer the question, there's always a risk that important information can be lost without a doubt and I think it's managing that risk as best you can for sure. Um is it possible to run sub-agents in parallel? Yes, you can um and this is async and cool code does it. I think Deep Agents does it as part of their recent release. So, yeah, and you should always want to do this for complex tasks. Let me give you an example, the web search I did, I don't want my sub-agents to go and run those web searches in sequentially, right? There's no point. They know their queries, they should all go out at the same time and get me the information in parallel and come back. So, of course, if there's a dependency, then of course, you'll need to wait for stuff but tasks that should can be paralyzed should be paralyzed as well. Um cool. So, I know that we're at time. Um so, I'll I'll pass back to the folks at Data Dojo but yeah, I hope you enjoyed the session and yeah, please tune in for the next one and follow the um the notebook and yeah, uh reach out to me on LinkedIn if you've got any other questions. Thank you.