Multi-agent context management - all about AGENTS.md | LLM context engineering bootcamp | Lecture 5

Key Takeaways

just confirm if uh my voice is audible, you can hear me, you can see the screen, all these three things. Okay, thank you so much. So, good morning or evening everyone depending on where you are and uh let's get started with uh session five. So, in the last week we discussed about six layers of context. We discussed about um what is the system prompt constituting. We discussed about rag in detail and also MCP model context protocol. Today we'll be primarily looking at uh two things. One is the compress part of the WIC or WSCI architecture. W is a right s uh then isolate select compress. Uh so these are the four parts. Select part is rag. Write is creating the memory persistent memory or short-term memory also creating scratch pads uh using task.md and compress is something where you have a context window which is getting populated as you keep using the large language model and then you want to retain certain information and you want to remove noise. So compress is an operation which can b basically do that and in cloud code for example the way it is done is using the / compact command and isolate is something which you'll be very interested to learn about because here we will discuss about sub aents. So if you have ever used cloud code or any other agentic uh coding platform you will see that sometimes it will automatically spin sub aents for different tasks. Sometimes the tasks are in series, they are serial tasks and sometimes they are parallel tasks and you know that it speeds up the execution by a lot. So we will discuss how exactly is context window constructed in these cases. What happens to the context window when you compress? Uh is there loss of information? If yes, how do you mitigate it or do you even want to mitigate it? And if you are isolating the context what does that mean uh in terms of what happens to sub aents. So sub aents have isolated context and some part of the context is alo also shared. So we will see how uh from a from the perspective of context window how these concepts um work. So just to reiterate a right part is something where we decide or we tell the model that if you are working on a subtask create uh individual task MD files and these task MD files will have to-do nodes and those to-do nodes will make sure that for each task the model is at any given point of time focusing on the subtask. So if the model is at the same time building front end and back end separately or let's say you are conducting you are working on a research agent where one agent is running certain certain experiments another agent is running some um uh literature review or finding the gaps in literature or something like that. Then they can have their own short-term memory. They can have their own subtasks and then those tasks will be finished separately independently. And select layer is selecting what you need into the context window from outside the context window. So from outside meaning it can be from a database vector database. It can be from a just a file system and retrieval augmented generation selects things out out of the context window to be considered to the context window. And uh in the WSCI framework compress and isolate is what we will be looking at today. Okay. And uh this isolate part is where we will discuss about sub aents. And this is something that we have discussed many times. The reason why we want to compress is um twofold. One is when you are compressing the uh in the next round of iteration the agent is getting smaller number of tokens in the API call and um oh the screen is blank. Do you all all of you are seeing the blank screen? Can anyone confirm if you see the screen or if you're okay thanks so uh so what I was saying was the reason why you want to compress the context is because you need to uh reduce the number of tokens that are going um into each LLM API call. So let's let's say I have a question. Okay, this we will discuss in detail after a little bit. But basically say we have um one agent and just let me just uh stop some background processes. My computer is becoming a bit slow. Just give me a second. All right. So I hope you everyone should be able to see the screen now. So you have one agent and let's say this agent is um making a API call. Can you tell me the cost of your large language model usage? What is it dependent on? Is it dependent on the number of API calls you make or is it dependent on the number of tokens which you consume? Let's say to one LLM you make 300 API calls to another one you uh make 20 API calls. All right. In the first case you consume let's say 10,000 tokens. In the second case you consume 200k tokens. In which case do you think you end up paying more? you're basically um build based on token consumption um for for input as well as output. So it doesn't matter if an agent is spinning sub aents who can make their own separate API calls as long as you are managing your total um token consumption in the form of an optimized uh context window really well. So it doesn't matter that you give your um sub aents ability to make its own API calls as long as the context window for that sub agent is very minimized uh it's it minimum number of tokens. So for two reasons you need to compress. One is of course to minimize cost. Second is to minimize the rot of the context because imagine you have only one agent for doing every single task. That agent will basically let me just uh allow certain people into the meeting room. Yeah. So if there was only one agent doing all the tasks for a project, what will essentially happen is that agent will uh eventually experience context ro. So beyond certain point uh there is too much information in the context window to process. Right? This we have discussed a lot of times. So there are largely four types of compression. First type of compression is the most obvious one or probably the most powerful one. uh not necessarily the most cost uh optimized one but it is summarizing the context window using a large language model itself. So what you could do is uh so one of the most popular context compacting or context summarization uh methods that is used in cloud code is LLM based summarization itself. So you make another API call to which you pass the context but ask the model to summarize the context into certain number of tokens. So if the context window size is 200k tokens uh and you have reached let's say 80k or 100k tokens already you could ask / compact you could run the operation / compact and then the context can be compacted to much smaller size. Often these uh coding agents also have 80%age context window uh limit where the if the context window reaches more than 80%age of its total limit it will start an autoco compacting operation and which will be based on LLM and which will be much efficient but sometimes if let's say in the next output in the next iteration of API call the output created is so big that context window exceeds 95%age of its capacity even more aggressive trimming will happen meaning some parts of the context window will be aggressively deleted. All right. So we will discuss this also in a interactive way when we are getting into the lab exercise. Uh do we have any easy way to calculate tokens for large uh context? Yes. Yes. So at the end of the day context will be uh having two types of things. Structured inputs from u JSON files or other file formats and natural language inputs. on an average um one word is equal to one token. So that is one way to see um I mean for hand calculating if you want to hand calculate how many tokens have you consumed just count the number of words otherwise charg itself has a context um um token size calculator. You can just Google charg or openai uh token length calculator. They provide a window into which you can paste whatever input you have or whatever text or whatever your context window looks like that you can copy paste and it will tell you exactly how many tokens are you at right now. And in cloud usually the context window size is 200k tokens which means if you are at around 160k tokens some you are in in some sort of not yet in the danger zone but you are at a at a zone where you would want to perform some compaction. So the way I work is in cloud code the auto compaction happens whenever the tokens have reached 80%. Meaning if it reaches around 160k tokens you will see that auto compaction automatically starts which is based on LLM. So uh it will make another API call and that API call will um semantically compress the current context into much smaller section. I will show you the details of what are we compressing. There are tool results. There are system prompts. what exactly do we compress and what exactly do we not compress and what are the after effects of this that we will see uh but often in my execution what I do is I do two things one is if I'm completely switching from task A to task B where I don't want any relation I don't want to be any memory from task A task A to carry to be carried forward to task B I just put the operation / clear and I have noticed in many type of tasks this provides significant improvement in the output quality. Especially if your quality you observe is to be degrading, you have to do a /clear operation. If there is a clear distinction between two task and they are not they are not sharing anything in common. However, if the task is slightly distinct and you think that lot of the things that are accumulated in the context window so far is because you have been carrying out a large like a long operation and the context window is having a lot of toolled results. Let's say you were running uh paper banana to generate images and you are having 17 images to generate and then in these images basically you have provided the descriptions of images but then by the time you are reaching for 15th or 16th image your context is so bad that the sort of uh images that are generated are very bad then at that point you can decide okay I don't want to lose the overall context just do / compact and then um continue generating and you will see a significant step function like a step function the improvement in quality of the output. So first type of compression is summarization. Second is called tool result clearing. Can anyone tell me what is tool result clearing? And why can we simply clear the results of a tool call? Why is not why is it not problematic or is it problematic? If yes, why? If not, why not? So and in simple terms first okay let's let's go step step by step what exactly is tool result you can refer to the discussion we had in the uh last lecture right so a simple example is let's say the MCP server is sending some sort of structured result as a result of operate uh as the result of invoking of a function. So a tool you can imagine to be a function or it can be an API call which executes certain function or fetches something from the database. At the end of the day there is some input going into the tool. Tool is providing you some output. Output might be in a structured JSON format or in any other format. Now you might notice that in many cases this output will have so many things. It will have just in time instructions. It may or may not have verbose results like which is like very detailed because often LLM for it to use the tool results it need description of structure of the result uh plus the result itself. So it will be very big document. Now think about this is the result of a tool execution is it stochastic or is it deterministic? Like if you run a function with a certain arguments, okay, do you expect the same result with if the arguments are the same? Typically functions are such that the result of the function is dependent on only one thing which is the input or arguments going in which means there is great deal of determinis deterministic nature when it comes to results. So uh of course it depends on the functions whether function itself has a stoastic nature but in general the functions if you want to mimic the results of the function which you got earlier and if you are in such a dire situation that your LLM context window really needs the result from a particular tool it can simply run the tool uh call once again right so there is no need of storing the last previous tool called results in the context window continuously you could simply simply entirely delete the tool result or you can simply compact it into few sentences. The tool result which may be in multiple lines maybe in JSON format can be compacted into simple natural language two or three sentence summary. In some cases, if you are aggressively trimming, which is what happens in a concept called priority based trimming, the tool results have one of the it's of one of the lowest priorities, which means true true uh tool results will be aggressively trimmed. Trim meaning entirely cut away. It will not be part of the context. But cloud code does not use this that often. The most common two compression methods used by cloud code is um LLM based compaction and tool result clearing. We will see this with an example how it how it visually looks like uh how the context window visually changes when we um you know uh clear the tool result. And the last one is so in priority based trimming we just assign um levels of priority P 0, P1, P2, P3 etc. P3 has P3 is something with the lowest priority and P 0 is something with highest priority. Highest priority meaning it will be the one to go last. It will be trimmed the last. Now based on all the things you know what could be P 0. So P 0 is um let's say you have examples a conversation history um system prompt then tool result then maybe uh some custom instructions for the agent. What could be P 0? So one of the highest priority items by default is system prompt. System prompt is the last to go. System prompt will often have rules, right? So system prompt is definitely not P3. What could be an example of P3 other than tool results? So tool results, yes, it can be it can be of low priority. What else could be P3? So we can define P3 in the following way. Um for example, let's say we had a conversation history from many context windows uh not context window many conversation uh windows before. So if we define one conversation window to have x number of back and forths, a conversation which happened much before can be significantly summarized or completely trimmed away to preserve the context. If it comes to trimming, by the way, trimming does not happen in cloud code unless you are at 95%age of your context window filling. If only if context window has filled 95%age, this will happen. Otherwise, trimming is not done. Okay. Can you give me one more example of something other than system prompt which you may not trim uh that I will come to the in this hierarchical compression. Um so in hierarchical compression using semantics you will decide what has the highest priority and what has the lowest priority and whatever is the highest priority will not be removed. Whatever is the lowest priority will be removed. So in hierarchical compression to decide the hierarchy you are using semantic meanings. Here in priority based trimming you are using rules to decide what comes in P 0, P1, P2, P3 etc. So can you tell me something that can be at the highest level of hierarchy meaning something that will be removed last which is not system prompt. Okay. Many people are saying user prompt. Okay. Uh for sure user prompt is something um valid. You cannot delete user prompt just like that. But my question is let's say user gave a prompt lm gave an output for example and that output is now add output tokens are now added to the context window. What type of output might come at the highest level of hierarchy? something which which you should not delete or something like that. It could be something like an output where after this output the user is supposed to give a preference based on the output. Right? So for example let's say you are building a website and your phase one is being executed. In phase one, you have told that you have explicitly told the coding agent that at any point if you feel like you need my input on um the user interface or user experience or the way you have to arrange your um cards in the front end or something like that and let's say the LLM is supposed to receive an input from the user after it produced its latest output. So LLM might have produced a um let's say a skeletal structure. Now to add meat to it, a user instruction needs to be provided. And for user instruction to be provided, uh it is very important that the LLM retains the immediate previous output. Right? So that immediate previous output which the user has not yet seen or the user is supposed to comment something on should be kept while doing compression. And how does the LLM know whether something is of such high relevance or something is of not high relevance? Simply by making another LLM call, another API call in that another API call. Semantic meaning can be associated to different parts of the current uh context window. Based on the semantic meaning, you decide the hierarchy and what you do is if uh you are at the top of the hierarchy, 100%age of the items are retained. If you are in the next level of hierarchy, 70% of the items are retained. If you are in the next level, 50% are retained. If you are in the last level, only 10% is retained. So if only 10%age is retained, this could be most likely tool tool called results. So if you are compressing your tool called results into 10 percentage, it simply means your tool result will become a one or two uh sentence summary. All right? So this we discussed already. So in LLM summarization if you have a long conversation history between the user and the um uh assistant you simply convert it into natural language summary. So structure might be lost. Some nuances may or may not be captured. But the goal here is most of the information or most of the facts which are there in this conversation should be ideally captured. And there is something called IRR information retention ratio which is if you decide that if you can say out in this initial context there are 30 information uh which are which are which can be mentioned in bullet points which should not be removed. If all the 30 can be recovered even after compression then the IR ratio is 100%age. If only 90% can be return retained so is the IR. So then basically it's a lossy compression right compression is always lossy sometimes the loss that happens is something that is not relevant many times you want to make sure that the facts which are quantitative things are retained uh in tool result clearance this is what I showed tool result will take you from large number of tokens being consumed to very very few number of tokens being consumed this we can uh visualize when we come to the implementation part and this is the most common thing which uh even if you are not using cloud code even if If you're constructing a coding agent yourself, u one thing you should define is how tool results are cleared or compacted. Um this we already discussed uh when to remove which are of highest priority. So system prompt is of highest priority never removed. Core instructions to your let's say which is there in the root folder agent.mmd file uh for the agent instructor agent orchestrator or core instructions for the uh which are defined in the um cloudmd file. they are not removed. Tool definitions which may be part of cloudmd file if the number of tools are less or uh tool definitions that are currently living because rag fetched the tool definitions from your database are also again not removed. Uh there is a question we are not counting thinking tokens at all because thinking tokens are not at all included in the context like they are just part of test time uh scaling but not really included in the context. You mean in reasoning models right? Um so one thing we can do is yeah uh so yes uh so typically thinking tokens my to my understanding it is also included in this context window because at the end of the day uh if it is a chain of thought for chain of thought to propagate you have to have the previous uh output which was produced as part of chain of thought to be added in the context. Um but one more thing you can do is uh in the last week we had a brief discussion last week I mean last lecture or last to last lecture we had a discussion about should you always have reasoning based large language models for uh react to be implemented there it's not really necessary that you need reasoning based large language models you can invoke behavior similar to reasoning with non-reasoning LLMs also just by relaying its output in the form of reasoning. So you you force the LLM to come up with a reason then coming up with an output uh coming up with a user output um and then go three or four loops of that before giving the final uh output. Is there any lecture around it? You mean a lecture around uh reasoning is itself? uh we don't have a specific lecture around reasoning but the basic idea is that the tokens that are consumed uh during reasoning tokens that are generated in the output as part of reasoning is also consuming the context window for sure um then yeah so the current okay first of all let's see this older history this could be part of conversation or part of uh let's say summary of the task that are completed you don't need tool results Stale tool results mean tool results which were produced produced earlier which is not used useful anymore not needed. This compress if needed is this is very very this is not done always. This is only in extreme urgent cases where you actually have to trim some uh recent conversation also. So you will clear the recent conversation and recent tool results only if you absolutely have no space in the context window. Right? Older conversation history and older tool results are cleared first. Then current task context, current user query and current action plan. These things are again having very high priority which means this will not be removed. System prompt is always kept. Um then in hierarchical compression this is what I said hierarchy is decided based on um semantics. So if hierarchy is of the highest 100% is preserved. If it is important 50%, um and if it is in the lower part of hierarchy 10 percentage or only 0 percentage is um kept um this I have already discussed. Okay. Is this how coding agents are designed today like cloud code? Yes, exactly. So in coding agents you don't have to define how compression works because these principles are already inbuilt inside the coding agents. However, coding agents are at the end of the day these instructions wrapped around an already existing LLM, right? So already existing LLM is Opus 4.6 for example. Around that these things are built and that is how cloud code is built. However, if you are using openclaw, you might have noticed that when you install openclaw, they give a long set of um available LLM options. So in openclaw when Pete Steinber Steinberger was building the open claw framework he has followed many things that are mentioned over here. However the only difference is there it's not primarily a coding agent it's primarily a uh agent which can act as a personalized assistance assistant. Um yeah yeah yeah. So hierarchy based assistance uh trimming is basically this inside the context window instead of saying hey tool results have lowest priority system prompt have has highest priority or old conversations have low priority new conversations have high priority instead of explicitly saying that you let an LLM figure out based on the semantics of what is there in the current context decide which part of the context is highly relevant. which part of the context is highly not relevant. In the previous one which was um trimming based on priority, there is no LLM involved in deciding what is recent, what is relevant etc. Relevancy is decided based on the structure itself. So in the structure if there are tool results which has a time stamp which is more recent you can decide a cutff tool results from the um last 10 invocions of the tool invokings of the tool are not part of the uh are part of the context window everything before that is thrown out of the context window. So you are deciding a cutff based on just some logic which is predefined. However in hierarchy based compression it's not trimming by the way it's compression. So you are not entirely removing nothing much but you decide that this is the hierarchy of different sections of context. If something is at the highest part of hierarchy, you don't trim it. You keep it 100%age. As you go down in hierarchy, you assign lesser and lesser importance, which means you summarize it more and more. So higher and higher compressions are applied. So finally when it comes to the lowest level of hierarchy, you may compress it to 5 percentage or 0 percentage so that almost nothing is kept. Maybe a one sentence summary is remaining and as you go up the hierarchy you will see everything is retained pretty much as such. Um so one more thing is when we are performing compression um it is expensive because you can't keep on compressing. See if you want a clean context window you could ask then why can't I keep on compressing every single time? Why do you why do you not do that? Why do you not compress the moment? Let's say you feel like your context is growing. Why do you wait for 80%age cut off to happen or 95%age to happen? Why not always compact or why not always clear the tool results? Always compacting is anyways not good. You will compact all by definition it means you are losing some information but at the same time you have to call the LLM you have to call some sort of LLM for for most common type of compaction which is LLM based summarization and that is expensive right so now imagine it'll be double as expensive because your context window is 200k tokens you are then calling the comp um LLM for compaction which means another 200k tokens are consumed And that LLM will provide you some output maybe that is like 10k tokens that 10k tokens are now used to represent your original context. So apart from your original 200k tokens which were part of the context let me just draw it. So let's say this is your original context and maybe so this is 200k and only 100k is consumed so far. Now you make another API call to the same LLM or to a different LLM depending on your how your structure is. Into that LLM you will pass the same context and provide instruction. So instruction will be maybe this much on how to summarize and then that LLM will produce the summarized version of this. Right? Summarized version of this may be only these many tokens. Now look at the cost associated with this to compress this much. You are consuming so many tokens. And then what you could do is then what you will do is this compressed context window will replace this 200k. Now this uh if 200k out of this 100k is consumed. Now this compressed section this will replace our context window right now. So yes you compressed by 90%age. You went from 100k tokens to 10k tokens but at a cost. The cost is you ended up consuming these many tokens in a separate API call plus you will introduce of course as somebody mentioned in the chat latency because uh each time you make this API call you're waiting for uh the results to come so that you can replace the results with this uh in this compacted uh context window. So you don't use this very very often. Um okay very good question about how do you choose the best model for uh compression. So typically for uh compression you don't need a uh a proper coding based model. You just need a model that is good at um language in general because llm based compression is basically compressing most of the things into uh summary sentences. So it needs to be really good at natural language. So even if you are using cloud uh code I don't know how it internally does this compression whether the model is different or not that I have to check but uh if you were to implement something like this manually you can use your best model for dealing with your original context window and uh for compression you can use a lightweight large language model so that at least API call you can minimize latency will still be there. Um sometimes you can also compress based on uh some sort of uh keyword based extraction. So you can uh from this context based on keyword you could extract the required portions and that required portions to be converted into a meaningful format. You can pass it into a separate API call which means you are not consuming like 120k tokens or something like that. you may consume like 30k tokens and then that can be compressed into 10k. Um then so this is the idea of compaction. But there is one more concept that we need to discuss apart from compacting the context which is splitting the context across different agents who are performing different functions. Let me just check if there anybody is waiting in the waiting room. All right. So in cloud code we don't have to mention explicitly what the compression technique is. By default it will be using LLM based compaction and tool result clearing based compaction. Um those are the most commonly used. You don't have to explicitly say however for whatever reason if you want cloud code to use different type of compaction because you you would say hey you are an auto research large language model. Actually why don't I show you something? Let me let me open my um Visual Studio Code. Just give me a second. I'll stop share screen uh sharing the screen for a moment. All right. So, do do you see my Visual Studio Code right now? Uh, just if you can confirm with yes, that'll be great. Are you seeing Mirroboard or Visual Studio Code? Okay. Okay. Thanks. So basically what you are seeing right now is one of my workspaces where I am using a continuous research agent uh to explore one problem statements that we are working on at the moment on vision language action models. Okay. So this agent I will just show you this agent. It was uh released by Andre Karpati uh in last week uh it's called auto research. So basically what this agent does is and this has blown up by the way in terms of uh number of stars and all last time it was 30k in one day it has gained 6k 7k more stars but basically what this agent does is so so he shows in this graph agent will conduct research and it will figure out how what kind of experiments to done what kind do to do what kind of model architecture changes to bring to better your results. So I just wanted to see if we are running some experiments if we ask this auto research agent to conduct our own research conduct its own set of experiments will the you know accuracy or will the metrics that we are tracing uh will it keep on improving and uh what will happen to the expenses etc. So here one comment which I had provided explicitly to cloud code while defining the its purpose was uh I am worried that you know my uh context window might rot because uh you are working on so many different um uh tasks one after the other sequentially uh in the process of running these experiments and writing the paper and uh because of that I'm worried that you may even have halluc hallucinate some results or hallucinate some experiments. So then what I did was this I explicitly mentioned like whenever you decide to do a new set of experiments where you feel like um the previous set of experiments are a complete in a completely different sandbox and the new experiment can be considered to be a different sandbox even if your context window is not bloated yet to 80%age just make sure that you are doing an auto compaction. So with this because of this in none of the sessions although this was running 24/7 for last so I started this experiment on uh sometime on Saturday around 2 p.m. or something like that although this was running 24/7 for the last almost 2 days never ever it ran out of the session uh limit. So as you know if you have used cloud code it has a session uh session based um usage limit and then after the session it will uh refresh but that never happened with this and I was keeping on and I had asked it to push it to my GitHub. So it was keeping on pushing the experimental results to GitHub and I was tracking the experimental results on GitHub and it not even once it stopped. Uh one way in which this could have stopped is I had given it my runpod access through runpod AP uh this access token I think API not access token and then uh I had around $100 in the run pod um uh um I had uh recharge run credits only if the $100 ran out for whatever reason it would have stopped otherwise due to context issue it would never stop or it would never rot. So yes, I was on to I am on a $200 uh cloud plan because I use it very much extensively for building products and running different experiments plus multiple people in my team use the same account. Um so we have total three accounts which are maybe used by seven or eight different people. So if my credits run out for a session, I just switch the account. So that's what I do. It's not ideal but that's the only option right now available. So what is done is if you have a big task a big project there will be lot of files which are actually relevant all those files obviously you cannot fit in the context and this is where you define sub aents by giving isolated um context to these agents. Um yeah yeah so how did you ensure context did not blow up? I simply gave an explicit instruction. The explicit instruction was that um whenever you are switching task whenever you are going from one experiments to another set of experiments even if the context is has not reached 80%age you compress the context or run/ compact operation. So this will ensure that when I'm running from one experiment to another all the baggage from the previous experiments are not carried forward. uh this will significantly keep my contacts clean plus it will also keep my price a little bit low price or token consumption a little bit low. I mean there are many different ways to do this by the way but this is what I tried cuz I was so worried that this may not run 24/7 simply because session usage might exceed because that has happened many times but uh not even once it did exceed the session although it was continuously running experiments. One thing might be because the experiments are actually run on runpod GPU and not on uh not by cloud code itself locally. That could that is one reason of course why too much consumption is not there. On the other hand, if you are building a website or something all the files uh node modules packages you are literally defining locally. So lot of lot more tokens might be constructed in let's say web development compared to in actual research because in actual research you almost never run your code in local Python. So the the visual studio screen which you saw is not my computer. It is uh an uh AWS EC2 instance uh with some 32GB RAM or something which I purchased because I needed RAM because I need to run multiple agents on that same computer. So the computer should not slow down but at the same time many agents do not run the experiments like building website or running Python script on EC2 itself. So I have connected the EC2 with runpod. So all the all such heavy experiments are running on runpod GPUs. So because of that reason one my Amazon AW uh my AWS EC2 instance does not slow down. Second my token consumption of cloud code is also not that high. But that's an artifact of what type of experiments I am running. But if you are running some experiments where you are running Python script locally in EC2 instance or your Mac mini or whatever device you use or your computer uh I'm 100% sure after some time your you know if you are running 24/7 experiments your uh session memory will run out. I don't I forgot what is the total usage total number of tokens that you can very easily see by typing slash usage. in slash usage they will see a bar show a bar whenever usage is about to exceed the session uh the session limit. Um then okay let me ask you a question uh this I think many of you may have doubt about if we um okay I'll answer one more question did so did cloud code figure out how to use runot API so that you can steer it from your EC2 project no cloud code did not figure out automatically I mean API key of course I gave cloud code but other than that cloud code it's it's very basic uh cloud code already um figuring out how to run run pod using its API is a very trivial thing. Uh I don't know if it has MCP or just API uh calling is enough. I don't think it's MCP. It's just API key based. So that part was very easy. You can just give the cloud code access to your run pod and ask it to execute the code. Store the results and keep the inference uh uh those things keep it on EC2 instance. Uh run pod you initiate the run the pod on run pod run the experiment. Once the experiment is done, once no im immediate experiments are uh planned, just close it, terminate the pod so that you are not accumulating cost. Uh one day it so happened that I was not using cloud code and I was manually running run pod instances. Uh and then what I forgot to terminate it. Then uh in two days we had some I think $150 or $200 bill for nothing. I mean we were not running any experiments at all because there is some idling cost also for storage. So if it goes for certain number of hours by default I think it's8 per per hour or something like that um for most of the pods but for expensive pods for more storage it'll be higher so even if you are not running any GPU instances any at any given point you will still accumulate some cost. So if you are not running anything you have to terminate it or pause it. So um so what I have done these days is I use the pod as I initiate the pod as and when I need uh for new experiments I so I tell cloud code one of your tasks is to minimize my uh run pod bill. So for that what and but at the same time it should not impact the quality of your experimental results because if you need to use the same pod again you can use that but make sure that you are minimizing my bill but if you don't have to keep a pod in idle state for 1 hour or 2 hour if that is what is needed in that case you terminate the pod you restart a new pod uh when your new experiment has to start but whatever results you have from pod number one store it in the EC2 instance so that it can be passed if needed into the um run pod now instance number two. So these are some instruction I have given. I don't think such extensive instructions are needed. I just gave this as a safety. But if you want to just run the run pod with minimum cost, you can just say that in simple plain English and cloud will most likely take care of it. Uh last week I remember one question was if you have two agents agent number one and agent number two. Okay, let's say cloud code is spinning two sub aents, agent one and two. According to your current understanding, what happens? Do these agents access the same LLM and same context window or do they make separate API calls or and they have separate context windows? Plus one more one more question is do they share anything in the context window? Is is some part of the context window of agent one also accessible to agent 2 directly like is there is there some way of sharing the exact same tokens which agent one is using and agent which agent 2 can also use. So ultimately all the sharing happens through agent orchestrator. The individual agents do not share context window directly. Why? Because they are completely separate API calls. So context window of agent number one and context window of agent number two are two different things. However, if for example system prompt needs to be provided to both agents, which means the tokens corresponding to the system prompt which are stored in agent one and tokens corresponding to system prompt which are stored in agent 2 may look very much the same because a tokenizer that is converting system prompt into LLM's tokens will will be applied similarly. uh so let's say the cloud MD file is to be entirely present in agent 1's uh API call as well as agent 2's API call then the way in which that cloud MD is tokenized will be the same because you are using the same LLM same tokenizer so the tokens that represent the system prompt in the agent 1's context window will sort of mimic the tokens that represent the system prompt in agents 2 context window. However, just because they are mimicking each other, it does not mean that this these two agents are sharing it. It may appear like that, but there is no direct sharing. If at all something needs to be shared, what happens is this agent orchestrator will decide what to be fed into the context window of both agent one and agent two apart from system prompt. So if agent one has bunch of results which are given uh stored back into cloud u into the orchestrator's memory and then orchestrator decides that actually that needs to be part of agent 2's context also then orchestrator can share it to agent 2. So I'll just show you one figure uh which you should remember this one. So these agents directly do not communicate via a shared memory but rather an orchestrator in this case you can imagine orchestrator to be cloud code itself agent A and agent B to be sub aents spun by the cloud code and cloud code should decide what files or um what database is shared between agent A and agent B. They don't directly do it. So just please understand this. Now coming back to why do we need sub agents the need for sub agent is very clear. Uh you have different tasks. Uh so if you are working on a research you could have an agent that is whose sole purpose is executing some code Python or JavaScript or some type of code which may consume a large number of tokens. But this this code agent need not let's say review the results of the code which it wrote. Maybe a code agent can write the code, a testing agent can execute the code and test it for various um you know use cases. A review agent can review the results and the code and then this review agent can its only purpose will be it will be provided a code. It will be provided the test case results and then the review agent has to provide a detailed summary uh which consumes maybe 2,000 tokens. So orchestrator agent will also have same context window size 200k tokens but it will not consume 200k tokens. It will consume only 10k tokens. Then it will delegate a task to this research agent. The research agent will consume maybe 40k tokens. This 40k tokens plus 60k plus 30k. So the 100k plus 130k plus 155k 165k. So roughly 165k tokens are consumed but they are not consumed within the uh individual orchestrator agents API call. They are consumed overall. Now this 160k tokens 165k tokens could have been consumed within one single agent. Okay. But then the problem that would have happened is if the same agent is looking at every single thing there is heavy context rot here that is not there because this 10k is now part of 200k. This 60k this 40k is now part of another 200k there is no no chance of context rot code agent another 60k as part of another 200k and the llm does not penalize you for making new context windows llm only care about how much how many tokens are consumed through its input and output so it's always better to consume these tokens in a modular way by making multiple API calls if that is what your application demands so this these deleg Delegates are basically sub agents and if you really notice what the cloud code is often echoing it provides an echo right as it is performing some some operations especially in this auto research I have seen many times it is spinning its own sub agents uh it will say four parallel tasks are running um right now if it is working on four different figures or something and then it will say uh four tasks are running I will give a summary once the task is done when all the all four agents are done uh executing the task So but the total token consumption is again not very significantly high. Uh so this I have uh noticed I have already mentioned then what the orchestrator decides this I will show you in a folder structure rather than uh in the text format. Okay. So let's see this folder structure when there are sub agents cloud.md all the agents ideally will have access to not the it will not pass the entire cloud MD into the context it will pass whatever sections of cloud MD from uh using rag or using keyword matching then what's the difference between this agents MD and this agents MD and this agents MD what what's the difference between these three first let's talk about this one what is this agents.mmd what could this be called as can you compare cloudmd and this agents.mmd they are different this agents.mmd is present in the root of your directory your root directory is my project this agents.md is present in that now maybe the purpose of this agents MD is to help your um cloud code decide whenever you when you need to spawn new sub aents. Now let's say there is a sub agent whose only purpose is to build the website. Now this agents.mmd which is one level below in hierarchy will decide that okay I am an agent who supposed to manage this entire website development task for which there are front- end tasks and backend tasks. Then this agent will decide okay I need to now create front end task. So this front end folder is there inside which there is another agents.mmd. Now this agents.mmd is the lowest level in hierarchy meaning it has only instructions related to maybe the front end schema or whatever the components inside inside the front end. Uh so now the question is let's say for whatever accidental reason you had given some front end related information over here you say I prefer uh a modern front end you are you mention here okay you should not mention but let's say you ended up mentioning but here you give some other instruction here you say the the front end should look like um uh monospaced uh um some sort of um you know legacy style should be followed or something like that what will be followed will this be followed or will this be followed will the root level agents MD's instruction be followed or will the sub agents uh instruction be will be followed so whenever there is a conflicting in instruction always the instruction provided by the child is used the reason is simple the child has more proximity towards the task that is being implemented at the moment so If front end is being developed at at the moment, this agents.m MD is more close to the front end related task. Whereas the agents.mmd on the root level of your working directory is not a front end specific agent. So it may have mentioned a few things but it would not it would not have been specific enough or it would not have been deep enough into one particular task. However, does it mean that the root level agents MD is replaced by the child level agents MD? Is is some sort of replacement happening? If not, if not replacement, what may happen? This we had discussed two lectures before. If it is not replacement, what is happening? What is happening? You can sort of imagine to be some some sort of concatenation. Conflicts are replaced. But otherwise, uh the system prompts which are there in cloud. MD are concatenated with sub aent level instructions and that whole thing is passed into the context window of the particular front- end agent. So here effectively four agents will be there. One agent at the root level, one agent at the web development level, one agent at the test level. Then under the web development there will be two more sub aents. One at the front end level and one at the back end level. This is how the folder structure will look like. So each of these agents will make its own API calls. Now the frequency with which the agents may make API call might differ. Uh so in most of the implementation steps whichever agent is implementing that particular task of front if front end is being built at the moment the frequency of API call of this child level agent will be higher compared to frequency of API call made by the root level agent. Uh so you're saying that if we are placing the code for the agent they will find agents.m MD at these respective levels. Exactly. In cloud code however you don't necessarily have to define this. The reason why I'm showing this to you is so that in case somebody is not planning to use cloud code for whatever reason let's say your company is not allowing to use cloud code but you want to construct agentic framework because agents.md is universal. All LLMs understand what agents.mmd is. Cloud.mmd is by the way not universal that you will have to explicitly define for the LLM unless you are using uh anthropics models but agents.md is a universal framework. So if you are defining it like this then it knows that okay there are four sub aents four agents here out of which two are child agents one is a root agent etc. What is the replacement for cloud.md in codex? uh I forgot the name codex I have been not used in the last uh many uh months but however um if you refer to lecture number I think one or two I had shown a table where you show uh the equivalent version of cloud.md is shown for uh codex but at the end of the day even if you are having the folder file name as cloud.mmd it is not an issue if you explicitly mention to the to the agent that my um system prompt level instructions are primarily mentioned in cloud MD file that if you explicitly mention it will refer to it. If not you have to use the uh default format. Yeah, in copilot it's called c-ilot instructions.mmd. Um in many uh agents there is there is a f file called soul.md. Sol. Um so this is how it works. So if root agents.md is there and in that if there is one instruction this will be taken. So use typescript uh strict mode. Then inside if src folder which is for web development if there is an instruction like follow the coding standards in style.mmd that will be taken. Then if there is a front end specific agent inside the front end folder and if it mentions react use react 19 no class components it'll be mentioned and within the front end let's say you are building components like cards or u models or something it'll have its own sub agent and that sub aent may have another instruction. Now the the context which this sub agent will use will include this plus this plus this plus this provided there are no conflicts there are no conflicting uh instructions. So what it may look like the context may look like something like this. It will have all these four instructions embedded in the context. In the context of which agent in the context of this um child of child or rather uh yeah sort of like child of child of child because this there are four levels of hierarchy here. So this is root this is the coding agent this is the front end agent and this is the component level agent. So this component level agent will have all these four instructions inside its um uh context. Now there is one question you may naturally have. I think one person had already asked this um in two lectures back. What's the primary difference between claw.md and agents.mmd? cloud.mmd is the primary document which you will refer to for your system prompt and this parts of cloud.mmd will always be used irrespective of what API call you are making within that project whether it's a sub aent level API call whether it's a root agent level API call or whether it's not even a uh uh orchestrator API call in at all levels System prompt parts of cloud.mmd which are part of system prompt will be used. Agents MD at the root level is used by the agent orchestrator. Meaning there are tasks which are outside of the agent orchestrator's job. Maybe the the cloud cloud MD the cloud code has to utilize the agent orchestrator for making sure that the front end agent backend agent sub agents all of them are performing his task but may there may be other tasks which are outside of that maybe to make sure that the overall folder structure is correct maybe to make sure that the phases that are being implemented which may include tasks which are not just web development tasks maybe there are other tasks for summarization or what whatnot for that cloud.mmd will be always referred to. So agents.mmd please note the orchestrator will be primarily using the agent and under the orchestrator there will be sub aents under the sub aents there will be even lower level hierarchy agents and all of them will refer to agents.mmd and all of them will naturally will also get access to parts of cloud.md or entire cloud md depending on the context why do I keep saying parts of cloudmd because sometimes cloudmd can be a huge file and you are not passing the entire file into the context that will bloat the context. So you are using keyword matching or rack you will fix you will uh fetch only required parts from the cloud.md. Um now I want to show you a few things but before that let me take you through some exercise and then we will come back to this. So I'll share a link with you. Just open this uh just a second. Okay. I just shared the link. I just want to make sure it's not shared in the waiting room. It's shared in the main Yeah, it's shared in the main room only. Just open this and put your um OpenAI or Gemini API key. Then yeah, collab link also. Let me share in a moment. So collab is very long by the way. So you will end up spending at least 30 minutes on collab. Uh so I will share the collab and please use your API key from the environment. But uh for the exercise purpose today I will be going through the interactive website. Yeah. Yeah. So if you give a prompt saying um produce I mean build this particular component level feature then uh automatically orchestrator will know which sub agent to trigger. Uh because orchestrator the role of the orchestrator is never to um run the code or build things directly. It is to make sure that the people who are supposed to build is building. So I'll give you one more example of my recent workflow which I have started. So for those of you have who have not uh used cloud code or any agentic coding framework know that often the execution will stop unless you very strongly explicitly suggest otherwise and then it will ask you for your opinion and sometimes you want to make sure that it's not blindly executing the task all the way. You want to give your opinion you want to give your review comments and stuff like that. So one of the better work workflows that I have recently tried and it's working very nicely is um on top of cloud code use open claw. So open claw is like a very nice assistant. So I will tell my open claw to use a very cheap um language model uh like uh gemini 2.5 flash or something like that. Oh by the way open claw was so expensive. I tried it with 3.1 uh preview. It was like crazy level expense like for 10 minutes of my open claw usage I ended up the the bill ended up being like $7 $8 or something which was like crazy but if you use a cheaper model Gemini I mean any model by the way um so I switched down to cheaper model in Gemini the expense was 100 or 150th something like that then I will tell my open claw personal agent as my assistant to take care of claw claw code. So I told him open claw gemini should never be used to uh never be used for you know um running any sort of experiment. So open claw's only purpose is to be an orchestrator. Orchestrator for whom? Orchestrator so that it ensures that cloud code is building the project and cloud code is actually doing what it is supposed to do. So whatever output is uh output you have um make sure that open claw checks the output cloud code produced and if it's going in the wrong direction course correct it so there is a question yes is it safe to use open claw so what I did is for my safety I did not install it on my personal computer I installed it on my EC2 instance on EC2 there is no personal data nothing the only thing which openclaw is probably getting exposure to is my API key but API key I have pretty good control over. So um so there there is no issue. Plus I have not connected openclaw to any of my um like calendar or Gmail or slack or anything. I only connected it to my telegram and telegram I have never ever used to text anyone like in the sense like I don't use it at all. So now I'm using telegram as just a tool for communicating to open close. So on my phone right now if I if I switch on telegram I can just uh text open claw as if I'm texting you know um my uh personal assistant then open claw will plan things accordingly and it will ask claude code to execute the task. So my Gemini API will not experience a lot of uh you know cost accumulation but rather uh claude which anyway I have a pro plan there the price will accumulate but yeah but please be you know please very be very careful with open claw especially with respect to price um price is it's it's so expensive especially if you are not careful with which model you use the number of tokens it accumulates it's like for each conversation after like 10 seconds worth of conversation you will see that the token has number of tokens have bloated up. So something is wrong with open claw. Maybe cloud code is not that terrible but open claw with respect to token consumption is so bad from at least from my experience. I don't know if it is universally true. Um so yeah so but that is a good workflow that way open acts as acts as you know if there is any issue like let's say your session uh limit uh ran out on cloud code how do you know cloud code cannot tell you once the API once the limit of token usage has run out. it simply stops. But if open claw is kind of uh supervising that work, it can automatically switch the account. It can take care of that. So uh and it can notify me through telegram if there is any issue in the execution. So I find that kind of workflow to be very fascinating because everything is running on uh my EC2 instance. I can switch off my laptop and go home and if any issue is there, I'll be notified on my phone on Telegram and I can I can fix the issue. I can ask it to take the next step. Um did you face any breaking point with open claw like context rot or forgotten memories? The thing is open clause open clause context is anyway very short. It's it's never executing anything. It's only orchestrating claude codes execution. So all the context that is loaded is into the um opus 4.6 context. Opus 4.6 context is anyway not managed by open claw directly. It is managed by cloud code. Open claw is only affecting the context that is there in my gemini uh API. But Gemini's context is not that it's not that precious to me. I just need it so that orchestration can be can happen very nicely. Plus if any issues happen, it can relay that information to me and I can take some measures in a timely manner. Even if I don't have my laptop with me, I can do it. But uh the context rot if at all it happens in open claw in in gemini it's not so important to me because first of all I don't think it will happen because the gemini context window is big enough and uh open claw if I'm using such that um I don't pass any heavy task to it. I haven't noticed any issue in in task breaking or anything like that. Yeah. So in my case open clause work is very lightweight. It simply ensures that cloud code is keeping on working. it's never stopping or terminating or if it's not detracting from its original task. Anyways, let's come back to this. So here I will show you a few examples um u based on LLM summarization, tool result clearing, priority trimming and hierarchical hierarchical compression. So let me show you first full transcript and what the situation is. The the the kind of um hypothetical scenario is there is a bite bridge company. That company is like a food delivery company. And uh there is some some sort of rush on a particular day. Uh there is a dinner rush where there are lot of customers who are not receiving uh orders on time etc. And um we need to check the logs check what has happened and there is a bite bridge API bite bridge AI uh chatbot also. So now if you look at the context there are a lot of conversations that have happened uh and there are a lot of tool results. So if I scroll down here can you tell me which one looks which one resembles tool result. So order dashboard this looks like a tool result right? Uh and as we discussed last time tools the way it looks like is the noun plus verb or here maybe it should be a slightly different way order order dashboard fetching or something like that should have been a better name but whatever does not matter. So this is one example of a structured tool result. Here is another one driver analytics the ordering the delivery person who is delivering this is these are the analytics of that config audit. This is also the log of a particular tool result. Then u this bitebridge bridge AI is basically the tokens consumed in the maybe in the observation uh process or reasoning process of the AI model. Then these are user instructions or operations manager Priya is sort of giving some sort of instruction. So in the full token uh that consume the context it's like 1.9K tokens and let's say for this time being the token size window size is 1.9K. Now in LLM summarization what we want to see is how many of key facts that are there in this is retained. So if there are 20 facts in this thing in this whole whole context window how much are retained when we do LLM summarization and how much compression it results. Right? So apply LLM summarization. So here an actual API call is made is being made to open AAI. I gave OpenAI API key and then OpenAI will uh the GPT4 mini I think is that's the model that I'm using that will summarize and let's look at the auto compaction result. So autoco compaction result looks sort of like this sort of like this and look at what has happened to the tool results especially. So one of the tool results was um bitebridge AI order dashboard. Do you see much about order dashboard here? There is not much about order dashboard at all because those total results are extremely compacted into natural language format. And from,97 tokens you you went down to 257 tokens. And you can see that most of the parts here bite bridge AI its own communications um or the summary of when the uh crisis began which are present in these timestamps probably all of those things are very nicely captured in this natural language unstructured uh compaction which can be later passed into the next API call and what I have done here is I have asked the LLM also to separately capture the key 20 facts from this original document. And so this original document is 1,900 tokens long. From that it has identified 20 facts which ideally should be uh captured even after compaction. And then look at this 18 facts are retained. Two facts are not retained. So sort of like instruction retained ratio irr is 18 out of 20 which is roughly 90%age. But the compression is very very effective. You went down from,900 tokens to 257 tokens. Now let me ask you a question. If I apply tool result clearing, what parts of this initial context do you think uh will be affected? Like will order dashboard be affected? So order dashboard will definitely be affected. It's part of the tool result. Will uh do you think this um let's say operations manager Priya's uh conversation do you think that will be affected by default? It will not be affected because it's not a tool result. So I will show you after implementing tool result how the result looks before and after and how many facts are retained. So tool result in this particular case has not resulted in significant compaction only roughly 550 tokens have been reduced. But that's also because out of my entire context right now tool results are not overwhelmingly high. Tool results are consuming some number of tokens. And if you look at what has happened to tool results. So order dashboard this tool result was roughly 200 to 300 tokens. And now look at it. It's roughly uh maybe two sentences with each sentences having uh maybe 10 words. So it has been compressed to like 30 40 maybe 50 tokens. Right? Right. So this order dashboard was very big and that has been compressed so much. Similarly, there was one more section where there was another tool result delivery metrics. See this is again structured data coming from the result of tool results and delivery metrics again you can see that will be compacted to just two or three sentences. But if you look at the other sections of the context um so what this operations manager Priya has mentioned there is absolutely no change in this section because this is not a tool result uh the bite bridge AI's own communication nothing has happened to it so this is the advantage of tool result and look at the facts that are retained 100% of the facts are retained 20 out of 20 facts however compression is only 1347 compression happen two is 13 47 token So if you were to kind of score the technique based on um retention rate versus score for compression, LLM scores better in compression because it has compressed to 200 tokens, 250 tokens, but the information retention rate is like uh a bit lower in LLM summarization. That's also because we are aggressively summarizing. We can also ask LLM to you know uh pay more attention to uh key facts that we could have done. So can we combine tool results and LLM summaries? Yes. Yes. Absolutely. So what you can do is yeah very good question. I really like the question. So that is exactly what we are seeing in the um 80% threshold. So what is initially done is initially do not summarize using LLM. Initially clear the tool results. Why is that? Because tool results clearing is very uh non-risisky. Cuz if you really wanted a tool result again, you can just call the tool again. Once the tool result is cleared and let's say you did not compress good enough. So right now compression has happened only 1347 tokens. Then the remaining items you compress using an LLM so that you are not immediately applying LLM compression. You see what happens to your compression after tool results are cleared. Maybe most of the facts will be still retained. But if compression is not good enough only then apply an LM uh summarization. Then priority trimming. So here if you apply priority trimming these are the four priority levels P 0, P1, P2, P3. So P3 is the routine conversation. You can see 1, P3 over here. So operation manager Priya has mentioned something like great summary. I'll create tickets for all follow-up items marking this incident as resolved. The driver distance spike was the key signal. So some some thing which this manager has mentioned it's a routine conversation which need not be retained. So this is entirely removed and as a result the number of tokens have reduced and let's see what else has been removed. I don't think anything else is removed yet. How about tools? No tool is kept. All of these are entirely kept. One thing I'm not so sure is 1686 tokens. So around 24 220ish tokens have been removed. But was this Priyas conversation actually 220ish tokens? Uh I can see this to be 58 tokens but it should ideally be 200 tokens. So yeah there is another one here 48 tokens. Then there is another one here maybe 30 tokens. Yeah. So if you add up all the conversations this operations manager has made yes probably they will add up to roughly 200. So you can see that what the LLM has figured that is a very you know low impact or low risky thing to remove is all the conversations made by Priya this operations manager. In fact not all the conversations some conversations some are kept and then the total result is around 220 um tokens being removed. Is the decision based on some kind of rule. It's based on rule. It's not not based on semantics. Uh here it is purely rules. Tools have the lowest priority then uh even lower priority is routine items. Then uh if a task is being implemented at the moment that will be at P1 and P 0 will be system prompt. So here only P2 is removed. P1 is not removed. Uh I think these tool results will come in P1. So yeah. So if more level of compaction is needed next what will be removed is these tool results from P1 they will be removed they will be trimmed aggressively but here again compression is very minimal and because of that reason it's not it's not very surprising to see there is 100% fact or information retaining this is not very surprising oh I think I wanted to show you one more thing um hierarchical compression just to show you how the semantic based compression works so this is hierarchical compression and here what is What happens is if something is in P 0 uh the P 0 will be found to be uh the 100% of whatever is there in P 0 will be retained. If something is there in routine P3 it's 0% retained. If something is in P2 10%age will be retained. And if something is 50% if something is in P1 50% is retained. So P1 is the tool results. So tool result will be summarized ideally. So here this order dashboard this is part of a tool result and it's compressed. Yeah I will share the pro prompt that are passed into LLM. Yeah I'll share with you guys. Right now it is masked behind the uh this website front end. You mentioned this earlier uh earlier it's not necessary to use top compression top model for compression in cloud code. Do you have to keep toggling the models for building any compression? Is this what you do typically? I personally have never explicitly mentioned the model to toggle between uh explicitly mentioned the cloud code to um toggle between the models. So I don't know internally what cloud code does whether it switch to haiku or cloud set or something like that instead of opus 4.6 because opus 4.6 is um the most expensive model right by far. So in the compaction I am not 100% sure whether there is a switching of the model happening internally or not explicitly I have never mentioned it but I have to check it. I will check and let you know. Um so this is how hierarchical compression runs. Again P 0 to P4 P3 levels are defined but in each level there is no complete trimming. Complete trimming may happen in P3 but if P3 is not removing everything entirely P2 is not entirely trimmed away. P2 is kept 10% 10% of P2 is kept. Uh yes, prompts should be there in Google Collab. Yes. Um for sure. Uh I I just need to verify whether the prompts are exactly same in the website or Google Collab that I have to check once um because I have made a few changes to the website uh recently. Then let's go into phase two just to see how all of this will appear visually. So here what we need to do is define a hard cutff for when compaction is triggered and define a hard cutff where the trimming is triggered. So can you tell me why do you need these two cutffs? One is at 80%age and one is at 95%. So if at 80%age of the context window size if compaction is already getting triggered why do you need another emergency trimming of at 95%. What's the logic behind this? So the question is why do you need two such uh thresholds at 80%age and at 95%. So it can so happen that let's say uh first of all you know let's say you are at the limit of 64%age um and let's say you are you have 100,000 tokens in your token window. Now 64,000 tokens are consumed. If the output of the LLM is another 30k tokens directly your number of tokens will jump from or let's say the output is not uh 30K. Let's say the output is uh 20K. Let me just draw it here. So let's say this is 100K and you were here for example let's say at 50K. Now you call the LLM and LLM's output is 20K. then LLM's output will fill this much more sort of then you exceeded 80% right so maybe you will this will be at 85 or 90% something like that then you will call compaction and what is the first thing which you will compact the first thing you will compact is probably tool results and let's say your tool results are not that heavy your tool results are only occupying this much right so even Even after compaction and even after compacting our or or even removing the tool results you still are above 80%age then what you what will you do you will continue calling the LLM and let's say in the next LLM call your context window ended up filling up this much where now this is 95%age. So now what will happen is this is an emergency situation because now you have barely any more tokens available to fill which means your output LLM producers cannot fit in your context window. So now there is an aggressive trimming that happens where tool results are trimmed and and P3 levels are trimmed or even direct after tool results are trimmed direct LLM summarization happens and why when direct LLM summarization happens your this thing will jump from 95 to roughly even 20 again. So I'll just uh let me erase all of this and let me show you visually how it will look like. This compaction continuous compaction will look like. So just click this uh start simulation. So here you can see system prompt, user instructions, uh assistant instruction which are maybe the sub aents and then the tool which are the results of tool call and it keeps filling. The moment you reach look at which part is getting trimmed the most aggressively. This this purple part is getting trimmed uh trimmed or compacted the most aggressively which are tool results. Tool results are compacted to very few tokens. Let me just play this once more by decreasing this compact at. So threshold I'm keeping to be uh 60% and emergency let's keep it at 95%age only. Now in first step system prompt second step user instructions. Third step the uh assistant maybe coding assistant tool call. So tool call there are results. Okay. And then you get the results back. Then again the assistant does something. Maybe it calls the tool. Then again you get uh tool results. Then again the tool results are passed to the assistant. Assistant uh consume some token again tool results maybe you inject a bit of user uh prompt here. So Priya has injected something then again assistant works tool call or maybe user instruction then again tool call again uh assistant again tool call. So the moment tool call is increased beyond a certain point not tool call the entire context is increased beyond a certain point compaction is triggered. And look at which part got triggered the most aggressively. This purple thing was uh was compacted the most. Neither assistant nor user nor system prompt. System prompt is very rarely touched. See, so tool results are almost aggressively triggered. And we can also see one more thing. Let me put this at maybe 90%age. and emergency at 85%age. Now what will happen according to you there should be like a proper trimming that is happening. Let's see what all things are trimmed. So here actually one thing that may have happened in the logic is I don't see the entire tool result being removed. Uh so in the backend logic I have to make sure that entire to result is removed. Right now uh when you look at the difference between compact and emergency it doesn't look like emergency is any different from compact. Emergency seems like it's also compacting the tool result. But ideally emergency is supposed to without any mercy remove the um tool results and remove anything else at P3 levels uh P3 P2 levels. But right now that does not seem to be happening. That's okay. Uh in most of your usage cases you will not come across this situation. um you will most likely encounter the threshold where tool results are uh threshold 80%age where tool results are bloating your context window and the moment tool results are compacted along with the rest of the things your context window will go back to maybe 36%age or something like that. You might notice also in cloud code often when a new task is being implemented cloud code will explicitly ask you um clear the context window and start execution. Sometimes it will ask you compact the context window and start execution. it itself will give you and then in the recommended section it will say recommended is clear the context window and start execution. Yeah. So P 0 P1 P2 in this case system prompt should be in P 0 because that has the highest level of priority. Uh P2 will will be uh sorry P3 will be tool results uh because that is at the lowest level of hierarchy tool results which you already have available. you don't have to keep storing in context because in case if you ever need the tool result in the future, you can get it into the context window uh by calling the tool again. Um then the assistance interaction and users interaction may or may not fall in P2 or P3 or P1 depending on the semantics because sometimes the user inter instruction at that moment is very important. You should not be trimming it. Sometimes the latest interaction that assistant had is again important. So if I if I re reset this back such that this is something like this and emergency is over here. And if you now run if you now keep looking at what happens to these colors, you will see that assistant assistance um length is sometimes getting trimmed but only after tool is getting trimmed. So tool has the lowest um lowest priority. So P 0 is red color system. Um purple color is definitely P3. The blue and green depends on what you are what the context is right now. So there you will have to use some sort of semantics. Uh and here just look at this the final tokens final capacity of comp uh context window that has reached after this is 54%age. Meaning at the end of all of this uh out of 2,000 tokens you are still at only 1,087 tokens. On the other hand, if the stream trimming did not happen, by the time 28 messages are there between tool, system prompt, assistant and user, you will definitely cross much much above 2,00. And um here we are not really uh looking at how many key facts have been retained or anything that we will not look at this in this exercise. One thing we should also look at is this multi- aent framework. So multi- aent framework what we decided was um for the same task if there are different types of subtasks multiple agents will be defined. So here this uh diff kind of file is showing you the difference between um before creating a pull request and after creating a pull request. So plus is the new lines that are added minus will be I mean plus is the new line added minus is the new line or old line which is removed just like when you make a pull request on GitHub. And here in this code file there are three different things to take care of and for that three different agents can be utilized. One is the style of the code. So this will review the code style. What are the naming conventions for variables whether anything related to documentation needs to be looked at and uh project coding standards. Security agent will scan for vulnerabilities in the actual code. Then performance agents will look at um perhaps the execution speed of the code. So these three agents are there and it has it its own budget and we can look at how the agent contract looks like. Okay, this is one topic which I did not discuss. This is where I exactly stopped. So coming to something like when you have two different agents let's say there is one agent working on front end another agent working on back end. There has to be some sort of commonality between agents may uh between what the agents may produce. So if the agent working on back end is producing an output which is in a different format from what the front end agent is producing there will be compatibility issues right. So here there is a get request get API/ API/v1/ user here / API/ users and so on. So front end agent may have a default way of behaving because front end calls and front end uh schema is usually different from back end. So they may have a default behavior but unless you explicitly say that the output you guys produce should be adhering to certain format or certain structured format they you will not be able to integrate them. So there is an idea called contact contract first pattern. So contract simply means the orchestrator will define a schema which both of the agents should sort of follow. So if there is an API called related schema, API specifications.json file will be defined by the orchestrator. By the way, cloud code does this automatically. You don't have to say explicitly that when you are spinning sub aents, define sort of like the contract that these sub aents should follow. You don't have to explicitly say that. it is implicit. However, if you are building your own coding agent outside of uh Codex or outside of uh cloud code, this is something which you will have to define explicitly. You will have to tell the orchestrator that make sure that sub aents are following or building based on a c certain predefined contract so that when those two things are integrated they will actually work. An example of this is I had talked to you about um building my new segregator agent or even building a central uh brain for visual. So the central brain for visual is an agent orchestrator and then there are many sub agents that we are building. Now one schema that we are now forcing these agents to follow is uh a schema that will allow us to have a two-way MCP. So two-way MCP as in um it's not like one part can be client and one part can be uh server. it can switch the roles. Um so both ways it can act and uh for this to happen we had to explicitly um mention it in the system prompt that system prompt I mean agent.m MD agents.mmd that um the output that is produced by this should follow the uh MCP format which are mentioned in the guidelines of my central orchestrating agents. Output of my email agent should follow the format mentioned by this agent. And uh same thing when when visual is acting as the client and this agent is acting as the uh server there is a specific way of communication from this client to the server. So this was again an explicit instruction not implicit. The reason why this had to be explicit is because we only defined the MCP ourselves which means there is a certain structure that these sub agent should follow. Um so that is what this section is about. Let me let me go back to the um this section. So the style agent has its own input schema and output schema. Security agent contract defines its input schema and output schema and so on. And when you run the multi- aent, this is just a simulation by the way. Um when the multi- aent uh schema is run, the orchestrator will decide when to call the style agent, when to call security agent, when to call when to call performance agent. um each agent the token consumption is completely different needless to say. All right. So this is just to visually show you how it will work. I want to show you couple more things before we move on to uh the agents.mmd file structure. Okay. To this I will come last. I want to show you one more thing. Yeah. So the way in which sub aents will communicate. So let's discuss a little bit about that. So there is a concept called fan in and fan out. So fan in fan out is nothing but how does a fan work? If if a fan is rotating if something is released from the fan things will things will go outside, right? I mean it'll be in straight line but it will go outside in this way. Right? So fan out is the idea where the central orchestrator will spin isolated agents which have and these agents have nothing to do with each other. Okay. And they barely share any apart from system prompts they don't share any base documents or anything like that. So this is this idea is in general called as fan out. But however sometimes orchestrator can say apart from system prompts there are a few more things which I want to share with the agents. So a good way to visualize that is in this way. So this is the orchestrator. This is sub agent number one sub agent number two and sub aent number three. By default, the system prompt is passed directly to all the three sub aents. So this is system prompt to agent number one, sub agent number one. This is system prompt to sub agent number two. This is system prompt to sub agent number three. Anything else? Let's say if there is a config audit which needs to be shared between all the agents or if some other file needs to be shared across all the three agents that is decided by the context curator. So now config audit may be so this these two green color lines show that config audit is shared between root cause analyst and remediation planner. So this file agent the orchestrator has decided that these two files are site of sort of like base documents shared base document for sub agent three uh one and three and this is system prompt. The remaining file will completely depend on the sharing of remaining files will completely depend on what the orchestrator seems fit to each sub aent based on the sub aent's uh task or sub aent's talent what it is supposed to do. So maybe architecture diagram orchestrator will decide that it needs to be only shared with impact assessor. So impact assessor will now get access to system prompt. It may get access to metrics dashboard or error logs and it will also get access to architecture diagram. So this is how the custom context is set up for each of the sub agents separately. Now if I go back this full isolation is the idea where every single sub aent is having completely different um inputs into the context. So there will be some base instructions. This base instruction can be in the form of agents.m MD. Here we are not mentioning the claw.md file that is implicit. I don't have to mention it. But let's say there are three agents. One for triage, one for root cause analysis, one for recovery. Um for for whatever event that has happened for this food delivery company. Now the the things which will go into the context of triage is its own base meaning its own agents.md file which is not shared. So please don't get confused. This is base a base with 200 tokens. This is base with 200 tokens. But this is different from this. This is the agents.mmd file for the triage agent. This is the agents.mmd file for root cause. And this is agents.mmd for recovery. Then there will be other documents which are other documents or other um tokens which are passed into the context which are relevant to those agents. And these are fully isolated agents. And in this case for example, okay, let me show you three agents then I'll ask you the question. Then there is another concept called shared base. So shared base visually you can look like you can see like this. So agent four, five and six in this case this green color shows that there is a common base that is shared between these uh three agents meaning common documents which the orchestrator decided to give. But apart from that there are completely different uh agent level knowledge which is passed into the context. So security agent will just get information I mean we'll get in addition to the shared base will get information regarding security performance agent will get information regarding performance and so does style. So this is the idea of shared base plus whatever is isolated between the agents. The last one is sequential sub aents where the output input that goes into agent number B will depend also on the output coming from agent A. So this is sequential because unless agent A is fully uh implemented agent B cannot work. A good example of this is for example let's say we are working on a research agent and the implementation can happen only after uh literature review is done. Then literature review agent agent A may come only after only before agent B and agent B can happen work only after literature review and gap identification is done. It cannot work before. So this is sequential pipeline. Now let's go back here and just look at shared base context how it look like. So here this is the shared context between the three dage agent root cause agent and recovery. There is a common shared base. Then apart from that there is uh agent specific instructions and this will consume certain set of tokens. In sequential pipeline the triage agents output is summarized and passed into root cause. So in root cause you will see there is the triage agents findings. Then that findings plus findings from the root cause agent will be passed into the recovery agent. So recovery agent now has in the context uh summary from tree age agent plus summary from whatever root cause agent has found. So recovery agent cannot function without these two things that are passed into the context. That's the reason why it's called sequential. Uh how the pattern defined how is it defined sequential and parallel? It is implicitly defined by the agent orchestrator itself. To give you a simple example, when you are think about it, if you are building a uh a website dashboard, you cannot run a components do components agent or you cannot run an agents.mmd corresponding to the components before you let's say fix the overall architecture of your website. So if the folder structure looks something like this MD and then the root level agents.m MD and here you have the coding agent. So this is another agents.mmd. Then here you have front end and within the front end you have components and this front end agent might have said a few things in the component. In the component hierarchy maybe there is uh navigation bar then and inside navigation bar let's say there are buttons. Okay. Now if there are agents for creating buttons, agents for creating navar, agents for creating separate components, etc. The agents.m MD corresponding to this button section cannot run before agents.m MD corresponding to navar has executed because agents MD corresponding to uh navar says that hey we are currently building navar in which three buttons are required dashboard login logout something else home button only then the agents.md corresponding to this will execute so here the hierarchy of this folder structure is kind of implicitly giving the um agents MD in the root and idea which agents will be executed at what time. But sometimes some things can be in parallel because when you are building the navbar you could also build your whole hero section. You could also build something else in your page and while you are building the front end you could also build some parts of the back end. Right? So if you were to manually define in sequence which agent comes first and which agent comes next, I think it will cause more confusion to the model than um bringing some sort of clarity. What is better is create a very nice folder structure in which the agents MD itself implicitly defines um what the order in which the sub aents are executed. Um if you have agents.md on top, what is the need for cloud.md? So cloud.md. So agents.mmd is only looking at maybe coding task and testing task. So let's say there are two tasks coding and testing. But let's say at the project level you have many other things to do apart from building website. Maybe you have to plan project road map PRP project road map planning. Maybe you have to assign task to your team members or something like that. Like once this is done you have to let's say push something to Trello or canban or you have to uh check the details of your project execution by the rest of your team members on GitHub. So at the central level there are many other tasks which cloud MD can do which let's say an orchestrator agent for coding cannot do or orchestrator agent for coding plus testing cannot do. So agents.md is specific to what all tasks it is supposed to orchestrate. CloudMD is specific to everything that your entire project in your in your workspace is supposed to do. So cloud.md will be corresponding to your workspace and then sometimes so this agents.md it might be confusing to you because right now I have shown you just one agents.mmd in the root file right. The easiest way to remove this confusion is I can literally move this out of my root. So I'll move it here and I will say that I don't have any agents.mmd at the root level. My only a my two aents.mmd is one agents.mmd is at the coding level. Then I may have another agents.mmd at the uh maybe uh project roadmap planning level and at the root level there is no agents.mmd but at the root level I will still have claw.md. So whether you have an agents.mmd at root level or not it's up to your project specification but at the root level you will always have a cloudmd file if you are working in cloud code. Um okay this let's get back. So this is how the um how the sequential pipeline will look like. Now just answer the simple question. There is a scenario multi-dommain restaurant onboarding. Three specialist agents are needed to verify a new restaurant joining the platform. Um in this case just read this and just tell me what type of agent would you do? Would you use full isolation shared base uh context or sequential pipeline? So there are three agents they have to do few things. they are analyzing the at the end of the day it's the same restaurant uh I mean same restaurant applicant application data each agent applies domain specific expertise so probably there has to be a shared base is what I'm thinking let's see so there has to be a shared base but there will be fan out fan out meaning there will be isolated context also how about third scenario let's come back to second scenario after this parallel city expansion analysis three agents Each research a different aspect of potential new cities for expansion of your food delivery platform. Uh demand agent queries food ordering trends. Then uh completion agent maps this logistics agent delivers uh evaluates the driver availability. So if it's a parallel city expansion analysis, it's a completely different city and uh maybe the things that you need to do in one city and the other city are completely different which means there is nothing to share, right? So it could be full fan out like full isolation. Then the last thing is rush hour incident uh response. An active Friday rush hour crisis needs urgent resolution. First is triage incident identifies the severity of the incident. Root cause agent uses triage to find uh findings to pinpoint to the So basically here itself it says triage findings are needed for root cause agent to analyze the issue and then the recovery agent uses both uh the triage results and the um results from the root cause agent which means it looks like a sequential pipeline because the output of one depends on the uh input of one depends on the output from the other. Uh then let me show you one more thing. This windows this part will be very interesting. So again look at this folder structure. So this will answer some of the questions which you may have. Your workspace is bitebridge incident. Okay, this is your project workspace in which there is cloud.md but there is no uh agents.md but inside the folder inside this agents folder you have an agents.mmd. Now this will be the orchestrator. Now inside this orchestrator I mean inside this agents folder there are three subfolders for one for root cause analysis, one for impact assessment, one for recovery planning. It's it has its own agents. Now let's just run a simulation of how this looks like which files will be passed into which agent. So orchestrator will have access to system prompt. System prompt says you are a specialist agent in Brisb incident response team. Follow the thing uh respond in JSON format and so on. claw.md has bunch of things about the team, incident response conventions, terminology, etc. Then incident briefing document. Incident brief is this document is probably passed into this working directory directly because this brebridge incidents incident is the folder to which some briefing might be there. So this incident briefing MD you can imagine to be similar to the ideas v2.md or something which I discussed a few lectures ago like a markdown document which kind of describes what the situation is and cloud.md will be independent of all of this claw.md will take care of all the incidents broadly incident briefing.md will describe the specific incident that has occurred and then there will be instructions provided to orchestrator. So orchestrator instructions are um provided in the agents.mmd this one and orchestrators delegation plan. So now this delegation plan will be a separate file uh which is not shown here but what that file will decide is what agents what sub aents now to run. So orchestrator will decide in the first step I need to run root cost analyst agent. So send the config audit plus driver analytics to root cause agent. Then after the root cause agent output is available impact assessment. So for that I need the impact assessing assessing agent to which I need to send send the root cause summary. Right? After that send have the recovery planner and then synthesize. So let me just run one step and see how the step proceeds. So the first step is root cause agent is called because in the orchestrators de uh delegation plan first sub sub first sub agent that is called is root cause analyst. So root cost analyst is al also getting system prompt. So it's the same system prompt it's also getting access to cloud.md. The same cloud it's also getting access to incident briefing. Incident briefing is the that markdown file at the root level. So this is getting access accessible to root cause analyst. Apart from this root cause analyst will have its own instruction in its own MD file agents.mmd file. So it will say your role is root has to act as a root cause analyst. Then analysis of the current incidents and then the config audit trail and driver analytics snapshot is something which orchestrator ended up deciding to send. So orchestrator had send config audit send driver analysis or analytics. So these two things orchestrator decided to add into the context of root cause uh root root cause analyst agent and which contributed to 150 tokens 160 tokens. So totally 310 tokens. These 310 tokens came here because orchestrator decided to add it. This 100 tokens came because it is part of the agents.mmd file. This root cause analyst in section this 85 + 140 + 135 tokens came because it is part of the there is a shared shared base plus system prompt is anyway part of the um um all the agents. So incident briefing is sort of the shared base it's shared between all the agents and now when we run now root cause analyst agent has executed its result are passed back as summary into the orchestrator. So now orchestrator has an additional 180 tokens. Ideally these 180 tokens some output tokens will be stored in the root cause analyst also. It's not being shown here. So please don't get confused because root cost analyst is calling its own API. That API will have its own results. Those results will be stored in this context window only. So this 770 should ideally increase to something. But then whatever the output is the output can be summarized and will be stored in the orchestrator like this. Now what is the next step? Next step is orchestrator already has a delegation plan. It has step two is impact assessor agent. So in the next step orchestrator will call the impact assessor which will get access to the same three files system prompt claw.md and incident briefing.md. But there are now new files. This new file is the agents.md corresponding to impact assessor. This new file root cause summary is the result of the previous agents execution which is the same thing. the same money tokens are here. Then this uh order pipeline metrics and normal performance baseline was added by the in uh orchestrator. So orchestrator had said send root cause summary plus order metrics plus baseline into the impact assessor agent. So this is these two files of 130 tokens and 90 tokens came to the context of impact assessor because orchestrator had planned it. So total 870 tokens. Now impact assessor will run its step which will produce its output and that output is now stored as a new set of tokens in the orchestrators context. So those new set of tokens are now 165 tokens. Now the third and last step should be orchestrator plan is um called the recovery planner to which these two things are sent along with some playbooks. So in the next step recovery planner will get access to system prompt cloudmd and incident briefing. Then this new file in yellow color is the recovery planner instructions in the form of agents.mmd. These two green color ones are the uh summary of the implementation of the previous two agents. And these two files recovery metrics and customer recovery playbook were passed by the delegation by the orchestrator when it delegated the uh task to recovery planner. And then finally uh recovery planner's uh final uh summary will be over here. recovery plan immediately activate tier two tier two and three refunds for all the affected orders 24-hour push notification so and so and so and that's it I think that's the last step yeah so look at how the orchestrator how many times it interacted uh orchestrators context got filled after three interactions one interaction with each uh each agents right and finally only the output that we need I mean the the recovery plan because this this whole workspace is about when you have an incident How do you recover from that incident? Incident could be large number of orders not getting delivered or whatever. So this is a a brief overview of how it looks. And you can also see this here how the context is getting full. So cloud MD plus briefing is is the 360 tokens that are part of all the uh or agents orchestrator as well as the three sub aents. CloudMD plus briefing cloud.MD plus briefing. Then this 205 tokens are contributed from agents.mmd of orchestrator. But then orchestrator will pass the conflict plus uh config file plus audit. Those are the 310 tokens. This 100 tokens are in the agents.md. So this yellow color defines the agents.mmd file. Number of tokens in the agents.md file. And these are the these files. These three this agents.md this agents.mmd and this agents.mmd 100 105ish tokens each. 100 100 110 and 105 tokens each. This one is the orchestrator agent which has around 205 tokens. All right. So this is roughly how it looks. Now what we can do is I think there was one section which I wanted to show you. Uh this I already yeah I think this will be covered anyways in the in the uh quiz part. Basically what we need to do is there are three sub agents. So sub a one sub agent will take care of order related items. One sub agent will take care of delivery related isuh items into the items meaning delivery related data into the context. One uh agent will take customer related data into the context and each agents we have defined to have a context limit 950 tokens 900 tokens and 1,100 tokens. And these are apart from the system prompt and apart from the task briefing MD which are already passed into all the three agents you have these many six uh not six these many uh 1 2 3 around 10 or 12 items to be passed into the sub aents the question is how would you decide that so uh we don't have to spend time on this but I will just show you the solution uh we can spend time in the quiz part so the solution is that you decide the context to be passed into the each agent based on what that agent's capability is. If one agent is really good at handling things related to order, pass things related to order to that agent's context. If another agent is supposed to handle things related to customer, pass only things related to that into the particular agent's context. Um, now let's go through a few questions that will make things lot more first confusing and lot more clear afterwards. Uh, please answer your options in the chat. Uh some questions are actually very very straightforward. So I'm expecting all of you to um you know give the correct answer. What is this information retention ratio? Fairly straightforward. What exactly is the definition of information retention ratio in context compression? Obviously it is the ratio of what are the total number of facts existing in the document compared against the total number of facts that were preserved after compression happened. Right? So ratio of key facts preserved to total key facts after compression. fairly straightforward. What about this? Which compression is most effective for large uh with uh conversations with large embedded tool results? Fairly straightforward. There is nothing confusing here. Uh do you do LLM summarization or would you do tool result clearing? Probably these are the only two options coming to your mind, right? So the most effective one for large tool results. If tool results are definitely your biggest uh item which is consuming your context tool result clearing should be the easiest one. But if tool results are equally proportional to other items in the context maybe then you will first do tool result clearing and then do LLM summarization. So LLM summarization can result in a lot of loss of information. Also a 15 turn operations debug session has 5,000 tokens. After LLM summarization, it's 400. It is 400 tokens with 16 out of 20 GX retained. What's the IRRa? I mean, fairly trivial. 16 out of 20 means you multiply by 5. 16 multiplied by 10 is 1860 multiplied by 5 means it's 80. So 80% should be the IR fairly fairly straightforward. In hierarchical compression tiers like critical 100% retention, important 50% retention and so on. What determines which tier a message belongs to? This is one question many of you might have doubt about. So here the question is is it D or C that's the question in in hierarchical compression do we decide the hierarchy based on some rules or based on LLM based I mean based on semantics. So in hierarchical compression it should be based on semantics. Uh but in the trimming in trimming it's based on uh it's completely trimming it's based on rules right but here it's based on semantics. Then why is priority trimming considered riskier than tool result clearing? What is the risk behind priority trimming? So this is a little bit confusing right just think about it. I think people are typically I mean generally consider uh confused between B and D. Okay first of all let's eliminate priority trimming requires more compute power to execute than clearing. Uh there is nothing like that. So A we can eliminate for sure. Priority trimming can only done by an LLM. uh while tool clearing is rule based and uh deterministic actually priority trimming can uh see trimming can be done by a nonlm because in priority trimming hierarchy is defined based on P 0 P1 P2 P3 uh P1 P3 is the tool results and maybe uh earlier conversation history which we can define based on rule P2 might be things which are earlier in the in the later part of the conversation like conversation that just happened recently. So we can define some rules. So here necessary LLM is not really required. Tool result clearing uh tool result clearing preserves reasoning context while trimming removes entire messages that may not be implicit. Tool results are always redundant. So clearing them is safe while user messages always contain unique information. Let's let's go with one of them and let's see. So answer is B. Tool result clearing preserves reselling context. But these two are very confusing, right? They pretty much almost feel like they are saying the similar thing. But tool result clearing preserves the reasoning context while trimming removes entire message that may contain implicit uh dependency. Tool results are always redundant. So clearing them is safe while user message always contain. Maybe the always word is not so true. It's not like always tool results are redundant. We can't say. Um and always user messages unique information also we cannot say. If this was rewarded in a slightly different way, it would have been even more confusing whether B or D is correct, right? So maybe this always word is not the not the right thing to say. Um an operations transcript has three tool results averaging 250 tokens each. So 750 tokens. Um oh in B it is saying let me go back to fifth. uh tool tool clearing preserves reasoning context while trimming removes entire uh messages that may uh contain implicit. So basically in tool result clearing if you remember LLMs can also clear tool results. Sometimes in the LLM summarization but there it's not clearing the tool results it's summarizing in one or two sentences. Um if it is complete tool result clearing it is um it is not LLM based but in LLM based summarization it can clear in LLM based summarization also tool results are cleared technically right tool results are summarized into one or two sentences. Um but yeah what this is saying is tool result clearing preserves reasoning context while trimming removes entire messages that may contain implicit uh dependencies. So trimming means um in trimming what may come in your P3 level can be r tool results as well as your conversation history. So your conversation history which is far from their most recent conversations may get cleared. So it may affect the context conversation history or or uh uh user inputs which are far away from the current user input may again get removed that may affect the context. That is what this is saying. Um okay you have three tool results each averaging 250 tokens then 12 conversational messages averaging 60 tokens each. So this is 12 * 60 which is what 720 and this is 750. If you apply tool result clearing replacing each tool result with a 30 token summary what is the appro approximate compression ratio? It's fairly easy. You can So basically tool results were originally 3 * 250 which was 750. Now tool result becomes 3 * 30 which is 90. So your tool results go from 750 to 90. So whatever that number is that much compression is there. So about 78% of original uh tool res from 750 to um 90 tokens. Let me check if first of all yeah this is this is not definitely 55%age. This is 90 out of 750 is closer to 78%age. Then in a threepass autoco compaction system, why does tool result clearing happen before message summarization? This I think should be clear. We have discussed this in detail like why is tool result clearing the first thing which you do in autoco compaction. The reason is uh as we discussed toolled results are kind of deterministic. Of course depends on the tool but in general when you call the tools with a certain condition it will always give you the same output. So toolled results are sort of deterministic. It lower it's low risk because if you need ever need it again you can simply call the tool again. So you can get maximum compression before resorting to LLM based summarization which can be lossy. You saw that in in the example which we discussed in the LLM based summarization that was the most lossy summarization because 18 out of 20 facts were retained two did not get retained. What what is the recommended compact at threshold for production autoco compact systems? This we have discussed fairly straightforward. You should know this. So there is um compacted and there there is emergency trimming right. So compacted is typically 85 80%age trimming emergency trimming at around 95%age. Um then what does context rot refer to? I think by now this is the only thing maybe you should be able to answer this in the sleep. It basically degradation of models uh context over uh time and usage. Then during an autoco compaction simulation yeah this question auto compaction simulation pass one fires which is tool clearing fires but the context is still at 88%age capacity emergency threshold is 95%age. What should happen next? First of all, is everyone clear why after tool clearing the context is still 88%age capacity? Why is it not coming below 80%age? Is everyone clear why? Because auto compaction triggers at 80%age if So my question is if it gets triggered at 80%age, how come the capacity is still 88%age? Wouldn't it bring it below 80%age anyways? Why why is this make sensible? I'm I'm just trying to recollect your memory. Why is why is even after autoco compaction triggering why does it stay at 80 88%age? Simply because in the whole in the whole context you don't know how much uh is the tool output. So when you clear the tool or um compact the tool results there is no guarantee that you may go from 94%age to 79%age or 88%age. You can you might still stay at 88%age or 80 above 80%. So what happens? So what should happen now? Uh I let me just check your answers. Um B A B most of you are saying B. Some of you are saying A. Okay. I'm not sure what what which one you're saying. What should be the next step? So nothing is between 88%age compacted and emergency. So system waits for more messages. Is it that or system retries pass one with more aggressive tool clearing? Okay. All of you are it seems like many of you are saying a some people are saying password retry password. Um so what see what what should happen is there are different levels of clearing. First of all in hierarchical clearing first level of clearing is tools and second level of clearing is um the let's say some things in P2 or or some other things a apart from tools in P3 which is like conversations that are very old or u um you know uh user instructions user prompts that are very very old. Maybe you can clear that because rest of the things uh things about user instructions are already captured in the rest of the context. So something should trigger right that's what ideally should happen because if we don't do anything we are guaranteed to reach this emergency threshold. You know when emergency threshold is used emergency threshold is used only when imagine that your context window is right now at 79%age. At 79% there is no compaction. Okay. Now let's say you your LLM produces an output and that output is these as these many tokens such that you jump directly from 79 to let's say 96%age at that point auto compaction which happens at 80% will not get triggered at that at that point directly emergency threshold will get triggered but anything before that if autoco compaction is not a the pass one of tool clearing of autoco compaction is not able to do anything it will try pass two so that is the next logical step should happen. uh if nothing nothing being done is not a good thing because then you are anyway see at 95%age anyway you are trimming without any uh you know care because your context window is almost dead so you have to trim so there is no no choice but at 88%age you may have some choice to remove P4 and P3 only do not touch P2 right so when you have more choices you you will execute that choice so it should be firing pass two immediately Pass two is compact at run the compact at but for maybe the next level of hierarchy. Uh okay another one why is the emergency threshold um emergency at is separate from compact? Why do we need these two things? So why do we need this? It's very aggressive. Uh so emergency threshold compaction uh I mean trimming is very aggressive. It will literally remove entire portions. Whereas compaction can be made using LLM. So if LLM compacts something uh from tool results, it can still compact whatever is remaining. But emergency trimming will completely remove tool results. So nothing will be available. Plus it will remove anything else needed in multi- aent code review system. Why is contract first design critical? First of all remember what is contract first? Why is it needed? So contract simply means defining a common schema for all the agents so that the results can be merged or or combined together to build one single product. So if you have sub agent for front end, sub agent for back end, whatever they produce should you know should be compatible with each other. Then what is the main advantage of fan out or fan in? So fan in simply means you have multiple sub agents you compile their results into one single orchestrator. So why is this preferred over a single agent approach for code review? So I think this is a bit confusing right? Okay, let's let's first of all look at what may not be the answer. Fan out is always faster because agent run in parallel. Uh not sure it need not be fast. It depends on the particular project. So it's probably not a right. Fan out uses fewer total tokens because the difference is split into smaller pieces. Uh there is no guarantee of that. It may even end up using a slightly more tokens. Why? Because the system prompt is passed to all the all the individual uh agents. So maybe the total number of tokens consumed may be a bit higher. What about um D multiple agents can use different LLM providers simultaneously reducing vendor lock in. I mean for sure that is possible for sure you could do that but that's not primarily the reason why we use fan out. The reason is um you have specialist agents. Some agent is good at code review. Some agent is good at security issues. Some agent is good at um perhaps um uh testing. So you need the specialist to invoke their special skills. In a multi- aent system, the orchestrator sends the full PR pull request diff to all the three sub aents. What's wrong with this? So earlier we saw the example where the PR diff was sent such that uh only the required parts were sent to required the the specialist agents. But what if we send the whole thing to all the three sub agents? What's wrong with it? Okay. First of all, let's think about it. Nothing wrong. No, it's not like there is nothing wrong. There is something wrong about it. So, it's definitely not a it violates the fan out pattern which requires equal sized input to all agents. There is no such rule. We don't need equal size rule or anything. How about B and D? B versus D. It weighs tokens. Each agent should receive only receive the code sections relevant to its domain. That's the first thing. D is it causes agents to produce duplicate findings since they all see the same code. Is D true? See, D is not true. Just because all agents receive the same input does not mean they all will produce the same output because their capabilities are different. Maybe these agents have different access to different tools, right? So there is no guarantee that they will all produce the same findings. But if the entire code is provided to sub agents, the sub aents can definitely see the relevant sections of the code corresponding to their expertise and they can produce the findings which they are supposed to find. But the thing is it will also receive a lot of tokens which it cannot do anything with because if there is a let's say a agent which can do some testing on a function maybe a code review agent looking for schematic of the code need not receive those tokens. So it weighs tokens basically but there is no guarantee that providing the same input to all agents will provide the same output. There is no guarantee different agents have access to different tools. Then a security agent finds five critical SQL injection uh vulnerabilities but its output schema only allows the maximum se uh sec severity warning of sever maximum severity of warning. Uh what design principle was violated. So output schema uh is not allowing it to produce an output which it is it's supposed to produce because of its restrictive nature. So um that's the reason why it will not this this is not a good structure. What is the orchestrator's role in the fan in phase of multi- aent code review? So fan out we discussed from orchestrator you go to different agents. Fan in is the opposite of that. So what what is the role of orchestrator? So this is where the dduplication idea comes. So earlier we discussed if we pass the same input to three agents they all will produce the same output. They will not all produce the same output but maybe some some findings may be duplicated between the agents because maybe two agents have similar some two agents some capabilities of two agents might be similar. Maybe then because of that they can produce same findings. So those findings we duplicate and then merge the finding from all the agents and determine overall review verdict. So the central orchestrator agent basically collates the information coming from all the different agents. So in the WSCI framework, what does the C specifically address? Very trivial question. So it's it basically compresses uh reduces what is already there in the context window while preserving critical information. So compress is basically compressing what is there in the context window already. Um okay this we have not discussed. So I will move to this. What is the relationship between compress and isolate in the WSCI framework? uh fairly straightforward nothing uh nothing nothing um confusing at all compress will reduce the context within a single window isolate just isolate is like multi- aent framework it will distribute the context so that different agents can have its own uh API calls so they can work in different context windows uh then in the context pyramid which layer has the highest preservation priority ity during compression. This also we have discussed many times. But which one has the highest priority system instructions and persona? They define the uh agents behavior and must never be removed. A production agent handles 50 turn customer support conversations. After 30 turns, the responses become increasingly confused and contradictory. What is the most likely cause? You know this already. It's the context rod. Um on Vijuara's website now we have a you know uh a calling agent. I urge you guys to try it if you really want to just go to our landing page and uh this there is like a phone icon. Sometimes it'll say it'll say like daily limit has reached but basically you can just um there is this ask AI option. So here what we have done is we have an agent with a specific context about um our programs and stuff like that and you will notice that after like 5 minutes or 6 minutes it's uh its ability to communicate in a very nice way will uh significantly reduce because we have not optimized the context or anything over there and after some point it will completely disconnect. So um basically its context window will become full. So compaction and compression we have not yet implemented. So it's a you will see a classic case of uh context rot if you just look at our current uh chatbot but uh I think now we have we are making some implementations not sure if we have um pushed it to production yet but that was the case 3 days back then uh so here context rot will happen then your multi- aent system occasionally produces duplicate findings the style agent and security agent both flag the same missing authentication decorator How should you handle this? This we already discussed uh three two three questions ago in multi- aent framework. So think about this sometimes uh sub agents can give findings which are duplicate. The only only place where you can analyze this is sort of um in the orchestrator because these findings are directly going into the orchestrator. So in orchestrator there should be a step a dduplication step so that you know these similar findings are grouped so that the context window is not unnecessarily bloated. You are designing an autoco compaction system for a coding assistant. Which type of message should never be compressed or removed? Okay, this is an interesting question. So what should never be compressed? This uh we briefly discussed during the lecture also if you remember right. Uh so the so if you if you read the uh structure okay let's read this D the user's initial request that started the conversation it's already captured in the ongoing context. So uh users initial request okay after the initial instructions when the when you are building something what you are building why you are building for what you are bu for for what's the reason why you are building all those all those things are kind of captured in the context although if it is part of claw MD file we should not delete the fire file directly right but if there are some initial instructions it kind of is there in the context already so if the very initial thing is removed it's kind of Okay. But look at this option A. The most recent code uh code edit the assistant proposed that the user hasn't yet accepted or rejected. So when you use a code uh code code editing assistant u you will see that the diff is shown in in the whichever ID you use right and that diff you have to either accept or reject before moving to the next step. So if user has not gotten a chance to say whether I accept this change or reject this change probably you should not delete that from the context yet right. So uh you should keep this uh this is like a very special case right. So this is a special case where you should not be removing this. Then a multi- aent system for food delivery crisis has agents for order flow dispatch driver dispatch customer support and communications. The communications agent need access to findings from all other agents to draft the status update. How will this be architect architected? In what way this will be running? Give the communications agent access to full context window of all other agents. Include the communications agents in the initial fan out so it can observe the investigation in real time. Run communication after the others are complete and pass only the structured output. Then have each agents send their raw context to shared database. So this is classic example which we discussed earlier. If there are agents whose input depends on the other agent it's like in a sequential way right. Of course sequential agents will also get other things in the context. They will get cloud.md or other markdown files which are relevant. So there will be a shared base also but uh the agent specific summary will be happening in sequence. When cloud code spans a sub aent how does the sub aent receive its context? It shares the parents context window via memory map IPC. It inherits full parent conversation history automatically. It receives a separate API call with a curated prompt containing only relevant context. It reads from a shared database that both parent can and some agent can access. It's not like that. The um relevant context is curated by the orchestrator. Here the orchestrator is cloud code. So this sub agent will have its own way to call separate uh LLM a I mean its own ability to call claude whichever model you are using using its own API. An orchestrator sends the entire 50,000 token conversation history to its three of the sub aents. What is the primary problem? This is again clear. The primary problem is context rot of each sub agent. So the point the whole point of creating the sub agent itself is lost here. if you are sending the same humongous 50,000 token history into these agents. So very very straightforward in the shared context shared base context pattern. What is the base that all agents receive? So there is one minor issue in the wording here. I mean answer should be B because a a common foundation plus agent specialized context. So agent specialized context which is agents.m MD it is special to itself. Okay by the way. So in this from this option maybe we should just delete whatever is written after this plus of course it is plus something but the common foundation will be system prompt task description and shared data like in the food delivery incident report MD cloud MD and any other uh shared data. So that will be forming the common base. Then why does the sequential pipeline pattern produce uh higher quality findings than full isolation? So there is one point which I want to discuss here which is see sequential pattern pipeline pattern it's like a double-edged sword because for reasoning it can do better better job because you you can validate what one agent has done using another agent but at the same time it can also cause error propagation. If one agent produces something erroneous, it is propagated to next agents also, right? So, it's like a double-edged sword. So, you will have to put some guard guard rails. Three agents have a thousand token budget. The shared base is 300 tokens and each agent needs 600 tokens of specialized context. What is the total context consumed across all the agents? This should be fairly straightforward. What is the total token consumption? So 600 is a specialized context plus each of them receive 300. So it is 600 multiplied 600 + 300 multiplied by 3 which is 900 * 3. It is not 600 * 3 + 300. That's not how it works. This 300 is taken three times. Shared base is uh although it's a shared base that same context is appearing as tokens in each API call. Okay. So this is common. So it's 600 + 300 multiplied by 3 which is 2700. Then a coding a assistant uses agents.mmd file in project subdirectories. How does this how does this relate to context isolation? How does agents.mmd file in the subdirectory relate to uh context isolation? So agents.md which is in a subdirectory is agent specific. So it defines the context which each sub aent receives. Right? Agents.md can of course be for a specific agent can be modified by the orchestrator for sure. Then when is full isolation the best choice over shared base or sequential pipeline? When do you prefer that instead of shared base or instead of sequential pipeline? This is a classic case where you don't need the agents to know much about what the other agent is doing. They are completely they're as if they are the only people who exist like uh sort of you know independent contributors in teams. So they are literally is in full isolation. Then uh what is the main risk of the sequential pipeline pattern? We discussed this like 2 minutes ago. So just like how if there is a reasoning the the remaining agents can verify the reasoning just like that there could be a risk of error propagation. If first error if first agent makes an error in sequential pattern it could propagate into the remaining part of the pipeline. So this this we have to put guardrails to make sure that there is an error and it's not getting propagated. So it may result in some extra token consumption to make sure that such a thing does not happen in a multi- aent system. What is the orchestrator's most important responsibility regarding context? Obviously the orchestrator's single biggest thing or rather two things are one is deciding what sub aents you need. Second is once you decide these are the sub aents you need um what should we feed to each sub aent. So it's deciding what context to include in each sub aents window. Then last questions question how do sub aents typically share their findings with the orchestrator. So this we already discussed there is a contract between agent and orchestrator sub agent and orchestrator agent based on that contract there is a schema that is defined and that schema will be in JSON format or whatever and then that that will be shared. So this is the quiz. So I need before we wind up the lecture there is one more thing uh some people have asked me personally regarding an extension to the weekend assignment. I can already see some assignment submissions that are good. uh but if you need extension to submit the assignment on MCP uh please take you know few more days if you want uh anyways I plan to um evaluate it after this course only so I will anyway need some more days to go through your submissions properly so if you if you need more time just take more time just let me know drop me a message uh then I can just take that into a note so uh this will bring us to an end of today's lecture so for the uh okay there's a question most of the examples are focusing on coding agents do these concepts work similarly in non-coding agent frame agentic frameworks like langraph crew AI Google ADK yeah so the uh framework of agents MD whatever we discussed today is kind of universal it works in almost all agent decoding platform as well as platforms like uh langraph but u I am not so sure about the same for the equivalent of cloudmd file because cloudmd And similar equivalents are there for coding based agents but uh for remaining frameworks I'm not sure how it actually works. My takeaway is if if you have access to a coding agent which is the most modern you should definitely give it a try. So in the last 3 days I was talking to three separate friends of mine who are working in uh in good positions and one thing that they have been telling me is like they they have been wanting to use cloud code or similar agent but then there are severe restrictions from their companies because one of them is working on Amazon another person is working in 3M all of these are like 100 billion plus dollar companies and it's very difficult for them to come to decision of how to use what to use. So it's very slow. So then what we thought was okay for at least to know what are the capabilities so that they can propose something to their team so that the team can form a policy which can which they can see how to implement in their in their at least in their team itself. Uh what we discussed was at least for some personal project which has got nothing to do with company try some of the most modern coding agents. Um and if the results are like day and night different at least try to make a proposal to your company about okay this is this is something that we should have many of you might have seen people posting on LinkedIn and x.com about their cloud code monthly budget like some teams are literally sometimes meaning like small companies like 100 people team $50,000 uh bill they are getting for like 1 month's usage it's like crazy and many people are literally stopping their operations whenever there is an outage in cloud code but yeah Sumit's question how is the experience with auto research are the results of experiment improving yes the results of the experiments are improving but one thing which I noticed is like uh auto research at least the way Karpathi has designed it has a tendency to oversimplify everything so it is going towards experiments that are too simple I mean that's deliberate karp you can literally see in the in one of the uh markdown files that he has written go for simple experiments which can be executed fast. So because of that one issue is that sometimes the model will not tend towards uh complex some if something complex needs to be done normally cloud code can do it but in auto research most of the experiments are are like 10 minute 15 minute 20 minute experime experiments but cumulatively they they of course they accumulate to something better I can see that the results are improving. So one in in the paper that we are working on right now we are seeing that um uh they are proposing like um uh from the from the embedding layer find out the deviation from the original data and subtract a certain number which sounds a bit trivial but uh it's working it's it's really working. So the results are improving but then my fear is okay are we trying something trivial or is there something logically amazing here right? So that that uh that issue is there but you should try uh auto research once in all of us in our team are running different research problems now to optimize some of our models see how good auto research can be. So the way it you can set it up is take any Amazon EC2 instance slightly better RAM would be better. So the free version has I think 1 GB of RAM or something which is too too small. get something with at least eight or 16 GB of RAM so that things are a little bit fast at least on the computer side. Then if you have runpod just recharge it for $10 or something $10 $15 and um then connect it with your github and you don't need to have open claw or anything in this case. Uh auto research uh instructions are very simple. Then whatever is your problem statement uh whatever your data set or model you have at the moment. If you don't have any model that's also fine you just need to mention it. But whatever your problem statement is mention it mention that and mention the direction you want to take and uh yeah in auto research only training train.py we can optimize on multiply multiple files. So the by default way he has mentioned it is only train.py will be changed but the way I have written it is in such a way that I did not directly clone auto research or anything. I gave the link of auto research to um cloud code and I gave my own specific instruction. I asked it to follow the philosophy of auto research but not follow it exactly because in my case I know that experiments some of them are not 10-minute experiments they are longer experiments but I just wanted to follow the philosophy the philosophy is continuous improvement and relentless experimentation in the sense like never stop until a human says you stop so that's the reason why it's 24/7 it's running uh just try that I also had used one more GitHub repo for research uh forgot the name of that starting with M Um but but I am primarily sticking to auto research for next at least few days. I just want to just see what what are its limits. How much how much can it optimize models but yeah uh you you guys should try it like at least try for some of your research try a modified version of auto research. Don't fork I mean don't clone the report directly. Just give the link to cloud code or whichever coding agent you use. then mention your your instructions like um you don't want to restrict to simple experiments alone whatever it is just mention that but follow the similar philosophy I think that will give even better results in any case if I get something amazing I'll share it with you guys uh in some days I'm asking it to run a few more experiments today and tomorrow uh all right everyone we can stop today's lecture here thank you so much uh for joining and staying back so so long uh tomorrow we'll discuss about memory uh on 8th Day eight session we will uh there are two projects option I'm thinking um I haven't yet finalized because both are equally exciting but I don't have time to time to uh show you both maybe I'll record one video one I'll show in the lecture live and one I'll record a video or something but u let me finalize it and um tell you my plan regarding that so that will be on Thursday or Saturday so the date I have not finalized u so the the build session will be s on my bill session will be on Thursday or Saturday Rad and Rajett's build session uh will be finalized. I'll talk with them and I'll let you guys know the date. Um uh share the explanation of software next week new course what it will contain. So that I think Raj will be a better person to answer than me. Uh all I know is like he'll show you some amazing um practices for modern software development with AI agent. So I I think it'll be a very very uh powerful tool anyone can tool as in powerful framework which anyone can learn from. So I know Raj's workflow in and out because we literally are working in the same office, right? But I think uh you guys will benefit a lot from uh learning the sort of workflow that we internally use at Visualara. So uh check it out. The link I can just find the link. Um just give me a second. Yeah. So this is the link. Yeah. Yeah. Thanks. Yeah, that's the link. uh just go through that um you should understand the structure. It's starting on next Monday for continuously 5 days uh Monday to Friday. So it'll also be again uh long lectures of building. You will build a lot of projects as part of it. Uh so it'll be like a pure workflow setting up um stuff like how to set up your perfect workflow for building using coding agents. uh because there are there are different things which you need to do for projects of different complexity level. So for some simple projects I never define like oh go with this project road map document with version one version two. I don't say that. I just want things in one shot. But if it's a very complex project I typically ask to go um first be in development mode develop the v1 iterate on v1. Once that is done um go to the production mode um oh sorry planning mode to development mode and then if it is if it is a good v1 then put it into production then go back to planning to v2 then uh planning of v2 to development of v2 and so on. So I have a specific iterative process which I follow for complex larger scale projects. For smaller scale projects I don't uh I don't follow any of that. It's it's it's more random. But anyways uh you'll you guys will learn all of that in detail on that boot camp. So see you guys tomorrow. Uh no it's not self-paced. It is uh live. It is it's live happening I think from 9:00 a.m. to uh 11 uh p.m. I mean 11:00 a.m. Just let me just share my screen. So this is the modern. I think somebody had shared the link right Shrihar you had shared yeah I myself had shared me just open that so this is the boot camp link and the goal here is like to have proper build session so you will you will have to be ready with your laptop and whatever your coding agent platform is uh it will not be a passive lecture where you can just listen so it will be very very intense and you will have to spend a lot of uh actual brain cells to to code. But anyways, just uh if you have any doubts, just drop an email to hello@visara.com. We will uh get back to you. But it'll be a very very nice course. I highly recommend it. Uh Raj has spent a lot of time designing this. So uh thank you so much everyone. Let's meet again tomorrow same time uh 9:00 a.m. And uh take care until then. Bye.