Context Engineering Simply Explained

Key Takeaways

Since the time Andre Karpathi posted about context engineering on his Twitter, this term has gained a lot of momentum over the internet. It went to a level where folks at Cognition called it number one skills for engineers building AI agents. The goal of this video is to explain context engineer in a most intuitive language such that even a high school student can understand it easily. Context engineering is a natural evolution of prompt engineering. So let's summarize what is prompt engineering. Let's say you are summarizing an article and you write this prompt. Summarize the following article. This is very generic prompt and it will produce a very generic output. If you are a good prompt engineer, you will give a specific prompt saying that summarize article in following three bullet points. You will mention your bullet points. You will also say use simple clear language suitable for high school students. So you will specify the audience which is going to consume this. So this is prompt engineering. Now when you talk about agents, you give an action to an agent such as book a flight between two places and an agent will have access to tools such as Xedia API and using these tools it will first search for flights. So let's say five different flights. Then it will decide which is the cheapest one. Let's say this is the cheapest one and it will book the flight. So an agent is nothing but a program which is using LLM along with tools, knowledge, memory etc to perform autonomous action. Previously when we were looking at the Q&A chatboard is just simple question and answer whereas agent will perform an action autonomously. Now internally it will take multiple turns. What I mean by that is when you say okay book applied between place A and B it will actually first it will make LLM call right so this is let's say first turn okay so this is Turn first turn where it will determine the plan that it needs to do all these three actions and it will perform the first action which is searching the flight in the second turn or a third turn it will look at the Xedia API It will find the flights, right? So it see it found all these flights. So this is turn number two. Then it will decide the cheapest flight. So that will be turn number three. Then turn number four, five. It might perform n number of turns which is n number of llm calls along with this specific prompts to accomplish a goal. Now here these prompts are generated by LLM. It's not like you are manually generating all these prompts, right? You don't have time for it when you're building LLM application. Let's say an agent, it will do this thing autonomously. Now, you run into this risk where these turns might get costly. You know, it might increase your LLM API cost bill. Also, if there is a problem in one of these prompts, it will misbehave, right? It will take an action that you don't want it to take. So there is certain risk in terms of generating these intermediate prompts or intermediate contexts right this is the context that you're giving to your LLM and in these terms whatever context is being generated it has to be really good and accurate now here you can see that every time I am appending to the previous context so if you give more information if you give the information of the previous context uh from the previous done it is going to be useful in general but the problem with in appending to this context is that you might run out of your context window right so this is the diagram I took from cognition blog where let's say the agent will break down the task for our flight scenario then this is a subtask one right and let's say it is appending that context then it is appending a new context now you might run into this context overflow. Not only the length of the context is a problem. Let's say even if you don't run into context overflow, longer context will result into some other problems too. I'm going to link this article in a video description. Please read through it. It says how long context will fail. You'll run into four different problems. The first one is context poisoning, then context distraction, context confusion, and context clash. I don't want to go through these in detail because then this video will become too longer. But read through this article. Just remember that not having a proper content in the context will result into numerous problems for your agent. Therefore, what you need is context engineering. As per Andre Karpathi who prefers the term context engineering over prompt engineering uh the Twitter says people associate prompts with short task description you would give an LLM in your dayto-day use that is your prompt engineering but when you're building industrial strength LLM app it can be workflow app or it can be an agent context engineering is the delicate art and science of filling the context window with just the right information for the next step. So you have a context window and you want to fill that context with the exactly the right information. So it becomes art as well as science as per cognition prompt engineering was coined as a term for the afford needing to write your task in the ideal format for LLM chatboard. Right? For the chatboard you are writing a good prompt which is good enough for that narrow task that you're performing. Whereas context engineering is the next level of this. It's an evolution of prompt engineering. It is about doing this automatically in a dynamic system. Dynamic system is your agent. You remember those turn one, turn two, turn three where agent is producing that that context. You are not writing it in that dynamic system. You want to do this in a very effective way. It takes more nuances and is effectively the number one job. Folks, if you are an AI engineer, if you are an AI agent engineer building agents, this is a number one skill that you need to have for anybody who is building AI agents. So essentially, if you consider LLM as a brain, okay, so let's say LLM as a brain or let's say CPU, then this context window is a RAM for your computer and if you have good quality data in your RAM, your output will be better. and context window you know context engineering is all about getting the right content inside your context and how do you get that content well through three different ways one is through tools so let's say when you are looking at Xedia APIs you want to get the right information on the flights by calling Xedia API and that information you got through a tool call second one is a knowledge if you're building a rag based system and let's say you are doing something with your database you pull the information from your Oracle or MySQL database that is your knowledge. It can be PDF file too. And the third one is instructions. So this is your prompt engineering. You write the proper instructions. There are four common strategies for context engineering as per lang chain. And we are going to go through each of them quickly. The first one is context compression. So when you run into this context overflow situation where your context is very very big one obvious technique you can deploy is you can compress the context. So whatever conversation and actions you have so far you see this green box now you that box has become little thinner. So you will remove unnecessary information from this context or you will extract only useful information. See key moments and decisions from that context and you will make that context smaller it each turn. So this is the first turn. In the second turn this particular context you make it you compress it. this one you compress it and that way you your context is not only good quality but you are also respecting that context window limit which every LLM will have different token limit. Then the second one is writing the context. So for writing the context you can use uh something called a scratch pad. So scratch pad is uh something you can implement with a tool call. So let's say you have a simple tool call which is writing to a file on a disk and this scratch pad is different than your LLM context. So let's say your LLM context window has a limit of 30,000 tokens. So that window is there. On top of that it's like as a human you know you have a notepad and when you're having a conversation you will make points in that notepad. So that notepad is a scratch pad but in your brain you will remember that information. So that remembering information in brain is your alarm context window. So this is an additional basically memory that you get and it can be implemented easily with a tool call which just writes to a file on a disk. This is useful for an agent session. The second thing that is useful for an agent session is a state. So if you have worked with langraph, we have a langraph tutorial on our YouTube channel. In lang graph there is a state that you will pass across all the nodes that is also within the agent session. So you can use that state to write information meaningful information for the context. And if you use chat GPT whatever your past conversations are chat GPT remembers that you might have noticed that if you turn off your computer today tomorrow if you turn on your computer which is a different session in chat GPT and if you say okay uh define my personality based on past conversation it will define it. So that is using long-term memory and it is valid across agent sessions. Okay, so these are the three techniques uh that you use for write context and it means saving it outside the context window to help an agent or a workflow application perform a task and once you have saved this information the second step is obviously retrieving that right retrieving that context it's called selecting the context and in that of course you are retrieving from long-term memory that GBD example that I gave then you are retrieving relevant knowledge. This one is a rag. So when you're performing rag you will have vector database from which you will retrieve necessary context that you can pass to LLM. And this rag is nothing this retrieving information through rag method is nothing but selecting the context. Okay you can do the same thing from scratch pad you are retrieving from relevant tools you are retrieving. So that is the second strategy which is selecting the right context in an efficient manner. And then comes I think the third strategy is compressing the context. So the fourth one is isolating the context. Let's say you are working on a big task and you you are building multi- aent system where you can divide the task across two agents. So that will reduce your individual context window for the agents. Right? So this is the context window for context for agent one. This is for agent two. And since you have divided them individual context window will not only be shorter but it will also be more relevant more it will be high quality for that agent one. Okay. Now building multi- aent system also comes with certain risk. So you have to be very mindful. You have to uh make sure that you can really parallelize this work. In some cases, you can't parallelize this work, right? You have to do things in a sequence because agent 2 relies on the output from agent one. So based on the situation, you can use this isolating context approach. So those are the four strategy. I'm going to link another blog from lang chain which will describe these four strategies in detail. So let's go through Andre's uh tweet in detail. So he prefers contextual engineering over prompt engineering especially for the people who are building AI agents. And what he says that it is an art delicate art and science. Okay. What you fill in your context basically. So context engineering is nothing but delicate art and science of how you fill the context window with just the right information and highquality information so that it is useful for the next step and it is science because doing this right involves task descriptions and explanations few short examples right you are filling right kind of few short examples because let's say you have 2,000 token context window it's like you have limited currency and you have to use those token very accurately. So let's say if you are providing too few short example you have to provide very good high quality diverse examples then rag okay related data then tools what tool calls you're making what kind of output tool call is returning sometimes these tools will return big output and 80% of this output might not be even relevant to the LLM so you have to tweak your tool calls and provide exactly the right outut put then state history compacting too little or the wrong information and the LLM doesn't have the right context for the optimal performance. If you have too much or too irrelevant then context then LLM cost might go up and performance might come down. Okay, doing this well is highly non-trivial. So folks there is no fixed set of guidelines. Therefore it is called art and science. Okay. And art because of the guiding intuition around LLM psychology of paper spirit. I hope this video helped you understand context engineering. If you have any question, please post in the comment box below. Once again, check video description for all relevant articles and information. Thank you very much for watching. If you like this video, please give it a thumbs up and share it with your friends. [Music]