Memory management | LLM Context Engineering | Lecture 6

Key Takeaways

Hello everyone, welcome to a new lecture. First of all, before we get started, I hope you can see my screen and hear me and see me all right. Thank you so much. So, today we are on to the sixth session of the LLM context engineering bootcamp and today we'll be discussing about memory. And this is going to be very important because when you are actually building certain applications using coding agent, many items which enter into the context of the large language model will be coming from the memory. And there are different structures for storing the memory. There are different types of memory. We will learn all of that today. And this will also come handy when you are actually implementing during the implementation lectures. So, first thing that all of us should understand is this I hope everybody by now completely realize that is that LLM is stateless, meaning when you are making one API call and then you go ahead and make the second API call, by default, there is no connection between API call one and API call two of the LLM unless we explicitly pass something into the context. By default, two API calls are not at all related. The way ChatGPT remembers your preferences or remembers who you are, etc. when you are chatting continuously is because of the explicit definition of what goes into the context through memory. So, we will be looking at first an example of the statelessness of LLM and how we can simulate just a simple example where if we explicitly do not pass the memory, how LLM behaves versus if we do that, how LLM behaves. So, I'll just in a moment I'll share a Google Colab notebook. Let me just first admit a few students. So, I want you to open this Google Colab notebook. I'm just sharing the link in the chat. Just keep your um OpenAI or Gemini API key handy. So, I have shown this to you in one of the lectures before. The way you can add API key in Google Colab is simply click this key button uh and then you can add it over here. So, you can add bunch of API keys and that will persist when you even when you move from this Colab notebook to another one. And it will not get shared. So, you can easily access the API key. So, just open this Google Colab notebook. Let's do a quick experiment. This will not take any time. Just to see how the behavior of LLM changes when you explicitly define the memory. So, first you import um google.generativeai as genai package, then openai. And this part this block of the code is simply to select which LLM provider are you going to use. Okay, so um so, this today's lecture is not necessarily about feedback. Today's lecture is about when there is a long history of communication between let's say user and the LLM or let's say coding agent and the LLM, how does the LLM remember stuff and what are the ways in which you store what the LLM should remember in the upcoming conversation. So, that is what we are going to cover in today's class. So, I have configured OpenAI API. So, here in this drop down you can select OpenAI or Gemini, one of the two options. If you really want Anthropic or any other API keys to be used, it's very easy to change the code. So, you can even ask Gemini to change the code. That part is easy. So, I'm not going to cover it. Um and then let's just load OpenAI GPT-3.5 Turbo. Uh please make sure that you are using some cheap LLM. No need to go with um you know, pro preview model or anything like that if you are using Gemini. For me, by default, it was like expensive LLM. I don't think that is needed at all. Now, what we are going to do here is we will just have one interaction. So, here we are designing a defining the user input. So, user input one is hello, my name is Alex. It is nice to meet you. So, this will be passed into the LLM. So, we are just printing the first interaction here. So, this part I'll show you when it prints how it looks like. Maybe I can put this in a different block. So, first interaction is um hello, my name is Alex. It's nice to meet you. Then we will be executing this else if condition because we are running OpenAI. And look at the OpenAI call. So, openai.chat.completions.create. And then to the model argument, we are passing the LLM model name, which is defined to be GPT-3.5 Turbo over here. So, this is defined LLM this defines the LLM model name. And then the messages is the role is user and the content of the message is the user input one. User input one is defined in the previous block, this one. So, this will be passed into the LLM as the message. So, let me just put this in a different block. So, now the LLM's response is to the question, hi Alex, nice to meet you, too. How can I assist you today? Now, the second interaction is we are in the second interaction, we are directly calling the LLM without passing anything extra other than the second user input. So, the second interaction is this, what is my name? So, right after you made the first API call, you're calling the same LLM using the next API next API call with user input two as the new input. Let me just admit few more students. And then you repeat the same thing. This time you again call the LLM and API provider is OpenAI and the message is such that the content is user input number two. User input two is this, what is my name? So, now if we run this, LLM will say, I'm sorry, I'm an assistant and I do not have the capability to know your name unless you tell me. So, basically, when we are calling an API, I mean, when we are using an API call to call any LLM, by default, it has no memory. You are just utilizing its weights, which are pre-trained. So, those weights have certain numbers. And when you are calling the when you are making any API call, what you pass into the context is the only thing that the LLM knows. And here in user input two, which is now what is forming the context of the OpenAI API API call, the only thing in the input is what is my name. There is nothing about Alex or anything like that. So, yeah, there's a question, when we are interacting with ChatGPT, it automatically adds previous conversation to the context while answering and applies compression whenever required. Exactly. So, OpenAI OpenAI in the chat in ChatGPT, I'm not sure how much like in the latest version that we have available, I'm not sure how often or how much does it compress because you might have seen this is probably a few months ago in the if you are continuously chatting in one same window, you will see that it has reached this the message has reached its limit, right? You might have noticed such errors happening. I don't know whether they have changed it in the latest not models in the latest way in which ChatGPT is implemented or not. Uh yeah, I have I have uh made everyone enter into the room now. There's nobody in the waiting room. Um but in the in the latest models, I'm not sure, okay? I have not used ChatGPT chat in some time. So, these days I'm mostly using it through um through my own terminal. So, so anyway, I have to set my own context. But this is the issue. The LLM has no idea about the context. You are the one who is responsible to set the context. So, now let's implement a simple memory. So, we will make an empty variable called conversation history. In this variable only, we will store the past conversations. So, we are defining a function called chat with memory. So, inside this function, a few things are there, but let's go to the section where we are calling the function. So, the function will be called over here. So, I'll just cut this and put it here. Yeah, I'll share the notebook link. So, just take a copy of the notebook and just make sure you put your um API keys over here, Gemini or OpenAI. So, this is the first interaction. Hello, my name is Alex. It's nice to meet you. And then instead of directly calling the LLM, we are calling the function chat with memory into which we are passing the LLM model, um and into which we are passing the user input, into which we are passing the conversation history and the API provider, whichever is our API provider. So, conversation history at the beginning, it's an empty array because there is no conversation history at the beginning. User input three is nothing but the variable which defines the same user input which we gave earlier, but we are putting it in a new variable called variable input number three. So, the user input three is simply this sentence. And LLM model name is whichever we picked. So, now let's look at the function chat with memory. So, The with memory has this variable called history. So, history.append So, when we are appending the role user and the parts is whatever user message we are getting into the function. So, the user message in the initial part will be whatever the empty uh conversation uh array will be whatever it will look like. So, initial conversation history looks like this. So, initial user message is nothing. There is nothing in the history, right? So, when even when we are adding history history.append, nothing is getting added in the beginning. But, in the second round, uh whatever was conversed before will get added. Then, our provider right now is OpenAI. So, let's go here. So, OpenAI messages is equal to an empty array. Then, role equal to user if turn in role. Else, the role is assistant. Because when we are uh using an LLM, the role can be that of the user or that of assistant. If it's assistant, it's relaying the output to us. If it's a user, we are providing the input. Then, content equal to uh an empty string. dot join. So, this will then uh concatenate different part different uh strings. part for part in turn of parts. So, turn is nothing but you're looping through your history. So, in your history, previous conversations will be uh appended one after the other. So, each conversation turn will be picked in this for loop. And then, in that for loop, you'll be looking at whether the role is that of user or the whether the role is that of assistant. And then, the content that we are passing into the LLM is the concatenated full context. So, concatenated context which contains the conversations made by the uh user as well as the conversation conversation made by the assistant, which is ChatGPT, I mean GPT. Um so, now this OpenAI messages will be append all these items appended into one single string. And what we are passing is response is equal to OpenAI.chat.completions.create. And then, we pass the model name. Then, we pass the messages. And then, if we just print it or if we are returning the function uh the value of the function, we should hopefully see see the full response, not just the uh lack of knowledge of the LLM. So, let's just define the function by running it and just print it. So, now this is the first interaction. Hello, my name is Alex. It's nice to meet you. Then, hello Alex. It is nice to meet you, too. How can I assist today? And then, in the second part of the conversation this time there is memory. So, user input four is passed over here. But, at the same time, conversation history is also passed into the function. And now when you ask what is my name, it will know that your name is Alex. Right? So, this is how you This is how ChatGPT works internally. Uh it considers all your previous conversations in the form of a conversation history. But, this is the simplest form of memory. I just showed this to demonstrate to you how memory works in principle. So, in principle, the way it works is creating uh enough context for the large language model so that it can become a useful tool for you. Otherwise, it'll be a useless one. So, now if memory was not present, every single chat will start from zero. So, it'll be a very frustrating experience for you. Uh recently, we were building uh as I was telling in the MCP lecture, we are building the um MCP framework for utilizing Blender for building our own style 3D models. So, the person who is working as animator, his name is Akash. So, Akash was uh basically trying to create um using Blender MCP 3D models in his own style. But, he noticed that when he whenever he's using the Blender MCP to create a new model it's not because he he did not explicitly define any memory. He had to redefine the style in every single turn. So, whenever he was trying to make a new 3D model or new scene, it was creating like it unless you explicitly mentioned your style, it was not able to do that. Then, what he did was he created a separate file system which served as the memory which will go into the context of the LLM so that when LLM makes the MCP server call, uh not LLM, the when the whenever the MCP client makes the MCP server call you are feeding the user style and the way in which Akash makes his 3D models also into Blender. And then, immediately the Blender started giving amazing output. So, it's so so important. You will when you are working on building any kind of application where your styles and preferences need to be followed again and again without having to explicitly define it you will need to define files uh which serve as memory. So, this obviously start starts learning from zero. Um There is no learning happening, by the way. No consistency or personalization. And this makes the tool a really useless tool if memory is not there. Now, there is one thing we should realize that there are even for humans different types of memory. So, we remember things like what happened. We remember things like information, right? Paris is in France. It's uh some information that we know in our brain. Then, we also remember or we also know things about how to do things, how to cook uh pasta or how to ride a bicycle or how to play football. So, we we know certain things. We have some knowledge. And we remember some incidents. So, in the memory architecture that we construct for LLM also, broadly speaking, there are three types of memory. It's not like we have to explicitly say something is episodic or something is semantic. But, the way in which it is stored in file will depend on what type of memory it is. So, episodic memory is something like if you are using a coding agent to build something the coding agent uh will remember user asked the so-and-so in the past conversations. It will remember user suggestion that uh or its own realization that use OAuth2 over API uh keys. If it's semantic memory, it stores the vague preferences of the user. So, if user says something like in general, I prefer dark theme over bright theme. You can store it as a semantic memory in a vector database. So, the vectors will correspond to it'll be a representation of the user's preferences given in natural language. Or, it can be even very specific uh facts or very specific um preferences that user has. For example, if the user always prefers just one page to show every single information on the website without having any uh hierarchy of pages that can be stored in a semantic memory. In procedural memory, what is stored is um how to do something. So, semantics is what uh what is or what I know. This is about how to. And episodic memory is about what happened. It's already written here. We will look at some examples of memory and we'll see how to classify it into episodic, semantic, or procedural in a moment. And when we are retrieving anything from the memory uh we are retrieving in one of the following formats. So, if the memory is stored in form of text. And you might probably be using keyword searching to find similarity between uh what is there in the context right now and what is stored in the memory. So, text similarity will be purely keyword-based matching. And this is actually not very expensive. You You have already seen this when uh we discussed that from cloud.md, we are not always fetching everything. We are mostly taking only parts of cloud.md into the uh system prompt using keyword matching. But, semantic embeddings, because it involves rag almost always, this is slightly more expensive um compared to text similarity. You will have to look at vector similarity. You may have to perform some re-ranking. And then only you will fetch from the memory what is required to be uh passed into the LLM's context. Sometimes, if you are considering the chat history in many times, you will say, I don't care about chats that happened before a certain date or I only care about the last 30 chats. So, in the email tool which uh I'm building I only care about the last 50 chats. Uh last 50 email threads. I don't care about the threads before that by default. So, that is sort of like a temporal memory where the recent recent um items from the interaction has higher preference to be part of the memory. The late the earlier ones are discarded. It won't be part of the context. Then, there are other storages for memory in the form of uh graph uh knowledge graph. So, knowledge graph is nothing but something that consists of nodes and edges. So, the way it is structured is nodes will have a bunch of um information in the form of a uh JSON format or something. So, key-value pairs. So, key and value. So, let's say the node is about information about customers. So, this the one node could contain things like um uh customer name uh preferences, the amount the customer has spent on the platform, etc. And there could be other types of nodes which describe products. And then the edges are nothing but connections between customers and products which may indicate whether a customer has added a product into the cart or whether the customer has tried to place an order, but it never went through or customer actually purchased depending on the edge type. So, I mean, this is a big field in itself. Graph knowledge [snorts] graphs or graph neural network where there is edge prediction, node prediction, entire graph classification, etc. or edge classification. There are a lot of tasks associated with the graph. But one of the tasks is from such a graph fetching the information that we need. So, this is also medium complexity because graph is not a very simple database. It is a complex database. Actually, in one of the recent industry partnerships which we had with a company, we use utilize graph-based data only for memory because it was very complex database with a lot of relationships between the different data points. You know, when we don't need a graph? When we don't need a graph is when we have rows database that are in the form of rows where row one and row two are sort of independent. In that case, we just can use some SQL database or we can use a some database that are structured like JSON, right? So, we won't I won't be able to talk about more details about this graph based projects because of NDA, but it was a it was not an immediate revelation that came to us that oh, we have to use graph for this. It came out as a result of so many discussions that actually we realized that the database is so complex that even if we tried to simplify it and if we try to represent it using a a simpler structured dataset, it would not work. But it's not Out of all the projects that I have worked on, only in one or two projects we ever ended up using graph. In most of the other cases, we did not use graph. And the most complex out of all of this is using some machine learning or deep learning algorithm to predict what item from the memory we need to fetch. So, obviously, it will be much more computationally intensive and I have never used it so far in any of the industrial projects. In most of the projects, I have used the first four. Text similarity is the most commonly used. Semantic embeddings are even in the email tool which I'm building, I'm already using it. Temporal proximity also is very often used. Graph-based memory is less frequently used and predicting what aspects of the memory you might need to fetch from ML or deep learning framework or some ensemble models like XGBoost or something like that that is even even in my opinion rare. So, now there are you can broadly say from the items that LLM context has interacted with in the recent past, there are three types of memory. One is called hot cash. So, hot cash is nothing but what the LLM context just recently processed. It is literally there in the API that just passed into the LLM or just to be passed into the LLM. And most often, this will be in the format of something like a JSON file. Because it will have a structure which look which looks like this. There'll be key followed by value. There'll be key followed by value, right? So, there'll be bunch of key-value pairs in a very structured format which is about to be passed into the LLM's context. So, hot cash is something where you don't have to explicitly do anything, but this is the highest or most um valuable or most preferable memory that LLM context utilizes because this is literally the one that we are passing. The second one is called warm storage. It's just a terminology. You don't have to really try to remember it or anything. But warm storage is equivalent to recent session history. So, one example of warm storage that we saw just now is this. So, this user input three was an example of warm storage because we stored it in the conversation history array. And then when we are passing the user input four, that user input four itself is in the is in the hot cash and user input three is in the warm storage. Warm storage meaning something that was really recent which was part of LLM's context. And this will also be stored in different types of memory formats. I will show you that in a bit. It can be Redis or it can be even stored in markdown files or TXT format. It can be stored as a vector in your Supabase vector database. Or it can be converted into vector in your file system itself using Chroma DB if you are local hosting some application. It depends on really your application. And cold archive is the least preferred one or the the least important one for the time being. This will involve things about you know, very big vector database consisting of let's say 10,000 emails from your past interactions. And almost always to fetch anything from this, you will require rag or you will require keyword matching. Right? So, rag is a very powerful technique which if you are building any kind of serious application, you will almost always use rag. And cold archive will exist in the form of structured SQL database or it will occur in the form of vector database or JSON objects. So, just remember that memory when we talk about memory, there are different tiers. Cold archive we did not discuss in this simple example that we just saw. We just saw example of hot cash and warm storage. Let me just check the waiting room. Just a second. So, now these are the different file formats. If it's a structured fact as we know, JSON or SQL or SQLite. And this is almost always stored in a structured database. If it is conversation history, we can store it in array. We can even store it as a text document or in some cases, you can also store it as JSON because the structure will be user which is the key. Corresponding value will be the interaction with the So, turn one, user, assistant. Turn two, user, assistant and so on. In vector database, it will be in the form of Chroma DB, Prisma. Doesn't matter which framework you use, you will have one vector of 768 dimensions or whatever dimensionality of your embedding is and that will those vectors will form your database. And if it is like a a memory that has to be always passed into an LLM and which is always persistent, it will almost always be in the form of TXT file or markdown files which is now becoming the norm or even XML files like what Cloud prefers. Where do we store episodic memory contents inside vector database? So, when you say episodic memory, you mean something that happened in the past, right? So, what has a something that has happened in the recent past, it will be stored as JSON if it's a part of a conversation history or in an SQL database. It need not be converted into vector, but something that happened in the far away past because in the far away past, lot of things would have happened. It's very difficult to store everything and fetch something from a document that containing 10,000 entries. So, for there, we need some retrieval augmented generation. In that case, almost always it will be a vector database. Amit has a question. Cloud.md is more of a warm storage. So, Cloud the document cloud.md, yes, it is sort of a warm storage for sure, but sometimes the items that we fetch from cloud.md are sections of cloud.md, right? So, from cloud.md, we will decide to fetch information regarding phase one or phase two. In that case, that particular section which is passed being passed into the context is called as hot cash. But the cloud.md file itself which is being referred to almost always will be like a warm storage, correct? Now, there are five types of memory management strategies broadly and today we'll be spending a lot of time on this through this Google Colab notebook which this is a second Google Colab notebook. I'll share this with you in some time. Plus through the interactive website. So, this is how to manage memory if you have ongoing conversation. So, we'll take exam as an example ongoing conversation between the user and the assistant. And we will see how each of these techniques can be utilized for that. So, there are five techniques. Details I will go through one by one. Windowing, summarization, key-value extraction, priority pruning. Some some of these are things you are already familiar with because similar ideas we have discussed in other context. But windowing, the basic idea is that you will decide that everything that belongs to the last five windows of back and forth conversations between the user and the assistant or the last 10 windows, they will form part of the memory. Rest of the things will be discarded. So, windowing is a little bit like yeah, exactly like sliding window and it will slide from whatever is the most recent conversation to back in time. And everything before that will be discarded. Can you tell me what is the What is the positive side about windowing and what is the downside? In windowing, what's the what's the advantage of windowing technique to decide what what should go what should be part of memory? Let's first discuss about about the pros, then we can come to the cons of this. The pro is that you don't need a separate LLM call or you don't need any special algorithm to decide what comes in the window. It's clear, right? Whatever come Whatever is the last five conversations, that forms the window. So, the definition of what comes into the memory is very straightforward and it is very easy to implement. The downside obviously is that you may have lot of useful information in these conversations also. Maybe user had mentioned a lot of their preferences in this turn one of the conversation or turn two. And if you are windowing, those things might be completely lost. It will be somewhere captured in the ongoing conversation to some extent, yes, but this will be lost. A better strategy, as Amit is saying, could be you somehow summarize this this whole conversations that happened before using an LLM into few sentences. On top of this, to not lose the details of the ongoing context, use window as such. So, the advantage of window is you have everything that the user has said word to word in this window. But in the summary, everything is not captured word to word. It will be an LLM-based summary, so it will capture semantics but not necessarily the nuances. But this is a good, you know, trade-off. So, the pro is it is the simplest possible implementation of what to store in the memory. The con is of course losing all the context before. Token cost is very less because very less or it's constant because what you pass as part of the memory into the context is the same set of tokens. So, if you say that last 1,000 tokens as part of the conversation is part of the memory, that 1,000 tokens, so whatever is the API cost for 1,000 tokens for that LLM, that will be the cost associated with memory. So, if the chatbot that you are building is very simple and you don't need any sophisticated memory or you just need quick conversational bot, this is the easiest thing to start with. The second and most, I would say, effective one is LLM summarization. Because here there is a proper intelligence layer. So, if you have large number of turns of conversation between user and the assistant consuming, let's say, 3,000 tokens. What the LLM summarization can do is it can summarize, as we were just discussing, it can summarize everything that happened from turn number one to recent turns and the last five turns of conversation or last 15 turns of conversations, which may be very few number of tokens, can be stored as such. So, the proper context of what is happening recently is not diluted at all, but what happened in the past is sort of diluted. So, this compression will result in some information loss, but that is the trade-off that we make. But this is very efficient in terms of token preservation. This is very efficient, as we will see in one of our exercises. So, the pro is that, of course, it captures meaning and semantics from the past conversations. But the con is that to construct this kind of a memory from the past conversations, you will have to make one API call. And maybe this one API call you are making once in every 50 conversations or once in a while. So, that whatever is the cost associated with the all the tokens being passed into the API of that LLM call, that will be your extra cost. And token cost will of course be whatever is the cost associated with recent five turns or recent 10 turns plus the summarized memory. So, this will be the total memory size. So, the cost associated with your ongoing workflow will be the cost associated with the ongoing workflow as far as memory is concerned will be summary plus the recent turn, whatever is the number of tokens for that. So, this is not fixed, of course. Then there is another one, which is really, really good at compression. It is called key-value extraction. So, key-value extraction is something where the entire conversation, which may be in natural language where there is no structure. The only structure will be user, whichever is user interaction, whichever is assistant's interaction, that will be the only structure. But converting something like this into a pure um super compressed key-value pair structured format. What might be an advantage of this? If you are compressing your past conversation into a into a JSON file that looks something like this, what is the advantage of it? Basically, the advantage of a structured format is to get any kind of data from a structured format, it's very easy. So, from this memory, if anything needs retrieved needs to be retrieved, it's very easy. This also results in very, very high amount of compression and this is also done using an LLM, by the way. So, we will explicitly ask LLM to extract this into a structure and convert it into a JSON format. The problem is that this extreme compression may result in loss of information like something like why the user even decided some semantic meaning like why the user decided to build something like this instead of something else. Because these are like very quantitative information. Language is Python, framework is FastAPI. Decisions REST over GraphQL. So, it's it's kind of the more you become structured, the more it is difficult to capture qualitative semantic information, which may which may be important when you are building an application. So, if you need extreme compression, um you can go with JSON kind of memory compaction, but you will also lose some information. In my opinion, this is the best, LLM summarization, where you kind of decide how much needs to be rolling summary and how much needs to be the recent ones. This is the best option. But if the conversation has itself inherently has a lot of structured items, let's say the conversation between the user and the LLM is already about some structured data or document format or something like that and not something in natural language necessarily, we can use JSON. I mean, we can use key-value extraction. There is something called KV cache used when tuning LLMs. Is it the same? No, KV cache is different. So, KV cache is simply something like this. So, the attention weights matrix will look something like this. So, let's say you have five five tokens. Cat sat on the mat. And let's say this is your token, so these are your words. And let's say one word is one token, so totally you have five tokens here. I don't know how many of you will relate with this unless you know the transformer architecture, but I'll try to make it as brief and simple. So, you have five tokens here. In transformer, what happens is these tokens are projected into three spaces. Query space, key space, and value space. And the way this is projected is your incoming tokens are multiplied by a query matrix or key matrix or value matrix. WVX. So, X is the input token, W is the matrix. So, then it becomes query, it becomes key, the the token becomes key and token becomes value. Now, what we do is if you have five keys, you will have one, two, three, four, five. So, you'll have five rows and five columns. So, this will be corresponding to cat query, cat query, sat cat sat on the mat. And same will be present here. The key corresponding to the token cat, key corresponding to token sat on the mat. And when we construct the attention weights matrix, the way it is done is such that there is masking. So, the future tokens are I'm going through this a little bit fast, so I know that for those of you are not aware of this, you will not understand, but that's okay. You don't need to understand. I'm just mentioning since KV cache, I don't want you to confuse this with anything else. So, this is how the attention weights matrix will look like, where all these numbers are zeros. So, this will be one. Sum of these two will be one, so let's say this is 0.6 and this is 0.4. Sum of these three will be one, sum of these four will be one, sum of these five will be one. Now, to make the next word prediction, what the LLM needs to do is LLM needs to know what is the So, this is the next word prediction. So, query softmax of query times key transpose divided by square root of D multiplied by value. So, this is the attention weights matrix. So, to to predict the next word, essentially, you don't have to calculate this whole matrix again to continue the conversation. Because then you are again computing the attention weights between cat as the query and cat as the key. You are again computing attention weights between sat as the query and sat as the key. Doing it for on, doing it for the, doing it for mat. You don't have to do it again if you want to predict the next word because you have already done it once. So, what you can do is these queries and keys, you can already store in the cache. So, these values you have already available, this you can store in the cache so that for inferencing, when LLM is trying to make the next word prediction, all these values are again not recomputed. It is taken from the memory cache and then only the last row needs to be computed. So, from a matrix multiplication point of view, having um KV cache will make the computation so much faster. Uh this is in the most simple words the way I can explain, but I'm I know that I'm skipping a lot of things here because um um you know, um Yeah, yeah. Correct, correct. KV cache is about multiplication between key and value. That's correct, yeah. So, basically then when you multiply with value, so you will have query dot key transpose multiply divide by square root of D multiplied by value. The end goal is in the attention weight matrix, you don't you just need to compute this one. You don't need to compute the whole thing. And then multiply query dot key transpose multiplied by value. So, value will be value vector corresponding to whichever is the next token to be predicted. So, the basic idea is that for LLM's inference to be fast, you do not um recompute the attention weights of the previous to corresponding to previous queries again. You keep it in the cache. But when we change the context every time for context management, we destroy this KV cache, right? So, if the context is entirely starting from scratch, yes. But if your context is elongating, let's say you have first right now I have five tokens. Let's say your context is now elongating to six tokens, seven tokens, eight tokens. You can create a cache for all the previous tokens so that the attention weights corresponding to them are not recalculated. They are they are available to you already. You just need to compute it for the upcoming new tokens. So, this saves a lot of computation time during the inference, not during the training, during the inference. But that memory is completely different from what we are discussing right now. KV cache is anyway something which the LLM does for inferencing speed. It only affects the speed. It does not affect the quality of the output. But here we are not talking about speed or anything. Here we are talking purely about quality. Right? So, for quality output, the context should not be a rotten context. Context should be of value, of high value. So, what are we now passing into the context? Right? So, that decision is about quality, less more about than about speed. KV cache is more about speed. Um So, this yeah. So, the pro of the keyword based memory is key value based memory is high compression, but the con is you will miss nuances from the past. The token cost is again minimal because the structured data format output is very small. Most often, the number of tokens associated with it will be like few tokens, hundreds of tokens. But uh this is used in very rare cases where the conversation itself has a very good structure, which means when you are compressing, you can follow a structured format. Then there is another one, priority pruning pruning, which we discussed in some capacity yesterday also. So, here an LLM decides from all the past conversations plus past items which came into the context, what are the most important? Um like cloud.md, user preferences, etc. They are eternal. If there are conventions with respect to what we are building right now, uh from a project perspective, they will be stable for months. So, if you're building an email response tool, most of the files which describe the user preferences like idea.md or initial.md, a lot of things will not change. Then if you have things in the um related to the phase one of implementation or phase two of implementation, it will keep evolving from a day-to-day basis or from a week-to-week basis. And if you are having errors or if you're debugging something or if um your context this or if your coding agent decides to try something before coming back to what your main implementation was, in that case, whatever is there in your memory will be very short-lived. So, these documents which are associated with your hypothesis testing or documents which are associated with debugging, they will not be used forever. So, the quality of those documents will be very low from a uh context perspective. It will never ever come back to the context once this particular task is done. Um So, the pro and con of this is also very clear. I just want to get into the implementation part. Okay, so we'll come to this uh in a moment, but let's open a Google Colab notebook to see a quick implementation of all of this. Let me share the link with you. Right, so this is the link. So, just open it once. Let's um go through this exercise really quickly. So, here all the five methods are being implemented, windowing, summarization, key value extraction, priority pruning, and the last one is semantic chunking. Semantic chunking I have not explicitly mentioned because you already know about it. You create categories into which you want to chunk your um conversation, and then chunk the conversation based on those semantics using an LLM call or using um some sort of embedding model. Here in this example, we are we don't have an embedding model. We'll be using a separate LLM call only. So, again, please set your API key over here, Gemini or OpenAI. And the first block of code is simply selecting. So, I have selected OpenAI here as the API provider. Summarization and semantic chunking are not the same. Summarization uh very good question. Summarization summarizes the entire history of conversation according to this example. I'm I'm speaking from this conversation example perspective. If there are 25 turns of conversations, summarization will simply summarize it into let's say 300 tokens. In semantic chunking, the first step that is done is from the conversation, each turn of the conversation you will label whether this is technical or whether this is related to some um let's say product or something else. And then whichever conversations are related to technical, you will group them together, and you then you will call an LLM to summarize it. So, in semantic chunking, in general, you don't necessarily use an LLM. You may use a language model like the sentence transformer to convert it into vector embedding after you chunk. But in this example, in today's example, I'm going to show you how to implement semantic chunking using LLM only, not using an embedding model. Windowing is done where there is just one LLM call. You pass the entire context into the LLM, ask it to summarize based on what your need is. In semantic chunking, you pass two things into multiple LLM calls. One thing is what the chunk is corresponding to. Is it about technical? Is it about customer? And then what are your preferences when it comes to summarizing this whole thing? Is semantic chunking the same as what we used in rag? Exactly. Semantic chunking is exactly what we used in rag. Um Okay, so now this is our data set, okay? So, look at the data set. This is the user. This is the assistant. And there is around 22 or something, yeah, 22 past conversations that have happened between the user and the assistant. Now, you will notice also one more thing here if you scroll a little bit to the right. Um There are two more fields called high intro, low intro, etc. The high and low are about the relevance of this particular conversation for the memory. Intro is about what this particular conversation was about described in just one word. Okay? So, high is about high and low are about relevance or medium. They are about relevance. Intro, the last item is about what the topic of this particular conversation was about in in just one word. My question is um Who created this? Like who made this high intro, low intro, etc.? Meanwhile, I'll also answer one question. Uh Uh semantic chunking using LLM only, not embedding. So, what's the difference? So, uh in this case, so when I say LLM, I simply mean um models like GPT where the output is in the text format, right? So, here in semantic chunking, in this current example, uh what we are doing is we first chunk the document based on whether it's in the intro category, whether it's in the text category, or deploy deployment category, then use an LLM to squish it in natural language itself into a small set of sentences. In semantic chunking, actually what is done is the same chunk is passed through an embedding model to convert it into a dense vector format. So, that vector will be bunch of numbers. It will not be natural language. Now, we can retrieve that we can retrieve the vector and retrieve the chunk corresponding to it. But by default, it will not be stored in natural language. By default, it will be stored as a vector. But in the current example, we are showing it is not it not in the form of a vector, in the form of natural language only. But yeah, coming back to my question, who wrote this? Like who decided this should be high or intro or or what the nature of this conversation is? Yeah, yeah. So, this data is synthetic generated by LLM only, no doubt about it. My question is if some if a data structure like this is there available to be, let's say, passed into an LLM for summarization, right? So, that memory can be created. Like who will do this job? Is the LLM that is being your, you know, main um orchestrator? Is LLM going to do this job is is it done by someone else? Or is it is it through let's say keyword based matching or something like that? Because think about what what this is. This is the conversation history that is continuously ongoing between the user and the agent. And This is this is constantly being developed, right? So you can have a a tool which can be something like a keyword based matching tool which based on the keywords can map the current conversation into a high relevance category or low relevance category in the cheapest way. So keyword based matching is very cheap. No LLM is involved. If you want it to be expensive but so accurate, you can pass this into a into a smaller LLM and say, "Hey, you should categorize this particular message into relevance it's not a relevant score. So high, low, medium is like relevant score and also into a category. Maybe the category you can give already like the categories that I'm interested in is intro this this this and this or it can also come up with its own categorization. Those are the two ways in which an ongoing conversation between an LLM and an agent can be categorized. So you make a separate LLM API call to categorize this conversation or you do it simple keyword matching which means it will be much less expensive to convert it convert or to add its relevant score and to add its topic. So these are the four items or five items. ID of the conversation, speakers whether it's an it's the user or the agent, content, relevance and topic. Now let us just print this. So there are 22 turns of conversations and 1097 tokens. How using keyword matching you get topic? So keyword matching we have to already define if the if the conversation involves hey so and so I'm so and so if there is if there are certain certain keywords, it forms in the category falls in the category of intro. If the keyword involves something which are from a dictionary of let's say web developers, then it can fall into category of let's say technical discussion something like that. So it depends on what keywords exist in your dictionary and how your conversation is compared to your dictionary. But it's a it's not a smart thing. It's a yeah, exactly. It's like a hardcoded logic which is fine. I mean, you don't you don't need to convert every single conversation happening between the agent and the user into some category by again making an LLM call. That's too expensive. So there are total 1097 tokens. Now the question is when we compress or when we convert this into a memory format using windowing or using summarization or key value extraction, how much will this 1097 tokens shrink? That is one thing that we are going to see. Plus how will the output look like when we are compressing this so that it can be passed into the memory of the LLM in the next call, how will this look like? So just scroll down to this section strategy one which is which is windowing. So in windowing we are saying the turns we are passing so the turns are nothing but it's this variable name conversation converted into turns over here. Where is my turns variable? Okay, I think I'm using a different variable name to be when we are calling this function. It's not turns. Okay, anyways, let's look at the variable that we are passing is conversation itself. The conversation variable is defined as these 22 conversations over here. All right? So and then when we say n equal to four turns of minus n means it will look at the all the items from the last item to the fourth last item. So last then n minus one item then n minus second item n minus third item. And then output of windowing is nothing but those four items and this n equal to four is what we are passing. This we can change. So then when we print, obviously you will see the last four items and last four items are three from Jordan who is the user and one from task pilot. So only these four items will be present in the window. And then look at the format in which it is present. This is for this is present as an array, right? So the the last four rows of this array are now present in your memory and this can be passed into the LLM which we will see in a little bit. And how many tokens are there in the windowing? So 1097 tokens is now compressed into 176 tokens which is roughly 84% saving. I mean, of course this will be high amount of saving and this savings is only going to increase with time because the conversation length is going to increase with time. But this doesn't mean much. This simply means that you are literally discarding most of your conversation, right? So we'll pass along with the speaker name, right? You mean when we call the LLM? Is that what you're asking? When we pass along with the speaker name? Yeah, we will we will pass along the four things in this case. Speaker name, the conversation, the topic and the relevance. Then let's look at the second strategy which is LLM based summarization. There was no keyword search applied here. In this case in the in the windowing there is no keyword search. There is literally you just say the last four windows you last four items you pick from your turns of conversation. You discard everything else. Then in the summarization you pass your conversation to this variable called turns. So conversation is over here. Summarization of conversation. Conversation is the 22 conversations. And you also keep say keep recent three. Keep recent three means the last three items will be kept as such. Everything before that will be summarized using an LLM, okay? So old is everything from conversation number one to conversation number n minus three and recent is the nth one n minus one n minus two those three items. And then here we have a function called call LLM then the instruction that we are passing to the call LLM is compress this conversation into two to three sentences. So this this conversation here refers to the the conversations from one to 19. So compress this conversation into two to three sentences. Keep all decisions and facts. Drop pleasantries. Pleasantry simply means the words that do not carry any decisive or factual information. And then we are passing the old conversations here. So this variable old consists of everything from one to n minus three. And then the return is summary and then let's look at the call LLM function also. So the call LLM function was defined here. So in the call LLM whatever the input is, input is passed into the LLM in this way. chat.completions.create model equal to model messages equal to role system then so it's a system prompt content equal to system role user content prompt. So here one thing is that this role user will remain the same. But inside the content because at the end of the day when we are passing something to the LLM, the role is the user. But in the content there are two things. Some conversations came from the LLM. Some conversations came from the user, okay? So please don't get confused. Here the reason why role user is mentioned is because we are calling the LLM as a user to provide us the next output. Correct? But in the content that we are passing implicitly which conversations was made by LLM and which conversation was made by the user is already provided in the prompt over here. So this um These 22 conversations contain the role. Here the role is that of the LLM. Here the role is that of the user. So that is very different. So please don't get confused there. Then there is no meaning you are a helpful assistant in a system prompt. Can system prompt be empty? That is a good question. Let's try that actually. Let's keep system equal to system prompt as null. Let's see what happens now. So now there is nothing that is passed that is as the system prompt. Probably it will work. Let's try. Okay, and then let's call the LLM summarization. So now the output is yeah, it worked. Actually because yeah, what you what you said is correct. Just saying you are a helpful assistant. I think it's already embedded in the LLM's instruction tuning. It already knows that it is a helpful assistant and it's supposed to give answers. So it's probably not needed which is why the output it doesn't look like the output is any different from when you leave it. So the output looks like this. This summary is that of conversations from one to 19. This is the conversation 20, 21, and 22, which is left as such. And this is the best kind of summarization because savings are huge, 78% of tokens you save. Uh but more most of the factual information are retained. And the latest context which are which is untouched will be also passed into the next call. Okay, so this is LLM based summarization. Let's look at the next one, key value extraction. Key value extraction is again, we are doing it using an LLM. We ask an LLM to convert it into a key value format. Okay, can we check if there is any difference if we add that to the system prompt? Of course. So, you mean absolutely there is any difference or not? Some difference will be there, of course, because temperature is set to be 0.3. Uh that setting is provided somewhere here. We can try playing with the temperature. Let's keep this as zero. Okay? Or let's keep this as 0.3 and then let's run this once more and see if the sentence how much the sentence changes. The sentence will change each time little bit. Um Where is Yeah, so just look at the summary sentence. This this will change slightly. Jordan is developing SaaS. That's how it started earlier. Jordan is developing a SaaS project called CollabFlow, a team collaboration tool. So, I'm just copy-pasting this in the chat just to keep it handy. So, this was the previous sentence. And run this once more. This is the new output. Let's see if there is even a single difference. Yeah, there is some difference as you can see. The two outputs are not exactly the same. If you read the two chats which I have just pasted, but you can just keep this as zero. And now after this we'll try with adding the system prompt back. Um and then run this. I don't think the first thing is yet executed. Okay, so now this is the new summary. Okay, just look at the structure of the summary and let's run run this once more. So, if you see this, the summary looks pretty much exactly the same. It's word to word the same. There is no difference. So, the temperature setting of the LLM allows it to select because at the end of the day when it does the next token prediction, um there there is a softmax probability distribution. So, if the if your library token library is of 50,000 tokens, your LLM says, "Here is the next token with the highest probability from your library." If the temperature is zero, it will pick the highest probable token always, which means the output will be very different. Maybe the last time difference was because of temperature. So, because of temperature, of course, there will be difference, but more importantly the question is if we keep the temperature uh the same. So, let's keep the temperature the same, but add a system prompt. Okay, so temperature let's keep it as zero, so that effect is not there, but let's add a system prompt saying you are a helpful assistant. Now, let's see how much the output changes or what changes. Qualitatively nothing much should change, but I'm just I'm just curious how much will the exact output will change. Something will change, I'm pretty sure. So, this is the new output where I added the system prompt, but kept the temperature as zero. So, this if I compare with the previous output, it's different, right? So, of course, the system prompt is playing some role um because it's also going into the context, which means the next word next token prediction will not be exactly like how it was in the absence of system prompt. So, it will have some effect, but because this is too generic of a system prompt, I don't think it's adding any value. Right? So, let's keep the temperature back to 0.3 because I don't want deterministic outputs. I want some variability. Uh oh, even with temperature zero output is not usually deterministic across calls. They were referring to some hardware entropy. Oh, really? I I am not aware of it. Um from my understanding uh next token prediction you have a probability distribution in the form of softmax. Uh temperature equal to zero means you end up almost always picking I mean unless you have uh unless you have a some sort of a random number generator coming into picture somewhere in the inference pipeline, which I don't think is there. So, I don't Yeah, I don't have any intuition on why if temperature is zero, the LLM calls can be non-deterministic. But yeah, for now the ex- the example that we just checked, it's of course deterministic. Um So, here this is one of the best ways of compressing information because you lose the context for the far away information far away conversations a little bit, but the recent conversations you retain. Just curious, is there any evaluation metric or benchmark to validate the accuracy of summarization? Can we use perplexity method? Yeah, yeah, we can use perplexity method or it can be even simpler. You just um in general, if you want to have some evaluation, just make one API call you where you ask the ask an LLM to extract facts and information from the last 25 conversations. So, you you just make that LLM call once in 25 turns. And then when you have the response, when you have the shortened summary that is constructed, use another judge LLM to check whether that constructed summary has how many facts from the facts which you extracted. So, if you extracted 20 facts and the summary retains only 15 facts, this is something that we implemented yesterday also. But yeah, perplexity score is also one way to do that. I mean, but it's it's little bit more formal way of saying it, but the most simpler way of doing it is just just extract the facts and see in the summary how many of those facts have been retained. Uh when summarizing the older conversations, somehow we should preserve some transitioning context just before the start of the window. Yeah, that is a good question and I think that can be done by just passing it into the LLM token I mean LLM context here. So, here we can say, "Compress this conversation." And I'm just going to copy your uh comment. So, I'll just paste it here. Okay, so I just literally pasted the same thing here. Keep all decisions. Uh then when summarizing like this somehow try to preserve then So, this is the sentence, okay? So, I'll just put it back so that the structure is not broken. And now let's see if we run this how the sentence summary changes. So, now this is the new one. Let's see how the new sentence looks. Uh summary Jordan is developing SaaS project called CollabFlow, a team collaboration tool using TypeScript. The MVP deadline is set for March 30 with a strict strict scope of doing so and so while ensuring integration tests tests are written for every feature using pytest. Is that the same thing that was discussed in the last Yeah, so in the in the uh item fourth item from the bottom, it was talking about pytest only. So, it is somehow providing the context corresponding to the latest conversation. But I think in the normal summary also it was doing the same thing. But in general, the answer is yes, you can do that if you want to preserve the context of the conversation that just happened before the third conversation so that there is some nice uh segue from the older summary into the recent conversations, you can do that. Now, for key value extraction also, it's pretty much the same thing where you don't split the data into older conversations and newer ones. You simply call the LLM where you say, "Extract all key facts as a flat JSON objects." Object uh then rules are maximum 20 keys and corresponding values, no sentences, no nested objects. So, basically this will preserve the JSON format. And then here you are just removing some characters to strip markdown format uh notations. And then just to print it just to print the conversation in a JSON format. So, let's just print this. This part is very easy. You are literally asking the LLM to convert it into a key value format like JSON. So, you see the massive compression that has happened. 1,097 to 150 tokens. And username, user role, project name, text stack. So, see testing framework is pytest. So, this this came from the fourth last conversation. So, here it was mentioned. Uh noted why test integration test requires is required for every feature. So, that is coming in the JSON format. But, all the conversation the the qualitative context of the conversation is lost. Only quantitative items are present here. Then, the last item is priority pruning. And this is where the fourth field or rather third field which we had entered will come into picture. So, the third field was that of relevance. And this third field what was constructed using a separate LLM call or using key keyword matching. So, high, low, or medium. So, what we are going to do in this pruning is we will keep the high and medium relevant items and we will drop the low items. So, in priority pruning, we will say high if the priority is high or medium, call it as in the keep array. So, if the item is belonging to the high or medium, it is kept. Otherwise, it is pruned. Okay? And then what we do is in the final output, you will simply see that these are the items that are pruned. They are simply dropped. There is no LLM call involved here. The only thing that only part where if you can if you want to say LLM call was involved was in deciding what is of high priority and what is of medium priority. But, once the priority is decided, you can simply drop the conversations that are of low priority. Here, I can also do something like this. So, here the the compression is from 1097 to 858. But, here I can just remove the medium priority items also. Let's see how many tokens are removed extra. See, it went from 800 and something to 539. Only high priority items are kept in the conversation. This is also one thing that can be done. That way, a lot of verbose is removed. A lot of unnecessary information in the conversation directly removed. And then the last one, semantic chunking. Semantic chunking is the most important one in my opinion, which you should ideally do using vector embedding model. But, here I'll show you how it is done. So, first, we have groups. Groups based on the keywords. I'll just run this and show you what are those keywords. There are I think five or six groups. One group is intro, one is text stack, then authorization, database, deployment, and I think there are two more or something, styling, planning, testing. Okay. Actually, there are 1 2 3 4 5 6 nine groups. Then, what is done is look at the summary. LLM is asked to summarize based on the group. So, we are passing chunks where one chunk is all the conversation belonging to intro. Then, another chunk is all the conversation belonging to text stack. Another is all conversation belonging to auth. So, here the total number of API calls that we are making into the LLM is nine because there are nine chunks. And each chunk is separately summarized by the LLM by getting all the conversations belonging to that chunk into one API call. And here the summarization is again very good. 1097 is summarized into simply 258 tokens. And if this is all again done using a vector database, it is the best because then you will again save a cost of calling an expensive LLM. Who came up with these groups? So, the groups either it can be through an LLM call as we discussed earlier. An LLM can categorize conversations into groups or if you already know what kind of conversations are happening in your application, you can already use keyword based matching to convert your conversation into groups. So, for example, Amazon I know from one of my friends, they were trying to implement for customer care before before LLMs came into picture using pure NLP, they were trying to do keyword based matching. TF-IDF we discussed, right? Keyword based matching to classify a customer's conversation into complaints or whether it's to ask information or whether it's to ask for refund, etc. But, if you use an LLM, it is of course an extra call. It's it's more expensive, which is why almost never you should use an LLM to convert your conversation into a group. That is very very expensive, which means for every single conversation you make with an agent, you're making one more extra API call to just classify it. Instead, don't make one more extra API call. Just use keyword based matching to match or map that conversation into one of the predefined groups. Or one of the one of the clusters. You can also use clustering to create new clusters which did not exist. So, you so that you don't have to predefine the groups. Um So, based on user's question, I can find out the topic and then only provide summary of that particular topic only. Exactly. Exactly. Uh Do we store these chunks in their cluster head in the key-value pairs? So, this is right now stored as just as a key-value pair document. I think in JSON format. This can be in any format. The key is the intro and value is this. Or it can be even as an array, two-dimensional array where you have one item which defines the name of the group, another as a string which defines the summary of what that group was about. But, the storage exact file format of the storage can be dependent on what your preference is. Uh but at at at the end of the day, when it is being passed into the LLM, it will be all concatenated into one single string. Uh how to make sure that clusters are always unique? You mean this? How to make sure that these are unique? So, that is basically by definition of cluster itself. When you are creating new clusters, like two times the intro cluster will not be created because then it will be combined into one single item. Uh when you are naming but okay, your question might be how can we make sure that one cluster called intro and another cluster called introduction, how can that not happen? If that is the question, uh we will have to put some hard rules. Hard rules saying um these are the only clusters. And then if there are two clusters of similar names, maybe just for that just for renaming and deduplicating the clusters, we can just make use of one LLM call. So, if one cluster happened to be named as intro, if another cluster happened to be named as introduction, and like that, let's say you have 15 clusters and maybe three or four of them are duplicates. Simply to rename the clusters, you can just make one one LLM call, which is not at all expensive. Uh do you attach one person's conversation to the group or whole turn? You attach everything that belongs to intro. So, look at this. Uh So, here here this is intro. This is intro. This is the user's intro. This is the LLM's intro. But, both of them are part of intro. Right? So, here Jordan and Task Pilot. So, both of these conversation, these are two different. One is user and one is the agent. Both are passed into the intro bucket. Uh I don't think there is any other conversation belonging to intro. Similarly, see, this is text stack. This is also text stack. But, this is the first one is from the user. Second is from from the agent. So, as long as you if you have five conversations here belonging to the category intro, it doesn't matter whether it was made by the user or the agent. All five of them are passed into the LLM for summarization. But, they are passed as one single chunk. If you are doing actual chunking and actual vector database creation, the chunk the definition of the chunk will be all the conversations whose marker says intro, then all those conversations will belong to one chunk. And all those you know, one such chunk will be converted into one vector. Um but in this case for these simple databases or simple 22 conversation long conversation, we don't need a vector database because it's an overkill. Does LLM also provide rag out of the box when we provide when we upload files, they should get converted into an embedding sent stored somewhere. Um That is a good question whether ChatGPT provides out of the box rag. I am pretty sure they have some sort of implementation of rag. Otherwise, when you are uploading, you know, very big PDF documents of 100, 150 pages, they are they are clearly able to extract information from specific pages in that PDF. If you if you ask it some information and in in which page this information is found, it can easily find that. So, yes, I think it implicitly allows allows as in it does rag to some capacity. Um Do we also store the results from key-value extraction into a long-term memory or are they only stored in the short-term conversational memory? So, here the idea is where do we want to pass this next? We want to pass this information into the ongoing conversation, right? So, let's say now Jordan Task Pilot has to respond item number 23, then item number 24, 25, and so on. It is it will form part of the hot cash. The item which is summarized because that is passed into the LLM's context in the immediate next call. The format of its immediate storage may vary. It may be in JSON format. It may be in markdown format or PST format, whatever the user uses. But, at the end of the day, it is forming part of the hot cash of the LLM because the summary is immediately being used. And when we when it is being used, it doesn't matter what format it comes from. When it is passed into the LLM, it will be converted into a concatenated string. Um Clusters are generated user specific or is it in general for all users? Um I mean, here here there is no concept of user and I mean, at least here in this example that we are talking about, here the user is simply just one person, whoever is using this app. The agent is the LLM. And then clusters are generated based on the conversation, right? So, if the if for the same person who is using this, if the conversation diverges into some other topics, of course, some new clusters. If you are creating a clustering algorithm, of course, new clusters can emerge. But, if you are using an algorithm where there are predefined 10 or 15 clusters, basically, what is done is classification. The conversation will be classified into one of the 10 or 15 clusters. Uh is the chat history maintained for each user session separately? Um actually, in ChatGPT, at least from my latest experience, that is not the case. I think between different chats also, the memory is maintained. Um I have I have many times noticed that because my wife and I use the same chat, sometimes LLM assumes that I am my wife. And sometimes, when she asks the question, that it assumes that it's me. Uh and we might be chatting in different chats, not at the same time, at different times. So, at least in ChatGPT, the way they have constructed it is so that it is ideal for one single user. I'm not talking about API, I'm talking about ChatGPT app. In API, there is no such thing. API is like it's like completely under your control. Uh okay. So, now priority pruning is done, semantic chunking. Uh these are the chunks, and let's just have a quick comparison between these two uh these uh six uh or five approaches. What did you achieve by creating the embedding for each chunk in this case? By the way, I did not create embedding, that's what I mentioned. Uh ideally, I should create embedding, but here, what I did is I summarized the each chunks into a small conversation. So, here, the chunk which which is called intro, which included two conversations between Jordan and the agent, is now com- you know, compressed into one small sentence. So, and look at the total number of tokens. Total number of tokens, uh if all these chunks were expanded, the total number of token tokens will be 1,097. But now, since all these uh chunks are compressed using an LLM, it is converted into 258. If you are using embedding, the chunks will be converted into vectors, then the vectors can be used for retrieving the chunk back in natural language. Uh if you embedded this, how would you retrieve and for what? You will retrieve the um Okay, the question is, if you are you Okay, for why um for what purpose would you retrieve? A good answer for that is, let's say your conversation history is 300 conversations, and let's say there are 30 separate chunks. And now, the latest question that the user asks is regarding database, and not regarding anything else. Then, maybe from the memory, you don't have to pass all of these chunks into your next LLM call. You just need to retrieve the chunk corresponding to database by doing uh vector database matching, and then find the vector corresponding to the chunk database chunk. So, basically, user query will be there, which is the latest conversation that has happened. Convert that into a vector. So, let me just uh draw it somewhere here. So, you will have So, let's say user asked a question to the agent regarding database. Now, for the LLM for to respond, it will get access to this particular query, plus some things from the memory. And what are the things from the memory here the LLM needs access? Does it require all the 30 chunks? Probably not. It just needs the chunk corresponding to database. And how is this database chunk found? The question of the user is converted into a vector. Then, in the in the database, maybe there are um 100 chunks converted into 100 vectors, right? And then, you will see based on cosine similarity or any other metric, how close is this vector to any of these vectors. So, if this vector is closest to this vector in terms of cosine similarity, you will retrieve this chunk. And meaning this vector, and then that vector will have an index, and that index will give you the access to the actual chunk, which is the conversation. So, that actual chunk will now be passed into LLM along with the recent conversation to produce the next output. Um But, how do I know the user question is about database and uh not about intro? Yeah, very good question. So, in typical rag, um you have to first um Yeah, one way to do that is the user's question is just converted into an embedding. So, you don't have to know whether this is regarding database or this is regarding intro, right? So, the same embedding model, whichever embedding model converted your memory into vectors, use the same embedding model to convert this user's question into a vector. So, you really don't know whether it is regarding database or not. All you need to know is to which chunk is this the closest. Then, if this is closest to the chunk corresponding to database, then that is the proof that this is something related to database. That's the whole idea of rag, right? In rag, you don't know uh whether the user is asking the the question that the user is asking is closest to something that is already there in our vector database or not. So, just by using the same vector embedding model, you can find whether this vector is closest to um one of the chunks in your database or not. Uh how do you decide which strategies you should implement for your app? You mean, whether you should use rag or whether you should use um some keyword uh key-value pair-based uh approach or not? If it depends on the seriousness of your app. In my experience, if the app has to be really good in handling the context, you must have rag. You must have a vector database because any serious application, you think about it. Any serious application will have a long history of conversations with a given user, or a long set of documents which will serve as the knowledge base from which the LLM can get the context. Now, in what format will you store this knowledge base if it has 3,000 lines? Are you going to store it as pure JSON, and then are you going to do key keyword-based matching, or are you going to uh store it as markdown file? Because if if you have a markdown file with 5,000 lines, right? Each line will be 10 words. Each word is one token. So, that is 50,000 tokens. Your your LLM might have only 200k tokens as its um uh token window, context window. You're already polluting your context window unnecessarily. So, what you have to in almost all the cases, any serious chatbot will require some sort of rag to handle its knowledge base and to handle the uh conversation to fetch relevant information from the database into your contact into your LLM's context. Uh for application rag would FAISS suffice most of the times. Yeah. So, Facebook AI semantic search, FAISS is Facebook AI semantic search. It is one of the fastest and most efficient one in terms of fetching the correct um vector from your database corresponding to the vector of your user query. It is it is one of the most commonly used. It is one of the fastest. In my experience, it is the fastest in in the all the projects that I have implemented so far. It is by far the fastest. Um Okay, let's look at uh the last thing in this uh Google Colab notebook. So, last thing is this. How did the compression work when we converted the conversation for memory? So, original item was 1,097 uh tokens. Priority pruning was the worst because in our conversation, very few items were of low priority. So, the compression only helped so much. But, LLM summarization was very good, very high um uh summarization. Windowing was very good, but also useless. Um Then, the next most useful thing is semantic chunking. So, either of these are very useful, windowing uh sorry, summarization or semantic chunking. But, the most useful one is semantic chunking because as your conversation grows, you need to um semantically convert it into vectors, store it in a vector data- database for your future retrieval. Uh all right. Let's also open uh this uh website once. Just have this ready. We can just see this in action in a slightly interactive way, okay? So, here, I have again another full conversation. This is a different example here. There is uh Jordan Task Pilot, but slightly different example. Uh the conversation might look almost similar, but I think it's it's there is slight variability. So, if we apply windowing, let's see what happens to the conversation in terms of number of tokens. The tokens went from 1,238 to 368. And this is what many of you were asking, how do you know whether the summarization actually worked or not? So, here, what I have done is I have asked the LLM to make a separate um list of facts and uh quantitative uh data from the conversation. So, these are all facts, right? The username is Jordan, project name is Colab Flow, etc. And then, you will make another LLM call to check, or you can make another keyword-based searching to check whether in the compressed context after windowing, how many of these 24 facts are present. In windowing, very few facts will be present, that is needless to say. The reason is in windowing, you are eliminating all the previous 19 or whatever number of conversations you have before your window, right? So, of course, this will be linear. Like, whatever facts existed before your window, they are all lost. If it is LLM-based summarization, let's see what happens in terms of facts retention. Most of the facts should be retained. So, you see, LLM-based summarization was almost as effective as windowing. So, windowing resulted in 368 tokens. Some LLM based summarization only has 280 tokens. So, there is more saving in terms of tokens. But, look at how many facts are retained by LLM. 21 out of 24 facts are retained by the LLM. So, the information retention rate is 88%. Then, key value extraction will be also amazing in terms of compression, but let's see how many facts are retained. So, in terms of compression, this is the best. 180 tokens only. It's even better compared to LLM based summarization. But, in fact, this key value extraction is also done by an LLM. Retention is checked by checking using an LLM API call whether in the summary document how many facts out of these facts. So, this is the now the facts list. So, out of this list of facts, how many of these are actually present in my summary? So, I'm making an another LLM call internally to check this. This can be done using an LLM call or using keyword matching. Both are fine. Right? So, that will LLM itself can give a score. Score saying 18 facts were found in the final summary out of 24 facts. And when it comes to pruning pruning is not the best, but here lot of facts are retained because here when if you look at the priority that LLM has assigned many of these conversations LLM has assigned as low priority. So, one thing I want to check is Yeah, so you see this? In the in the first conversation the priority is not low, it is high because it defines the user's name the name of the project. I mean, name of the project is collab flow. Name of the project, my name is Jordan. So, all these things are defined in the first conversation. But, in the second conversation, look at this. The task pilot's response is just verbose. It's just saying it's just being nice to the user. So, this is given low priority. So, this is completely removed. So, you can see that many of the responses by task pilot is just mirroring. It's just relaying what the user has said. And then it's saying this is what I'm going to do. So, many of them are given as low priority and because of that pruning will eliminate all of that. So, most of the things that are retained are Jordan's communication. Very few related to LLM's responses provided. And see, most of the items which are part of the checklist are provided by the user only. And because of that, even after you do pruning a lot of the things are retained. Only two items are missing. So, the score is 22 out of 24. Even better than LLM summarization. But, this is depending on a case-to-case basis. In semantic chunking these are the chunks that are retained. Intro, text stack, database, deployment, MVP plan, practices, etc. Each chunk is now summarized through two or one or two or three sentences. And look at the amount of information that are retained. This is very very high information retention. Look at the compression. 74%. So, compression is very good. Memory this retention of the information is also very good. So, semantic chunking in my opinion here performed the best. Right? So, this is just a illustration of what type and this is exactly what we want, right? In the when we are passing context into a new LLM call you don't want to pass some random useless information which are just nice communication between the LLM and the user, but you want to pass facts and quantitative information. And here this semantic chunking is working to a great extent. Let's check one more thing. We discussed about memory tiers like episodic memory, semantic memory, procedural memory. Okay, there is a question. Semantic chunking using vector DB is conversation will be grouped into different categories using rag by converting conversation into embedding and then comparing similarity against vector search. So, conversation is grouped into different categories. That is not using rag. Okay, I think there are two things to to discuss here. First, to create the vector database how do we create the grouping? Before the vector database is present in the right because when we are performing rag, we already should have a database, correct? And to that database, we are comparing the upcoming conversation. So, to create the database in the first place there should be some intelligence in the embedding model. So, the embedding model will decide um what the vectors will look like. And before passing something to the embedding model, if we ourselves are doing the chunking how do we know which conversation belong to which task? Whether something belongs to intro, whether something belongs to deployment because to perform chunking when we discuss rag, we did few types of chunking. Fixed size chunking, we did windowed chunking, we did semantic chunking. So, there were different type of chunkings. So, similarly here to chunk the conversations into different categories, you need one layer of intelligence. That intelligence can be rule based. It can be purely based on keywords or that can be used on using LLM. Now, once the chunks are created the chunk is converted into a vector database. So, now you have the vector database which stores the information corresponding to the past conversation in the form of chunks in the form of vectors. Now, when a new conversation happens, how do we know whether these vectors in your which of the vectors in your embedding is corresponding to your new conversation? For that the new conversation also is converted into a vector using the same embedding model. And then that vector is compared with all the vectors in your database and whichever vectors are close to each other whichever vectors this particular user query vector is closest to, those chunks or those vectors will be retrieved. And they will be passed into the context. So, this is how rag works in in this conversational chatbot. Okay, so let me ask you few questions. How about this? Always format API responses as JSON with a consistent envelope. Is this an episodic memory, semantic memory or procedural? This is about [clears throat] a procedure, right? This is about how to do something. So, procedural memory. Last Tuesday, the deployment failed because the Docker image exceeded 10 GB limit. What is [clears throat] this? This is something about something that happened in the past. It's about an episode. This is just exactly like how humans store information. It's it's an episodic memory. The development team has four back-end engineers and two front-end engineers. This is not episodic. This is about semantic, right? It's about what is the truth. What is that what is that the LLM knows. When debugging production issues first [clears throat] check the error logs, then metrics, then deployments. This is again about implementation. It's about procedure. So, it's about how to do things. The user reported that API returned a 500 error when uploading files larger than 50 MB. Is this episodic or semantic? So, this is a little bit confusing. Is this something which happened or is this something which I know? Let's go with episodic because this error which occurred is something that happened and it's just that the user has reported it. Then, the user prefers Python over JavaScript. What about this? Is this procedural or semantic? Because is this defining when we are implementing the code, I should use Python. Or is it saying is it talking about some sort of user preference? So, this is about something that the model knows about what user's preference is. It can be about language, it can be about color theme, whatever. So, it's it's semantic. When the conversation gets technical automatically switch to shorter, more direct responses. This is about how to do something, right? How to do how to implement a certain procedure. So, it's procedural. On February 18th, the user asked about pricing for the enterprise plan. What about this? Is this episodic or semantic? This is again a little bit confusing because here is the question about is it about the fact that user asked something? Or is it about the fact that um user had had made a request regarding pricing for enterprise plan. So, this is a bit confusing. Let's try with episodic. So, episodic it's like what happened. So, user asked about something. That is what happened. The project deadline is Q2 2026. What about this? This is a knowledge. This is what I know about the project. So, this should be semantic. When the user asked for a code always include unit tests along with implementation. This is again regarding how to implement something. So, this is sort of the memory sort of procedural. How to do. The user decided to switch from MongoDB to PostgreSQL after comparing query performance. So, this is It's about an incident, right? This is about I don't think this is episodic. I think this might be semantic. I might be wrong. I am also equally confused whether this belongs to episodic or semantic. Let me think about it once more. It's about user's decision. User decided to switch from MongoDB to PostgreSQL after comparing query performance. Uh, mhm. It's about an event. Or is it about what happened? User decided to switch. I'll go with what most of you are saying. Most of you are saying it's episodic, right? Let's see. Okay. This is because it's a it's about a user's decision. It's not about the knowledge that user prefers PostgreSQL over uh it's not about the knowledge that user prefers PostgreSQL over MongoDB. It's just that in that moment user made a decision. API rate limit is set to 100 requests per minute per user. This is about knowledge. Then for this user, use markdown tables instead of bullet lists when comparing options. Is this procedural or semantic? This is about when you are doing something, do this, right? So when you are following a procedure, do this. During the sprint review, the team agreed to delay the payment feature by 2 weeks. What about this? So it feels like it's episodic. Then the user's email is Jordan at thisthing.io and they work in the Pacific time zone. This is about the knowledge. Then two more questions. Uh three sessions ago we refactored the auth mode module to use JWT refresh tokens. Episodic. Then before suggesting database schema changes, always ask if migration need to be backward compatible. This is again how to do something, procedural. And last one, company uses AWS for all cloud infrastructure. This is about knowledge. So it is semantic, right? All right. Fairly straightforward. Not fairly straightforward, mostly straightforward. Some of them might be confusing. Let's look at one more thing. Um about uh memory assembly, like how the memory gets assembled from different um Yeah, very good question. Where there is no user interaction. How the memory is used in agentic application when there is no user interaction. In fact, most of the uh coding agents are like that only, right? Coding agents have so Cloud Code has to decide by itself not just based on conversation, but based on other file structure, how to assemble the memory. So let's try a simple quick exercise on assembling the memory. Uh first is what what aspects of the memory is hot, what is warm, what is cold is decided by Cloud Code itself because of it has intelligence, right? So now the way it works is you have a token uh context budget just for memory and from this the way it is added is if there is a conversational window which is stored in some format, it can be stored in a you know, JSON format, it can be stored in a markdown format. Cloud Code will decide to add this to the um current context window. So here is where the LLM smartness is coming because Cloud Code So this LLM context window is different from the context that the agent is using. See, when Cloud Code or any other coding agent is implementing its decisions it has it has multiple context window. It will have its own context window to decide whether which files are relevant for the main context window. The main context window will include all the information regarding what the main task is. So if the main task is to produce a response based on all the available context, then that main context window should only be populated with what is relevant. The purpose of the other API calls is probably to decide which among these files should be part of your memory um um to be passed into the LLM's context. So here conversation window is added. Then this is the episodic memory. So there are there are conversations which are or episodes that have happened in the last week and last to last week in the format of JSON. So 150 plus around 100 tokens or 254 tokens. Then semantic memory. This is about the user profile. So user profile will be stored um in some format. So semantic memory can be again in vector database format or in any other format. So if an LLM made the compression into a JSON format, it could be in the JSON format. But it can also be in a vector database format. Now I have one question, okay? If I click this these vectors are getting added. Is the vector getting added directly like this? Or is it something else? So if you have a vector document is it directly getting added or is it something else? So here the way I'm showing it is 267 tokens corresponding to this. These two vectors are getting added. See, vectors are just numbers. If you look at a vector, it will look like an array of numbers, -1.2, 1.3, etc. If you just add that vector directly, it will not make any sense to the LLM. What you are actually passing is when we say you are passing certain item from the vector database, what you are actual actually passing is the chunks corresponding to the vector, right? Those chunks are sentences. They are being passed. If an entire vector database is passed, it simply means the entire chunks corresponding to that database is passed. If only some vectors from the database is passed, it means only some chunks from that database is passed. So now this is how different files are assembled. Um and these tokens corresponding to are Now one more question. This 124 token, 267 tokens are these the tokens corresponding to the the dot vec file? Or are these the tokens corresponding to the chunks corresponding to the vector? They are always corresponding to the chunks corresponding to the vector. Why? Because at the end of the day, tokens are about what the LLM is seeing. If LLM is not directly seeing the vector array uh if it is directly seeing the chunks, then the tokens are calculated based on the chunks, not based on the vector the the numerical vectors, right? So these are based on the the chunks which are in the format of um I mean, which which are in the format of natural language sentences. So and then all these uh tokens will be added to contribute to the context budget. And now this all these will form uh part of the context and these are from the memory. Um For the question in this exercise, do we need all these contexts? For this one, yeah, I mean, it is constructed in such a way that uh we have to select everything to construct a a really really good context to get an answer. Uh I did not walk through the question. I just wanted to show you if you had a certain folder structure or file structure how are things assembled to be part of the context? At the end of the day, context is about concatenation of different items in natural language to be passed. Here one token is not equal to one chunk. One chunk is equal to multiple sentences. Multiple sentences equal to multiple words. One word can be one token or multiple tokens. So here if there are 124 tokens, it doesn't mean there are 124 chunks. It means there are some sentences in which maybe let's say on an average each sentence is 10 words, which is which roughly approximates to 10 tokens. There are around 12-ish sentences here. There are around 26-ish sentences here corresponding to all the chunks that are passed here. But one chunk is not equal to one token. Typically, one chunk is a longer sentence. Otherwise, it doesn't make sense to just store one word as one vector, right? Because one word does not carry too much semantic meaning. But a sentence carries a lot of meaning. So usually chunks are multiple sentences converted into one one vector using sentence transformer or similar um uh vector embedding models. Um okay. So this also we have covered. I have a few more questions to ask you in the form of quiz. So let's go there. Okay. So let's go through a few questions just uh to test your understanding so far. Why do LLMs need an external memory system? You can just uh say A B C D. The LLMs need external memory because LLMs have no state. The API each API call is unique, right? So LLMs are stateless. That's the reason. Then what happens when a multi-session agent has no memory? So what happens is if this agent has no memory and if it is configured to ask some questions while onboarding a user, it will keep on asking the same things. It will not remember anything from the past sessions. The past context is not not will not be there. A user tells the agent their project deadline is uh Q2 2026. In session five, the user asks, "What's our deadline?" But there is no memory. What does the agent do? So, in this case, what does the agent reply? So, what will most likely happen if it's uh if it's just an agent um I mean, if it's not an LLM, if it's an agent, it's because um you would have put some sort of um framework for agent's information uh absorption, right? Because agent has to absorb some information. And typically, agent gets information from the user. So, agent is usually trained to ask questions to the user rather than directly give an answer, unlike LLMs. LLMs may directly give an answer, but even in the case where I asked, "What is your name?" to the LLM, it did not hallucinate a name and gave a random name. Uh it gave it said that I don't know your name, right? Because it's not provided. So, typically, it asks for information which it does not know instead of directly hallucinating. Why does human memory differ from LLM LLM memory in a How does human memory differ in a fundamental way? Fundamentally, LLM does not have memory. Uh fundamentally, LLM has uh its weights and biases to produce a response during inference. Um so, humans, however, we have persistent memory, which is maintained automatically what not. But LLM memory is explicitly engineered. LLM itself is like a RAM, right? It does not have storage. Um Which memory tier stores learned behavior rules like always include tests with code? Whenever we say, "Always do this, always do that," it's about some implementation of procedure. So, it's procedural memory. Then, a user tells the agent, "I prefer Python over JavaScript." This we already discussed. Which memory should be should this be? This is user's preference. So, what should it be? Semantic memory. Then, which memory fits best fits? Three sessions ago, we refactored the auth into JWT refresh tokens. So, this also you guys answered very nicely earlier. It's about what happened. So, episode. For this user, use markdown tables instead of bullet lists when comparing options. This is best classified as dash type of memory. It might be a bit confusing semantic versus procedural. So, this is a command that is given to the LLM. So, it's like when you do this, do that. When you do something, do something else. So, it's like procedure. What is the main disadvantage of windowing? Fairly straightforward. Windowing simply eliminates all the information uh which came in the context early. Then, which memory strategy achieves the highest compression out of all the strategies we discussed? So, in general, the highest uh compression strategy, the best compression strategy is key-value extraction because it is storing everything in just few key-value pairs. Look at the key-value pairs. They are just words. They will look like they are long, but it is mostly empty space. It's most It's few words. So, the number of tokens in key-value pairs, although the way it is printed, it feels like there is a lot of it occupies a lot of space. The actual number of tokens it consumes is very less. What is summary drift and why is it dangerous? Summary drift is as we keep on summarizing um what happens to uh the uh summary our after some time? So, what may happen is uh after few rounds of summarization, you will lose uh start losing information because each summarization is loss of information, right? It's like any compression algorithm. There is always a loss of information. So, it will keep accumulating. A coding assistant needs to remember project facts and specific debugging sessions. Which strategy combination is the best? So, project facts will be stored as what? It's like what I know, what is the tech stack, um uh and uh what kind of uh architecture should I use, etc. So, I think it is semantic chunking plus key-value extraction. Then, in the three-tier storage model, hot-warm-cold um storage, what does warm storage contain? So, warm storage is everything about the things which LLM did very recently, not not stuff in the current context window. So, compressed summaries and recent history that can be quickly loaded. Then, which stability tier should a temporary debugging hypothesis be assigned to? Temporary debugging hypothesis is when the agent decides to uh spin maybe a sub-agent or something to test a temporary debugging hypothesis, perhaps. So, this is Uh we did not explicitly discuss about stability tier, but uh uh I think it should be some sort of experimental in the sense that debugging is almost almost always the moment the bug is fixed, you don't care about that whatever debugging hypothesis is. So, it's sort of experimental. You don't even have to store it the moment it is done. A memory's access count uh drops to zero over 30 days. What should the decay system do? Basically, there is a concept of memory decay in the sense like you assign lower and lower priority uh an LLM or keyword-based uh uh mapping will assign lower and lower priority to items that are too old. Right? So, what typically happens is uh after 30 days, if the number of times this particular memory is accessed is zero, you can literally reduce its priority because if something is not used in a 1 month, its priority is uh very low. It can be kept in cold storage. You don't have to delete it. Why should memories include a confidence score alongside the stored information? So, we did not discuss the confidence score of the memory. What we discussed was the score not the score of the memory, the relevance of the the uh raw item is what we discussed. We never discussed once we convert it into a memory where we put a confidence score or something. So, low low confidence uh so, low confidence memories are um useful when in some later point of time, if you are ever utilizing something from a low confidence memory, you can have a you can you can err on the side of caution, right? Uh because let's say in your memory, something is stored that the user always prefers um JSON format for structured documents, something like that. And then later, this this has changed. So, if there is a low confidence memory that is referred to, um that score will help the agent ask a question like, "Is this still true?" Something like that. So, but this is something which we did not discuss, so don't worry about it. Ace framework, we can skip for now. A production coding agent starts giving advice that contradicts the decisions it made 3 months ago. The user is frustrated. What's the root cause? This can very much happen when you're working on something. So, what happens is the files which are stored earlier, the LLM will assign or whatever the coding agent's framework is uh framework for annotating the memory, it will keep on assigning older and older ones with lesser and lesser priority. So, the uh past documents will have lesser and lesser relevance. Claude codes uh claude.md is a exa- is an example of which memory architecture concept? So, claude.md is loaded in almost every single uh API call, if not fully, parts of it, right? So, it should be something like uh hot storage because uh it's not warm. Uh it's working memory temporary instruction. It's not temporary at all. Claude.md is like always See, it's it's literally uh consisting of bunch of system prompts. Uh so, items from cloud.md will form part of your system prompt. So, cloud.md is the is always loaded. Not the entire file, parts of it, depending on whether you use rag or you use keyword-based matching. What is the recommended memory strategy combination for a production multi-session agent? I mean, this is very simple. I mean, when you have a production agent, you have to use um all the tiers, episodic, semantic, and procedural. All everything will be there. Then, so it's a very straightforward. In production chatbot, who or what is responsible for deciding which message which past messages to include in the next API call? Um so, think about it. It's a chatbot. And in the chatbot, who will who will decide what comes into the context in the next API call? Of course, it's not user. User is not repasting or anything. Um because that is going to be very difficult. If I'm using a chatbot of a company, LLM cannot manage its context window. Context window is managed elsewhere. API gateway, it caches recent messages and injects them transparently uh into each request. Or application developer, they write the code to select, compress, inject history into. So, we have to whoever is designing the system to make the API call to the LLM, we have to decide it. It's our decision. Right? So, we have to decide how the past messages are uh being passed into the LLM. It's not decided by the LLM. Um You are building an AI tutor. Between sessions three and four, the student's context window is completely cleared. What does this mean for session four? If context window is completely cleared, um there is pretty much no knowledge existing from previous sessions. If it is completely cleared. So, then session four starts just like a fresh API call. What is the fundamental difference between context window and persistent memory? So, persistent memory context window is like short-lived. Context window um short-lived or it's it's uh dynamic. Once it exists for one API call. So, the current context window exists for one API call. For the next API call, we need to fill the context. So, but persistent memory is depending on external actual stored files. Jordan told the agent, "My company uses monorepo setup." Uh then 3 months later, the agent recommends setting up a separate standalone repo. What type of memory failure has caused this? So, this is some sort of knowledge, right? Um so, "My company uses monorepo setup." With this is like a it's like a knowledge. So, it's not about implementation. The agent recommends setting up a So, I think it's a failure of semantic memory failure. Because it's about knowledge. Then, an agent observes that the user always asks to explain like I'm 5 year old whenever they are confused. Should this be stored as episode or procedural memory and why? It's about procedure. When an LLM produces a response, when you are doing this, do like this. Right? So, it is about procedure. Then, the minimum viable prototype deadline is March 30, 2026. Which memory tier should be uh should store this fact and why? So, this is semantic. Because it's about Oh, this is not semantic. My MVP I think this is semantic. It's not about uh Wow. Which memory tier should this store? Mhm. I think it's semantic because this has not yet happened. I'm just confused because uh because this is too crystal clear instruction. Okay, this is the deadline. I think it should be semantic. Um this might be a mistake. Then, let's go to the next one. You compress a 30-turn conversation into three sentences. The user asks, "Why did we choose TRPC over REST?" The agent says it I don't it doesn't know what happened. So, summarizer, usually, as you know, it keeps facts. It keeps um uh solid quantitative things, but not the why behind certain decisions. Uh KV store has auth, clerk. The user says, "I'm considering switching to auth.js. What do you think?" The agent enthusiastically enthusiastically agrees without hesitation. What went wrong? KV store has no context. Then, priority pruning drops all turns scored below three. A turn scores two and says, "Quick side note, uh the client is based in Germany, so we'll need uh GDPR compliance. What is the consequence?" So, what may happen is just because the user said it's a quick side note, maybe the LLM or keyword-based matching might have given it a low score. And because of it, priority pruning will drop it. In priority pruning, low-score items are dropped. But the problem is GDPR compliance is very important. Um so, maybe this constraint is lost just because we in the pruning it got dropped. A memory user prefers vim keybindings. Hasn't been accessed in 6 months. The decay system triggers. Um what's the best action? Move it to the cold storage. Uh and no need to delete it. Then, semantic memory says uh "Users prefer PostgreSQL." A new session contains "I migrated everything to MongoDB last month." What should the memory system do? Uh this is a little bit confusing, right? Should we keep the uh PostgreSQL semantic memory? Or because this episode has happened, should we also update semantic store? Um I think we should keep the older preference, but still update the semantic store. That is what I would do. Um because otherwise, there will be again conflict later. If one part of memory says user prefers PostgreSQL, the other part says factually that MongoDB is what is being used right now. Uh in the hot, warm, cold storage, which of the following belongs in the cold storage? This belongs in the cold storage. It's too too old. It's not deleted, but it will be in some sort of markdown files or txt files or JSON files which are barely referred to. This we don't we can skip. Um After deploying a production agent, users report it keeps recommending "Add an index to the users table." even after they have already done it. What's the What's likely the memory failure? Basically, there is no memory which says something was already done. And because of that, LLM will keep on referring to um the same thing again and again. Why is a well-engineered memory system considered a competitive mode for AI products. I mean this is more like a qualitative question. The main thing is that see why do we care about AI chatbot and all it's like highly scalable and highly personalizable. So if something is highly personalized um it means it has a strong memory behind it. It knows the user preference in and out. So you have to be able to somehow house store the user preferences to be utilized. Uh we will do this when we are building the mm application two days from now two days or on weekend. I'll confirm it in some time. But um this is uh about memory. You should know how to explicitly give instructions to the LLM. We will explore that a little bit in the building stage. But we should give explicit instruction to the LLM um how to store in what format to store where where to store etc. If you are utilizing coding agents not LLM I mean coding agent. If you are using uh cloud code cloud code um inherently does bunch of these things without you having to explicitly define it. Uh okay based on query how does the agent decide whether to retrieve information from episodic or procedural memory? Does it perform this retrieval for every query? Yeah so uh in coding agents like uh cloud code in every single call that you are making or every single action that you are taking things are coming into the context from various different places. Uh some are coming from memory some are coming from the immediate user instruction. So uh cloud code through its API call a separate API call it can decide from where to query uh the latest information to be passed into the um context. So when whenever you think of cloud code you should not say okay I am using cloud code which is making one API call to Opus 4.6 to implement this task. It's not like that. It has multiple agents. Some agents are good at handling file system on your local computer. So if you ask it to create a file or delete something move things rearrange it it's good at doing that. And how do you do it? It it has access to maybe your file system MCP. It also has access to LLM. So there one set of LLM API calls are being um handled. Another set of calls this cloud code might be making to fetch the relevant information from the memory. And then finally all of this will be compiled into uh your final or or your main task API call. So if the main task is to reply to your user or to implement a certain line of code for that the LLM would fetch the information using other set of API calls fetch the information and pass it to the context. So it's like cloud codes or codex operation is like a a series not series it's like bunch of LLM calls happen in parallel um towards constructing your uh context window for your main API call. Right? So we'll wind up today's lecture here. Uh tomorrow we'll come back and see some some sort of uh patterns that we have to follow in production. Uh I have uploaded the yesterday's lecture but it will be live by only today 3:00 p.m. 3:00 p.m. IST so it will be available at that time. Um and please get ready for the implementation session. So I have discussed the dates of implementation for Dr. Raj and Rajit session so I'll upload that in the portal. Uh then second thing is please be ready with some sort of um coding setup. So please install Visual Studio Code that's the first thing. Uh I will I will put all the prerequisites so just be ready with it. And if you can if you can manage just I would highly recommend at least for one month this one month just take um cloud code subscription once. There is a separate course on modern software development where cloud code will be extensively used but in our build sessions also we'll be using uh cloud code extensively. So at least for one month if you can just take the cloud subscription uh so that uh the pro subscription so that you can use cloud code. It will it will truly change your life. Once you use it I think if you use it seriously you will get addicted. It's very empowering. The dates I will upload on the portal. I'll I'll let you know. So we'll meet again tomorrow same time uh 9:30 a.m. Uh thank you so much everyone. The implementation session so this course has three implementation sessions. One will be handled by me one by Raj and one by Rajit. So that will be separate but other than that there is a separate course of for five days which Dr. Raj is handling. It's called um modern software engineering. This is the link. So this will start next Monday. It will run till Friday. So it will happen around the same time I think slightly earlier 9:00 a.m. to 11:00 a.m. IST or something like that. So just check this link. Uh so in this it will be purely building like you will be um entirely building various applications um using cloud code framework. You will learn how we internally use the uh framework for building initial document files uh initial instructions for cloud code for building proper softwares. Um Put coding session class with uh with the software course. There will be separate coding sessions. Yeah. Is there a free version for modern software developer course? I don't see I'm not sure if it exists or not. I am not I'm not 100% sure. Probably it's not there. All right guys uh let's meet again next uh tomorrow and then tomorrow I'll let you know regarding the schedule for my build session as well as Raj and Rajit's build session for this course. Thank you so much. Take care. See you see see you tomorrow.