Overcoming Agentic Memory Management Challenges

Key Takeaways

So imagine memory as a tool, something which is still debatable if the AI agents actually need a mechanism to forget. Imagine it's not us who are controlling these tweaking parameters, but it's an agent. The vision memory should be able to communicate properly with the textual memory that it has access to from day to day. >> You dove into the memory world for the past 5 6 months because you built out Cortex. Can you give me a lay of the land at the high level? >> So when I was starting I think uh at that time uh there was not anything very specific. There was like me and there was uh lang me which was like quite popular at that time. What piqued my interest was kind of came out at that time. It's like around I think 5 months ago five or 6 months ago. So uh oh also like Emilops community had a very nice reading group discussion on AMM and we just like discussed the whole paper there which was very helpful to understand things even better. One of the key ideas I saw in AMM was uh let's say you're inserting something into the memory or like database or whatever a sort of a collection you're saying which the agentic system will have access to. So the key differentiating factor was it sort of forms relationships. I'm having this talk with you right now and currently we going to have this talk uh like I knew that in my mind. So let's say few days back. So uh currently I'm connecting this current state with my past memory because it's sort of interconnected and if you see there's a relationship between these two memories. It's like we can also like name these relationships something like is a result of that. So current state is a result of the previous state that we were discussing about previously. So something similar like uh amm does is it takes different different memories that you want to store in inside the whole memory collection or whatever you say and it sort of forms relationships between them. So if I dive a little bit deeper, uh let's say you are inserting something in the memory, you take that and if it's like the first thing you're inserting, you just insert it directly. But before inserting you just analyze it a bit like sort of come out with a summary of like what it's about. So we are calling it like context. Then we also come out with like keywords and tags. So like what are the main key things in this? Now by the way this small memory chunk can be as big as possible or as small as possible. That's on the client side that we're giving like and that's like on how you actually use this memory system. That's what AMM was trying to do. And >> so in a way it's it's like a knowledge graph. You've got this chunk and then you have metadata around it and that metadata connects to other pieces of metadata. >> Yep. Yep. So the metadata doesn't directly connect but you have these sort of metadata and there just so currently let's say you have like few memories already in place then a new memory comes in it sort of does a semantic search between all the memories that are in the collection and let's say it takes out like top k of the memory now once it has like the top k memories it asks llm with their uh memory representation so we don't use the whole memory we use the summary and the key words or like tags and whatever other mater uh do you see any sort of relationship between these existing ones which we found to be similar to the current coming memory. Now if you do then just connect them. So what I mean by connect them is uh update the existing metadata and also give a nicer metadata to the current coming memory and that's how the whole graph keeps on forming. So it just sort of makes a connections in between these two and at the time of using using of the memory. So like let's say a query comes in now we have to look for like the most relevant ones from the memory. So what we do is we just analyze this query with an LLM. So the one differentiating thing I think in AMM is on every insertion and before like every retrieval we do an LLM call >> because we are trying to pre-process and AMM doesn't really focus highly on the latency side but more on uh the quality of the memory that you're getting out. >> I was going to say it sounds very akin to a search problem. Yeah. Yeah. For sure. So currently we just take out the metadata thingies and now we just literally we do a semantic search on using two things. One uh will be the what the query was and the other thing will be the keywords from the meta. So from the query that we extracted. So we do two retrievals parallelly at the same time >> and then uh we take those two results and just feed it onto the context and do whatever we want with it like pass it to another them to answer something or >> that's very much on the personalization side right >> yes I mean you can literally do anything with that and this is like the most basic core part of AM >> um where it didn't really perform that well compared to the exist existing methods that was there. But what was striking to me was the relationships forming in between. >> Okay. >> And I was kind of trying to think very deeply on like how humans think about memory >> and how we can actually improve it using this relationship foundation because I the only place I saw this relationships uh in place was in graph rag. So, but I I currently feel that in graph we have all of these like different nodes and we compress down information way too much. So, that's that's one of the issues and that was not there in the AMM thingy. >> Then you got inspiration from AM. >> You thought deeply about what and how memory works within humans and then you said let's take it another step and let's create our own memory and >> Yeah. Yeah. I was I was like brainstorming with u few of my friends and like whoever's was like in this space and was reading papers basically and saw that so currently I was trying to I think I I use like deep research on like to understand basically how human also thinks specifically from the memory perspective like on a very high level and also a bit low level. So they have this thing called like long-term memory and short-term memory. So I mean we have recently seen in the AI space or the LM space the short-term memory. It's it's nothing. It's just the recent uh window of the chat that you're having with the AI agent or the conversation assistant or something. That's one form of short-term memory. But I wouldn't say that's a short-term short-term memory because it has to be a bit bigger than that. So but about long-term memory is this is something crucial. I think that's where all the magic happens in the long-term memory. >> So the relationship forming and like everything storing nicely and this whole big graph that we are storing in inside that is part of the long-term memory. So and we don't expect to long-term memory happen like as soon as possible. So I mean it can always keep running on the background. It's it's something we don't expect to happen instantly but it will it will take its time but as you as we having this conversation we are building onto things constantly in the back of our minds >> and it's not like I mean I'm using certain parts of it but that's from my short-term memory. So short-term memory is sort of like a window. It's constantly updating, updating, updating. But in long-term memory, we are just storing everything like kind of dumping and while we are dumping things are processing parallelly in the background which is extracting lots of merit out of it and interconnecting things in between. >> Yeah, I remember taking a online course a while back now and it was called learning how to learn. >> Wow. And it talked a lot about this, how we can create memories because that helps us learn. And some of the things that it mentioned was the more that you access from the long-term memory, >> Yeah. >> it the more solidified it becomes as something that you now know. And so the frequency and that's why you do like flash cards when you're trying to learn something because you try and remember it and you can or you can maybe you can and then you flip it over and then you remember it. Okay, cool. Yeah, there it is. And so it accesses it again and you basically do that until you now know it and then you can spread out the frequency. So, if you're trying to learn something and you're struggling with it, maybe you're doing these flash cards every day or twice a day, three times a day, but as you then don't need to flip over the card to remember something, >> you can space it out and do it once every 3 days, once every 5 days, once every week. And so, it feels like what you're talking about, it has a little bit of that inspiration. But the other piece that I think is interesting is that you're you're also mentioning we've got this long-term memory and we're able to throw things in it. How do we weight it? And how do we know that things are important from it? Because for humans, we have something that is like, oh, the more frequently that we access it, the more important or the the more solidified it becomes in our memory. >> So, I have used these like Anki flashcards. I think it's quite popular on like everyone who's trying to like learning something. >> I I remember like trying to learn >> Japanese. So I was just like >> learning hiragana characters through like Anki flashcards and I mean the concept of space to repetition I think that's that's the term. So yeah, people say so that's quite popular and uh this is something very much seen in humans specifically uh because uh we tend to forget we tend to forget things a lot and this is something which is still debatable if the AI agents actually need a mechanism to forget or not. >> So there are a few people who actually believes that yeah AI systems need to forget because information gets outdated. We have to prune the things out. >> Yeah. >> And there are like lots of strategies to remove things out. >> And also some people believe that no, it's an AI agent. It doesn't have to exactly act like a human like just um dump everything and just use the most important ones and archive maybe in a some way. Yeah. Well, also the other piece I remember that you were mentioning too was how humans sleep and sleep is a huge factor in how we are able to bring things into our long-term memory. Right. Exactly. Exactly. So, uh yeah, I mean there there are papers like where they say that um human keeps on processing when they're like sleeping. It's >> the thing like the term is kind of consolidation. So like human memory consolidation happens when you kind of are resting or like when your stress levels are like quite low. So and your brain is like not I guess multiunctioning and doing a lot of things together. But yeah, so AI just doesn't need to sleep. So it can do so many things uh in the background. >> But when you're thinking about how to consolidate different memories, how did you go about that? Because as you were just saying, you're inserting memories in with their certain metadata and then you're constantly updating this knowledge graph. >> So cortex is something which I built on top of AM at first. M so I took all the inspiration from AMM on how it was happening and I thought hey we can actually improve this thing and there's so many possibilities in here. So what I did the first thing was uh oh before I mention this I have to also mention this that have you ever used uh obsidian? >> Yeah. >> Yeah. >> Yeah of course. >> So they used uh one method called I think I think I'm pronouncing it wrong but it's like zetal kisten something like that. So what it is is uh you can reference things in any page from anywhere. So at the end Obsidian gives you this whole interconnected graph where you can check that check like different clusters on what all things you have mentioned and which page connects to which page. So your whole writing journey is kind of captured in a graph and what all are the interconnected webs in between that comes out for you to see. >> Mhm. So you can get the whole bigger picture. So AMM paper like mentions a lot about Zel Kiston and sorry if I'm pronouncing that wrong but whatever. Yeah. So uh AMM just forms like relationships in between them. So what I thought is currently we value things differently all the time. >> So our values and everything like depends constantly on lot of factors. It might be our own biases and what our previous memory was. What have we seen our life? What we have like gone through every day I think in our life which is totally different for every single person. And by person in this case I mean a memory. Okay. So uh yeah currently when we are like what the key change I think I made was um I sort of named the relationships that each memory is being connected by. So what I mean by that is I had some key terms. So initially I was just like experimenting with what if we just name it extends by or something like uh definition of or something. So I had some hard-coded key terms in here and uh I tried to extend it extended more. So that was there and also like there were key terms like reciprocal of something like defines. So a memory defines this memory >> and now but we don't really know that how much it defines is it like so if we talk about it mathematically zero being the there's like no definition between each other and one being like it fully is the definition of another. So one being the definition of other. So it can be scored. So we can actually score that what is the relationship strength between these two. >> Nice. >> So and this is something we don't really do by like we don't do like manually or like any sort of instantaneous uh thingy in the pipeline. So while we are forming these metadata connections and relationships between these memories through an LLM when we are asking the LM process we ask it to also give a score and only score it if it's like very confident that there is some sort of like highly weighted relationship is present. So that kind of uh gives you the whole overweight I mean whole weight of the whole thing. And the other thing is also we ask it to like name the relationship out of these possible naming schemes. So that's just a big set that we have and it just um yeah basically chooses that how it uh is connected to each other. So does it extend or is it a definition? And there's one more other option that is uh >> should you be merging two memories. So sometimes in our uh memory we actually tend to even merge things into each other. So and I I kind of feel like that it's it's very important. So that's some feature I also added there. And >> yeah, sometimes we can falsely remember things because I think a lot of times we will merge two memories and then we'll remember, oh that was maybe if I've been to a place twice. >> Yeah. >> But it was a long time ago. I mix up which time I did something. This is all experimental. This is I I just wanted to see like what happens at the end. So there were like few few types of merge that we had something like an update which you just said. So in update what happens we just concatenate different memories into one and it just becomes a new memory. >> Yeah. >> Yeah. By the way we are maintaining all the history in the metadata. So >> okay if you need to reepparate them at a certain time >> if we can do that. And did you take any different approaches from AIM on the LLM calls and the search and retrieval style? >> Yes. Uh in in from AM I mean uh they just uh had a simple prompt where at the time of uh inserting a memory they used to process it first and then insert. And what I mean by insertion is like it forms relationships between the top most similar memories that it could find in the database. >> Mhm. Um and at the retrieval it just used to take the query and then just take the query and do a semantic search on the topmost and it takes the topmost K memories and also goes one level deeper because now we can see for all of those top K memories what are the connected other connected memories and you just take them also into account. >> Obviously they will be like less similar to the original query that you have. Um yeah I think uh these two were like the key factors but uh something which we added in cortex was we added birectional uh connections between these two so there was only a single connection uh between the memories I I as far as I remember so I I changed that to like birectional connections because it doesn't make sense if one is going that to only one side because uh for uh every connection that is represented in the graph uh it shouldn't be only that it defines this I mean we can even have a uh backward connection saying uh it is defined by so that at the time of uh retrieval if this one comes up so this also has some thing to uh give like more context to the model >> so that at the time of retrieval we can use this and this together this is what we call depth in graph rag So we can actually customize like how deeper to go, how many connections deep to go. So generally in practice like it's it's nice to use just a depth of one or two because three gets like way too much sometimes and too much noise might come out come out. >> Were you vectorizing this also? >> Yes. Uh it's a it's a whole uh vectorzed data set only. So I mean we just use a single vector DB. There is no graph rag there. Sorry there is no graph DB that we are managing. for this specific use case >> because at the time of retrieval we are just doing semantic search >> so there's that and by the way what I'm talking about right now that's that was like the intermediate part of cortex like when it was like just coming out and I was experimenting there >> oh there's a v2 >> ah yeah there will be improvements on top of it yeah >> all right let's talk about that a little bit >> sure so um currently there are people are like coming out with these AI assistants or agents where uh they have to do multi-dommain tasks. So the domain is not really connected to each other. It can be connected somehow. It might even not be connected. So let's say you have chat GPT right now, right? Or like any sort of AI assistant which you use to ask for a lot of different things. Now it can be about your work, it can be about your personal life or it can be somewhere in between because your friend might be working as a coworker for you. There has to be some sort of distinctions or like collections I would say that uh we currently automatically have as a human in our mind and we know how to separate them and also like sort of form connections between them. So I kind of visualized this and I like did some research on the same that uh this is sort of like hierarchal collections that's that's what we are like going towards right now. So the existing systems whatever we are like seeing in the AI agent space for memory we're only focusing on the flat uh memory structure that's what they're calling. So what happens in a flat memory structure is uh there is no sort of like hierarchy or so when you say flat hierarchical structure are you talking about something that is less like notion where we have all of our different files or any file structure where you've got your file, you click into another file, you click into another file and it just keeps going down the line. that's not what you have right now and that's where you think it's going or is it because right now as you mentioned it's more like obsidian where everything is just on this flat connected space and you have clusters over here and clusters over there but there's no hierarchical way of doing it. Exactly. So uh yeah exactly you explained it very nicely. So just like how a file system works. So in file system there is always this like hierarchal structure. So first of all like if you're searching for something in your laptop or desktop you're looking for uh let's say a folder which has a very high level overview of like what you're looking for then you go inside of it and then you look for maybe a very specific things. So sort of this is like the concept of categories and like subcategories. So imagine in high level you have like lot of different topics in your mind like work, it can be personal life. Now all of these different topics have like subtopics internally inside work. It can be um your current company that you're working at. Maybe you have a side project that you're also working at and it's making money for you. And money also maybe is connected with like finance. It's like a bigger topic in general in your life. And finance even connects with uh money. >> Mhm. >> But it has also a sort of triangle for each of these like top to bottom structures. we can see sort of like a triangle forming which has like multiple things connected to each other. It's it's like a tree basically >> 100% >> just a tree. This is something uh that humans I think use like deeply because uh when we are thinking of something we sort of think we we think like very fast but uh we sort of think through hierarchies first and kind of connect things together. So we go down from top to bottom. Also we are taking into account our short-term memory which which creates some sort of biases on like how we are kind of traversing the whole thing. But this is something which is missing on the current systems. It doesn't really represent these hierarchal collections or the topics directly. Okay, so this is something we should uh represent uh also and take into account and sort of build like a hybrid search which uses both of the both of the previous way we were doing like retrievalss and storing and also it makes use of the whole hybrid thingy what I just told about the collections. So in cortex we like we saying it it's smart auto collections. So, >> but what does that enable? I is it just that it's faster search? >> No, it's it's not faster search. Uh what it enables is higher quality search. >> So, it gives you much lesser noise compared to what I mean flat search will give. Just to give an example, uh let's say you search for something like fix this. Now, fix this can mean literally anything. It can mean like fixing your car or it can mean uh >> yeah what is this >> exact so fix this like what is even this so this it means so many things in your life and if you use a flat hierarchial model it can just fetch out the fix your car or like I don't know fix your friend's brain or something like that >> anything with fix in the metadata it's going to grab >> exactly we are kind of uh providing like the whole we are providing an option to give context also So this actually needs this is ultimately a context problem. >> Yeah. >> So uh when we are kind of doing retrieval there's an optional context parameter but overall uh it should first of all go through the what are the top level collections are what are the top level uh keywords are I will talk about it in terms of two two ways. One is like let's say you're inserting some sort of data. So memories these memory systems in general I think kind of has like two key functions. One is insertion and another is retrieval. So imagine you just have like something which you're constantly inserting or it's like happening in the background >> and at the background you can constantly keep on retrieving and use that however you want in your agentic systems. Keeping these two things in mind uh if we talk about insertion for let's say this auto collections thingy which we have in cortex. So when a new memory comes in uh let's say you have some existing memory already in place and when a new memory comes in it kind of categorizes based on what the user persona and what are your priorities are this we can actually write down initially. So based on that it kind of categorizes that if that memory represents just on high level like work or if it represents work then kind of subcategory can be um first job and then some more there can be some more subcategory like python maybe it's something related to python that you have uh are trying to like store in your memory now u sort of in this way we like every single memory when it like comes in it sort of creates like all of these uh topics and then it's subcategories only if it means something I mean it's optional it can just be a very high level thingy also that uh I hate my job >> so that's like just something which is like connected to your work >> and it doesn't really have any other smaller subcategories to it might by the way so something it can have like let's say emotional aspect of it. So job then emotions. >> Now we form all of these like small small categories and after it reaches a certain threshold we check like what are the frequencies of all of these different uh memories and what are their categories are. So if it exceeds a certain threshold we form a collection out of it. So now this is the collection and anything whenever like we we by the way I didn't mention that how we are kind of forming this categories. So at the time of um getting the metadata out of a query if you remember so in AMM and or or in cortex >> you're making the LLM call. Yes, in LM call we are also asking to give it a category specifically and with some context which is optional uh to take into account that hey it already exists with these try to keep it minimal and noisefree and so that it doesn't really uh give you any kind of category because you can name one thing in many different ways and that that won't be good for us. So how it started it should kind of like um keep on going on that direction and not really diverge a lot. After this uh what happens we ask the LM for the different categories and their connections. I mean not connections the categories and the subcategories just name them basically. So we check the frequency we form all of these like small small collections and when we are following when we are creating these collections what we are saying that hey now we have this uh let's say 15 or 16 memories together we're forming this collection let's give it a description like what the whole thing is about. So now the description will be sort of like what are the key things that we are keeping in the memory. So this is something that happens in the background by the way. It's not something is actively happening when you are like using the memory system. So it keeps on running on the background which keeps on checking that if it has reached a certain threshold or not. If it did then start a background process to create a collection out of it where um you give it a summary which we are also calling as a description and give it a query helper. So what query helper is this is something that will come at the time of retrieval. Uh it kind of it is a prompt. So it's it's a meta string you can say which will help you to create a prompt on how to query this. >> And so when you're making that background call and you're putting it into you're inserting it into memory you're doing both at the same time. one for the hierarchical structure and then also all of the stuff that you were talking about before with is defined by and what the score is and so it's one one LLM call that will give you all of that and then you can parse that out and say all right we have everything we need now for both of these structures >> these two are happening u parallelly in the background so it's not really synchronous so don't they don't really like affect the latency together >> so one we are calling is the global search. This it searches things like globally and the other we are calling as like auto collections search. So this is like a very constrained and narrowed down search if you look at it. So at the end we take some of this and take some of this and only take the most uh important ones that matches the most with the query like what actually we are like looking for and just feed that or like return that as a memory retrieved item you can say. So >> yeah to continue about I think how I was uh talking about the retrieval of the autocolctions. So we sort of like form this uh query helper which is like uh how how should you even go about querying this uh collections because now if the queries like fix this now if it matches with uh the collection python sorry collection work let's say now in work uh the query helper would be uh based on the description does it really help to even query this so there's like two options. One is to query or even not query. So now if you actually want to query uh this whole collection then how should you modify this query? Because in rag there is this concept of like query expansion or rephrasing where you actually rephrase the query based on what the context is. Because sometimes like fix this might be related to the previous 10s that you are talking about because if you don't provide it some sort of context it can't understand. So query helper takes into account like the context and this new query and it checks if it's relevant to this collection or not because it has access to a description also. >> Mhm. And uh since it has all of these things, it will generate an answer in yes or no. Like let's say if it's yes, then if it will uh modify this query with fix this uh thing about Python, let's say. So it will kind of modify that whole query to uh be retrievable or like searchable. So okay. So till I since I mentioned this point so there is uh one key thing that is at the time of retrieval we are taking this and we are doing two things one is uh on the smart collection side we are doing two things uh one is we are taking this query and sort of doing a semantic search over the different subcategories and collections that we have I mean over the whole collections that we have let's say you figure out four top collections out of it the most similar ones based on his descriptions. Now you give it 30% importance only. So now we form sort of a composite score because currently we are giving it only 30% importance. Now we are giving 70% importance to what are we querying inside all of these top collections. So inside all of these topk collections like each of these collections might have like 15 or 16 memories or like whatever the threshold you have set. Now among them we do a semantic search uh through the modified query for each of them. By the way now we are giving this 70% importance and finally by two important scores like 30% and 70%. we are finally coming up with a composite score for each of the memory points and then we sort it and then we see like what are the most relevant ones uh let's say you only select top J out of the final composite scored memories that you have retrieved and then you take this into account from the autocolctions retrieval and you take into account the whole global search retrieval also and then you just select I mean however much you need and you just give it back as a as something like which LM systems can use. >> Mhm. >> Yeah, that's that's like the whole overview of it I guess. What about like when you want timebased questions or recency and I say tomorrow I'm doing this but then tomorrow becomes last week. Currently I realized this a bit late while we were like building cortex but it has support for the same thing. So there are two things I can see right now. One is uh when we want to do these recentbased queries and when we want to do a particular date range based queries. So it might be something like uh do you remember that what happened on March 20 from between March 2025 to uh April 2025 like on the first 15 days. >> Yeah. >> So uh >> so that's like one one way and another can be something like uh hey what did I talk about >> uh what did I talk today something like that. So there is one step that we are we are wanting to do. So for all of these like date range based queries where you're expecting them have two options. One is to like provide a date range and another is to kind of uh I mean yeah we are calling it like temporal weight. So what temporal weight is uh now the weight can be anywhere between 0 to one and what zero like what temporal weight being zero means is uh you don't give any sort of uh bias to recency. So uh currently it just happens like normally how it was happening the whole hybrid search but uh if the temporal weight is like 1.0 zero. Now what you do here is you you do the retrieval B from just STM. So what I mean by STM is like from just the short-term memory that you have and since short-term memory is like a window and now you give importance to the most recent ones only. So that's that's something and now if the temporal weight is like let's say 0.7 sort of. So uh in 0.7 what we do is uh we at the time of retrieval we are going to the long-term memory and uh we are taking the whole query and if you remember we have always a limit at the time of search that how how far to look back how like what should be the size of k so currently we are kind of multiplying it by some some uh factor so based on like how what your weight is we take we create a factor and we multiply that so that we get even a bigger window. And now we basically uh take all of this into consideration because at the time of uh autocolction search we have this uh composite score mechanism if you see so in composite score it doesn't matter if se I mean if you if a uh collection gets like higher score uh because it will only have like 30% importance there might be 70% more importance on like something which uh was like lower down in the line. So overall the composite score can give you like a better distribution. I can say >> I think I'm understanding this. Is it keywords that are triggering how much weight you're giving to that long-term versus short-term >> with keyword? Yes. So we have like all of these set of keyword checks. >> So if I say last week that's a keyword or if I say last year that's a keyword and that determines K. Yes, exactly. So recent would have like something uh 0.6 as the K. So recent I mean yeah we shouldn't like keep it more higher. So last week we can have something if higher. So the uh the temporal weight is getting decided based on like what the keywords are. We have like a huge set of keywords. I mean this we basically did because uh it's efficient but an even better better method will be to ask an LLM if you're okay with having more uh latency on the whole system but that's something I think we all can improve and like how we are connect currently deciding the weight but the functionality is there and there's also one more parameter where we can pass the date ranges so it will constrain the whole search between only these two ranges >> and it will use all of these auto collection thingy and the hybrid the global search between a certain date range only whenever like you're taking a query you can always um extract some date range out of it using the whole context before querying the memory system so that you have some date range to look around for. So, it seems like you've got a lot of data that you're pulling from, whether it's the short-term context and you're deciding how much short-term versus long-term versus the hybrid search or the file structure, for lack of a better word. I can't remember what you were calling it. And the knowledge graph style. Yeah. Do you find that it all kind of collapses upon itself if there's a right or wrong answer or because I imagine it all leads you to the same place if there's one answer but maybe when it's very fuzzy you can just get this bloat of a whole ton of noise. >> Yeah. Yeah. First to check the fuzziness uh we are kind of doing like when similar things are coming up we obviously are doing like duplications there. So because at the core the memories are represented as like an ID in the database. So you can do drelications there. Now how much uh noise to actually incorporate and that actually depends on your limit and what I mean by limit is like how many memories you actually want to retrieve. >> So you can always like tone it down. So it can be like um I just want like two memories and that reduces the noise itself. Yeah, I think if if you're like losing out on something. If you are like losing out on something uh just increase the K probably so you can get like more memories out of it >> and you can totally like shut down the global search. That will be super constrained then. So you can just uh remove the not >> you you can do that on a case by case basis or you do that just Oh really? >> Yeah. So it's it's just a flag in the code base. So it's like it will be totally shut down and only the auto collections thingy will be working. >> So there you can get like a much constrained and if it's like a uh like a case where you don't want any false positives. So that will be the case probably. Have you messed around with I feel like there's potentially some cool stuff you could do where it's cascading where first you try with one and if it doesn't work then you can go a little bit deeper and a little bit deeper. Maybe you try with just the short-term memory plus uh hybrid. >> Yeah. Or if you can't find it, then you add in the knowledge graph >> or and if you can't find that, then you just say, "All right, well, let's try again with everything and see if we miss something or let's increase K." So, uh, this this idea, this is something that I have in the plan that is currently you see the whole cortex as something that human is using right now, right? And it has like lot of these different parameters in it. But what I want to actually convert this into is sort of tool where it's not humans that will be using cortex. >> It will be the agentic systems that will be actually using cortex. So imagine memory as a tool. >> So I mean currently the MCP is like MCP trend is like skyrocketing. So I guess uh it makes sense to call it as an MCP tool but imagine it's not us who are controlling these tweaking parameters but it's an agent because it so if it's an agent there it is automatically like processing your request and it is forming that hey what the date range should be while calling Cortex. So it has Cortex only as a tool. So in that way it can uh tweak all of these things and see like like what parameters give it the best result. >> Mhm. Okay. Let's get some coffee from Tiaro >> and then we can uh All right, we're back. Good. >> Maybe we can talk now for a second about context engineering and how this all fits into that. >> Yeah, sure. We at Prem are thinking very deeply about how to kind of create agentic systems. And we are as we are like building one internally that uh we realize that giving all of these agents these like different tools. But making sure that the um the tool usage and like the number of tools that these uh agents actually have is like highly highly important. and making sure that there is only one thing on the memory side. So that's that can be something like very crucial because uh now you're you're eliminating noise in a way. If you only have one 204 memory then you don't have to go and use up all of the this memory kind of search aspect if it's not needed. >> Yeah. Yeah. that that actually reduces like a lot of uh noise on the tools number of tools it has access to. So there is like a paper out there that uh the number of tools that you use the accuracy of like the task completion rate of an agent goes actually down. So the graph is kind of like uh this I would say. >> Yeah. >> And I've heard people talk about how much more expensive it is if you now have to give access to all these different tools. Your input tokens go up. Yeah. Yeah. So like uh there's just getting like more confused and confused with like so much noise of like different tool definitions and all. >> So I mean there are some strategies where uh we can eliminate that. So there there has been very nice blog about the same context engineering by Manus also. So where they talk about a prefix caching and how to uh how to uh just block one tool from the LM to see so that it doesn't really use them when it's not really required or it doesn't even have that in its context. So all of these like new patterns that are emerging like as a whole the whole context engineering thing. So I mean it it was all prompt engineering before but it's so much bigger right now because currently >> the way you use these tools and how you feed in all the information for the whole completion of a task is like has become the more thing and like more and more people are working on the same. Now giving the whole memory as a tool to an agent makes very much sense because we don't have these like static flows anymore where like a human will like for only a certain cases like a human would uh retrieve on what it's required because agent can do the same thing and we are constraining on how much freedom to give to the agent by working on the memory tool aspect itself. So but while how we are like building cortex is like we are reducing on how much freedom to give uh the agent who is using this as a tool like we can give it like lot of freedom with lot of different parameters or we can constrain it down to very minimal parameters that's one way I think how we can uh reduce down entropy overhaul. >> Yeah last thing you're interested in vision memory also. >> Yes. So uh this is a coming soon feature I would say in cortex that I'm working on right now. So I think there are some nice uh implementations done by a lot of different people. uh currently people are thinking a lot about how to represent all form of uh senses or like signals I would say as a part of their part of the agentic memory because currently if you see it's all text like everything that is people are working with are just text but what I'm seeing the trend right now is people diving so much uh more deeply into the whole video vision and audio also. So like just to see the bigger picture I think we have to target the whole five senses like how human see and feel and uh can experience different thing through all of their five senses. it something similar has to emerge for the Asian tech ecosystem also because all of these has to merge together and have there should be a way that it's able to intercommunicate with each other like the vision memory should be able to communicate properly with the textual memory that >> it has access to from day to day or it can be just some audio. Obviously, you can uh compress these down to only one domain like compressing down a video to only a text and it just interacts with everything is a text at the base. Then >> same with audio. Yeah. >> Yeah. But you're losing context there. So this is not a lossless compression. You're just compressing down the video and you're losing lot of compre lot lot of context even if like you are explaining it very nicely on what the video is about. So uh there is this nice uh thing that got released by memories AI. So they are working on uh large uh vision model something it was large uh language memory vision model something it was like that >> where where they are kind of processing the whole processing a whole video to make it more searchable more indexable. So now this indexible uh thing is not like that it just it's just converted into text but sort of like an embedding but uh feel free to like check out memories AI but I think that's like the start of the things that we are going to see in the coming months like let's say in next sixth or 12 months where we are able to properly index and store video data or like any sort of vision data or audio it can And at the core it will be just numbers. It will be just vectors or like however you want to represent them because it shouldn't be text because text should be a more highle version of it. So at the core it should be something all same and which is which has like most of the context without [Music] having any sort of like uh lossy compression I would say. [Music] Now you me and