How memory makes AI agents more effective

Key Takeaways

[Music] Welcome back to Real Terms for AI. We got a lightboard. This is going to let us do some amazing deep dives into designs and architectures and control flow and all the things that y'all have been asking us to go into more. But Jason, I heard that we might be graduating from prompt engineering to context engineering. Are we? Uh, I thought that we were just here to talk about sessions. Specifically, sessions in the context of application development and data management. >> Okay, that was a lot of words, but I like the idea of sessions. So, let's go with that. >> So, today we're going to deep dive into sessions in the context of using them in agents and other types of memory and stateful things that we may want to include as a part of building a better agent. And to do this, we're going to use our pet shop example, which we talked about earlier this season, but you could do this with anything else, like building a taco bot for finding tacos or any other type of example. Fun fact, I got a new cat, so I'm super into this pet store example because I've been looking for things that a kitten needs. I haven't had a kitten before. So, like kitten food, kitten toys, kitten beds. How does memory and sessions help me with this? >> Let's map out a simple flow and then we'll talk about how we can improve that flow using sessions and memory. So, let's start with our agent. You may want to ask the agent things like what types of toys are great for kittens or things like food. Okay, >> in this example, we have an LLM that will also help the agent query our product catalog, which resides in a database. And doing this, it may return a few results for things like toys that are great for your kitten using a semantic search, which just means searching for things that's similar to kitten toys. Now, in this flow, which you may have stored in some place somewhere, we're actually creating a session with both your user information and things that we're retrieving from the LLM. Okay. But in this example, the LM is only aware of my intent and my current question, not the fact that I actually have two cats because I have an older cat or that I've ordered from this pet store before because I have or that I have certain preferences about cat food and I may have already provided those. So, how do we get that information into the agent? This is where memory is actually one of the most critical things we can provide for our agent. To recap from our prior show, if you'll remember, we have typically type we typically talk about three types of memory. First is working memory. That is memory used during a specific task. Our short-term memory or our session, which describes things that are happening in the current state of things that are going on in our agent. And then long-term memory, which may be things like attributes that we store about our users over time or things like the order history for things that you've purchased from the pet shop before. All these things can actually help us make our pet shop agent more useful. Okay, so let's start with long-term memory. If I was running the pet store, I'd be storing the stuff you listed there. information about the user and their pets, information about their orders, like recent order history or past orders, and definitely information about preferences they've potentially set in their user profile. And you put it in a database. I get you that makes sense. When the agent starts up a specific session, I need to retrieve a subset of that information, though, because that could be a lot and maybe not all of it is relevant to this session. Maybe we only want the most recent couple orders and just some of the information about my preferences relevant to the quer current query like types of cat food that my cat likes. >> And you probably want to recall all of this right from your long-term memory into your short-term memory or your session as you might say if you're an application developer. Now, for each question the user may ask, we can give the long-term information as a part of the context that then we use with our large language model to get the best answers to our user's questions. So, now that it knows I have a preference for like a specific brand of cat food, um the agent when it looks up the information in the database can use that to return responses that are relevant to my specific preferences. That's super cool. But there's another big piece of information here that we haven't spent enough time on. Jason, >> is that that you actually want a different brand of cat food moving forward? >> No, it's that I got a kitten and I got a kitten this weekend. So, it's not going to have that information in the database because it's new information. So, how do we get that into the system for later so that you know if I log on in a week, it knows that information? Now, >> can AIS generated AIS actually get cats? I don't think so. Let's talk about how we can bring our new CAD information into the database. So, this is where we're going to actually do something with our short-term memory and our long-term memory. What we want to do is essentially when our session's over, we want to send this information to a processor. And this processor may actually use an LLM to help summarize that information about any new points. In this case, that you got a new cat. And then we'll actually send this over to our database and we'll update the information that we have about you. So that way the next time this is retrieved, our agent will know the most current things about you, the user, to answer your questions the way that we want. Now, let's not also not forget that we want to provide some interface so that way you as a customer of pet shop can also manage the memory and any preferences that you've provided because we want to be compliant but we also want to make sure that you have ultimate control of your data and that we can be good stewards about all the things that we know about you. For example, we may want to say change your preference and so we may have an interface where you say food don't want. We would use the same process in flow where we send information from short-term memory through the processor and then we would store that information in long-term memory except the thing that we wanted to redact from that. >> Okay. >> So we can use memory and state for other parts of the system too. For example, like one thing I've run into is sometimes agents like they'll use the wrong tool or my opinion the wrong tool and I want to be able to say hey don't do that again. Can we use memory for that? >> Absolutely. So, one of the things that's so great about memory is that we can actually in addition to storing things about the user, we can also store things about how the system works. >> Okay. >> So, in the system case, what we're storing is is also things like, hey, this tool shouldn't be used for this instance, even though it sounds sounds like orders is not orders. >> Okay, >> perfect thing for us to consider as a part of that. Now, this is context engineering, but it's also just software engineering or application development. >> So, you're trying to make sure that the tool we use is the right tool so that we can get a high quality answer as frequently as possible out of our agent in terms of recommendations, preferences, things like that. >> Exactly. And also, this is where we can use things like model context protocol or MCP to help us manage all of our memory and state. We can actually use MCP servers with tools to do things like create and update memories >> or even ask itself questions for how the agent can perform better with all these different components. >> That sounds like evaluation. >> It it is. And we can even use these things like MCPs to say, hey, only send a subset of memories to a specific tool or even a sub agent to be able to use in the in their processes. It's really not unlike how we think about handoffs in different applications. Maybe from a database to an API or from a service to another service. >> It's all software engineering, Jason. >> It is all software engineering. AA. >> Okay. So, we got a pretty complex diagram here. So, I'm going to summarize to make sure I followed and make sure everyone watching followed as well. So, we have memory and state and that information can be used to improve the operation of our agent here. Some of that information memory is in our session which is you know this interaction. Some of that information is in long-term memory, potentially in a database, and some of that information or memory can be used to actually improve the behavior of the agent by giving it information on specific tools to use or not use in specific situations. >> Exactly. And we can do all of this with prompts and other things, even using different contexts to make the system work better. Now, if you'll also recall, we always put helpful links below so you can try out different components of this to be able to see how things like short-term memory or long-term memory work together. With that, this is Aza and Jason for Real Terms for AI. >> Happy prompting. [Music]