Open Source Friday with Mastra

Key Takeaways

[music] [music] [music] [music] [music] [music] [music] [music] [music] [music] [music] [music] >> Hello everyone. Good morning. Good afternoon. Good evening. Wherever in the world you're joining us from. Welcome to Open Source Friday. Thank you for being here. We made it. We made it team. We made it to Friday. I don't know about you all. I had a nice and spicy week. >> [laughter] >> Very eventful, but very good. Lots of good work done. Welcome Brandon. Thank you for being here. Welcome Sumi. Welcome Crepa. Welcome Mr. Landini. I appreciate you all so much for joining Open Source Friday. And for those of you who are just tuning in for the very first time to the stream, welcome. This is Open Source Friday. This is a show sponsored by GitHub, produced by GitHub, and hosted by an amazing team from the DevRel and Open Source teams at GitHub. And we're here just to bring you amazing projects. Not just the projects themselves, but we get to talk to the contributors, the creators, the maintainers, and all the folks who are making Open Source keep on going. And boy, do we know we depend on Open Source. Welcome Mr. Israni. Welcome. Welcome. Welcome. I'd love to know what people are joining from. So, this is my favorite. Thank you for doing that. I'm prompted. That is my favorite thing to read when I see where folks are joining from. From Jamaica, Ms. Roxanne. I'm so jealous. From Cameroon. From India. Thank you everybody for joining. What an amazing We got a great team watching today. So, I am joining you live from the West Coast of Florida. Where is very warm. So, salute and warm hugs for those of you in parts of the world where it's still very cold. But, as I mentioned, Open Source Friday is a show where we get to highlight amazing projects. This project in particular that we have today has had an incredible story. A masterful pivot, really. So, it's like really interesting to me also. It's like teams get started, they get to work on things, and then they they make adjustments. But, they're doing something that is extremely timely. And I'm delighted to have the CEO of the company joining us today. For those of you who are not familiar with Mastra, you're about to learn everything that there is to learn about it. It's a great framework. And welcome to Avi Iyer. Who is the CEO? Hi. Thanks for having me. Thank you for taking the time for being here. I I appreciate that the team has been trying We've been trying to put this together for quite some time. I think I met someone from your team at one at a conference last year. Maybe it was Render. Or it's been It's been a while. It's been a long conversation. I'm in your Discord. I've been lurking. I've been using lately. But, thank you so much, Avi, for taking the time to be here. I cannot wait for you to share what Mastra is about. So, I know that it's sort of it came from the Gatsby team, or maybe some of y'all were from Gatsby at one point. Maybe? I'm Yeah, okay. So, if you could tell us Yeah, a little bit of the origin story. Like, what problem were you trying to solve, and how did this all get put together? Yeah, so me and my co-founders, Sam Bhagwat, Shane Thomas, and myself, we all worked at Gatsby for quite a while. Sam being a co-founder of Gatsby. And, you know, Gatsby was a For For those who don't know, Gatsby was an open source static site generator with with React. And one of the first of its kind back then. Static sites were all the rage during the Jamstack era of our development careers. And, you know, we kind of expanded from static to doing other things, and eventually got acquired by Netlify. Another prominent company in the open source world and Jamstack world. Um Once we got to Netlify, it was a little, you know, it takes some getting used to joining a new team, being part of a new product. And slowly, we all started to fade out and tried to to think of what was next. Surprisingly, I was the one who left last from Netlify. I actually enjoyed my time there. Um And Sam had pretty much stolen us out of Netlify with this idea [laughter] of uh I want to build a better Salesforce. Um which is Surprisingly how we're going to get to Mastra. It's a very surprising journey there. But, he wanted to build a better Salesforce. And I guess we had the arrogance to think that we were the guys to to do a better Salesforce. Um we tried um We tried building a linear-like experience for doing a CRM. And we spent a lot of time building a CRM when this CRM was supposed to be an AI-powered CRM. Aha. >> we kind of didn't We didn't want to touch the AI parts because there wasn't any movement in TypeScript in AI at the time. Um and we were kind of too busy focusing on things that we like, user experience and product and things like that. So, when it came for us to go raise a seed round, um pretty much all investors were like, "Hey, this is a AI CRM, but I don't see the AI part." Where is the AI? >> [laughter] >> And we were like, "Oh, yeah, if you squint a little, you might believe that it's there." Um So, you know, taking rejection on the chin, uh we started trying to integrate those AI things that they expected, and we found it to be very difficult to get far fast when you're trying to iterate and like, you know, experiment. So, um Funny enough, I was visiting a friend in New York City. I went to his office WeWork in Financial District. I walk in I was supposed to have beers with the guy, but I walk into a hackathon, an AI hackathon that was happening in the WeWork lunch room. And so, I, you know, turns out he can't hang out. He's participating in the hackathon. So, I figure I'll participate, too. Um In doing so, it took us 8 hours, the whole day, to build a chatbot that can read PDFs. Now, that might seem so really contrived today, but it was really hard to do. Mainly because you're educating yourself for the first time how to do something, as well as putting yourself under a time pressure to do it. Um so, we did it. We got it working. Pretty much most people were submitting Google Colabs, Jupyter Notebooks. Yep. If you've been to a hackathon, that's not what's going to win a hackathon. You need to have something cool. So, we built like a Next.js app, and thankfully having the web experience from before, you know, our our hackathon project looked very good. And we were able to win. And that actually made me really think that what if other people could get far fast? And something that took me 8 hours should take the next person 1 hour, or even less. So, Sam, Shane, and I, we looked at each other. We didn't want to do open source again. No offense to open source. We had just had done it for so long that it was kind of tiring. But, then after going through this experience, we were like, "We have to do this again because I think there is a just a missing library or framework in the market to help people build AI." So, [snorts] that next day, Mastra was born, and the CRM was dead. Um And since then, we started, you know, from that moment we started building what we like to call the primitives of AI engineering. Um And when we started, the primitives were not as much as they are today. It started with, you know, LLMs with tool calls in a loop. And then it has expanded to workflows and memory and MCP and skills and you name it. We are building the primitives for doing that. Um I'm sorry if this is a long story, but we got into Y Combinator shortly after, you know, our first release, met so many agent builders in San Francisco. It helped really focus what the like what Monster as a open source framework should be. And, you know, since then we have, you know, our 22 cases GitHub stars, we have over a million downloads a month, like people are really resonating with what we're building. Yes, you have been on fire. I'm going to share the repository here and for my friends that are joining me on LinkedIn, hang with me. I'm coming with the link that way. So, everyone go ahead and go to the link and start the project, please. Let's get the project trending. I love Gatsby. I I think one of the I mean multiple of the very first things that I ever did, like my first web pages were on Gatsby. And then funny enough, I became like a Netlify fan after, too. Like obviously like that that that that transition. Um that art of the pivot is so interesting to me. That's a whole conversation. We need to sit down and talk about founders and making those decisions because it's hard to shoot the puppy when it's like you're like, "But no, I love this thing." Um and then making the decision that open source is the way to go. Um I salute that. I know it's exhausting and having had a project like Gatsby that, you know, such a long time and so impactful in the community. I know that comes with a lot of baggage and work and and and time and effort and tears and sometimes blood. [laughter] So, I can totally get the hesitation there. So, you got into Y Combinator and then based on this hackathon experience, you build basically you're like, "Okay, there's an opportunity here to build this framework. We're going to do it." And then you pick your stack and you build it I guess you build it using um is it Vercel or no? The next definitely the next JS framework, right? Yeah, for the AI pieces we uh we started with using AI SDK. Okay. And that was a really good way to jump-start everything. But AI SDK is a great library, um but there's a lot of other components that needed to build an agent for production. And so then we started layering other open source libraries um and then started building some of our own uh to to kind of complement the LLM piece of the the puzzle. Amazing. So, now you're Okay, so you're you're actually have a lot more going on now. Um and so if I'm a developer and I'm watching this and I'm thinking, "Okay, I want to build an agent. I want to get started doing this. I need a framework that's going to support my scale, right? Like maybe this is going to grow into be a business." Then they want to start with Mastra. Um can you show us a little bit about the framework itself in action? Like I think as as we kind of see a demo, we'll be able to, you know, dig into some more questions. And friends, if you're watching this and you have questions, please don't hesitate to post them on the on the chat. I'll be sure to read them for you. Um go check out the Mastra website, too. It's so beautiful. I was just complimenting your designer. Like has it. Yeah. So, let me I'll do a the canonical weather agent example. I've I've prepared some other ones as well as the conversation gets a little deeper. Perfect. >> But uh let me share my screen. And I'm going to show just how people get started with Mastra. And I might have to like unshare to put a API key in or two, but uh we can get started anyway. That's totally fine. While you're doing that, I'm going to share actually and this is when I met and my gosh, I'm I'm spacing out the name, but cuz you're you're either your co-founder or maybe it was it was all of y'all wrote a book, right? Or Sam wrote a book. Yes, I'm going to share it because friends, I have it. It's it's it's a fantastic little like it's a good comprehensive look of how this works. Um and it is free. Thank you for that. I love that. Uh I shared it on the link uh on the chat so that you friends can go ahead and grab it. And Mina, thank you for being here and to answer your question. Correct. This is open source. It's an open source library. Here we go. So, I got your screen now up. Okay, great. So, to get started with Mastra, you can run MPX create Mastra at latest. And I'll zoom in for everybody. Thank you. Um okay. I don't know what this NPM thing is about right now, but uh um open source Friday. And I'll do open source. I could pick a my provider. Let's use OpenAI, that's fine. And we'll skip the API thing. Um and this is kind of my first discussion point. When we first started Mastra, um LLMs did not know about us. So, we had to figure out other ways to help users use their own tools to write it. And at the time, WinSurf was very popular, Cursor was very popular, and MCP was a very new thing. Mhm. >> And so what we did was we built an MCP doc server. So, you know, once installed, Cursor and Claude code and all these coding agents would know about Mastra via your questions to them. Um and then now we have skills, of course. So, you can add skills to this and we have different Mastra skills that allow you to write Mastra or let your agent write Mastra. Because when we started a year ago, people were still writing code by hand, but as we know, things have changed, right? So, now our agents are writing code and so you need to be able to do a lot of good context engineering for that and skills are a great way to do that. So, let's just go with that. And I'll initialize a new repository. Now we'll install the Mastra CLI. One very interesting feature of Mastra is we have what's what we call the Mastra studio. It is a all-in-one place to build and test your agents. Um it's a nice companion to the code. And you know, you can go and chat with your agents there. You can go experiment and you know, have a lot of fun. So, cool. Now we have this. So, it's Friday. Now let me not let me not leak my key real quick. So, I'll go put >> [laughter] >> open AI key in and then we'll be off to the races. I love this. So, you have your own interactive UI then. Studio actually allows you to like visualize this and do all the testing. That is pretty cool. Correct. And maybe I'll take two of these keys. You can try all of them. Um okay. >> pretty cool. And something fun about and thank you for having the MCP also and like um MCP is not dead, friends. >> [laughter] >> Yeah, definitely not dead. >> I'm going to put it ON A T-SHIRT. BUT YOU'RE ONE of the few um open source or rather TypeScript frameworks that you ship both sides of MCP, like I you have it like both as a client and a server Yes. uh in the framework. Yeah, so yeah, MCP is not dead. All right. You tell me when you're ready for me to share. >> And then maybe oops. I am I'm ready again. There we go. We're in. >> So, so we're back in our project. I'll run NPM run dev, which starts the Mastra development environment. And you can then go to localhost 4111. And now you'll see that you are in the Mastra studio. And we should with this very canonical example of a weather agent, which I guess, you know, the reason why people do weather agents, for example, is the tool call to get weather is a open source free API, right? So, it's a very much easy thing to get started with. Um I set up OpenAI, but in Mastra, we have this thing called the model router and it allows you to pick other models. So, while I was off-screen, I added API keys for Anthropic and open router. And so, you can see here that we support many models. You just got to bring your own key. So, it's not necessarily tied to a specific model provider. Um let's ask, "What is the weather in Miami, Florida?" Let's see. It'll call the weather tool. You have this diagnostics here. You know, Miami, it's partly cloudy. Um cool. And it should return soon or I'm having some internet problems, but uh I could also try >> Listen, it's live. That's like we we know. We know this happens. But while it's coming up, so you charge you picked up two different providers, right? Yes. Um you can add I mean, you can add any of your model providers that you want. Um but then are you able to pick what provider you use for what task? Yes, so everything in Mastra is dynamic. So, I'm going to open up the code now. So, I can do open up Cursor. And oops. Let's go into an agent here. This is our weather agent. I move this here. And I'll talk about the primitives a bit um while building an agent. So, our agent class is the main agent primitive and what it does under the hood is it runs agentic loops. And how you configure it is one, you give it instructions or the system prompt. Then you can give it the model. Here I I we have magic strings in our model router, but you could also pass an AI SDK model. Or what you could do is you could pass a function. And then return this. So, you have we have this thing called request context. And you can imagine that if, you know, let's say if the request context that the user Sorry, that get >> and based on, you know, let's say if it's Andrea I'm going to then return Anthropic or that. So, like and this request context is where you can put this uh kind of uh contextual information to then make those calls. A lot of this a lot of these parameters are uh configurable um dynamically. And I think that's very much uh a necessary thing in today's uh game. People are not using a single model for everything. Makes sense. Yeah, so I'll go put this back for now. And if we go back to our demo it did respond and that's cool. The next thing that Monstrous does because what we learned early on is if you're just chatting with an agent, you really need to understand what's going on behind the scenes. So, you can see in the top left here we have traces. So, this is what I just looked at. What is the weather in that? I can look this up and now I have a full tracing view of all the things that happened. So, this was the input. This was the output, other metadata that occurred, and different attributes on the agent. I can look at you know, the LLM call itself. You can see how many tokens it uh inputted and outputted. You can see the raw input and then the raw output. And I'm using GPT 5.1 mini, so it has reasoning state in it and, you know, that's cool. You can see all those things. And then finally, you have these different steps of the agentic loop. So, this is you know, just returning the raw data for that. The next thing that we learned is with observability and traces, you need to be able to constantly iterate on your agent. And so, that's when we built scores. And scores are a way to run evals. And evals are very important. Not a lot of people do them cuz I'll be honest, they're very hard to do. But you should be doing them. And if you are doing them, you can run scores based on the inputs and outputs that are going to your agent. So, for example, this is a tool called accuracy score. Um it's written so, you know if I'm asking for the weather, does it call the weather tool? It should. And that's what this score is trying to do. Then we also have other scores like completeness. Is the response complete? You know, these are all what we call code-based scores. They don't use LLMs to to evaluate things. You can, you know, to score anything, you can just write a function. You can decide what that means to you. You can use NLP libraries, etc. But there are some types of evals that are done by LLMs and we call this in the industry LLM as a judge. So, this translation quality uses an LLM to score the response from the agent using its own, you know, model itself. So, this one scored one cuz usually in evals you want to score between, you know, either pass or fail, zero or one, and you want to leave less ambiguity to it. Um to take that Oh, yeah. We can stop for a question. No, no, yeah, for sure. And let me take a look at the chat, but I'm wondering because obviously you just mentioned like evals are hard to do, right? And like scores like this looks really simple and letting the LLM sort of be like the signal. Was this good or not? Uh doing like its own evaluation in a way. Uh but are you seeing like people is this something that people are using actively for their own evals? Like what are if they're making mistakes while they're setting up scores? Cuz I would imagine like there's a lot of noise to that comes with it. That looked beautiful though. Like that was like great, but I don't know if that's something that's implemented in the way that scores work. Would it actually help you keep more clean information data like show you what matters? Yeah, so it really depends on the type of company that is really going deep into evals. We have many big logos and customers using Monstrous now. And the range of people doing evals is a spectrum. If you're in an industry that requires compliance like law or med med uh health care evals are being written uh religiously. Mhm. Mainly because you have people to answer to when things go wrong. But on the other end of the spectrum, you have many people building coding agents and personal assistants where the eval is the user's taste, right? If I built an email agent for myself and it did something wrong, I know that it's wrong because it's my email. You know, I have a relationship with it. Um so, there's like a big spectrum. Scoring is not an exact science. It definitely takes many iterations to get right. And uh so, to do to kind of build what we we like to call it the scientific method of, you know, having a hypothesis on how to change your agent for the better modifying your score to actually collect that that score and then over time seeing are you increasing score to from zero to one? Um and if you are consistently getting ones, then you're doing pretty good for yourself. Beautiful. I love that. Okay. So, you register your scores, they're in your instance, and then you also have like the historical data that you can continue to >> Yes. >> Perfect. Thank you. >> Yeah, so scores here, you set them up on the agent very similar. You can put all these different scores. You can say, you know at what rate do I want to do these? We consider these live evals. Live being they happen asynchronously after a turn with an agent. So, all of these will run 100% of the time cuz we we ratioed that that way. Um and but you can also do offline in the sense that I can come here and I can create a data set. I'll just make a fake one. Ooh, let's do it. And within this, I can just add items that are the inputs and outputs like what is the weather in Paris? It's hot AF. Let's just >> [laughter] >> Uh this should be message I believe. And this also should be I don't know, result. Let's see. And I might validate these things. Must be valid JSON. Well, you know, it's always a live demo when it's a live demo. This is how we know it's it's real. >> [laughter] >> Um in any case, if that works for whatever reason, um you could have a bunch of inputs and outputs that uh you want to add or you want to test over time. So, actually I can go to our trace view and I can add this to our data set here. And I will add this to the fake data set. And and you know what? Let's let's go add some more things to our data set. So, what is the weather in Paris? Let's use a different model. Let's just use 40. Why not? RIP. Let's see. It's thinking. What did the chat say? Didn't like AF. >> [laughter] >> That was it. Okay. >> That's not too hot. And then what's the weather in San Francisco? So, you know, as you're you know, you're communicating with your agents, you're shipping them to production, you're going to gather a lot of different traces. And some traces will be signaled good and signaled bad. AKA signaled good means, "Wow, the agent really responded well. I should use that for a test item um going forward so I know that given the same scenario, it's always going to respond well in that scenario." But then there's also some example bads where it did not do well and you want to make sure you have those. So, here now we have items in our data set. We can run experiments on them. And I can just run this agent. I can select my weather agent and I can select the score. Let's just use the translation quality and I can run that. And so now, it's going to take those inputs and output or input items and then run the agent on them and then maybe I I change something in my agent. I can iterate on it. And then I can just start running these over time. So, it scored one. It'll probably score one on all of them. You know all of them were scored. So, that's kind of cool and this is what people are doing now um with us like we just released these these these experimentation features. And so, people are getting really excited about building their data sets up, getting non-software engineers involved because a lot of companies are you know, their software engineering teams for a subject subject matter expert company. So, for example, if you're a vet clinic, your engineers are not veterinarians. So, maybe the veterinarians should be the ones deciding what is good and bad in uh the data sets and what is worth testing and what is worth not testing. What? That is very fair. I was going to ask you and you actually just answered that question cuz there is a lot of like third-party tools that do evals. But you made it part of the framework. Like it's in the framework. Yeah, the reason we wanted to and, you know, no hate on any other tool. Yeah. But when you're using those tools, you're not actually as close to your code as you as possible. Um like you're let's say you wrote a Monstera agent, but then you're doing evals in a different product. Well, we thought, you know, these things should be connected, especially if, you know, you're running the agent loop in Monstera, well, the traces and the observability and the evals and the data sets should all be part of that loop as well. And so that was the decision we made, but you can use products like LangFuse, LangSmith, BrainTrust. We have exporters to everything. And this might be a good little thing we learned about open source. We may want to build our own things in open source, but we have to meet users where they are. And so, for example, when we first started Monstera, we only supported lib SQL as a database, mainly because I thought lib SQL is cool, and I still do. But when someone asks you for Postgres, you can't say no. A lot of people are using Postgres. You want to meet users where they are. Not everyone wants to use Monstera observability. If they want to use LangFuse or BrainTrust, we should be 100% in support of that and make sure that they can. So, the way we designed Monstera, I can go just do a little quick view here. This is the Monstera monorepo, and I'll just give you an example of how we do our code in a way that it is available for anyone to fork or extend. And the best way I think to show that off is storage. Monstera supports many storage adapters because you need to store data when you're running these applications, whether that's messages or traces or what have you. And so, in Monstera, we have Let me go to the base. We designed everything as base classes. So, we have these base storage domains. And so, workflows have the storage domain, scores have a storage domain, even observability, etc. And if I go to one of these, you can see that it's all class interfaces. They're [snorts] abstract classes. So, if you wanted to bring your own database, you just implement the workflow storage with your own flavor of whatever the heck you want to do, and then you're off to the races. So, we designed kind of all of Monstera to be extendable via these class interfaces. I appreciate that. I think that that's where a lot of platforms go wrong trying to lock you into your choices, but you got to you got to let developers choose what they want to do. I That's fantastic. I love that. And you're doing that with evals, with observability, with storage. Fantastic. Yeah. Um some other things we can note is our agents have memory by default. So, memory is the ability to you know, it may seem kind of simple, but a conversation history is one way of to do long-term memory. So, as you can see, this conversation has all my messages. We have like a memory section here where you can add different types of memory, semantic recall, which is the ability to use vector embeddings and search through the message history. We have working memory, which is a, you know, a memory type where an agent can use either a string or an object to write little notes for itself. And then we just shipped a new memory, which is called observational memory, and I'll go and add that right now. Observational memory, true. What does that do? What is observational memory? This is new to me. I have not I wasn't familiar with that. So, observational memory is a new memory system we just released over a little bit over a month ago. And so, couple things we learned, actually was prepared for this, but uh if you want we have this like this obsession right now with building agents that never forget. Yes. And I just think it's such an interesting topic. I'm no neuroscientist or anything, but I really love how the human brain works. And right now, before we were working on observational memory, we were all using Claude code, and every time we hit compaction in Claude code, we felt like Claude got lobotomized. It just did not know what was going on, and we have this million context window, and at the time it was a 200k context window, and then you just feel like after compaction, the journey you went on with Claude, it's always all for loss, you know, cuz you're kind of starting over. And many people had the I'm just going to start a new session. Every time compaction happens, the old session doesn't matter. And we just like, okay, this is a weird this is a weird problem. Let's like think about it. So, we know that LLMs can process so much information, but like the memory over the long session is actually worse than a human's. Cuz I can still remember what happened this week, even though it was a spicy week. Like, you know, I probably remember it more than my coding agent does. And so, we think that these things are kind of related, you know, like there's so much stuff that goes into agent context, and there's so much noise, but what you really want is the agent to make sure it knows and keeps the signal, right? I may have a bunch of tool calls to book flights or search for flights, but the main observation here is I'm trying to go to New York City, and that's really what matters. All this other stuff is just polluting the context window. Mhm. And so, our the common solution to this, which many people do, is you extract the signal from the user message, right? I say I'm trying to go to New York City, it's going to go search the conversation history to find the right messages to build the context window dynamically. And that's cool, but it invalidates the prompt cache. And you know, what we what we want when you're Well, if you want to save some money, you want to make sure that your your prompts are cacheable, so you're not paying the same price. There's a different price for prompt prompt or cache tokens versus input and output. Sorry, input. So, we figured like LLMs never build rich continuous pictures, especially if you keep like uh starting new sessions. And if you're always dynamically injecting uh this information, it's always like in a single turn, situational for that one moment, but you kind of want like a a like a a holistic view of everything. Mhm. And so, and then if you have too much signal, how do you decide what doesn't matter anymore? You know, for humans, a lot of biggest uh thing that allows us to compress is that time, right? Things that happened 5 years ago, obviously, maybe not don't matter anymore. Um and you know, time is one way you can decide what doesn't matter, emphasis, you know, your emotion, etc. So, like this is like a typical retrieval pattern that people will use. You know, you get a user message, you embed it, you get it put into a vector database, you rerank it, you inject it, and you hit an LLM. And you know, this is kind of how it works. So, you're just chatting with your agent. Every time you are just retrieving context from your database, and then you're injecting it, and you know, pretty much this may be efficient, but it does not it's not cost-effective, right? Especially if you're paying input tokens for every everything you're doing. So, that's one example. So, Tyler Barnes on our team, he really dove dove into this problem, and it's honestly shout-out to him. I'm just a figurehead for talking about it right now. Um we had he had this insight like, you know, humans are really good at like filtering out noise. And what we can do is we can thrive like as humans we thrive with lossy memory. Um we don't necessarily need all all of the data to then figure something out. We just need, you know, high-level pointers or clear events that have happened. And agents right now, they need a lot of like hand-holding there. So, we thought, you know, it's forgetting a feature. If you look at your common day, you do all these things, but the things that stay are the events that matter. And the things that you forget are things that don't, right? Like I don't really remember tying my shoes today, you know? I probably did, or, you know, it's just it's not important. But I do remember I had to come on this podcast, and I'm going to remember later that it was a great podcast, right? Or live stream. So, those are the kind of things that we wanted to model it. So, we wanted to model agents this way. So, we built it. It works pretty well. And how it works uh is uh we have three agents. So, your main agent, like the one that I showed in our studio, this is the main agent. We call it the actor. And these other two agents are the we call the observer and the reflector, and we think it's the subconscious mind of your agent. It is this agent these two agents that are always watching the conversation history, making observations, and then compressing things into events, so you don't necessarily need full context. So, this is how it works. I'll just show this version of it, and And this is three separate agents, right? Three different agents tackling the same memo prompt. Okay. Correct. And then it's all running in the background. So, as you can see, as messages come in, these observations get buffered up. As soon as you hit a token threshold, usually we set it around 30k tokens, cuz we kind of do more with less, um it will then take the message history, compress it into these event-based observations, and then now this observation list is in your system prompt, and you only need to need to take the messages that you have not observed yet. And so, what this allows us to do is you have something that's very cacheable, and it actually works very well in terms of recall. Um and how we kind of figure that out is we did a benchmark called the long mem eval. It's a 500 questions, and it has tons of data, like 57 million tokens of data. And essentially it's like a question answer with an agent, um and the model has to find all the right information. And so, we benchmarked it against other memory products that are on the market, and we did really well. This benchmark is quite old. It uses GPT-4o as the base model, which I did say RIP GPT-4o, even though we just used it. Um Yeah. People It's like a nice baseline though, because if you can score well on a bad, quote-unquote bad model, old model, then what what what what would you be able to do in a new one? So, in our results, we got around 84.2%, which is like, you know, pretty solid. Um this is across the board. But then we thought, okay, what if you start adding really good models. So, we did GPT-5 mini, scored like 94.9%, Gemini 3 Pro, and everyone in this list is good. It's just we were, you know, our new strategy is kind of helping out in different uh areas. So, these are the areas, and once again, it makes sense that these are the areas because this feels more human to me. Temporal reasoning is thinking about things in the aspect of time. That's such a human thing to do, cuz we're always looking at the clock, right? So, I think that was really interesting that how we scored high. >> [snorts] >> Knowledge update, when we learn new things, we should unbias the old things we know and start biasing new things we know. And then multiple sessions, like you are, you know, as a human, we are multi-session in our lives. Talking to you right now, talking to friend, I didn't change as a person. I have to maintain context between these sessions. So, it was really cool to go through all of this, and uh yeah, and Gemini 3 did pretty well there as well. That's impressive. I'll stop for questions. That was a lot. >> No, that was that's super because honestly, this is so timely cuz that the problem of persistent memory, like especially now that there's a lot more people tinkering with agents for their own things, and they want to be able to offload the things that don't matter. Like what the exercise you just showed us of like, I'm not going to remember the door I came through, but I know I came in into the office. Like is it important how I walked in, kind of thing. Uh that's super super interesting. And so, you have three agents on this job now. Um you mentioned something that is super relevant to me right now. I've been reading a lot about this and doing my own experimentation with memory because of cost. Yes. >> And you mentioned something about the cost of cash versus input for tokens. And I guess that is the reason why we should be thinking about this problem, especially when we're building agents for scale. Like you want your agents to feel human when you're especially like a support bot, I'm much rather than that I have to repeat to you, I'm trying to find a flight to New York. Yeah. [laughter] Like we had this conversation already. Um so, thinking about cost, obviously like what's one massive application, but the human angle of it and thinking about approaching it like from the point of view of how a person does is super fascinating to me. What are you seeing people experimenting with this? How old is observational memory? Like you said it's about a month old. Yeah, we released it in February. And uh this is our research post. You all can uh take a look. Yes, please. >> Um and it's a deep dive on how it works and everything, and you know, people have built their own versions of observational memory in like Python. Um we're going to be building out a more generic library, so like anyone in TypeScript can use it. You don't have to be a Mastra user. Um it's completely open source. Um >> Beautiful. And it's super fun. Like uh I've been in the same Oh, I guess I don't want to get too far ahead of myself, but like with this, I've been in the same coding session for a month. No way. Yeah, it's pretty cool. Okay. So, because see, you're we're moving beyond compacting, and like what you mentioned earlier, and I'm going to drop a link on the chat about how Co-pilot handles memory, cuz it's a new thing for us as well, like because of the same reason. Like you don't want to lose the context of sessions. Being able to resume sessions in compacting for expenses, for whatever reason, uh it makes a lot of sense. But you're been coding the same session for a month using observational memory, and it's working out, obviously. You're not feeling any kind of degradation, and you're you're still you're still coming out fresh. I love that. >> to be so into doing Git worktrees, yeah, and I still am. I just wanted to test myself and see how far this can go, so I don't even use Git worktrees anymore. I just try to accomplish one task at a time as fast as possible, then switch context within the same uh session, fix that, and then, you know, over time you're overlapping in code areas, right? Um and I have not missed a beat. Um So, it's been it's been pretty pretty amazing to use. I appreciate that. I think if you would have said that to me 6 months ago, I would have hopping a so box about Git worktrees myself, but I find myself not using them as much anymore because of this, because now we have the possibility, and like the linear path might be the best path. This is awesome. Let me take a look at the chat and see uh if there are any questions from folks. There were questions about whether it's open source, of course it is. When you were doing the demo, uh the studio is is is part of the project. Yes, that is open source. So, when you're testing out Mastra, if you want to go in and and create your agents, like yes, this is this is the the framework, what you're seeing here is what you're working uh with, and it is free to use. Um a comment about being TypeScript first, and definitely cuz frameworks are for sure shaping the way that we're we're doing things now. Uh I don't think there's any questions that are super relevant right now. Friends, if you want to ask a question about it, please drop it in the chat. Uh otherwise, we can carry on with the demo, but this has been fantastic. Awesome. So, I want to, okay, switch gears now. I mean, it's the same gear, but we're going to go into a different perspective. Now, agents today, where I feel like we're in a coding agent era right now. Um when we got into Y Combinator, many agents that were people were building were for human human for workforce. For example, replacing the back of office of a small business, or trying to replace accountants and these things. And I think, you know, as I mentioned earlier, writing emails is very hard. To truly replace a human, you have to come correct. You cannot have any mistakes. And so, it's a very hard task, and what I think has happened is coding agents do not require the same rigor because we are the tastemakers of the thing that we're using. Plus, we can decide not to push the code or commit the code. Mhm. And so, we've seen this huge expansion, uh it's like a Cambrian explosion of coding agents and coding products, Co-pilot, obviously, Claude Code, Open Code, Droid, all these different ways to write more code. So, it almost feels like they're trying to replace us first. Uh it feels like. And whether that be the case or not, um in Mastra, we want people to be able to do whatever they want to do. So, I'm going to turn this weather agent into a coding agent. Um just a couple lines of code and I'll turn >> Yeah, just a couple lines of code and I'll turn this into a coding agent. So, we have some primitives to turn your to turn yourself into a coding agent, or your your agent into a coding agent, and we can do that with a local file system and a local sandbox. And I can also get a workspace. So, workspaces in Mastra are a collection of a file system and a sandbox put together to allow your agent to read and write files, execute untrusted code, as well as leverage skills, right? Uh Claude skills or agent skills. Um so, let's make this workspace happen. Once again, workspace is an interface. You can extend it. And let's do sandbox. And let's do your file system. Oops. I haven't tabbed in a long time. Uh what is this? Base path. Workspace. So, simple as that. Um I can also have a skills path. I think this is an array. Let's just say it's dot agent skills or something. Is it skills dir? I forget. I forget all the time. That's why TypeScript's great, cuz I can just uh see what's in here. There you go. Um if it still is agent skills. Cool, it was in the array. So, take this workspace and I'm going to pass it to my agent. Okay. >> I have a studio, oh, go ahead. No, no, no. I I just wanted to clarify like the parameters were there. You just gave it within the workspace the tools that it needed. And then you gave it the execution, right? Like I missed that bit. If you give it the What's there a Let me see. File space workplace. Okay. Okay. Okay. Okay. Okay. Okay. Okay. And then of course sandbox. Yeah. [laughter] I could get a sandbox from E2B or Daytona or you name it, modal, etc. In file system, I could mount a file system from Google Cloud Storage, S3. But you know, for the sake of the demo, we'll just use my local machine and let it yellow. Perfect. Should I pass the agent workspace agent? Cool. And then now, if you see on the right here, we have all these workspace tools. Um and these just magically get added. So, now your your agent is is able to read and write files. So, let's actually say can you write the Tokyo weather to an MD file for me, please? Call the weather tool fund. Okay. Now it's going to write the file. So, you can see here. And it saved it. Okay, that's cool. Well, I want to go look at this. So, let me go to the workspaces tab and here we are, Tokyo weather. That's awesome. Um why don't we add some skills, huh? So, in Monster, we support skills. Once you have a workspace, you can go and find skills that you like. Um for example, this is all the skills that are on a website called skills.sh. And I believe there's like something called a skill creator. We can try it out. I don't know if it'll work, but here's a skill creator. You can see what it does. It helps creating a skill. Let's install it. Cool. Now we have that skill. And let's go back to our weather coding agent thing. You can see now we have skills here on the right. So, this is the signal. Um help me write a skill for telling the weather. Let's keep on brand here today. Okay. So, what you're doing now is you're using the coding agent that you just created to create a skill to do the thing that it just did. Yeah. Yeah. Which uh They're meta, I think, but uh I love it. I Listen, if you show me that you publish that skill to skills.ai, well, actually it's not I think it's not automatically, right? Like it's it's like an automatic thing. Yeah. >> Um but you should be able to, right? If you have MCP in there, you should be able to like create a repo, publish it, send it on this way. Look at that. >> is making it's making like a a script to fetch the weather. So, now like you know, this is just like the next level. It's like, "Whoa, like how crazy is that?" That's just amazing. Let's go back here. Let's see if we have our skills. Here's like our skill there. And uh that's all all cool. I should say you need to write it to the skills path. dot agent skills. Oops. I forgot another API token. But as you can see, this observational memory was running. Um and you can see it was trying to do um stuff, but I forgot a API key, so that's my bad. But at least you can see that it's it is running in the background. Cool. It's writing the file. It's all good. I think if we go back to our workspace, should be somewhere. Oh, it's right here. weather skill Pretty cool, huh? That is very cool. So, let's take it a step further than this. Okay, so um as I said, you know, if you want to build a coding agent, you should. Um we have a coding agent um and you can get that from npm i uh monster code. This might have some problem cuz I don't really use npm that often. I use pnpm. So, I'll probably do something stupid. We'll try it out. Try it out. Let's try it out. New what? Um I have to just have to delete this path. Yeah, I don't know. That's what This is why This is the reason I don't use um npm. Cool. Cool. And now, let's just run monster code. Cool. So, now you have a coding agent that's more focused than the weather the the fake one I just made with real system problems on how to be a coding agent powered by our observational memory. So, like when I was saying that I've been in a session for a month, I'm in one of these sessions. Um so I can say, "What's up?" Actually, first let's set up some observational memory things. Here you can set up your observer and reflector models. I'm using Haiku. That's chill. And What are those thresholds? The thresholds are for you? Like what you're expecting is or The threshold is like I'm going to wait I'm only going to start observing if the conversation history >> reaches that. Okay. >> reaches that. Got it. >> to compress it even more, um it has to hit 40K of observations already made. So, you're constantly trying to compress down um and then, you know, uh keep the context window uh like without rot. Um so, then what I usually like to do is like, "Please deep research this code base, um understand what we did, and finally make no mistakes." >> [laughter] >> All right. With a threat. >> So, you can see you can see that when I installed this, I installed the Monster skill. Which allows So, this is imagine if you were working on a Monster project yourself, there's a Monster skill here that's teaches the agent and steers it in a way to know what the framework does and then, you know, um so, this loaded the framework guide. And now it's reading stuff. It's reading the practice.json. It's going to go and understand what all is in this. Just reading a bunch of files. And it has a complete picture. It's going to give me this information. And I think I'm going to have it build me another agent. for fun. Okay. Listen, but this is very interesting though, because you're like you created a space where you can build the agents or become an agent itself. My my brain my mind is blown. No, this is very very interesting. So, what surprised me, you just gave it. Okay. Surprise me. Um and so, as you can see that we've built So, this Monster code is open source as well. Nice. >> It uses a base class and that's the theme of the talk today, base classes and interfaces. It uses a base class interface of what we call a harness. And a harness is a very popular word in AI engineering right now. Um and so, if you want to build your own coding agent or Claude code, you could extend that harness, use Monster agents, and you know, you'll get a lot of this out of the box stuff. For example, the harness allows you to have a task list. It'll ask you to you know, you can ask questions. This whole Claude code or Claude co-work UX gets built in by default when you're building with this harness abstraction. Um and uh yeah, it's just been really cool to like take open source to its fullest because, you know, a lot of these other tools are closed source. Um and we just kind of wanted to like pull back the curtain and >> Thank you. Yes. And expensive. Yeah, for sure. But this has a really nice buttery feel to it. I love it. I love So, completely open source and basically when I I can create this and create as many agents as I want Yep. in Monster. Amazing. >> And you can deploy it to your favorite hosting provider. Mhm. U