Making Claude Code Actually Remember Things

Key Takeaways

Here is one week of my work and every note which we have here is a conversation with a cloth code. Every link is a file it touched. Here's an example of ordering me Instacart groceries and all of the recipe files which I created and I could go back to this at any point and by the end of the video you'll have this too. A clo code already has a memory but it doesn't search across your vault. doesn't know what you worked on on a different project yesterday. And QMD fills this gap. And QMD is a logo search engine for your knowledge base. It serves as a retrieval layer where you can find anything in less than seconds. And here is what I'll show you in today's video. How to turn your vault into s searchable uh memory. How to load automatically context before every session. and how to index your knowledge base uh so that no conversation getting lost. All right, let's get started. That's our memory stack and what we have here is our obsidian vault. I am right now inside of Obsidian and here I have a bunch of files, bunch of folders, all of that are Martin files and we want to somehow be able to uh find and search for relevant information and the way we do it is through QMD and QMD is a CI search engine for your dogs uh from Toby that's a CEO of Shopify and it uses combination of SIM semantic search and keyword search and it delivers amazing results when you search across your world much better than grap in specific cases. So that's the tool which enables this and then we have a skill which is called recall which leverages this QMD tool and loads into the context the information which is relevant for us right now and then we can have access to this through cloud code or open claw from anywhere. Now the way I set up QMD for myself is for each of the obsidian vault folder I have a QMD collection and this is one to one mapping for me and then for each collection I can perform a focused uh search topic. If I want to look through my daily notes I can just go there and scan um and scan those daily notes. If I want to look through my sessions with a cloth code, I can do the same as well. And I'll show you how to embed your obsidian vault into QMG collections after this demo. Here is um example. We can go to Claude and talk to Claude and Claude knows how to use this CI tool uh QMD. We can run it ourselves to to get information. here all of my collections listed here and here we have files and the time since it's been last updated. Now I can go to cloud and tell um like perform a search um I want to perform a search u about my research uh about graphs can you find those one conversations and it's going to use u uh this um QMD command search across my notes and here are the relevant results where I was performing research about different knowledge graph approaches another Another example, I can search u I want to search about notebook lm and find relevant files uh and sessions about notebookm which I had in my vault. Now and this is a basic way to use QMD and it can quite fast get us up to speed to find information about relevant u topics which we want to recover into the memory into the session uh into our context right now and I can compare that with the standard way how you do it. I want to search uh all information about u notebookm which I had in my obsidian bold uh the sessions and uh the files and this is how you would typically go about it telling the clot okay go go wild and search my vault and it does that using this haiku model and this one is just using this brute force approach and and you can see that it takes It's already quite some time. It's been almost a minute whereas this search was almost instant. So, we're getting into much better results in terms of time. How fast can we search? And also the results are more meaningful and we are also saving much much in terms of tokens. It's been already uh almost 2 minutes and it's still searching, searching, searching and I'm interested in comparing the results which you will get. Okay, it's finally I think it's finishing that. It's been 2 minutes. Uh it's trying really hard. It's trying really really hard and I'm already bored. So, and typically during this time I would just go and uh scroll Twitter or or LinkedIn. So, and this becomes really a waste. So, I need to somehow like spend this time and focus on something else while waiting and that like distracts me. But that's one dis advantage of having fast search. So, you don't have to like go and do something else. You can really focus on one session on one context and work on one thing now. Okay. So, it's been 3 minutes. Wow. Um, okay. So, and we got this response. Um, damn [laughter] 300 files. So, yeah. And here is the results. And I'm actually not so sure that's better than than we have here. So I I think here it's it's uh the results from QMD just from the standard search function of QMG is much better and they're almost instant and they're much more more context efficient. You don't have to use sub aents to do that and we are saving on tokens and they're much faster. Now that's one basic usage. Let me show you how can you embed different collections to QMD. So that's the commands which we have here. So there's a lot of stuff going on and there are different commands. Uh there's a standard one called search that's a full text um BM25 search and there is also vector vector search only. And this this one we'll explore the explore next. And for collections, we can have like those commands. We can at least remove the name. And let's try to create our collection. I want to create a new collection uh for my transcripts uh for the last 7 days using QMD. And I have those transcripts which um transcribe um everything I say and I save those into my vault. And here's example of the files. Okay, here's example of the files. It just uh transcriptions of me saying something. So, and from that this is very very valuable insight. We can try to embed it uh into a vector database and then ask like semantically meaningful questions in a way we couldn't actually capture through a standard keywords. All right, that's cool. We are here. Yeah. So go ahead and add to a new collection called transcripts. Okay. And here we're using the command collection add transcripts and we are adding the files uh from this folder. Okay. So we just added all of those files and we can search across our transcripts. M let's say yesterday I went to a meetup and I had some brain brain dump from this meetup and my thoughts and I want to ask a question about it like maybe some cool cool ideas I had during this meetup regarding the this discussion was about uh the browsers I I remember so can you search that now and here are the results I asked to not to expose any private information and here are like my ideas from my brain dump after the meetup and this is one of the ways you can use it but now you can push it even forward. What really excites me is semantic search and semantic search is something that uh which would enable surfacing and finding information which you couldn't find through a standard search by keywords and going beyond grap. And here are some cool cool cool ideas which we can try right now. Um let's say um I want to search across my daily nodes and find uh the days when I was happy and what was the reason for that and this is quite non-trivial query and clone is very very smart because right now it tries to adapt this query and uh searching for like you know happy grateful excited energy great day and find sematically relevant connections across my daily knowledge. or my transcripts. Here's example. [laughter] Uh mood is good. Glad I was able to sleep at all. Not great sleep but functional. And there are some other other ways that come up. Yes. Yes. And here's the example. Um high energy happy days. Um where I finished the seminar and I finally was done with everything and I was and I was good. Um and here is um pattern analysis that my happiest day when I ship something and I had a good sleep recovery uh like sona or 9 hours of sleep and this is just example how you can try extract um semantically relevant information which you can't find through grab yeah I also want to find um like the points of frustration on our own or uncertainty from my transcripts or from my daily notes. Okay, we're just getting uh all of this um information like using keywords, frustrated, uncertain, stuck, confused, finding relevant relevant files. Okay, [laughter] here [clears throat] are the patterns which we surfaced. Uh recognize tendency to seek quick fixes through coffee, overeating, sleep, social media when hitting roadblocks. H sounds familiar. Yeah. Um here's a session which I had about actually trying to unblock myself and here is the uncertainty uh when I was trying to find if u I'm doing the right thing to get those results. It was unclear for me what to do in that situation here. I I don't know what happened. It was day before my defense. Um what else do we have? Um, focus on most uncertain parts. Strategy working well. Uh-huh. And here's some concerning patterns. H. And here's a pattern that frustration comes from uncertainty about direction more than from the difficulty. H. Yeah. Okay. And here here is a real insight which I had but I actually forget about this right now that my best copings insight was um back then in October when I was writing my PhD thesis and I was about to give up uh and I need to like do it like day by day come in and write something. But what I realized back then is that I just need to see it through discomfort instead of escaping to quick fixes and I want to find this file actually. Um yeah, that's actually quite interesting. I didn't expect that it could surface this. That's so amazing that you can just explore that. Okay, so I had this like daily report back then. Um I had uh like you know this PhD work I like rewrite chapter 4 um what else we did and here's yeah and here's the exact citation which we found that's just amazing just amazing now it's been like a quick introduction into I don't know that's probably more than 90 seconds but that's just basics and then we can go into the actual memory on how to build something meaningful with that. So here is how to actually stop reexplaining all day long the new context to clot. Okay, so that's the one that's the one. Yeah, that's a call start problem. I think we should all face that we have a session we open a new terminal and then we start working working on something and we are good like cloud knows okay what are we working on it remembers what decisions we have made in um in this session it has a full conversation history it knows what to do next okay cool now so [laughter] but we are hitting at some point the context limit and when we are getting to let's say 60% % here in terms of context. So then we need to compact or do a hand off to another agent to continue working on that and then we kind of need to like decide what do we do like we start a new session then like half of the decisions have been lost or even worse if you want to continue next day I I don't remember what was happening back then and there is I just open new terminal and then what's going on I need to somehow like find all this context like what was the projects um what is the decisions and I need to start from zero like all the time and you just like tell CL okay find all the information and it like goes for 5 minutes and this is what we've seen what happened that it's not efficient it takes so much time and spend so many tokens and the results are not that great because it has to read every file and it can't really negate the complexity of the work which we're doing in our vaults so there you go and that's the problem at the core problem which you're trying to solve here. Now let me explain how do we go about this. So we have we have this like recall skill and this recall skill enables us to scan our clock conversations or open clock conversation given a timeline from yesterday or from last week and it parses and like finds all our our conversations from this timeline and then we extract the session timeline. Okay. So that's cool. So let me um just try in a new session um just run this uh sorry call that's the command and we can have a yesterday today last week this week or topic or graph. And this is I'm going to explain later. Let's say just yesterday. Let's reconstruct yesterday. Um yes I want to reconstruct yesterday uh a precise timeline what I was doing matching the conversations and the files create. Cool. It loads the skill right now. Okay. It loads this workflow. Within this recall skill we have this recall workflow and runs this Python script. It references all of our conversations. And here they are by time by the number of messages. Uh the first message. Yes. 39 sessions and how do you not to lose mind in all of this like uh craziness like indeed busy day [laughter] and now I can go back at any point in time to previous time and understand what I was doing back then. We are using haiku right now to like understand uh what was happening in that session and it's going to go back to us with the results and this is just one way to interact with this uh skill with this memory system u that's a temporal path that I want to like know what I was doing back then um and what was it about okay and that's that's very very cool and we [clears throat] have a fully full timeline reconstruction what I was doing back back then like time by time and just the full timeline and I want you to get me the session ids so I can also easily resume those. Now this is one of the values that you can have clear understanding what you were doing yesterday and go back to any of those conversations. Now cl just going to uh get those session ids and I can resume actually information back. Okay. So let's try this session. So that's the session ID and then if I open a new new terminal instance I can go back to uh cloud and tell like resume and that's the session ID which I can use here. I think I need to have a full session ID here to make it work properly. Right now it doesn't recognize it as a command. Okay, here we got a full sessions ID. Um I can just tell okay using this command to to resume and I'm back to to where I was during this conversation. That was example of a like temporal path where we recall what was happening at some time in the past. Now and this part which really makes me excited is using this um topic path where we use the actual uh QMD and we use this BM25 search and it's been performing very very well for me. So let's try that. Let's try that. Just start new conversation. Yeah. So let let's see how how it working. Um we're going to use this recall command and uh we're going to recall something. Um by topic um let's say graph graph rug. Uh I'm really interested in in graphs and I'm like have a bunch of stuff in my world about graphs and I want to continue working on this. Oh okay. So it actually decided to use uh like search for QMD video because I have this context sharing with code. It can see what's been selected. Um and okay, so we're going to go like through this QMD video. So what do you have about that? Nice. And we are back. We are back. So I've been working on that for last two days. And here is a her current state. So we have all of this. We have uh like all different files each cloud can surface in less than a minute. We have um dashboard, we have a production plan and here is a to-do list and um now once you have this into your context so we can ask what is the next high highest leverage action. Okay. And you can try it with your projects, whatever you do in your obsidian is it like life, business, work, literally anything. So you can clearly reconstruct um any information in efficient way in less than a minute using this uh topic recall command. And this is all like single skill. And there's also graph way to do this. Um so let's try using graph and this is the one which I showed. Um actually that's a graph for today and here is a graph which we have and we have like different uh sessions and each blob here is a session. So you can see it here those are different sessions and they have um color code by by the time of creation. If the session is older, it gets dimmer. And if it's a recent one, it's um it's highlighted in purple. And then we have a different files which are created here. Um let's say this one. So we have like this um QMD video where I was planning for that. And we have um here we have this uh file called slides which I'm working on right now. So that's the file and you can see that it's it's been I've been working on that in in a couple few sessions. So you can see the connections here and here is a command which I tried just now like recall command and that's only for today. Now we can go a bit deeper. We can try to explore. Okay. So what we've been doing in the last week okay so let's try to do like something more interesting. Let's try last week. All right, the graph is loading and right now it's much more rich and complex. And so that's very exciting. I really love this. It's kind of fun. So what is this? Um Uhhuh. And this one is I was running this blind spot analysis where I want to have understanding on what are the blind spots that I have in my uh life or my strategy. And here I have a skill for that. And we created um and it has a different workflows um analyze like retrospective analyze and report and here is the plan which I created. Uh I can go back to this. I can copy this path and tell cloud open this one. Okay. And right now it opened it for me. So that's kind of all of the information. So we can also use graph view to uh clearly surface what's going on here and see connections between different sessions. And this one is is is kind of fun that I was exploring um a lunch places and I just tell okay I want to have a [clears throat] great lunch and uh we actually analyze like different different places to go and I store those as activities I might want to try and if I want to get back to this I can just copy those path and um and continue from there and copy and paste it into cloud code um and then we can like tell to read like these files and let's understand what's happening that's interaction with graph which is quite insightful for me and I really love to see collections um and this rich visualization and that's been a three ways to interact and recall what is going on that's been temporal path that's been path by topic if I want to know okay I had this thing which I was doing and I want to know I want you can stack the context about it or the graph view. So whatever you prefer if you're a visual person maybe graph is the way to go for you now and this whole memory system is open source. So the link in the description let's continue let's continue with more excited stuff. So the actual difference what what's actually the point like oh okay cloth can do grap by default we also have this BM25 uh approach and we also have like this vector search so let's try to understand um let's try to understand what the is happening here let's let's go and and try uh to to benchmark this okay so I want to benchmark different approaches Um, so I want you to run those queries and present to me results in a clear way so I can compare the benefits and the performance of each each of them. Okay, good. Let's make cloud work. We'll compare those three approaches. So right now we are using the GP that's the one that's the native tool which clo has and uses by default. We also have a BM25 that's Q& search. We have uh semantic only. That's QMD varch. We can search for like cool couldn't sleep. And this one is actually um I think the hybrid like this one hybrid it's it's it's supposed to be the best but it takes right now 21 seconds. So that's a bit problematic. I think the QMG research took 2 seconds. That's fine. We are good. And we are getting back after 35 seconds. Cool. Cool, cool, cool. Um, and here are the benchmark results. I want to also know like exact like row results which you're getting from the queries and please perform the analysis on this. So yeah, so this table is good. So let's look through it. Uh, we have this grap approach like query sleep. It has 200 files and there is a wall of noise um like research workshops workshops templates. Then there is also BM25 which has a more relevant information and we also have a semantic search. Okay, sorry just got got a bit off track. It has a relevant uh more relevant information about sleep experiments diagnostic and goals regarding the sleep. Then there's a semantic search finds meaning aha couldn't sleep. Nice. And then there there's a best ranking sleep interrupted at 3:00 a.m. Okay, nice. I want to look at the actual data. That's the most interesting part. So here are the parts and let me see if the code did a bit better. Cool. And here's the exact analysis. Uh so here we have uh grap where we search like across we found 200 files and uh it finds all the files which contain staying sleep. It it has like all all over the place all over the place. It even finds this like sleep uh sleep [clears throat] system call that's completely unrelevant. Now if you try to do a QMD approach um what we have here that's actually the response. So what we found is um a reflection about sleep quality. We found experiment with tracking sleep fragmentation patterns. Uh sleep interrupt at 3:00 a.m. Um that's much better. Nice. I load that. Then there's a vector search when we search for couldn't sleep. And then we found um this much um sleep pattern. This one I don't get really. And this one I don't really get. Uh uh this one is because it directly about uh couldn't fall asleep at 3:00 a.m. Uhhuh. And there was a goal about bedtime discipline. Uh-huh. So it kind of goes beyond the keywords and explores the meaning. That's the point of that. And then there is a hybrid approach which took us 35 seconds to run and it ranks this note as the highest one. And this is um another one with a higher relevance. Couldn't fall back asleep. Uhhuh. Nice. So here is a comparison of each of these. So I would say grab from the worst. uh it introduces a lot of noise. So, BM25 is fast but it doesn't find the meaning but the results are quite good. Then there's a sematic search which finds meaning and then there's a hybrid approach uh which like actually finds the best results and does a comparison. uh showing how to go beyond graph and and find find relevant information even if you don't use specific keywords. That's that's the unlock. Um here are some other cool ideas. You can also embed different collections. So I have this YouTube CLI skill. Um just find videos uh about u investment. I want to learn about investment. I don't know for some reason I want to learn about investment and it's going to get those videos for me. Okay, investing for beginners here all of the all of the videos and then you can tell okay so can you get transcript for one of these videos from me please and then embed it into a new collection so you can actually like get a bunch of videos about the topic which you want to learn and then embed them into QMD and then ask questions about it. Now we are creating a new collection. Yeah, the collection should be called different. It's not transcripts. It's about learning for investment. Let's direct cloud a bit. Um, okay. So, we are creating this folder. Okay, cool. We got the summary. We got the summary for that video and we're right now adding it to a collection and we are embedding it um to enable sematic search. Okay, so it's embedding also the previous chunks I guess like all the other documents which I had, but you get the point. So you can just download bunch of materials, bunch of articles and ask semantically meaningful questions like it would be for notebook. And if you're using notebookm so you might also find this v this u video useful on how to connect cloud code to notebook lm and get the answers with exact citations to a passages to actual transcripts um passages and articles paragraphs and how to create this like knowledge graph for all of your notebook topics sources and citations they all become connected and the cool thing you can also do audio or reviews uh podcast which you can save and listen in your obsidian and there are many many possibilities okay so let's get back so it seemed that we actually embedded this video script and we can query and get the information about or we add we asked about index fund and here is a index fund make sense okay so you can go beyond just your vault you pull external sources. Nice, nice, nice. Oh, and this one I think is also cool ideas. Uh how you can uh use it um to let's say find the ideas that I have never acted on. Uh let's see what we got here. I'm really having fun with this. Uh so it's kind of new direction, new way to um to interact with cloth and in the memory with all your context. Um okay so what we got here is um oh CL can you summarize what's going on here it's a bit hard to parse the actual row results and having clo to help us to synthesize insights from from this data is is very very useful okay uh cool on October n 19th uh I wanted to build a PhD writing dashboard but never did it okay what else a undraw applications and never follow through. Sounds like me. Uhhuh. I had an idea to record a screen recording about full obsidian workflow but never committed. Uh yeah, just there's a lot of fun. You can try different queries, different searches and find information in a new way. Explore what's happening with all of my notes and it's happening all locally. all of these embeddings they leave on your computer. All right. And to conclude the section, so how how do you start? So I would start with BM25. It's it's actually like very very fast and it gets you 80% there. And if you want to add semonic meaning, I would recommend to do it for your transcripts or for your brain dumps where you can surface those u connections uh because you don't have like um specific terms there. You can just like oh free floating you have like your brain dump and you just throw stuff on the wall and uh you can find the relevant pieces and the connected dots and if you really really want to push the performance forward you can try to use both but it going to cost you half a minute. All right and moving forward. So moving forward that I touched upon that like this cloth code it says all your conversations as um files on your computer and I just had like the 700 sessions in last 3 weeks and each one of them has decisions questions okay here all of them here's all all of the sessions and here are the statistics here are the files which I created here are the skills oh here yeah this is the files which I created and here is the example of the session. So I open this one open claw and I used open claw skill here to manage my open claw instance and uh it has session ID so I can always go back to it. It has a date and uh it has the skills which I used and this is the link to my open close skill which uh controls my open call and helps me to manage it. For the artifacts, it's the files which I modified or created and have a full conversation history saved right here. And those conversations I embed into QMD and I can ask anything um about those. And this is um they're stored in this um collections called sessions. Now that's cool. So we have all of these. we can keep track of our conversations in our world and we can search across them so that I partially um showed that I won't repeat myself here and you can also do like sematic search across your sessions across your interactions with cloud and find maybe frustrations with your workflows where you couldn't capture the exact words and that's the way it works and that's um part of memory system which I'm sharing uh and open open sourcing so we have those convers sessions are stored in a doc folder and then we have this skill which I run only once and it exports all of my cloud conversations into my obsidian as a markdown files and they leave there so I can change the folder doesn't matter and at the end of the each session when I close my terminal I have a hook which exports and embeds those sessions into QMD. So I don't have to reindex it all the time. And then I can use recall command to load this context and it's always fresh. So the index always always fresh. So I can get back to anything I had even from today. That's kind of the point and that's a pipeline for the sessions extraction if you're interested. So I think the key key here is that we parse a clear markdown the actual signal the actual user messages because row files they have like tool users the system prompts the roles we just want to have a clear markdown and then we embed this into our QMD index and then we search across this now that's cool yeah and that's kind of the whole stack um not the whole memory stack we have our obsidian we have We can run this recall skill to reconstruct information and we can access it from anywhere. Yeah. So, let me connect um to my computer so I can share the screen. Okay, that's cool. Okay, I just connect to it. Um here is open claw ready and let's wait. Okay, cool. So, that's how how how I go about stuff. That's my open claw. So okay now I'm just using my phone. Uh here I tried the reconstruction from from yesterday and I can ask so what kind of skills do you have? What are the skills available? And the key here is that it uses all the same skills which cloth hood has. So my obsidian bolt is shared with open claw as well. Now it's it goes and thinks about stuff right now. So, it's going to respond very very quickly in a minute. Well, there's a lot of stuff. So, that's kind of the idea that I have uh all of the those skills in my obsidian. I have this agent skill base um which tracks all of my skills and they're available also from open claw and how this like the most like you see like that that's that's the one uh most powerful for context. Yeah, this recall graph it's also available here. So that's uh the pipeline. Yeah, the whole memory stack and that's um open source. You can download it in the description.