Building an AI Code Assistant: How Cursor, Antigravity & Agentic IDEs Work

Key Takeaways

Hey, how it's going guys? In this video, I'm going to show you that how you can build your own AI code assistant. Lately, you would have been working with a lot of agentic idees like cursor, anti-gravity or wind surf. But have you ever thought that how these guys have been you know building these ids or what kind of underlying tech or techniques they might have been using uh in order to make your life easier when it comes to V coding right that's what we're going to see in this video so if you look at here on my screen of course this is going to be a completely CLI based experience uh but I'm going to basically break down this in a way that you'll understand this and if you want to build something like this you'll be able to build it right so first of all this entire project is open source and available on my GitHub repo. This is called code assistant AI. I'll give the link in description. Now if you if you have been seeing cursors of anti-gravity uh all these aentic ids they use different techniques like they use rag they use a like abstract syntax tree like how do they parse the code how do they search something uh how do they map repositories and everything like there's a lot of things that happens within uh underlying these ides right now that's what we're going to see in this video so there are four major thing that we're going to see one is the code parsing which is a based. So when you mostly work with rag, you do some kind of text splitter, right? So either you do uh character text splitter, you do recursive character text splitter. But when you work with code coding or programming languages like let's say Python for example, you cannot do text splitter or recursive character text splitter whatever because then you will lose a lot of information within functions and classes and whatnot. That's where the a comes up abstract syntax tree and that's a very intuitive thing. Now that's what written here also. If you look at it says unlike simple text splitter that breaks break code arbitrarily this uses tree sitter to parse code into abstract syntax tree and this ensures that our functions and classes remain intact right that's what that's what it does right so the code parsing happens through that now the vector storage uh they perform these IDs perform different kind of you know vector storage that you're going to see the retrievalss behind it. So uh for this demo we're going to look at chromb but in reality they don't work with chromad like they they work with other vector storage and vector databases uh for semantic search uh the system understand the get user and getting the user list means the same thing and blah blah blah like just for the semantic understanding like let's say I have get users within a function it basically means that we're finding out the user list when you write a prompt in like let's say cascade or chat or wherever right so that's what it is now when we are working in small scale scale. Let's say when you work with small scale because we are not going to build here like uh winds of a cursor or something but just to understand like what kind of underlying techniques they might have been using but when you work with larger code bases you have to work with repository mapping the repo mapping thing which is very important. Basically it creates a compressed tree structure that shows the file hierarchy that shows the class and functions definitions. This gives uh the AI a global view and that's how it works. Right? So that's what we're going to look at as well. And then uh in the end combine all of these things into a chain kind of a retrieval chain that will help uh the model to basically look at your prompt and then find out context aware responses or generate context aware responses. That's what you're going to do. And the key feature of it is that aware code parsing that basically helps us prevent uh not prevent preserve the semantic integrity. And then we're going to use MMR maximal marginal relevance search for diverse relevant result. I want to show you that repository structure mapping for global codebase understanding interactive QM mode source document tracking like you source the document uh like through git version control or whatever and support for multiple languages like Python, JS, Rust and whatnot, right? And here is the entire GitHub thingy that you can have a look at that. Now I'll show you the code here right and if you look at this architecture uh this is the system architecture it says AI code assistant system architecture. you as a developer like developer is me here right we questions we get answer how there are few things one is the code assistant core which is your CLI interface which is my main.py pi file. So if you look at this main.py thingy over here, this is what I'm running right now. It takes a lot of arguments that you see uh and it's currently running as you can see it over here in my terminal. Uh the moment I ran this, if you look at this, I am first activating the virtual environment and I then I'm saying okay. Uh let's run this. It says initializing AI code assistant and I have a test repository. So this is my test repo. Now in this test repo I want to perform everything and see if it's able to do it. the agent table to do it right uh you can look at the repository the extensions and then I already have a vector store which is saved in this folder called chroma db you can look at sqlite 3 that's what chroma uses it for all the serialization and decentralization thingy that you see uh for all for persisting all the vectors uh and then we have uh repository you can see building repository map initializing rag chain and code assistant ready now here is a question so if I ask show me all the authentication methods uh and uh you know it's going to take a little bit of time to do that. So let's wait for it and then meanwhile I'm going to show you the code over here. Okay, let me first show you this diagram that I was explaining. Now if you look at this diagram now you can see it's it's able to show me. So I'll first show you this show me all the authentication method assistant. If you look at this the authentication methods in the provided context are primary located within the o manager class in the o login.py Pi boom right this is how you find it out I mean if you look at cursor and windows or any other thing that's how they find it guys right I mean of course their scale is large they have been using lot of engineering behind it but that's how you can get started now if I show you this file uh which is uh let's say this test repo and in this test repo we have an o folder within the o folder we have a login pi and if you look at this login pi we have our uh you know the o manager now if you look at this o manager it's able to find out all the authentication methods Right? So this is fantastic. That's how the code gets read. And then of course you can keep on chatting with it. You can have a memory here. You can have an episodic or semantic or a long-term memory. It all depends how do we engineer it. Now this is this is what it is. So let's go back to the architecture here. You already saw uh this. Now if you look at this architecture the AI code assistant system architecture. As a developer you will ask a question. Now how the gets the answer? We have a code assistant code. Now in the code assistant code as for this particular project we have a CLI interface but you can have an IDE as well a lot of open source ID go forkit and then build on top of it. Now then we have an orchestrator there has to be an orchestrator because it has to go and do a lot of things. So we need an orchestration framework to do it. So if you look at the orchestrator here is my code base. Now it has three things processing layer storage layer and the intelligence layer. Now the mapping happens here. The repository mapper which is a structure builder and I'll show you the code in a bit. Uh then we have a code parser which is the tree a abstract syntax tree that I talked about and then we have the storage layer where we have open AI embeddings and then we store our vector store in chromb right and then the intelligence layer is pretty simple. So we're using maximal marginal relevance for search which is here MMR search and then rag and everything happens over here. This is a high level uh thing. Now rack pipeline over here as if you look at MMR we are using MMR because it ensures diverse result and not five identical code blocks. Okay, we need diverse result when it comes to these kind of uh problem statements, right? And repository structure map, you can see the get repo map, uh the build context context includes retrieved code, repo structure, function signatures and class definitions. All the arguments, all the signatures, your hierarchy, the file directory and everything becomes a context. And this is how the rag works out over here. If you look at this, I have given you everything that you can go and look at yourselves. pretty much self-explanatory if you understand the uh crux you know behind this problem statement right of how a building an AI coding assistant now the AD based parsing versus simple text splitting why a matters right let's say if you have a class which is payment processor now within this class if you have two functions like let's say for example if you have an init function and we have a process function and then if we just go with a simple text splitter and have some kind of maybe a character limit like let's thousand or 500 characters and if you break that out you will have function function splits in half right and then you lose a lot of information. So you have a one chunk then you have the other chunk both the chunk will have maybe uh individual functions or maybe the functions uh might lose the return statement or might lose some other variables and whatn not. So that becomes a bottlenecks also even if you have a chunk over size that becomes again a problem right now why ASD is fantastic for this kind of problem because it's split by logical units so you have a chunk one and chunk two completely uh each chunk is semantically a whole and intact so you have the complete information it basically keeps the classes together maintains the entire context and AI sees the complete code no broken signatures that's what a does so you if you're working with coding assistant make sure that you are working with HD based parsing and not not simple text splitters that's the first thing now this is already there I I'm not going to go into deep already talked about it here is a complete interactions I'll show you the repository mapping which is important so repository mapping basically gives the AI a global view when we talk about AI here we are talking about the agents by the way I mean all these idees nowadays are agentic powered right now without repo mapping okay uh you can look at here it says I only see login and user class with repo mapping you can see the full o module structure and everything. The challenge is when you have let's say a million plus lines of code right and vector search only returns few number of page document or few number of chunks like eight or 10 or whatever you have called it but AI need to know the entire overall structure the where files are what exist and so on and so forth and in that case you need a repository mapping so basically if you look at the mapping process here it is scan direc directory tree so you'll have a directory tree for example if you look at my directory tree I have a virtual environment a chromadv folder are diagrams and output. These are all my directory tree, right? Uh then it extract the classes, functions and imports and compress into tree structures and that's why it's good, right? Uh because it can it can have a look at the generated map which is like generated map look like this. If you look at this uh I have a source folder, I have a O folder I have a login pi all the classes all the definitions everything is there and then AI now knows what files exist can suggest files can find out files can uh understand the architecture that's how you know winds of cursor if you look at that the repo mapping they find it out right and how it's used so for example if I ask the questions how do I authenticate like for example if you look at this here okay let's say I'm going to ask in this question how do I authe Authenticate. Right now once I ask how do I authenticate it retrieves chunk login functions and user class and build the context chunks plus repo map and LLM sees the full picture and then it check that file the O manager and token service and basically synthesize a response. If you look at here it gave me a response now. Okay. Now the question was how do I authenticate? So it says to authenticate in this system you need to interact with the O manager class located in that. It also tells you where it is located because now it's know where the file is because it has the directory tree, right? Uh so that is important and it it basically goes into it and look look at all the functions and classes and basically gives you a very good answer. So this is right now I'm doing into terminal but when you're working with like let's say if you really have a gill and a passion to build an agentic ID then you have to do a lot of things on top of it. Of course, first you have to take an open-source uh tool like VS Code and on that you have to start building it, right? It takes it takes a lot of effort guys to build it. But something like in CLI you can definitely build it, right? An extension or a CLI or something is easier to build. Now this is the diagrams for TSON. So I'll just show you the code now. Now in this code you can see I have few files. Forget the test repo. Test repo is a repo to test it out. Now in this source folder which is the motherload. So in the source folder we have code assistant.py. We have code parser, we have rack chain, we have a repo mapper, we have a vector store.py. So first let's look at the code parser. Now I'm using lang chain right for this thingy. So lang chain uh if you look at it has language parser. It has uh this generic loader. Let me show you that I have a class called code parser. Now it says parses code files using a to maintain semantic integrity. Now this is how it happens. I load the repository. You can see uh file extension. We are loading the Python, the JavaScript. You can add more if you want, right? Uh you can see these are the language map. I have language.python. Now this is the mapping happens, right? Because langen has that in community modules. You have for Python, you have for JavaScript, TypeScript, Java, C, go, rest, whatnot. Right? Now I have a list. You can look at it. Then later we'll append that list of course. Uh and you can find out this is how we are checking the extensions mapping it trying it out the generic loader repo path glob everything within it parser loading