LlamaIndex Webinar: AI Coding Assistants with CodeGPT

Key Takeaways

hey everyone uh welcome back to another episode of The Llama index webinar series uh today we're excited to host co uh code GPT uh and Daniel one of the lead authors will be presenting this this project um and so you know it started off as a vs code extension for uh just you know a code co-pilot and I think it's s evolved into a general platform for AI systems and co-pilot Generation Um so I'm super excited to see you know just the uo of it all how really helps people become more efficient in their day-to-day productivity whether it's coding or anything else uh and especially with platforms like you know the rise of Devon uh with autonomous agents there's been a lot of interest in how do you actually use these things to really automate away and not just automate away like make your own personal day-to-day lives more efficient um so I think Daniel will talk a little bit about the platform potentially go into a short demo as well as an overview of how it works and then we'll do a Q&A session towards the end so without further Ado uh passing it over J thank you so much thank you for the invitation thank you for having me H I'm very excited about to talk about CBT while we are working about like more than one year working in this project so um let me introduce myself my name is Danielle Aila Aras I'm computer engineer from Chile but I have been living in Michigan for about four years so so English is not my favorite language but I'm going to try to explain everything ER I'm going to try to do my best if you have any question I can re reans the question with more ER details and another words but I'm going to try to do my best so um what is cod gbt what is we are doing so everything start with um a full ER extens from a free extension for Vis Studio code so you could connect your llms and you could start to write code with llms in Visual Studio code with this extension we have integration for a lot of providers right now we have Microsoft Google Amazon Bedrock we have the anthropic interraction too so you could connect Cloud 3 gbt 4 and uh it's it's really exciting to start to grow ER code with with llm so we are going to try to explain everything that we are doing with this extension and we we have more features uh but everything start like that so um we have the extension right now and also we have the marketplace and we have a Docker with API I'm going to explain everything I'm going to try to explain everything in this in this session and the extension is pretty simply to use you have to go to the tab in in Visual Studio code you have to go to the marketplace Tab and just have to install the extension that have more than 1 million downloads in Visual Studio code and when you install the extension you have everything that you need to start to H use llms inside of Visual Studio code so H I'm going to explain what we are doing with the this extension if you want to connect different kind of providers you could choose your your provider and you have to connect your own account to use the free um the free plan from the extension to so it's this this part is totally free you could connect your own account and you're going to be able to use your account with your tokens inside of Visual Studio code so this is pretty amazing right now in this video I'm using clo three um and you just have to select the code and we are going to be able to get the full context that you are working ER what language are you you using what libraries are you using in this H in this file and when you select the go we have more context for send to the llms and get more information that you are doing inside of Visual Studio so um we have H the vision to so you could work with image for example here you could select any uh component from any web web page so in this example I'm selecting a bar chart and I'm using like a context inside of the chat in code gbd and so you don't have to just write a code you you could use image to get more context for your work and in this example I'm just using an image and a couple letters and I'm going to be able to get the full um chart in Python and I can run this this script and I'm going to be able to get the the the same component that I take the picture in the web page and this is pretty amazing because you you right now you could use image and and text for coding so er you could use different kind of image for example here I'm just drawing some tables and the code gbd is going to be able to know that you are using this kind of H tables and you are writing H here we could see that the the models they are pretty good ER using image and getting the full information from the just a drawing and get the full H script from SQL so you could you could use that kind of features this is another one um the name of this tool is react sandbox so if you are using react we could build the full component from the same the same thing that I showed you so for example if you use this tools you could get um any component from any web page and you're are going to be able to use that component like a context but the the most amazing thing that you want to be able to watch what is happening in this case the gobd is writing the full code and but you could you could see what is happening and you could itate you could uh send a text to the model and change things in the component so in this case I'm changing the title to the black letters and with one click you're are going to be able to have the full component in Visual Studio code so that's pretty amazing to start to write code with llm so in this case I'm using AMA so if you want to totally privacy you don't want to pay for the llms you could use AMA inside of your computer and you can run a an llm in this case I'm using I think I'm pulling the V2 model from Microsoft let me check oops sorry yes I'm pulling Fe too so you could download the models start to run the model inside of your computer and then you just have to write the the name of the model and you're going to gobd is going to be able to connect with AMA and start to use the the full local model inside of a visual studio code so you don't have to pay you're going to be able to have everything private in your local computer and this is pretty amazing because it's really fast you could use this kind of mod models inside the visual studio code and it's going to be totally free and you could use like chat or you could use like a autocomplete code so if you want to H use more kind of models you could ER read the this article I I wrote the the full article to connect with AMA and you will create your own models inside of with um mod file I think it's the name of the file and then you could connect this this model inside of Visual Studio code and start to use it um how we are doing these kind of things um so we need to know the full structure of the code so er you when when you use this kind of model inside of Visual Studio code you have a lot of information from your code editor so we are getting the full information from your project we are getting the the class the the function the the everything that you you have in Visual Studio gold and we are using that context to send that information to the llm so um when you use the chat models you just have to use that kind of model but when you use the autocomplete models you have to select different kind of models so we are H using completion models we need struct models to get the full information to to pass the full information to to the llm and then we are using fill in the middle to autocomplete the the information inside of the autocomplete mode so for example in this case we are reading the full information from this file and we know what what we have to autocomplete in this line so you could use use CBT for autocomplete to like GitHub co-pilot so this is happening inside of CBT this is the basic mode so you could send the query and we are going to be able to get the full context the full codebase the the code selected and the package that you are using and we send all this information to the llms H with the prompt engineer of course and the llm is able to get the the correct answer so the the other H feature that we have is that we are using experts agents to ER to work with um with llms inside of CBT so you could select different kind of agents in this case we have a lot of agents in the platform so you could use this this agent expert in different kind of Technologies and you're going to be to have more information uh for because right now we have the information from Visual Studio code but what happen if you need information from example from an API of documentation from another um another um another web page or another uh libraries that you are using inside of your project so we are using Lama index for get that information we are using rack H to get all the the uh the information from Vector database and send that information to the llms and but it's not enough so we are working with um with repos too so when we when we work with documents or or libraries if we have examples uh or documentation that will work for example here um in this example I'm using the H stripe API so you just have to select the expert agent and you are going to be able to ask um question about the stripe API so er that's that's pretty amazing because you don't need to get the full context you just have to H select the agent and you're going to be able to use that that agent with that context H with that information and you're going to be able to use that code from the ER stripe documentation what we are using with h the repo so that's um I think the big challenge that we have in CBD we are working a lot with h repos to get the full context from the code so right now we we build this code GPT a code base index and basically is an extractor of information from bu repos so if you charge your repo in CBD we are we are using that um that function that we we build and we are getting a lot of com component from that ER that repo I'm going to explain ER that that kind of H thing that we are doing but we are using um this this new H this new function from Lama index 2 so we are testing what is what is better right now but uh we have a lot of work in that in this kind of features but for example when when we get information from a repo we are building different kind of components for example here we have the the class the method the function and the files and everything is connected so we cannot use Vector database with um chunks to get the full information for a repo because you could qu some function or some code and the llm is not go is not going to be able to get the full context so what we are doing here is we are H scanning the full repo and we are getting all these connections with class and F functions and we are sending this information to the llms we are not using rack so we are just scanning the the repo and for example this is LV index repo we scan the full repo and we get this kind of H graph um this is an example that we are using the L index repo and you could get the full answer from the repo and in this Cas I'm asking how can I split and our ER function is going to be able to get the full information it's going to be able to send the text splitter and you could start to use the text splitter context and you're going to be able to get to have the full information from that function and if you need H examples ER the gobs is able to get the the example from from the repo and send you the example to get the context so the other thing that we have the other feature that has is that we we are we work inside go Visual Studio code but we know the the full developer development ER work is not just in in Visual Studio code so er we have integration with slack with Discord with GitHub and you could use this expert agent or your agents inside of different kind of environment so in this example we are connecting a the agents inside of a slack so you could select your workspace in slack and you could you could connect this agent inside of your slack environment so for example here we are selecting the agent and you are going to be able to have the the the same question that you are using inside of Visual Studio code but in a slack with a more people with with another environment with another information so the last the last thing that's I'm going to show you is that we we have everything on premise so if you want to use the full H CPT platform inside of your servers you could get the full platform we have a Docker with all this H I just show you and you could install everything in your server and you will have the full integration of these tools inside of your data with your privacy with your your security and we have a python and JavaScript ER libraries to to use to if you want to ER work with these experts and that's all I don't know if you have any question um let me know great thanks so much for the presentation um I think we have a few questions in the chat and then also uh you know we also there's just some general Q&A questions I'd love to ask um maybe just uh go through some of the questions in the chat first uh one of the questions is uh do you plan to support other IDs as well besides vs code yeah yeah yeah um it's a a little bit a little bit difficult H try to work with a lot of IDE but we are working H with Microsoft now so we are working with a visual studio code but we we plan to to get more idees in the future um one of the questions is in terms of retrieval um uh it's specifically about like reranking do you happen to know any like reranking capabilities that are specific to code based data um or you know maybe the more general question is uh are there just like General things around retrieval you have to think about with respect to codebase data that are different than um you know like standard retrieval over tax yeah we we are using R ranking but um when we work with repos it's a little bit difficult because you have different kind of function in different kind of um level of your project so um when we work with documents is is it's working but um it's different way when you work with with repo so and and that repot that I I just showed you is is a good repo we have the doc tree we have the the the the good name of the the function we have the the class everything really good ER structur it but when you use another kind of preot that's not really good ER structur it h everything explode everything is different so um yeah I think for for rer we we are not using rack so we need to change everything when you you think in in code I I think it's a challenge we are just testing this these things that just show you so um probably in in in a couple of weeks we are going to have more more details that we we we are doing in these kind of examples okay no re yeah I was about to ask and and you know it's it's okay if if you um aren't able to share some of these things because because you know you have some inflation under the hood and you're releasing some stuff in a f um I guess maybe at a very high level I I know you mentioned you're not doing rag for the repo but given that like the repo can be very big uh what are some like General thoughts or intuitions that might be able to help guide some of the listeners here yeah the things that you could use rack but when you have documentation inside inside of the repo you could use it for example in in Lama index repo you have a lot of documentation uh so you could have that kind of um files and when you when you read the full repo you could select different kind of f folders and files and you could use rack for that kind of folders and it's going to be really good but when you have code and is is complicated it's different so that that's changed everything um one of the uh questions was basically you know how do you um make this like local secure in terms of like proprietary code I think you basically answer that because you said that you have like private deployments um but if there's anything else you want to add uh feel free to I know you support like AMA and you support like Docker deployments yeah yeah yeah exactly so so when you use the the free um the free extension we are not H using Oro servers to connect with the the provider so you could use the the extension and you could connect your your own API so you could connect with your your provider and we are we are not a we are not a g gway in this kind of call so um but when you use our uh platform with our experts um yeah that that's that's happen in in our servers but does happen with a big big companies that we are working right now for example a bank that we are working right now they they cannot send that kind of information from uh that that um that kind of information to our servers so we have Docker so you you could use everything inside of your H service and you can understand everything that just show you H and you're going to be totally private y with your security with your data and everything is going to be yours um the next question I'm actually pretty interested in the answer in your mind like what is the best way to chunk and split code um I I know you defin that graph of like a l index code based as well as over codeb codebase um what have you found to be just effective in in that regard yeah yeah we are we are H testing this kind of this kind of functionality right now so um we are not using an llms to get the the we are not using embedding we are not using chunks so I think this is the the the better things that you could use to get the full H context of a repo because you have classes you have function you have a the documentation so you need to connect everything and probably one class is going to be connected to like 100 class so and one class is going to be using like um the same function in different kind of parts so if you have the full connection you could have the the the the the full note and the full way to to get the full context um it's different to explain this but it's difficult to explain this but I think this is the the way that you could scan the full repo and get more information and then you could use rack from the documentation and you're going to be able to have a full context of the the question of the user I I think maybe just for listeners I I feel like when we think about rag it typically is in like a very traditional setting where you have like a lot of text and then you chunk it and then you put it into a VOR database um but it seems like from here just from code your you're not really modeling it in in that type of pipeline like at all to start with uh you're really just defining like some sort of abstract like syntax free right and then for any documentation related things you are splitting it up and chunking and and doing some vectorization um but for code itself you mentioned you like scan the entire repo you just like Define this overall graph uh and I assume like there's some algorithm you're running on top of this graph to basically fetch like relevant information from the codebase um and whether or not that's like vector search like you could be using like knowledge drafts you could be like doing some sort of like keyword entity lookup and then traversing the connections and the links um I think these are all possible so so I think maybe you know I'm sure like not not all the details are are public yet but like generally speaking I feel like the like maybe for code specifically like some of the existing rag principles just don't completely apply yeah yeah exactly um what are uh so actually I I have a question um like you you mentioned um you have like a chat ux uh and then you have like this autocomplete ux um what are uh some of the general uxes you're thinking about in terms of like how to best assist the user um and which of these uxs is the most popular among your users yeah ER the I I don't use a lot of the autocomplete in in my project so I I use a lot the the chat ux I think is is is easier to use that kind of a ux um it's ch gbt so everyone could use it um I think right now the the the next step is the agents or the llms could read your code and then they they can send you notification from something happen in your code and you don't have to start the conversation if you have this kind of features in the future you're you're going to be able to have not notification from one agents and go you don't have to start reading everything every time uh I think H when you have that kind of um notifications you could have more um how to I say that more independ from your project I don't know if I saying these kind of things correctly but I'm trying to say if you don't need to H start a chat and start an ask a question with your code and you you don't have to send the information fears um it's going to be easier for you because you just have to wait for the that notification from the llms um I think I I I explained ER if you can understand that yeah know I mean I I did want to ask your thoughts about like just um overall directions for agents as well as like what how you think these Evol yeah but actually right before that um in terms of just like the chat based ux um one one one question I want to ask is like when you first look at it it looks very much like the chat gbt interface um what what are your like maybe at a very high level like how do you explain to users like what this is doing say Beyond just like a raw LM call um because you know from one way of looking at it it could just you people might um think oh this is just calling like trbt or something and getting back an answer and then you're just like copying and pasting the code uh into vs code is that how that generally works or are you also doing um like context augmentation inserting like the repo context into that like TR interface as well yeah yeah we are we are getting context let me show you I'm going to show you the the extension really quick can you see the extension now yep okay so for example here we have the the the the extension open over here so you could select different kind of models so right now we we have this this H ux so if you want to use ER the the marketplace with more expert ER you could go to this way and you could select one of these experts so right now we are trying to get this kind of a context or this kind of Agents uh more easy to the user so they can start with one of these H experts so if you are um working with typescript for example you could start with that kind of agent so in chat gbd you have to um send the full context so you you have to write you um I'm working with typescript h you have to act like a really good programmer with uh typescript and you have to write that kind of um that kind of prompt so right here we have the the the expert with the full context with the full documentation so you can start working with that the with this kind of expert in in your project and you don't need to ER select or write that kind of prompt to start to work with so I don't know if if you if you start everything from one just one prom it's going to be difficult for the user and we have a a lot of new programmers that they are start to work with llms and they don't know how to write the the correct prom for a start ER asking question to um the llm so we we could do these kind of things and they can start working with one agent or with one expert right are you you also injecting the context from the repo into the chat interface or is that mostly for the autocomplete functionality yeah from the repo too I see yeah um in in terms of uh you you ask is like be Beyond um I'm actually kind of curious to get your thoughts like um so you have a currently like chat interface and you also have like an autocomplete interface uh what are ways that you're thinking about extending like Co gbt um especially if we look at it like 3 to six mons from now and also a year into the future uh what are some like overall interfaces that you want to involve in this into yeah the the other the other feature that we are working on is that you could run your code and you could get the the full contest that what happened with your code so for example if you are running a python code we could run your code we can get the what happened and we could get the context of a the the console so we could iterate in that kind of features in that kind of um functionality and we are going to have more context from what happen if the co the code have box so we could fix it so we are thinking in something like that we are working in something like that right now and we have a lot of new feature that we are thinking is you you could do a a lot of thing with this this technology and inside of Vis you have a lot of context so you could just use it for for get your GitHub repo for get a new commit for example uh I know GitHub copilot have a lot of feature too that they are working on but I think ER we are we are different because we are using a lot of models uh so if the a new model is is in the market you could just change the model in inside of gobd you going start to use it so I don't know I I think uh we're we're going to be able to H publish more features in the future but we have a couple in mind in mind I think one of the uh comments actually is basic there's some potentially interesting features Beyond just like autocomplete like you could have uh well you know you said your mention or you mentioned like being able to execute code um but also having like some sort of bash mode where you can you know execute like cly commands uh maybe run like get commands to and then basically integrate with your overall like Git Version versioning like GitHub management right uh in addition to just like the rock hode itself yeah we are working in in more integration because we know that the llms they can write a really good code but we could use it in the full um in the full um software developer environment so we could connect with the slack we could connect with tro with h jira with another tools that we use to H developers software so we are we are thinking in a lot of integration too um I you know last week I think one thing that got very popular was Devon which I'm sure you know is like the like autonomous agent that can go off and and do things you know and maybe takes a while to do it for like 10 to 20 minutes but then it comes back and and is able to just build like a full stack web application for you I'm curious to get your thoughts on like um whether this is like a direction that you would want to evolve code GPT to or you thinking about like a fundamentally like different ux where the human is maybe interacting with the code a little bit more yeah I think dein is pretty amazing um so what we are trying to do is like uh the connection from the the the basic ER software development right now that so we are trying to connect this kind of um this kind of this kind of um um right now we we have one way to to to developer software so we are trying to connect this kind of way to the this new way with llm so um I I think Devin is is you is is is doing something amazing with the full control of everything but I need I I think the the deps need more control of what is happening in in your work so ER if you need to know what is going what the llm is gonna is going to H do with this kind of code or if the llm is going to ER write a different kind of code you need more control so in this way we are trying to ER to to we are trying to I don't know it's different I think it's a different tool tools we are using the llms to write code we are using L llms to to get more context from your project if you need a if you have more context more information from expert more information for apis you're going to be able to write a code faster and better but H you need more control I think if you have the full control of your project you're going to be able to write the the the the the full uh Pro by yourself yeah what one thing to maybe maybe not full autonomous agents but one thing to make it like um automate a little bit more of the workflows is to kind of add like pockets for some Asian is able to do certain parts of a developer workflow um I think this is a com a comment in the chat as well which is like um one thing you could help automate for instance um is like unit testing and code testing um so that you know given some piece of code it like it's a interesting exercise to think about know how do I just generate like a bunch of unit test and put it in the right folder um and then you know kind of go back into trying to run over these unit tests having like understanding where it's failing um and then even like suggesting to user to start with like this is where the code original code might be failing and here's how you could potentially fix it yeah we're not working in that right now but probably in the future we're going to have more integration like that H but yeah it's very interesting that you could you could do with this kind of technology and maybe a general question is um so you support a lot of different LMS like what llms have you noticed to be the BB code o really good question a I think um gp4 is is the the best llm that we I I always use gbd4 for code right now clo three is pretty good but um I think for for different kind of different kind of context but um what another one is that's really good is a deeps ER coder I'm using that that model in with AMA in my local machine and it's really fast and and really good and C Lama to is really good um I think that's that's for llms that they are really good with coded yeah I think it's interesting from like a developer perspective do you see a lot of developers like picking between LMS or do they mostly stick with like grd4 as a default yeah I I'm I'm doing that so I'm changing from gbd4 to Cloud three every time that I don't have the the correct answer so I think in the future developers going to is going to try to different llms but I think the the when you have the the the correct context and and the full information H that you could have from your project uh the llms is is is not important if you have a really good context you're going to have a really good answer answer so I'm I'm using gbt 3.5.5 from for some um for some task so if you have good context you could use whatever you want right now because all llms they're really good yeah Mak sense yeah I think um you know may maybe some general thoughts and then before we conclude I yeah I think it's interesting I do think we a lot of developers these days um honestly are just straight up just copying and pasting code into trt or I've just been copying and pasting code and J Pro clad because you can just drop entire folders uh into jav I think um and then you can also just upload like some code files with like the 200k context window that cloud 3 gives you um and then just asking questions and getting back code and you just copy and paste back into your editor and so add a very very basic level of things like code gbt at least everything's kind of integrated in in VSS code so that friction time between like trying to copy and paste code is is way shorter um but because it's more integrated you can and you have that chat interface and assuming you are actually pulling context from the Rebo you might have you can maybe reduce the time it actually takes for users to just like upload full context into that UI interface because it's pre-baked in and so you can just use that to basically uh ask answer questions right over like any amount of data in your repo um and I do think like a lot of people are still using chat to help them with development and this could potentially by being more integrated with their development environment be more efficient yeah yeah right now that's we are thinking to use that kind of a big H window context because we we have more more um Windows to to use that kind of a code but the thing is when you have the the full repo is different because you have a lot of files and folders you have a lot of uh information in different H in different um source so yeah you're going to need Rack or that kind of retrieval information to to work with the full context of uh software developer project so yeah like you could still do retrieval I think but it's interesting I think with long contact models you could probably do retrieval but just like uh on the level of files so you just dump like entire files into context window um so like given any sort of question just like fetch like the the top like five to 10 most relevant files and just like stuff everything into the context window and see what the response is um of course that that's going to cost you a lot more money but um you know maybe as these models get cheaper in the future this becomes more feasible yeah or prob free yeah um great okay I think that's basically most um most of the questions so far um and then we can conclude the webinar yeah for for everyone in the audience thanks so much for joining um and then just Daniel any last things you want to call out before we conclude um you could use code gbd for free so you could start to use that ER that Tool uh if you want so you could use with Alama too so if you want totally privacy and your code and create your own H co-pilot you could use it and no thank you so much for for this invitation thank you for much so much for having me here and that's all all right thank you Daniel and thanks everyone see you next time thank you