Model Training and Autocomplete Functionality // Varun Mohan // MLOps Podcast #195 clip

Key Takeaways

hey I'm Veron CEO and co-founder of exf function that builds the product codium which we'll be talking about and I like my coffee as as a lat I'm pretty basic I like just a lot of milk in my coffee CU I I really want to hear about codium and I I really like this idea of like you built something knowing that you were already in this gigantic scale cuz you were with EXA function you had scale that you were giving to customers and you were helping people with the serving layer and then you were like okay well let's just do that but with our own product now right y so I'm guessing that because now give me the like fact check me on this but you're serving a lot of people like hundreds of thousands of developers yeah so yeah break that down let me go into the story so we start we started building codium last year we realized the product needs to be free we would go bankrupt if we used opening so we then started saying hey we got to like train our own models um we have to train our own models to make this work so we actually pre-train entire models from scratch to run codium for our developers um we train our own models we evaluate them we deploy we AB test them all this other stuff and we learn a ton by the way and a lot of the conventional stuff of like prompt engineering yeah we had to prompt engineer in interesting ways and I can go into that but a lot of it was domain specific and I can also go into that because for our problem I'll tell you why training our own models provided us a lot of value for our problem when you write code codium one of its main features is autocomplete and autocomplete is maybe a little bit of a devious term because it sounds like oh this is like providing me like a couple characters we sometimes generate 20 30 lines of code for the end user no way that's awesome and the reality is when we when we generate code we need to look at both code before and after code in different files too and this this doesn't look like the data set of this doesn't look like any of the tasks that exist right now for these pre for these generative models generative model is generally speaking the way it works is like you have some existing text and it just keeps adding on to it it doesn't by default it doesn't know how to apply context from other files it doesn't know how to fill in code in between in in between your cursor and so we had to train models explicitly to do these toss well and to tune them to to do well on these toas so we had to actually go out and trainer own models to actually do [Music] this