Deploying a Private Coding Assistant in 5 Minutes, Yann Leger, Koyeb

Key Takeaways

training will be deploying a private coding assistant in five minutes together um if the network is with me um so hopefully it should be good uh I'm Ian I'm the co-founder and CEO of coep um I've been building Cloud infrastructure for a little while um and basically we're going to focus on the last steps the rest is the past doesn't matter much um we're doing an AI servess Cloud infrastructure uh today um what our car business is to provide fully managed a fully managed Global servess platform to let you deploy both high performance apis um and inference workloads uh today we'll be focusing on the inference workload side uh with an LM which is deployed and basically um what we are going to do is we will deploy a private uh coding assistant on GPU with cot St from mol um and use it in vs code uh with um a tool called continue dodev which is doing like the aut to complete and giving you the plug-in basically to query the all um so if I click on the link at the bottom basically I will a would L uh here more or less uh to deploy AMA uh and I have a huge like deploy button um and um so I can actually technically hit this button whoa this is like super bright um and uh I'm going to deploy it on a GPU to make it faster so we have a few like kind of machines um but basically what it does is it deploys an uh server which is packaged in a Docker container um and um I'm going to pick a GPU to make it faster uh here and uh it's in frankf for so that's okay and we'll just hit deploy and um so the good news is the network seems to be working so what it's going to be happening um in a few minutes is that uh we're going to create a virtual machine with a dcker container running inside of it and the GPU which is passed to the to the D container um so it should take something like um actually I can maybe zoom out a bit um it should take something like um one minute uh basically um and because I don't want to make you wait this minutes uh we'll switch to one which is already um ring so we have the one which we are which is spawning which is starting here um we have the at the top we have one which I created before um and we have an open web UI which is actually like uh curing like the theal um and so I previously pulled like like two Ms so this is a service which is exactly what we just did so if I go to the uh right tab uh I will be quering so wow this is also too big okay so this is just a web UI to query the M directly um and I can query like a a tiny Lama mod which I I I pulled on this Cal M um so I previously asked hey do you know how to repit python maybe I can ask like and um what about uh Ruby and basically it should work uh if everything is with me and use this remote mod and I'm going to do the use the exact same um remote instance and so the end game of this is to have your own private M which is not train on like third party uh which is not reusing your data and that you can trust uh um so yeah apparently it also knows how to run Ruby so it's good news um and you can also use the exact same Mall uh inside of uh VSS uh card so we are using a a tool called continue. Dev uh it's the one which is open like here on the left you have a chat uh and you have uh basically also uh a simple config so it's using like this uh app which is deployed on co. co. app so it's a remote server um and uh it's going to provide autocomplete but with a fully private model so uh here I can basically interact with the mod uh directly um and I will ask like I want to uh because what I actually want to do uh today is like to create like a uh service on coab uh with our own Pyon SDK and actually I don't I'm not writing much software this day so um the model is going to help me out here uh so I wrote like some lines before so it's basically importing our SDK uh creating a basic Mall uh and um if I go basically uh for one normally it should like suggest me code using this remote mod and actually it works um and so this is actually covering like the remote uh AMA instance which is running C cutr so um it's not using so it's not on my laptop um and it's like using High Performance Machine and can be using used like in a in a professional context so um yeah and so I can actually like also ask the all on the left something so um I'm new to uh V to cab uh and need to uh create a uh server uh with the python SDK wo and now the mall doesn't want to work anymore that's interesting so I have no idea what's happening uh oh just like it decided to take some yeah it works um and so yeah that's going to conclude the demo for tonight um but basically uh the end game is to show that like you can run like something which is usable in oh and actually there is a feedback here so that's interesting to know for the next speakers um but um yeah so the end game is to uh um was to show you that it's like not really complex to have something which is an llm but in a completely private setup and production grade um so yeah so we have more gpus coming up um with the same exact developer exper experience and optimized uh boots and so if you want to try it out um you can uh go on c.com and we give free credit and that's it for me for tonight wooo thank you Yan