Demo: Building cloud-native, AI-powered applications with GKE

Key Takeaways

[Music] hello everyone my name is morphy I'm a Dev Advocate at Google cloud my focus area is mostly on kubernetes and running AI workload on kubernetes um so quick hands on how many people have heard of kubernetes or used kubernetes before all right keep those hands raising for a second how many of you have used gke if you haven't drop your hands h still quite a few number so GK is uh Google's managed kubernetes so we're going to talk about spe kubernetes how to run AI workload on kubernetes and specifically GK in this case when you talk about building Cloud native applications um there is kubernetes probably the first thing a lot of people uh think about when you think about Cloud native application and when you're talking about AI workload on kubernetes I specifically think of three separate use cases number one is you have a model that you want to just make sure that other people can access so the serving inferencing use cases number two is you have a model but or don't have a model but have data that you want to train the model with so you want to run fine tuning or training job both full fine tuning or low fine tuning of that case and the number three which is the probably the most common used case you have an application and that you want to sve somewhere right and that application these days might be talking to something so they access AI related uh apis and other things and that API could might not even be running in your cluster that API could be a Gemini API or something like open AI API that you're calling from your application so we want to make sure that all three of these use cases are well served in that regard like why would you want to even use kubernetes for things like this so if you look at a spectrum of things that are fully self-managed to fully like managed for you uh something like Vex that my colleague just talked about would probably in the Fest end where you just care about your application AI data and models everything else is taken care for you your own data center or something like a bare metal server probably will be all though in the other end where you manage everything from updating your like Linux distribution installing wiring uh figuring out how much Network bandd you need all that stuff kubernetes probably sit somewhere in the middle right you have a lot of things you can tweak and control but for the most part A lot of the upgrade and update gets taken care of so you have the flexibility of bringing the tooling that you want to bring in like the library that you want to use the software to use and once you build a container that kind of becomes that portable unit of operation that you can take from this cluster to other provider to your own data center that's running kubernetes performance- wise again kubernetes just talks to the underly infrastructure so at GK we can give you all the necessary resources like GPU and TPU for you to get the best performance for your application that you want to run and finally the efficiency is both efficiency in terms of getting the right resources as well as being able to manage an autoscale environment that can scale down to give you the best cost efficiency to be able to run things at Large Scale now at Google if even before kubernetes we have been doing things with containers for a while now uh we at Google we have been running containers for roughly 20 years for pretty much all the big application that we run and all that institutional knowledge have been baked into our manag service for kubernetes which is gke we can provide unparallel scale compared to the rest of the industry at this point as well as a new mode of operation for kubernetes called GK autopilot where you don't even have to think about the nodes you're running you just Define the application you want to run in a yl format and autopilot takes care of provisioning the hardware running your application and scaling them down when you no longer need them so we talked about the three mode of operation I think about when we talking about running AI workload on kubernetes so the first one we're going to talk about is taking a model like Gemma in this case and fine tining it I think we saw an example that uh kilin just showed to like the notebook but when you want to take that same experience and somehow package that in a container the task itself is not that much different we want to just take the same code package that in a Docker file get a container out and that container then can run in a kubernetes environment so we're going to look at an example of fine-tuning Gemma on GK before I start the video which is going to load in a second uh uh you in the screen you what you see is the Google Cloud console for gke and I have a kuties cluster with three note pools uh one of the note pool is just your general CPU nodes two of the other node pools the G2 standard 96 and G2 standard 8 one of them has has eight L4 gpus and one of them has one L4 GPU the one that has eight L4 GPU that's the one I'm going to try to use to F tune a Gemma model using some data set from hugging face and the reason I'm trying to fine-tune this model I'm going to take the pre-train model of the 2 billion parameter model but fine tune it using a data set called SQL context so what that does is if I give it a question about some table and give it a table structure it should be able to give me some SQL query that I can uh then run against my database so I'm going to start the video uh so this part goes pretty quickly so I'm going to select again these are the two note pools that I have with gpus and the code is taken from a notebook I just copied all the codes of the notebook blocks and put in a python file and I'm setting most of the variables that I need like my Laura config if I'm using a float Point 16 some optimizers I'm also uh going to set up the uh so the data set that comes in I'm going to change the data set into some different format that I need as well as uh the the Laura config I'm going to have some training arguments all of this stuff you you have seen if you went through the notebook yourself but again all of them are now in the single data set file also run the trainer. train after the model training will finish I will then copy the I will then combine the model into a single fine tune model and finally upload that all up to hugging face I will need a Docker file to make that in a container so that's you see the docker file there and put that into emo to talk to kubernetes after all of that is done only thing left to do is to run the command in real life this command end to finish takes about 33 minutes but for time sake I have sped up the video so this whole thing is going to end in about 30 seconds but what we'll see if you can read the font which is not super big I I notied now is again the trending is happening I have 5,000 total data set to Total uh rows in my data set which is about 5,000 steps but because I have eight gpus you're going to see my total step is actually 625 because each GPU kind of breaks it into each step is about eight step in total in real time uh in about 33 minutes this I have my loss function I have my model created after that is done my model weights from the Lowa weights will get combined with my original model and then pushed back up to hugging face and in a second I'm going to also go to hugging face the last time I had pushed this was about 4 four days ago but when the video was recorded the last time I had pushed this was 38 minutes ago and which updates to now less than a minute ago so after I finish uploading if you were to go back to this link which is mic code JMA 2B SQL you should see that the the model file was up updated about 4 days ago so that was the first part right we find two our model and then pushed it back to hugging face now anybody on the internet can go and grab that model and set up their own serving that will give them a model that is now equipped to handle SQL queries now before we go do that let's just talk about what all of that kind of as a big picture looks like at the very bottom layer we have our accelerators that we want to use we have our gpus in this case we also have tpus which are like custom built hardware for doing AI workload like training and fine-tuning and on top of that we have our kubernetes platform that is gke that you can have not only your application running but as well as AI workload running on top of that you can think of different types of AI workload you want to run you have your training fine-tuning that's one type of workload that requires a lot of resources for kind of a long time and the other side inferencing which is more things that needs to be running constantly as deployment so in that case being able to optimize Resources with that if you're using a ag GPU constantly that's probably more expensive than having smaller gpus that can do fine tuning uh do serving constantly as like a deployment and finally on top of that you can bring any kind of tool like your pytorch your caras other tools in the open source Community as well as close source that can run on kubernetes that can give you the complete picture of your AI ecosystem and as for this release for Gemma from the GK team we released four different guides that allows you to uh figure out how to do serving on gke using open source tool that exists in the community so we have for GPU we have VM hugging face TGI as well as 103 llm on Triton server from Nvidia and for TPU we have the example with sax ml that he can run to see how serving the same model can be a little bit different across these four different options now that we the last thing we're going to talk about is the finetune model that we just pushed up how do we go about serving that model in GK so the four guides I talked about the one of them guides is serving open models using gpus on GK using hugging face TGI TGI stands for Tech generation inference I think a previous speaker already talked about that kind of world of serving large language models so this guide kind of goes through serving Gemma 2B 2B it 7B and 7B instruction tune version of the model on TGI so for us to get started all we basically need is that yaml that we are using here but in this case instead of using the Gemma 2B model directly we're just going to swap that out for the model we just tuned the model we have under muy codes SQL something something so we can copy this whole Yo over to there and we're just going to replace this value with the value here muo Gemma to be SQL and after all the if the all the sec in the right places all you need to do then is run this application slowly type that and again this process to download and set up serving takes about 4 minutes again to the magic of uh video editing we can get this done in about four to 5 seconds but the basic idea is the same you deploy it goes to hugging phase downloads the model sets up all the weights and token in the right place so that it's ready as a server for us to go then make HTTP query against it now I set up like a port forwarding over there so that I can call it from my Local Host you could also set it behind the load balancer if you want to get the same experience across the internet now I talked about how this data set was about SQL so the user prompt I have is some question about a data and I'm also providing the context of what does the database look like so I have a database that is create table uh Farm of some data set and then when I make this query against my table the model that previously that was just giving random answers now it is it can answer things like give me a select average working horses uh from my table so again I haven't actually tested but like from the looks of it first glance it looked like it's going to work but if it doesn't I can always go back and find T it even more to fit the need I have so in this short 78 minute video we kind of saw the whole flow of you have data you take that data and F tune any model that you want to and then push that model up to hugging face and then also set up serving so that not only you but other people also can take advantage of a fine tune model to use in an environment they want to so if you can set up like a load balancer against it other people on the internet can be using it also locally from your cluster other applications can hit that endpoint to get data out so we're super excited to see what you'll all be building with these kind of tools again GK cluster you can have a node with like a huge amount of GPU not just l4s but h100 A1 100s to fine-tune really big models not only from like fine-tuning with Laura but also like full fine tuning or training and get similar results and great [Music] results