Deploy Gemma 2 with multiple LoRA adapters on GKE

Key Takeaways

[Music] hi I'm V from Google cloud and I'm abdell today we'll show you how to deploy Gemma 2 with multiple Dora adapters using TGI on Google kubernetes engine that sounds complicated can you break it down for us sure let's go over some terms first Gema is Google's open large language model and TGI is an open source Next Generation inference server from huging phase now low rank adaptation or Laura is a fine-tuning technique used to adapt a base model to specific tasks without retraining the entire model instead Laura introduces small adapter layers that are trained on a targeted data set while the original base model remains Frozen so why would you want to deploy Gemma 2 with multiple Laura adapters on jaki well there are a few benefits in a multi Laura deployment you deploy this single base model and then load multiple Laura adapters and you can dynamically select a different adapter for each incoming requests so you don't have to create multiple deployments which is more cost-efficient that sounds great how do you actually deploy Gemma 2 with multiple Laura adapters with TGI DLC on GK um I was hoping that you would show me that sure well it's a multi-step process first you need to create a GK cluster we already covered this in previous videos check the links in the description box after that you need to deploy Gemma 2 with multiple adapters using the TGI deep learning container on the cluster that you're right that doesn't sound complicated at all is there anything else people should know well first of all let's look at some y files first we have my deployment. yl file this file describes our deployment including the number of replicas the container images we are using in this case TGI and any environment variables see this section here this is where we specify the GMA 2 model and theora adapter we want to deploy next we have our Ingress yaml these files Define how our deployed container will be exposed to the outside world it basically creates a load balancer so we can access our Gemma 2 model enough enough jaml files let's get this show on the road can you show me how to actually send a request to it sure using curl you can send request to the model and specify the adapter layer to use in the particular case I am using the SQL adapter okay it looks like you got a response back but I was hoping you can show me how a developer would use this in the r of course using the hugging face Hub is dek I have to import the inference client specify the model endpoint that is the IP address of my load balancer and invoke the model using the magic coder adapter layer to generate some Rust code that's amazing this is this is rust code so you're saying that this is one deployment of a base model and then TGI selects a different adapter based on the request that's pretty much it awesome well there you have it we showed you how to deploy a Gemma 2 based model with multiple Laura adapters don't forget to check out the description for links to all the resources we use today including those yaml files and if you have any question drop them in the comments below thanks for watching and we'll see you in the next one [Music]