vLLM: AI Server with 3.5x Higher Throughput

Key Takeaways

this is amazing now we are going to see about vlm it is easy fast cheap llm serving for everyone so why we need VM if you are planning to host your own large language model in a GPU which requires High throughput parallel processing streaming output open AI compatible API server then you need VM High throughput means you are able to send multiple requests to this AI server so VM is nothing nothing but hosting your AI on your server which you can run privately in a cloud or locally in your computer by the end of this video you will learn how you can install VM how to run it how you can integrate this with your own application also we'll be seeing how this differs from llama file and O llama that's exactly what we're going to see today let's get [Music] started hi everyone I'm really excited to show you about VM the key difference between llama file and VM is that llama file uses just one file and with crossplatform compatibility also LL file focuses on CPUs if you don't know about LL file I've covered that separately on another video which I will link that in the description below so VM focuses mainly on hosting a large language model or serving a large language model which has GPU support even though this supports CPU the main focus is on GPU just because of this high throughput and parallel processing it is able to perform any task quicker one more key difference between llama file and VM is that VM has a list of supported models as you can see here so only these models are supported but in regards to llama file as long as it is ggf format you are able to run the model you can convert any open source large language model to ggf format which means you are able to run any model you using llama file so here are the key difference llama file CPU focused and you can run any models VM GPU focus with limited models so when you see o llama it is similar to llama file because ol llama and llama file uses llama CPP at the back end to run it on a CPU but VM does not use llama CPP it has its own optimization techniques I'm going to take you through step by step how you can install run and integrate but before that I regularly create videos in regards to Artificial Intelligence on my YouTube channel so do subscribe and click the Bell icon to stay tuned make sure you click the like button so this video can be helpful for many others like you first make sure you got graphic card I'm using RTX a6000 I'm using M compute so you can use mvin praison as coupon code to get 50% off mainly for a6000 and a5000 now I have logged into that machine here first step pip install VM and then click enter now it has installed the main package now VM serve then we need to provide a model so if we go to the models list I'm going to use quen 7B chat just copying the model name and then pasting it here then going to mention trust remote code that's it only this line of code and then click enter this will automatically download the model and it will start running it you can see it's loading all the checkpoints and now you can see it's running in this URL with port number 8,000 now we have successfully started the server so you can even run this on any GPU supported private Cloud instance or virtual machine you can also run this as a Docker file and it supports multiple GPU now we have installed VM and we are running VM the final step is to integrate in your own application I'm going to use open AI SDK because this is open AI compatible so in your terminal pip inst open Ai and then click enter now let's create a file called app.py and let's open it inside the file from open AI import open AI then setting up the client with the base URL Local Host 8000 version one API key as a fake API key next client. chat. completion. create and then we are asking a question to the large language model that is give me a meal plan for me today model name quen quen 7B chat that is the model name which we downloaded and running after this just printing the response like this that's it only this bit of code and we have integrated VM in a python application now I'm going to run this in your terminal Python app.py and then click enter and here you can see the response from the large language model quen using vlm similarly you are able to use post request without using open AI SDK just by using post request you are able to call the quen 7B chat model and ask question like this I will put put all the code in the description below you can also run this using prais chat just type pip install pra chat and then click enter next export open a API base Local Host 8000 version one and then click enter Then export open a API key as a random fake key that's it now praise AI chat and it will automatically open this user interface in the settings I'm going to mention open AI slash and the model name click confirm now I can ask who created you and here is the response I'm a l language model created by Alibaba Cloud that is correct so finally the key difference and when to use what if you want to host your models on a GPU use VM if you want to host your model in a CPU then use llama file and if you want to build and test a AI application as a developer then use ol I'm really excited about this I'm going to create more videos similar to this so stay tuned I hope you like this video do like share and subscribe and thanks for watching