Considerations and Optimizations for Deploying Open Source LLMs at Your Company // Oscar Rovira //

Key Takeaways

our next speaker on this lightning round um is Oscar from Mystic Ai and Mystic AI is actually a y combinator company I think they're based out of the UK and bath um and he is going to talk to us about something that I'm personally really curious to hear more about um open source llms hello Oscar how's it going hi Lily thanks for having me very good yeah of course all right here are your slides in all of their Glory I will let you take it away awesome thank you Lily so yeah so we're going to basically talk about some of the considerations when it comes into deploying these open source llms um so I'm sure like many of you are probably familiar with this scenario in which uh you know management team goes to the data scientists or the ml engineers and it's like oh this is all these modes are amazing let's see how we can make it valuable for a product but you know how do you get that and so really the the question that everyone is kind of asking is how do I go from this llm big model into getting a fast secure scalable API endpoint so something like similar to like open air it does with their own private models how can they do the same experience but with open source models or the llm that maybe I've trained myself and so I don't know it all actually becomes a very similar software engineering problem to other ml models that we've been deploying it's just maybe potentially more you know memory required to actually deploy this model and so if we go through some of these challenges um so you know we know that each model actually is different and so because of that variability one of the things that happens is that each of them require different libraries different packages to run so you need to have a system that is able to manage whatever is the environment required to run this model now as we mentioned llms are one of those uh beefier models so like they require actually a lot more GPU memory than other models that maybe we are familiar to have in production and so having a consideration for gpus are able to run those models is also very important having a system that is able to take care of large uh mem one there's a lot of memory required to run this model and so with that in mind you actually have to think about which is the best GPU that is going to like deploy this model but you know where we are heading towards the kind of like solution is one that you don't even have to think about all of these things but one actually has to think every time they deploy a model is exactly the requirements from a library's perspective and memory as well and so one of the classical uh pitfalls and and drawbacks that we're seeing right now is the amount the limited amount of GPU availability to actually run all of these models because they consume so much memory and those are actually very expensive and everyone wants them and so it's how do you access a huge pool of gpus when maybe one cloud provider doesn't have access so like how can you easily go to a different cloud provider that maybe actually has the gpus that your model requires then when you're actually thinking about like scaling this model is one thing is the point it in one instance on one GPU but then you actually need to consider you know how many users are going to be using these models is it going to be just one user how many requests is going to be handling per second and then you actually have to consider potentially hundreds of requests per second and so if it's running concurrently that means actually hundreds of gpus running at the same time and so you need like a software that is able to actually take care and dynamically scale according to how many requests that given model is asking for and then at the same time we obviously want to make sure that uh you know we don't spend thousands and thousands uh per hour on running some of these models and so you need some kind of software that is able to dynamically optimize for you know how many uh what is the most optimal GPU to actually run this and and there's different different techniques like using spot instances leveraging multiple Cloud providers in which one of them may be cheaper than the other Etc now obviously uh you want to make sure that the end user experience is as optimal as possible and so we are targeting or like ideally in the industry should be targeting 50 milliseconds of API latency which would allow a much more um easier experience for something like streaming then other optimizations that one has the thing is how do you make sure that everything is ready on the GPU uh when the user is about to do an API call to this model that has been deployed and so something that we call preemptive caching is how do you make sure that the model is cached ahead of time of the request being processed as well uh you know as I mentioned if you actually want to like be able to run the maybe the nicest experiences that we are seeing with llms is that kind of like streaming as if you were like talking to someone or chatting to someone it's like the word comes out uh as soon as possible and as opposed to having a chunk of tags being thrown into you and so like you need to have a software that is able to actually provide that kind of streaming as well as being able to deploy these type of models in the easiest way possible for the data scientist team and they don't have to go through massive amount of layers of maybe software development or actual teams to ask them hey I have my Jupiter notebooks or my models please deploy them make sure it all works how can we bring that barrier closer into the data scientists and empowering the data scientists to do it themselves then obviously you need to make sure that you're monitoring the full system so because things break so being able to quickly resolve whatever it breaks and of course you especially with open source that's one of the reasons why it's so exciting for many many companies at any scale is to actually be able to run these things on your own uh premises or on your own cloud VPC whatever so you want to make sure that you optimize uh privacy and security and so you need software that is able to be deployed whichever other premises that the company that you're working for uh requires and so how all of these challenges how are people are currently solving them I think there's like three kind of like uh higher uh overview type of approaches the option one is let's look at like what is the current software engineering team and what are the skills is that they know how to use how to leverage Classic kubernetes Plus Docker and let's just make sure that each circle file what is Docker has a different model and we just leverage case for all the scaling and that's what we are used to because we run a lot of microservices mostly on CPU compute and we assume that potentially this is going to be the same for GPU workouts so price is actually not the same but you know there's a lot of like uh challenges that come when you actually try to deploy things on kubernetes for specifically uh um GPU compute and so it's a good thing to actually approach it this way because it gives you a full control and you're able to actually deploy it in the infrastructure that you maybe already have in place but also requires the experience or an expertise of knowing how to build this at with all the challenges that I've just mentioned and how to make sure all of these get you know solved through the infrastructure the first software that you are um that you already have in place in the company and obviously building something like this from scale from the beginning this ammo internal ml platform actually does take a lot of time the second option is you look at a cloud and then you just use whatever is the solution that they have like vertex AI AWS Edge maker Etc problems with that is obviously you are you're gonna be like you know limited to whatever is the cloud provider that you're going to be choosing from the beginning and so there's a lot of like uh Cloud login of course and then you require still the expertise on that specific Cloud vendor and it obviously will take still time and and uh the time and the resources to actually maintain all of this infrastructure In This Cloud Venture that you've chosen and then finally the the third option which actually is is now Rising a lot more is is near different competitors and players trying to give you a solution which gives you all of these infrastructure out of the box right so like going into the first question I was saying like how do you get given this model how do you get this API endpoint out of the box is that is the goal right and so this is actually one of the solutions that I'm going to be talking about is which is the product that administic we have built which really allows companies right now to actually get fast it's the fastest to go to market because everything is there all the challenges that I've mentioned are solved allows teams to actually deploy these llms immediately whatever they wanted in any cloud provider etc etc the only challenge the only problem with that is obviously you're not building the infrastructure yourself so maybe you are not getting those learnings but companies have so many other challenges that maybe they should be focusing are supposed to be focusing on the infrastructure of deployment same that we don't focus on infrastructure for payments stripe handles that maybe we should allow other companies to handle infrastructure for deployment and so this is what pipeline core comes in which is the product that we've built at mistake and and really it's um it's very much built for the data scientists with the data scientists in mind it's how can we empower the data scientists to deploy these models in reliable infrastructure immediately and so it all starts with a simple amount of decorators like for instance in this one I'm showing here is just you decorate different functions that you need to Define your machine learning pipeline the beauty of this is that it's not limited to any framework you can actually do whatever you want you can you know run the combinations of different Frameworks you can have some pre-processing code post processing code whatever python code you want to run this supports it right it's not limited to a specific file or or tensorflow for instance that we see some other softwares and so this is for for loading the mobile in this case it's very simple I'm just loading uh uh actually this one is for the Falcon 87b um then you basically eventually so there's another function that you you find actually how the inference pass should be built identifying the step-by-step of the um machine learning Pipeline and then finally you know I wanted to show you a bit of a demo while I'm running a bit of time but basically it's just a load test allowing you to suddenly as soon as you define this pipeline you upload it and you get an API endpoint to actually hit scale with this llm that you just like uploaded into your platform and out of the box you get a a dashboard to actually monitor how you weld your platform is doing and so this is one of our customers where they are getting like a P95 or 25 milliseconds so what you can think about pipeline core as this with a simple SDK we really give the companies the power of an experience that we've seen team for ML workloads uh with like Dynamic scaling cost optimization you can you're able to deploy across any Cloud on-prem instantly run streaming batch on an inference and a bunch of other amazing things and and yeah so we are definitely like onboarding companies right now we are hand-picking them so please go for a try uh book a demo I'll be in touch one of our team members will be in touch with you and we'd love to be able to show you more about what is that we build with pipeline core and help you get deployed those models that you have as soon as possible into whatever infrastructure you desire and yeah thank you so much for your time and if you need anything from my end you can reach me out on LinkedIn you can reach me out on email or feel free to reach out on us at mystic.ai or please to book a demo and I can show you a lot more about pipeline core thank you so much awesome thanks so much Oscar I'm so excited to hear all the new all the new giving us kind of a preview of what's going on thanks Lily all right so we'll send you to the chat if people have questions for Oscar