New course with Predibase: Efficiently Serving LLMs

Key Takeaways

I'm excited to introduce this new short course efficiently serving large language models built in partnership with prabas and Tau by Travis there this course presents a stepbystep deep technical dive into how the text generation process of Transformer networks is implemented this turns out to have a significant effect on the time to First token that is after you've input a prompt how long the user has to wait to get the first token of output as well is the overall throughput of an LM Travis who's an expert on AI serving infrastructure will show how LM inference is actually made efficient and cover key technical details like KB caching which caches computations from early steps to speed up the generation of later tokens and you learn about precisely how the computational steps are carried out whether you're serving your own fine tune LM or one that someone else has pre-trained this course will give you a much deeper understanding of what's actually happening under the hood on delighted in news the instructor Travis there co-founder and CTO at prabas Travis leads an elite engineering team developing prab base's platform for training and serving LMS prior to that Travis led the team that built Uber's Michelangelo's platforms deep learning capabilities and also led the development of popular open- Source machine learning Frameworks horovod and LX thanks Andrew I'm really excited to be here in this course you'll learn how large language models generate text one token at a time and how techniques like KV caching continuous batching and quantization can be implemented to speed things up and optimize memory uses as you serve multiple users at once and you'll learn how to implement your own llm inference server in pytorch by implementing these state-of-the-art algorithms from scratch and measuring their performance this sounds great and in addition to serving a single pre-trained model you also learn techniques like low rank adaptation or Laura which can be efficiently employed to serve hundreds of different custom fine Tu models on a single device without sacrificing throughput that's right Andrew it turns out there's a lot more to it than downloading a model from hugging face and putting a web server in front with the knowledge from this course in hand you'll better understand the trade-offs that you have to make as you think about the performance of your application and you'll be better positioned to evaluate what a potential vendor is offering you and whether their promises are realistic this will help you make the best decisions for your project and for your company learning the technical details of how OMS are Serv will help you become a better developer I hope you enjoy the [Music] course