Simplifying Training and GenAI Finetuning Using Serverless GPU Compute

Key Takeaways

Awesome. Hi everyone. Uh Tis here from the data bricks product team and hi I'm Irene from the data bicks engineering team. Uh thank you all for joining us today. We're really excited to share the latest around GPU training and fine-tuning here at data bicks including a new product we're launching this week called serverless GPU compute. Okay, so just before we get started, just a standard uh safe harbor statement. We have many new features we're excited to show here today, but some of them are forward uh looking. And just a general reminder, uh please to complete your surveys if possible. The feedback is very valuable for us to help improve and add to these sessions over time. And so today we're going to cover several topics around training and fine-tuning. We'll start with giving an overview of the state of training and fine-tuning as we see it here at data bricks and learnings that we've had over the last year or two. Uh secondly, uh we're going to introduce serverless GPU compute which is a new product we're launching to make it easier to do training and fine-tuning of GPU models on data bricks. Thirdly is we're going to showcase this new product via a pretty detailed demo. We're going to cover one on fine-tuning and then one on training a recommener system using the serverless GPU compute. Then fourthly, we're going to cover roadmap and future direction, including a beta that's coming out later this week on AWS and later on to Azure as well. And then finally, we're going to conclude with some takeaways and some next steps. So, we're going to start by first recapping a few trends that the industry has been seeing as we see them here at Data Bricks. At data bricks, every customer we speak with wants to do data intelligence and maximize their organization's potential for data and AI. Core to data intelligence is to really build AI systems that can learn, understand, and reason on an organization's data. And then part of that is why we see so many customers train custom models on data bricks. The types of custom models we see are pretty diverse. You know, there's forecasting use cases. We see retailers who for example will want to forecast demand and supply for their supply chain. On the recommener system side, we see customers building recommener systems to personalize user experiences for their users. Uh NLP has seen massive growth and adoption with generative AI and we see customers using that to streamline customer support or understand customer feedback uh manage sales processes and so forth. And then with multimodality, customers are now investing in things like voice, image, audio, uh, new models in that domain. And over the last few years, we've seen customers also invest in fine-tuning to help bring agents into production. And some of the key drivers of this is one, accuracy, two, cost savings, and third, latency. On the accuracy side, fine-tuning can help make your agents reflect the natural tone and structure of your data and industry and language. And examples of this would be, you know, tailoring product descriptions to your company reflecting your own brand voice. We also see customers looking at fine-tuning from a cost savings point of view. So where they can leverage smaller models, especially for larger bash use cases and we see examples of this in document extraction and customizing image models on larger data sets. And then lastly on the latency side, so improving latency for real-time applications is another motivation to fine-tune. And we see customers power userfacing features with strong latency guarantees. using fine-tuning approaches. And then over the last few years, we've been really tracking three big shifts in deep learning and fine-tuning. One is the broader adoption of neural networks for a variety of tasks, not just unstructured data, but also structured data. Second is the Cambrian explosion of open source generative AI, which has been awesome to see. And then thirdly is rapid progress in techniques and tools around generative AI. So first across a variety of domains, customers are leveraging deep learning more and more to get state-of-the-art results. So we see this commonly in unstructured data for things like computer vision, audio uh and natural language processing, but also even for tabular tasks, things like recommendation systems where transformers are making their way into that domain as well. and things like forecasting with models like DPR and Kronos and so forth and even so like in forecasting we're actually seeing great quality gains by using things like DPR in our own serverless forecasting product autoML product and then secondly is the Cambrian explosion of open source generative AI we really feel open source gener generative AI has hit escape velocity over the last year uh so actually on the right is a chart from hugging fac's 24 year review where they actually hit the 1 million model mark hosted on their platform which is an awesome awesome result. Um and then on the on the left just we're seeing great results from models like llama mistrol and so forth and a broad set of models have really gained adoption. So this is also from the hugging face 2024 year in review. We can see the variety of models that are being used for both NLP tasks, multimodal tasks and so forth. And then lastly, trend three is new tools and techniques. So data scientists have more tools than ever to train and fine-tune models especially on GPUs. Uh one on one side tools are making it more efficient to do fine-tuning and do more parameter efficient approaches. And we're seeing amazing uh adoption of frameworks like onslaught for that. And then on the post- training side, we're also providing um there's also the industry is converging on tools to make it easier to improve the quality of these models. And so new techniques like RL or a technique that database has developed called Tao are all kind of making uh room headroom there. Um and you can just see on the right just the rapid adoption of different uh frameworks that are there. Uh this is just the GitHub star count uh for each of these repos over time. So the result of these three trends in terms of setting our road map here at data bricks is that we see the demand for GPU compute growing. Um, and all these different trends are kind of creating additional uh need for GPU compute. And so to that end, today uh and this week at Deis, we're going to be announcing serverless GPU compute as a way of making it easier to do training and fine-tuning of models. So today, dealing with GPU infrastructure is still pretty hard. uh on the cluster creation process, we've kind of gotten customer feedback that it's tough to they get trapped in quota issues, provisioning issues, slow provisioning times, and oftentimes the data scientist who's doing the work doesn't really know how to build and maintain the cluster. And so, as a result, organizations are losing developer velocity, project get projects get slowed down and ultimately deadlines are missed. The second issue we see uh is management of a cluster. So when you have a cluster uh you're constantly having to worry about things like idle time or time when your GPUs aren't being really used for the fullest and so that leads to unnecessary spend for customers. And then lastly is GPU infrastructure itself is really errorprone preventing customers from really scaling to drive the best business outcomes. So we often hear customers who feel like they have to subsample their data on a smaller data set just to get it to fit to run on a single node. And then they also have to face things like node failures and other issues and inter node communication when it comes to GPU training on multiple nodes. And so these are all pain points that we see. And so today we're excited to announce serverless GPU compute which is a scalable ondemand compute coming to data bricks that makes it easy for you to do deep learning and fine-tuning. Uh there are kind of three key aspects of this product. One is it's easy attached to serverless GPUs starting with A10s with H100s to come soon. And you can see on the right hand side just a quick gif of how this looks in your data bricks notebook. You can go into your environment tab, select the GD A10, connect to it and then you can seamlessly get started with running PyTorch code or other frameworks you'd like. The second key thing is these uh this GPUs are provided on demand. So you're only really paying for the GPUs as you use them and you don't have to worry about clusters that are kind of left running afterwards and so forth. And then lastly is these are all uh scalable and optimized for GPU workloads. So this natively supports critical performance enhancements like RDMA for that inner node communication. Yeah. And just to kind of and just to share kind of uh what's launching this week is a beta for single node A10s. It's available in AWS now and it's coming to Azure in the next one and a half weeks and it's great for things like forecasting, deep learning of smaller models or parameter efficient fine-tuning. But compute is really step one on the journey to train a model. And so some feedback we've heard from customers is that they want to go from data to training to deployment with really strong governance all in one really unified process without having to worry about working in one fragment system that doesn't really talk to another or one workflow that doesn't really connect with another. Um and the second pain we've heard is like actually managing the dependencies especially when the the frameworks are changing so fast is actually a pretty difficult problem. And so we want to make it easier for customers to come in with code and just get started quickly without having to worry about installing every single thing they need to do. And then lastly is getting distributed training and orchestrating the workflows is actually hard to get right. It involves many different steps under the hood. Uh making sure your GPUs are utilized well and communicating well and all those are things that are adding friction to the developer process. So with serverless GPUs, we're also going to be launching a new software experience built in optimized for deep learning and fine-tuning. So one is this framework is lakehouse native. So it seamlessly works with the GP the GPU seamlessly work within your data bricks unity catalog and lakehouse and it connects with things like system tables for example for cost observability. The second is a runtime that's customized for deep learning to give you both the flexibility to use different frameworks you'd like you know frameworks like uh composer hugging face ray pietorch without having to install everything kind of from scratch. And then thirdly is a simplified orchestration layer for GPU workloads. So we're actually uh as Irene will show in the demo launching a new decorator that makes it easy for you to go from smaller GPU workloads to larger GPU workloads all within the same notebook with minimal developer friction. And this will also seamlessly work with things like data bricks jobs and workflows. So you have one single platform to manage all your different training workloads. And then all this is natively built into the lakehouse. So it you know combines your governance with uh the data engineering that you do alongside with the GPU compute. So all of this kind of comes in one simple kind of streamlined workflow under hood you know using ML flow to track experiments and manage model versions and streamline the path to deployment as well. And then we're about to get into the demo soon. So really excited about that. But one last slide before then uh is that you know coming soon will be H100 support. And so Iron will be as part of this demoing both A10s and H100s and also uh we'll be talking about the launch decorator that makes it easy to go from uh a single function to uh a single GPU setup to many GPUs. And so with that, I'll turn it over to Irene to go over the demo. All right, so for this demo I'm going to go over how to train and develop models on serverless GPU compute on data bricks. And I'm going to go over a few different use cases to show you well first what it looks like but also how it integrates really nicely with the other data bricks features such as MLflow Lakehouse and model serving. So let's say I'm an NLP scientist at a manufacturing company where there's lots of industry specific jargon. I'm in charge of launching our new chatbot and in beta testing we got a list of sample questions and answers from our experts and we now want to make sure we incorporate this chat uh this data into our chatbot. We tried techniques like rag but I feel like fine-tuning can deliver better domain specific quality. So I'm going to experiment with a few different fine-tuning techniques and model sizes today and take you along for that journey. So the first thing I'm going to do is I'm going to inspect my data. So you can see here I have my data. I have a table in Unity catalog. I can see an overview of my data uh different columns, their types and I can also go to sample data to get a preview of what my data looks like so I can understand it better. So once I've understood my data now I can get to developing and training my model. So I'm going to do that in a notebook. But of course the first thing I'm going to need is GPU compute. I have a deadline to meet and I don't want to go through the headache of, you know, having to talk to a cloud provider, having to get quota, and then once I get those resources, having to manage and make sure they're set up correctly. Instead, I can simply go into my notebook, I can go into the right hand side here into the environments panel where I can select what accelerator I want. So here, for this demo, I'm just going to be using an A10 and I can just click apply. And this will start to connect to GPU compute. So what's happening under the hood is we are grabbing that compute and we're also setting up our notebook ripple so that we can attach to it. Uh we're also setting up integrations with things like workspace so you can have access to all of your workspace files seamlessly from your serverless GPU compute notebook. Uh also integrations with things like unique catalog so I can access things like that table I showed earlier. So great. Yeah, we're connected to GPU compute and I can start training and developing my model just like that. It's seamless, it's fast, I don't have to go through the headache of, you know, all the things I said earlier going through a cloud provider and I can focus on the problems that I really care about solving. So now we have our compute. So let's get into actually uh fine-tuning and our models. So we're going to start with a smaller model for the A10s. So we're going to start with Llama 3.2. 23B and we're going to use Unsloth. Unsloth is a popular library for efficient fine-tuning. And the first thing we're going to do here is we're going to start by loading our model and tokenizer from Unsloth. We're going to create a pft model and then we're going to process our data. So you can see here we also as I mentioned earlier we have integration with uh our table. So I can just view my table here directly and run SQL commands. Um, I'm going to process my data here. Uh, I'm going to format it into the proper chat template according to my tokenizer. And then after I'm done doing this data processing, I can start training. And so I'm going to create this SFT trainer from hugging face and I'm going to I have all my different uh parameters here. And right before I start training, what I'm going to do is I'm going to start an MLFlow run. And to this MLflow run, I can log all the metrics I care about um and monitor my run as it goes. So here I started my ML flow run and now I start training. So you can see in the cell output I also get the different you know the training loss as it's going. But I can go to this view in MLflow and I can view the model metrics and I can see these different metrics like learning rate of course loss and I can also go back to the overview panel where I can see which notebook created this run and I can also see different parameters that I've logged as well. Um, for example, what model I'm training on um, and an overview of my metrics too. Cool. So once I'm satisfied with the uh with my training, I can log my model with MLflow for easy serving on data bricks. So here I merge my model weights uh my pft adapters and here I log my model and I also get a nice suggestion here on how I can use agent evaluation to further improve my model. So once my model's logged, I can see it in my MLflow run. So I can see it here under registered models and I can also see it in Unity catalog right over here. So I can see all the past versions of the models for previous fine-tuning runs I've done. So I can kind of track the progress and different training versions. And I can also see that it's governed in the same way that my data is governed. So this helps enforce the right permissions in our org and help create uh enforce the right trust. So once my model is logged here, I can serve this model with this button and that will take me to this page where I can just create an endpoint. Uh I just need to name my endpoint and I can set different settings here and with a button I can create my endpoint. So without any configuration or extra infrastructure I can go from the model that I was training to a production ready endpoint and once this endpoint is ready I can run evals on it and I can also chat with it on data bricks playground. So if you see here I can just uh this is playground and I can submit I can start talking to my model. So you can see here this we kind of taken you from inspecting our data to uh connecting to compute with serverless GPU compute training our model and to a uh production ready serving endpoint. So next let's say we want to start scaling. I want bigger models. I want more quality. So what I'm going to do is I'm going to start full fine-tuning on llama 8b and I want to scale up to 70B eventually as well. So, so I'm going to do this uh I'm going to show this uh fine-tuning example today using LLM Foundry. And when I do these larger workloads of full fine-tuning on these larger models, I'm going to need more powerful compute. The A10 that I I showed you earlier is not going to be enough. we're gonna need to actually scale up to using H100 nodes and even multiple H100 nodes. So here I have my Foundry LM Foundry YAML. I'm going to be training on llama 8B and I have my configuration here and I have my training code here where I'm just passing this config to our foundry train function. And but how do I actually access the compute that I said we needed? we need H100s. So what we're going to do is we're going to import from this serverless GPU library. We're going to import this distributed annotation and we're simply going to annotate our training function with this annotation. So I have at distributed and we need to specify the number of GPUs, the type of GPU, uh H100, and I'm going to say remote equals true because we want to scale beyond our notebook resources since on this notebook we only have an A10. And so just like that with this annotation once I run my function this will provision those resources for me. So eight H100 GPUs and it's going to launch this function in a distributed manner across all eight GPUs. So you can see here here's what the output looks like. Um you get this status bar that will show the status of your workload as it's going. You'll also have access to the full logs. So what does that mean? The full logs will show you each of the logs from each of the GPUs. So per GPU rank for easy debugging. So you can see what each GPU is doing. And lastly, we also get the rank zero standard out logs streamed back to us in the cell notebook. So we can also monitor our training there. So this was 8B. Now, how do we get to 70B? So, what I'm going to do here, I'm just going to copy this code. I'm going to create a new code block and I'm going to make two changes. The first change I'm going to make, of course, I need to change my config to uh to llama 70db. Well, this to 70B. And then the second thing I'm going to do is I'm going to uh change the number of GPUs to 32. So with such a large model with 70dB, we're going to need 32 GPUs. So I'm going to change this to 32. So two changes, number of GPUs from 8 to 32, model from 8b to 70B. And then now I can start training. So you can see here it's grabbing the job status and we can see here we have the live status of our workload streamed back to us as it goes. And of course we have our full logs and this would show you the logs for each of the 32 GPUs. And yeah, so we can see this is what the output would look like for a completed run once it starts going. And we also have integration uh similar to before with MLflow. So you can see here here's the MLflow view for the AB run. You can see the different model metrics that we might care about loss uh parameters and of course also a registered model here for easy serving. So this is distributed training made easy. You can see how we easily scaled from just using eight GPUs for an 8B model. Uh with just a couple of changes, we can scale up to a much larger model and provision much larger resources. So I didn't have to go, you know, make sure that get a cluster and make sure that I have my all the right nodes and make sure that everything's set up including RDMA. Everything just works out of the box. And lastly, a lot of our customers really love Ray. So we provide a nice integration with Ray as well. Uh so you can see here similar to before we have an annotation. Uh we have this ray launch annotation where we can annotate a function that just has ray code. And with this annotation we specify the number of GPUs. In this case we'll do 16. we have the type H100 and remote equals true because we want to grab remote resources outside of our notebook. So in this case I just have I just put any array code in here. Um so I can easily port over any projects any Ray projects that I previously had and this will create array this will provision those resources and create a ray cluster on those resources and launch array code there. So you can see here I'm also printing out the available array resources that we have and you can see here in the output we indeed have uh we expect to see 16 GPUs and 16 H100s in particular. Cool. So now you see how we can uh go from llama 3B to uh llama 8b and 70B. So we can train and fine-tune um a chatbot model and try out different techniques and different model sizes as well. And we can also have integration with Ray. So if you come with a Ray project in mind, you can also run it very easily using our uh our distributed our uh serverless GPU library. All right. So for this last demo, I just wanted to show another uh example that's kind of outside the genai realm. So show this two tower demo. This is a two tower model. It's commonly used for recommener systems. And here for this example, we're going to be using torch recre um pietorrch and composer. So here again we have very nice integration with unique catalog. So I can just access my tables directly and do any data processing I want. Uh I have my different imports, specify my training hyperparameters and in this example we also define our full model definition here and we provide any all our integrations into composer. We create our model and optimizer. We add any MLflow logging callbacks that we need with composer. We prepare our data from our delta table and finally once we're done preparing our data we can start training. So similar to before we have we are logging to MLflow. So right in the same workspace we have our MLflow run. Uh we can see the different model metrics that we might care about. We see our loss here and we also have our logged model here that we can use for serving and we also have our parameters here. So we've shown you how we can use GPU serverless compute on a variety of use cases from training a chatbot on different model sizes llama 3B with unsloth pes fine-tuning to full fine-tuning on 8B and uh 70B and scaling up to 32 H100s. We also showed that we can use we have integration with Ray and we also showed that you can use other do other use cases as well including recommener systems. So with that, I'll uh bring things back to TIS. Cool. Awesome. Thanks, Irene. Awesome demo. Great. And just to recap kind of the demo Irene showed, we showed three things. Uh one is connecting to serverless GPU compute, training model, and fine-tuning models on that compute and then evaluating and deploying those models all in one seamless experience. And we're really grateful for the support of early adopters like Faxet and Rivian who are using this to help power deep learning and fine-tune models. And now we're going to cover the road map and future direction briefly. Uh so one uh thing we showed in the demo is the interactive experience. So using notebooks and connecting to other serverless GP GPU compute but the same compute is coming to jobs as well. So if you have workflows or uh retraining pipelines or other types of pipelines you're looking to run on serverless GPUs, you'll be able to do that really soon. And then the other thing Irene touched upon was bringing ray code. Uh so if you have an existing ray project, it should be seamless for you to bring it into serverless GPU compute and use it uh as needed. And so that I'm going to conclude briefly with some just brief takeaways. So one customers are delivering data intelligence today using GPU training and rapid developments in deep learning and open source genai has created new tools and new methods to actually get better quality and better results for the business. Second is serverless GPU compute is a new product that we're launching here at data bricks this week to simplify that GPU training process. The goal is to save you time and effort from having to manage your own infrastructure, maintain clusters and get to results faster. Thirdly is you can use serverless GPU compute directly in your lakehouse to streamline training and deployment of models and you can still leverage MLflow Unity catalog for governance and the broader data bricks platform to keep everything in one seamless flow. And then lastly is you're able to get started today uh uh you A10 GPUs are going to be available in AWS this week coming to Azure really soon and then H100s are also coming really soon. And with that I'm going to conclude with just a brief QR code here. This is going to link to documentation on the data bricks website that just launched and you should be able to then learn more about uh the feature and would love for you all to start trying it out. And with that, I just want to thank everybody for their time here. Really really appreciate you all joining us here today. And if I Irene and I will stick around here at the podium for additional questions as needed too. [Applause]