Post-Training and Deploying Open Source Reasoning Models in Foundry | BRK232

Key Takeaways

All right, thank you all so much for joining this afternoon. I'm going to speak loudly because I know the room is a little loud, but uh really appreciate you joining our session and hopefully you're going to walk away here an expert in post-training open source models and deploying them in agents to scale your AI across your company. As you all know, agents are extremely impactful. They're changing the way we all work. And I've had the pleasure of working with many of you, many customers both outside of Microsoft enterprises and startups alike, as well as teams inside Microsoft, shipping some of the agents that you might have used in your daily work. And one of the things that we are consistently learning is that deploying a model into production is just the beginning. We all just like every software application we deploy need to focus on continuously learning and improving them to make sure that they perform at scale on all the different edge cases that we might not have thought about. And so what we're going to do today is show you exactly how to do that to learn and improve to drive down your costs and improve your quality to get faster, better, cheaper agents using Foundry. I'm really excited to walk you through this because one of the other things that we've been learning is that agents are extremely token hungry. As we offload all of our cognitive processing, our thinking, our work to the cloud to agents, we're seeing a substantial increase in the token consumption as compared to regular chat bots. And this makes sense because agents are doing multiple iterative tool calls. They're rational rationalizing over the data that they fetch from our enterprise systems, our our databases and validating our policies. And they need to think about what to do with this information. And in doing so, the costs increase. So as you scale your agents, you're scaling up your bill as well. So one of the things that we're going to show you is that how to substantially reduce that cost by learning from your production agents using the power of reinforcement learning to make sure that your costs remain flat while your co quality m is also flat or in even increases over time. Now to do that one of the powerful techniques that you can employ is post-training or fine-tuning. People use these terms interchangeably and fine-tuning is extremely important because it teaches your models to think and act the way that you want them to. Not only can you teach them your business domain, you can teach them how to use your tools to interact with your systems that are required to take action on your behalf. And in doing so, you can eliminate potentially erroneous interactions, which are costly. Now, we've all been leveraging the power of the latest frontier models, and they're extremely intelligent. They can learn how to use your tools and usually perform pretty darn well. But while they're rationalizing all of the different potential tools that you might have given to your agents, their thinking and their course correcting is costing you money. And so by fine-tuning your models, you can ensure that they are choosing the best tools for the job in the right order to achieve the best outcome at the lowest price. And one of the ways that you can do that by using Foundry is to facilitate not only selecting the best models but then also uh ensuring that you have the right tools in your agent and you bring in the right eval. And this is critically important. And we're gonna explain how all of these come together by using Foundry to facilitate getting faster, better, and cheaper fine using using fine-tuning and deploying the models and agents to production that continuously learn. So the first step in this whole process is figuring out which models are appropriate for your workload. And that's getting increasingly complicated because there are new models coming out all the time. The good thing is Foundry has all of the models you could possibly want to choose from and maybe even more and we're adding to them continuously. The key to making sure that you've got the right model for your agent is the scientific method which is guess and check. Now we're going to show you how to do that checking process which is the most critical step and that's in scientific terms here is called evaluation. So by evaluating the the models you can ensure that you have the best one for the job. Now let me show you precisely how we can go through and select different models and then we'll show you how to evaluate them and then improve them over time. So here I have Microsoft Foundry and I'm in the context of an agent for a customer service uh retail scenario and a customer wants to be able to return a particular product that they purchased recently and in doing so we need to not only sure ensure that uh we uh interact with the agent or the customer in a friendly way. We need to make sure that all of our policies are honored. maybe do order lookups, make sure that they actually got their product, uh that that sort of thing. And so we can do that with a Frontier model like uh the GPT5 series and it's extremely good at figuring all these things out. We can also do that same thing with a smaller model like the Quen 314B model, which is an open- source model, and it's also capable of thinking and figuring things out. It may not necessarily be as powerful as the GPT5 model, but we can see that it can still kind of reason over what it should do. Now, the question is, how do we make sure that these models consistently do what they're supposed to do? If we look at how these actually work, we can dig in and Microsoft Foundry collects not only all of the telemetry that is required to continuously monitor how well things perform in production, but we can drill in and we can see the details of any given particular trace. So we can see that for the particular request to the agent, we can see it kind of went back and forth multiple times. It executed some of the tools that I gave it which are needed to understand the context of the customer the order and what's appropriate and then it rised over that data in it did a loop a couple of times and we can see for each one of these chat scenarios each one of these tool calls the costs are increasing and so the smaller of a model that we can run with to achieve the scenario achieve the quality bar that you're looking for, the higher the probability we can do so in a costefficient manner. Now, what I'm can show you as well is how to evaluate the quality because this is the key thing. You should only choose the models that are going to hit your quality bar. You would not deploy any sort of application to production without ensuring that's true. Now in Foundry we have this collection of evaluators that are built in as well as custom evaluations. And so what are eval exactly? Let me explain. So evaluators are the product spec. As a product manager, this is extremely important to me and it's incredibly important to everyone to make sure that you define your success criteria before you deploy anything to production. It doesn't mean just hitting your quality bar, but making sure that your policies are honored, making sure that the right tools are called and more. This is a multi-dimensional optimization technique. You can use the built-in evaluators or create custom evaluators in Foundry to facilitate complex analysis of how all of these things are interacting. But first, in order to do that, you have to define what does good mean to you. And that is what an eval is. So I want you to think about this a little bit differently than teams have in the past. In the past, people have thought about this as when I deployed a production that I'm going to monitor whether things are going well. I want you to shift that on its head. Define what going well looks like to you. And that enables you to continuously hill climb to get to a high quality output. And we're going to show you today why that is extremely important. Benchmarks for any model, whether they're provided by a frontier lab, an open-source initiative, doesn't matter. Benchmarks are only information that someone else has decided they should publish. It is a good starting point to learn from. But you have to test the models on your own scenarios, on your own data to make sure that then you can hit your success criteria when shipping to production. And that is what evaluators enable. We have a bunch of them in Foundry that are built in as a good place to start. But your custom evaluators are what is going to help you ensure that you can continuously hill climb in a way that matters to you. And so the way that we do that in Foundry, you can see we've got several of these custom eval jobs that I've run. And I'll open one up. And we can see that my definition of quality, these different evaluators. I've interactively tested these against multiple different agents that I've deployed in production. And I've created a unified metric, the retail quality that we want to optimize for. And we can see that my first set of agents that I've deployed and tested the versions weren't really that great. the model quality was good like that on its own can be used for many things but in the context of this agent and in the context of the way I've defined quality with my eval I can actually tell that it's not doing that great and so the question is how to improve it and that is the whole point of the talk today is we're going to go through how exactly to optimize these agents based on your custom evaluators using the power of post training on these open source models. So what we're going to show you is to how to we started with in this particular case a particular frontier model that was pretty good out of the box. Didn't have to do any work at 76% quality. Not bad, not great. We want to get to 90% or better in order to ship to production. And the question is how? We started with choosing a very small, very costefficient, 10x lower cost than the Frontier model, but the quality was a little bit worse. That's going the wrong direction. Then we'll show you how to use the power of both supervised fine-tuning and reinforcement learning based fine-tuning on that open- source model to eventually get to the quality bar that we're looking for. And to do that, I'm gonna ask VJ Asky to come up to the stage and he'll walk you through how to use Foundry to achieve this scenario. >> Okay. Uh thank you everybody for coming. So uh we'll learn couple important uh keywords today. Um postraining this is a word that's uh popularized by labs uh like OpenAI, Anthropic, etc. and you don't get to hear more often. So usually you talk about uh fine-tuning and reinforcement fine finetuning or RHF postraining is an umbrella term that encompass all of those things. But let's talk about post training in the context of how a model is trained. So anytime a model is trained take any model that you have like uh open AI model, open open source models, MAI model, anthropic models, they go through three distinct stages of how the model is getting ready. uh one is called pre-training, mid-training and post- trainining. There are three different cycles independently varying from each other. It's not a continuous thing. Uh you can start from anywhere and continue with anywhere and what you do in each of the stages matters and how expensive they are matters. Right? So there are three things that you vary when you're training a model. Uh data and you have your uh compute and you have your model architecture or parameters. So if you look at what's happening in the pre-training is orders magnitude more data is orders magnitude more compute and the parameter is pretty like the model's parameter can be pretty huge and all of the weights are being changed. That's where the model is learning about the word and it knows how to be intelligent so to speak. The mid-training is where you infuse specialized knowledge. Uh the specialized knowledge could be anything from language to coding or anything of that matter. That's what you do. Post-raining is where the model gets good at doing the work that you need it to do in terms of agentic workload for example. So when you're trying to get your agents to work, the agents need to know how to kind of let's say get everything right. So and most of the models are inherently good and they're trained in the post- training phase. And today we're going to talk about uh post- training as a specialized topic. So in post- training what's happening? So just like how the models are being trained in the super uh the labs um you can do post training by yourself. What does it mean to be post training? So you see on the left side you have uh various different ways where you distill your data. Your data can be distilled from let's say agent that we have uh in production the traces which is what we're going to talk about or they could be uh human preferences or we call uh sometimes uh sometimes RHF uh we have synthetic data that you could be picking data from. So you could get the data from any different ways. And then on the right side you see what do you want to get the model to do? You want the model to be let's say safe safer than what it is uh as a base model or you want it to be good at a particular task be it coding or your agentic workflow. Even though the base models keep getting better and better they don't know the rules that you are engaging the model for. Let's say if you want to have a support ticket uh that you want somebody to handle or you have any kind of workflow in a manufacturing unit or a healthcare unit or anything for that matter you the model trainers or the base model does not know your specific uh sequence. So the way to get that right is kind of do the same techniques that the labs do in your setup. But when you're doing that you have you can have two objectives. One is you want to get the model to be like accurate and then task complete better for you guys and also cheaper. So when you take a large model they're pretty good at doing what they're doing but because they're trying to kind of discover the right thing they can be pretty expensive too and the model size parameter size is very uh huge. They need like lot more compute to kind of serve. So they can be very expensive. So as you kind of get into CLI or any of these things when you start coding and when you start doing tasks you realize that your token cost can go up pretty fast and you heard a lot of examples in different uh uh these days where somebody unwittingly spend like a million dollar or some cases $5 million without knowing that they're spending that much because the token cost is that high and you want to basically reduce and posting is a way to do that. So what we're going to talk about here so posting has multiple recipes and the recipes so let's let's go back to the previous slide. So, so the post- training um here you can combine the data in any of these data and try to optimize for any of the skills on the left side on the right side and what you're doing there is you're basically doing one at a time and when you're doing that you're picking a recipe so today we're going to talk about one recipe that includes fine distillation from the traces finetuning and RF so fine-tuning this is something that most of the people are familiar with so what's happening in finetuning is you basically bring bring in an expert and say, "Hey, here is a pattern that I want it to mimic." Uh, so basically when you bring in a customer response agent, you want to say, "Hey, you want to have the tone to be this way. You want to have the format to be this way." You're kind of basically helping uh the agent know how to kind of mimic and the mimic can be the expert here can be an expert, a human or the data which basically is formulated by an expert. Right? That's an example that we're choosing here in the box one which is demonstration. And what you see happening in the box three in the tokens, you see the ping grading that's happening there. It's basically grading the loss. It's trying to figure out how close to the actual expert answer the model answer is. And it's grading it. It's it's a loss function which says hey if it's basically exactly the same or it's like partially same and it rewards both of them, right? Um and then you basically have uh the model weights get shifted based on what's the right behavior. That's SFT. So SFT is basically telling you like mimic or like kind of what to do right. So in terms of u RFT this is a order magnitude more complex process and but it has the biggest bang for the buck uh because these tasks are for verifiable task all the coding that you see the when the agents are doing good at coding or math for example anything that is like uh something that you can actually go and verify and prove RF is a way to teach it. So let's say in in the bottom here you're showing let's say to get a refund going the model needs to go get lot of sequences right lot of policies right and then make sure that each one of them is hitting in the right order and the way to do that is by basically having uh in the reinforcement fine-tuning each of the steps be rewarded so that the model learns not just to mimic but to also navigate paths that helps it get to the right answer intuitively. So that is what reinforcement finetuning is. So we're going to combine both reinforcement finetuning and uh finetuning with a distillation to get us to where we need to. Uh so It's got locked on this one. >> Yeah, I got it. >> Yeah. Okay. Okay. So, I have kicked off some jobs here. Um, so before that, let's let's go back to uh uh agents. So Chris talked about the agents. So let's say we have multiple agents. So we see a agent GPD52. So this is a powerful model and the agent is using this model. Now you're more likely that GPD52 is probably giving the best performance or now 55 if you were to use and you want to learn from how the best models do. So you take you go and then figure out the production traces and then production traces basically uh allows you to kind of uh create a data set from that. Let's say if you want to do reinforcement finetuning or fine-tuning uh you basically say hey I want to uh distill expensive large model and then I want to potentially kick off uh fine-tuning itself I can start a fine tuning job right here. So I've done something similar um where I've created um these uh data sets previously. So you could create so that's that's a continuous loop that we're talking about closing the loop. You put the agent uh in production, see what the agent does, take the traces and then kind of start fine-tuning those things. So now if you um so how do you basically start your job? So we we'll show you two two things today. One is a custom jobs the other one is a API that we are shipping low-level API before that if you want to do a custom jobs what we you want to get yourself a box in the cloud what we call workbench or compute instance where you come in the machine is set up for you the workbench is set up for you so that you don't have to worry about pip installing all the dependencies and everything else it's all connected so you go in and say connect and then you it opens up a VS code uh for you and then you basically start iting on your code right away and uh I already have uh something set up um here that so this is a recipe that we talked about. So here um we are trying to teach the model how to do u your task better. So we are starting with the base model and the traces from there you're getting the agent traces and then you're doing SFT and then you're doing RFT and then you're going to monitor and evaluate. So in the traces, so you take all the production traces, you filter down, distill down exactly what you want, you pick the right traces and then figure out your data set. That's what the data set creation process that I just showed is doing default, but you can do it by yourself. And then RF is basically uh each of the examples, the prompt that you have, you're rolling out, you're picking like multiple samples based on uh different uh what the model responds, pick one of them and then follow the trajectory and then reward the trajectory and you can kind of do that. Um so and then the ingredients that you have in your setting up is you for example the business APIs that you want to call um the graders that you want to choose the GPU clusters that you want to set up how many nodes that defines how fast the job runs how quickly the job runs. You can do all of those things and then you kind of u install the SDK um configure the job and then you kick off the job. So let's do that. I have done that already. But let's um I can run them in any order. Um so we have the setup is complete. Then I can kick off uh a SFT job. In the SFT job that I'm doing, you see that you have the data that comes from the data set that we just uh trained and the code you can bring the code to um so if you're an expert coder or a data scientist, you can bring the code. You set up your compute clusters. You pick the model of your choice and you set up the SFT job and the SF job generally creates a model that you feed to the uh reinforcement fine-tuning. But for the sake of demo, we're not going to wait for that to happen. So once you kick off, it'll actually show you the job link that you can kind of go navigate to. U in the RFT, uh it's something different in in reinforcement finetuning unlike SFT like we talked about the it's more sophisticated. So you can basically configure how the reward verification happens and how basically how to arrive at the best answer. So there's a concept of uh sampling, training and rollout. So what that basically means is you take a prompt that's a customer agent prompt one of the prompts that you have and then you create multiple samples, pick a sample and then have multiple trajectories in which the model responds like different orders, different uh sequence of calls, different ways it responds and then each one of them you pick and then you grade them like we saw in the slide deck. So you can set up all of those things in here. you can figure out how how many GPUs you want, how many um like what kind of a topology that you want for the clusters and then you submit your um job and then basically it'll give the link for you to kind of go navigate. So I have done that previously. So let's go back to our jobs and uh let's pick uh RFT job in here and we have a authenticated array endpoint where uh it's a ray cluster which is running u uh your uh all the GPUs and CPU nodes and then you have array as a distributed uh engine that's basically taking your Python code and distribute across the uh different nodes and you can basic basically run your array dashboard. You can see that um all the nodes are active and the resource are being used both CPU resource and GPU resources. Let's say if you go look at your uh jobs in here, you can look at the jobs and see what's going on in the job. You see for each of the nodes, each of the actors, um everything that's being run. So you can kind of go further deep and then basically see what each of the actors are doing, what methods are calling, what functions calling. So you can have complete visibility into what your job is doing. It's not just a fire and forget API. So for advanced users, data scientists and uh uh developers, you can actually be extremely meticulous about knowing what's happening at each node in each cluster. So you can actually go find that out. Um and similarly for the cluster if you want to see hey how many nodes there are they're all live. This is at a cluster job level monitoring you can do that. Now if you go back like what we talked about the more interesting stuff is the rollouts that we talked about like what's happening with the rollouts like why is the model behaving the way it's behaving or grading the way it's grading. So let's say see these are the rollouts in progress. So we pick uh let's say the the rewards is kind of increasing and some somes are rated uh rated well or graded well rather someone ons are not. If you want to see why something is not graded well if you look at let's say the prompt eight so we've chosen eight here. So in here um you will see um for this particular sample that's picked what are the tools that it called and in what order. You see clearly it kind of missed calling each of the functions in here. So if it missed obviously you're going to get lower rewards and uh it's graded lower. That's why the grade was lower. Uh if you want to basically do a sideby-side comparison with what a good run will look like. So you basically have um something here graded uh uh higher with u and this is not a good example. So you can actually see a tool comparison with what something went well like let's pick a different one which probably highlights it better roll out 16. So here you see one of them is u uh the the reason for cancellation expected out outcome is to basically look for the reason for cancellation. Um here you will see the tool calls a particular one got it right the other one uh didn't. So you you can actually see for each rollouts why a particular roll out is getting rewarded higher or lower. So you kind of understand how the model is behaving. So that's basically um the power of using let's say foundry where you get complete visibility into what what's happening there. And once you do that, um, so like Chris said, you want to eval um, so like I have a eval uh, in here which basically shows that each of the samples that you you evaluated what's the pass uh, rate or the quality of your evals that's uh, happening against each of the samples. So you kind of get so you picked a particular model that you trained and you see that this is looking like a good number for you and if you want to eventually compare uh across the different models that Chris talked about of saving cost. So here is this shows a demonstration of the fact that 5.2 the GPD 5.2 uh does better base model than you uh the quen 14B would do. uh but then as you start taking the smaller model which is like significantly cheaper and faster and then you start basically u doing post- training on it that includes distlation fine-tuning and uh RFT it you can start seeing the score go up uh subsequently right so it's it's getting better you see that the pass rate uh of the models which are post-trained are better than the ones that are not the rewards are better as well um and then here let's say if you particularly pick one particular sample here. It'll show that the score it got like the 14B initially uh didn't score as well u or it almost zero and then failed. 52 did okay. Uh it still failed the better score. So now you start doing SFT of the smaller model you start seeing this go up. So basically you get your score eventually to where you want it to be. So this post training can be done in a loop like every step can be done individually and it can be done in a loop and you monitor and you kind of get it right. So that's that's where your evol evals is. Um now so this is all of this is fantastic. This is a code first experience that we saw in the uh in foundry. So here you're provisioning a cluster you like manoj will come and talk about the cluster as well. So you're provisioning a cluster, you're managing your array, you're bringing in the code, you're doing all the infrastructure. That basically means when you see that the node is not talking to each other or is the network uh topology across the different nodes not as good, you have to debug the um data streaming uh performance, the network performance and all of those things. So it's it's it's expert heavy duty work. But if you want to do something um more easier, we are launching something called u we we we not named it yet. Um and I'll show you a notebook where this work. So the previous one you need to have GPUs. So everybody in the organization may not have GPUs but everybody has a laptop. So here um this uh API that we're doing uh is uh a lower level GPU constructs where you are uh able to just bring in your RL logic on your laptop on your CPU and kick off a job and the job is kicked off in the server and the server manage all the infra and everything else it's doing right so that's um so it it in this case it's a code word we're calling it loom it manage the GPU cluster the images the environments like everything it manages for you and uh the networking etc. And all you have to do is uh understand four different GPU constructs uh when you're doing a model uh training you most of you guys know it's a forward pass backward pass where you're optimizing for a loss function. So you have access to those core GPU primitives just like those four using and that's all you need. You don't need to worry about how the cluster and everything else works. So that's something that you can kind of outsource to uh the service and then you basically kind of like so everything on the left side runs on CPU and your client on the right side it runs on the GPU cluster and uh these are the four calls that you basically have to export and uh manage. So now so the reason why it helps for you to kind of have this control is let's say if you're trying to teach uh uh a model linear algebra or mathematics right you don't want to throw the hardest problem to at the model the fastest you want to teach the model how to do basic arithmetic you you want to show how to do addition subtraction and gradually kind of climb up the hill to kind of show linear algebra calculus and all of those things and you kind of uh do that um in in your client and uh The code sample if you were to do use this higher level API would look something like this. This is a pretty basic uh to understand and but it gives you the maximum control because you all you have so no cluster no nothing you're just bringing this code in your laptop in your notebook and then kind of variation of this and you can do both SFT and RF using this API and then you set it up. So that's a notebook and then u so the jobs when you submit those jobs using this API it'll show up in found so like the previous jobs that I showed you it shows up under train because it's a custom training uh so this the training uh harness that we have and the canvas that we have it allows you to do all pre-training mid-training and everything else very powerful but for uh the higher level API these show up in fine-tune it's like fine-tuning so even if you just submit your uh CPU uh like script It just shows up as a fine tuning both SFT and RFT and then it everything about it is like exactly the same. You can look at the logs you can see all the checkpoints that you have. It takes some time to load but every single checkpoint. So when the model is training all the checkpoints you can deploy these checkpoints they all show up here and you can deploy those checkpoints and similarly just like how we did with code first you will have a similar experience for you to dashboard and then figure out how to basically compare uh your runs across uh different things. And you see here u you use any of the higher APIs create a job the reward function goes up um you see the entropy going down um like this is like what you want to see the group composition climbing up so you basically can monitor even again here you can basically get all the lowest level of u visibility that you want to get for your job it's as rich a telemetry and monitoring that you can got in the code first you can get that in the API first as well uh so that's basically the um uh demo and then u so if you look at now the surface area of foundry uh so what we are so what we are showing you here is the fact that we had the higher level API today in foundry you have it today uh where you take a job fire and forget we are introducing two new capabilities which is code first and the lower level APIs they are coming in we have a link at the end of it for private preview you guys can sign up so anybody who wants to basically come and train using any of these methods. You can come and train. Now Chris will take over from here and then talk about uh desktops >> BJ. So we talked a lot about training the models, improving the models. Now the question is how do you go from a model that you trained to be able to incorporate it in an agent? And the answer is you need to deploy it so that it can run as an API that gets integrated with your agent. And one of the key things that we're proud to announce uh at build today is our foundry manage compute. What this enables is deploying uh models both the uh serverless trained models, models that you might have trained on your laptop or in another model uh platform and bring the weights to Foundry and you can deploy them using Foundry Managed Compute. It's a serverless scalable solution that enables deploying any open-source models or custom models and we automatically optimize those. We scale them out. So you can bring your own weights regardless of where they're trained or you can bring custom models with custom containers that have highly optimized run times using things like speculative decoding or uh draft models uh to facilitate speeding up the the inference of these custom models as well. And so to show you how to do that, I'm going to ask Manoj to come on stage and walk you through that. Hey everyone, I'm Manoj. Uh so uh we saw how training is done. I'll now talk about deployments. Uh you need to deploy both base models and fine-tuned models. Deploy base models to do evals and like see where the performance is and then improve that performance and do fine-tuned model deployments. So here this is uh the model catalog. Let's take a quick look at how you deploy base models. Um, so I can probably like uh go to collections and look at the fireworks models here. And uh the fireworks models like they let me uh deploy to a provisioned throughput unit which uh which is our managed offering that gives you predictable uh performance at a guaranteed uh uh throughput SLA. So that's the uh way you deploy uh you know the models that are available through the fireworks and uh the other option is like what Chris just now introduced is the managed compute. So this is basically um yeah so this is this will enable you to deploy models to uh foundry managed GPUs. So let me uh pick up like a uh a model and uh try to show you how to deploy it. Um I'm trying to find the quen 32 billion. Here it is. when I try to deploy this model, uh I'll pick the deployment option as uh managed compute. This is what we are talking now. Uh since the since you are paying for these GPUs, uh you have couple of uh options on like how you want to uh manage the deployments and how you want to pay for it. So here there's a option where you can like run this model on two GPUs and this deploys it with like uh uh enough KV cache to maximize the context length. Uh and the different option is like you you could probably deploy it on only uh one GPU uh which is like uh you know if you see here there's just one GPU is being used and this is the smallest footprint the model can run with and that's like just the 40k uh context length. Uh I can select both of these and uh go ahead and deploy. Uh but for now uh let's switch context a little bit. You you saw how to deploy base models. Uh what about fine-tuned models? So the fine-tuned models they show up here in this list when your job runs them. The job automatically registers the model here. But if you have trained your fine-tuned model somewhere else in a different environment, you could import it into the project. And uh we have a command to do that. And when you import the model to the project, the most important thing you need to tell us is uh what is the base model associated with the model architecture that you are bringing. This helps us to identify the right containers and the right uh deployment configuration for us to deploy these models. So you specify the base model um you tell us uh the uh model name um model version and then you tell us whether it's a full weight model adapter and that generates a command and you can copy paste and run this command and that model will be imported. So now that uh I have the fine-tuned model, I can go ahead and deploy sorry I can deploy this model. Deploying the base model and the fine-tuned model is identical. No difference uh uh whatsoever. Um and I can select global managed compute and again the deployment templates and the same flow where you select the accelerators. Uh it's the same flow that you uh I showed you for the base model. So uh I can uh go ahead and uh deploy this uh model as well. So this deployment is going to take a couple of minutes. So let's uh uh let's go look at a model that I have uh already uh already deployed. Yeah. So if I come back here to deployments, uh if I come back here to deployments, you see that this got kicked off, but this is a model pre-eployed. Um I can click this model and uh try out this model in the playground. This works great. Uh I can call this model programmatically. Uh here's the code snippet. This is a chat completions API which means uh it is stateless. Um so you have to manage the state on your end. U but basically we want to use these models in agents right side what we were showing uh before. Uh and to do that uh you could uh add this model to an agent where you get like state management and all the other capabilities. Uh I have added that model uh to a agent here. So um if I if I come to this place um the same fine-tuned model that I showed you I have already added it to an uh agent and uh now uh I I can again chat with this but the benefit here is that I get like tool calling. So, uh I can say like NBA uh games, then it should be able to like pull up um real-time information and answer uh those uh those uh queries for me. So, let's look if that happens. Yeah. So, looks like uh there is a uh there's a game uh today evening. Yeah. U now you can see here that this model supports uh uh chat completion uh like the responses API which means uh it'll be able to uh track uh your responses. For example, here was a sample query I had run uh where I just said like remember what's my favorite color and now even though I don't pass the chat history I'm just passing the previous response. It is able to recolct that what my favorite color was uh and inform me. Uh so once you have deployed what becomes important is observability monitoring and cost management. So u uh the cost management with managed compute is like available out of the box. Uh all deployments can be uh you know all metrics can be tracked. So here is a chart where I I am tracking the total tokens consumed uh by the by the quen model. Uh I can u add these charts to a dashboard and get uh rich monitoring as well. And finally, when it comes to cost management, uh I I am able to track and monitor uh costs for this model. So here I have focused on the Quen model. Uh I have filtered by the Quen model and I'm able to see the costs. Back to you, Chris. >> Clearly training models without deploying them is a waste of time. So having multiple ways to deploy those models to do so in a costefficient and scalable manner is critically important. Today I hope you've learned several things. One, there are many models available for your workload. Foundry has them all. Open- source and proprietary models. You can choose them, add them to your agent, and evaluate them on your workload. So this is critically important. If it's if it's good enough and it's hitting your price for performance, fantastic. deploy it. If not, you can fine-tune it. You can iterate and improve continuously. You can leverage the new reinforcement learning capabilities, both the lower level APIs, full control code of any open- source framework like slime, ver TRL. Leverage the power of Ray for distribution and monitoring. You can browse and monitor the quality of the rollouts, the samples. You can deeply understand how well the models are performing and and deploy them in a costefficient manner and then you can operate them with control either leveraging the new serverless foundry managed compute offerings on one or many GPUs autoscaling with different context lengths and Foundry makes it easy to configure, deploy and scale to meet your goals and then most importantly we're automatically collecting all the telemetry you need to both monitor the costs to create new data sets to learn from once you've deployed in production to continuously improve the models. So you can get started tonight. We've got all the repo samples published. You can get the slides later uh from the same location that you can get from all the sessions. Thank you so much. We've got many other sessions on Foundry to learn how to deploy, manage agents and models all available here. Thank you for coming tonight. Appreciate it.