Model Optimization in Microsoft Foundry: Deployment and Evaluations

Key Takeaways

On to my favorite part, production. Before we take our model to production, the things we have to do, right? We have to do evaluation to make sure the model does what it's supposed to do. And we already saw the statistics in terms of during finetuning, we're able to reduce the loss and also increase the accuracy of our model. But then now, how do you evaluate the model is doing as it's supposed to do? I'll show you two demos on that. And then we also have to think about deployment. How do you deploy this models? What are the different deployments options? And you want to deploy in production. How much will it cost? So the last thing will be cost management. What should you look out for when you're fine tuning Azure OpenI models in Microsoft Foundry in terms of cost? So let's get started with evaluation. First things first is in terms of evaluations and one of the best places you'll want to explore evaluations is in the fine tuning process itself. So we have the details the monitor and the logs and we check out your logs. You'll realize that it has the model evaluation passed. That means there's evaluation being done during the finetuning process to just make sure it's safe. It's not violating any policy. Right? And then the next thing will be for the agentic tool calling we have a custom grader for us. So this is a custom grader that tries and create a function a python function that uses precision and recall to calculate the f1 score of our model. So for example if um how many pass exact matches were there how many had only name only matches and then we try and see what the precision is what the recall is and calculate the F1 score and with that we round it off and we give our model a score right so for this we will have that grader by the back of your mind so we'll come in and evaluate the GP41 model the 41 mini and the one that we've already fine- tuned. For the evaluation, we'll be using the Python grader that I just already showcased. Then we'll just use our retails tools JSON file to give it the definition of the different tools that we have. And once you've already configured all that, we'll have our faint tuning job running in the cloud. You'll see the evaluation will take a bit of time. So it will take about for for this it took about 44 minutes 11 seconds for the evaluations to be done but it now depends with um the different metrics you've set and also how the grader works that will now determine the time it takes for completion. Once that's done you will be able to see our evaluation output the metrics. So we can also see the same in foundry. So I'll just select the evaluation that we already done and you'll be able to see what the rates are for the difference ones. So you've seen we just use a small data set that's only 100 values and you see the increase for tool calling is about 6% from 59 to 65 which is not bad considering GPT4 is at 74%. So you can see that improvement that we've been able to make. So at the back of your head, you might be wondering how much that does all this cost. Head over to the Microsoft Azure pricing page for OpenAI and you can search. This is the easts region. So you can search depending on the region you're in. So let's say Sweden central, you'll find the fine tuning models. So for reinforcement fine tuning, this is the cost. So it cost probably uh if you're training per hour this will be the cost and then the inputs and the outputs tokens that also counts into the cost right and then also you have to think about hosting. So how much does it cost to host your model? And this is the other thing you also have to think about is how you doing your training, right? You can do your training in a very specific region or globally depending on whichever region you're in. So it just goes to where there is compute or you can use the developer training where this is where you wait for compute to be available for your training to take place. Right? So the other thing you need to do is uh remember for different models this is O4 mini which has reinforcement when tuning for this other different models supervised fine tuning. So for supervised fine tuning, it only takes the input and the output tokens and then the trading uh depending on now where you're located. You can see the it decreases depending on like the size of the model like GPD4 is cheaper and 41 nano is a lot cheaper as well. And then you also still have GPD 4 and 4 mini available for the time being. And yeah, that's basically how you can be able to figure out your cost for training. So, for example, if you're training on developer, you will only it will only cost you $12 per million tokens. But then the other thing you should know is when you're deploying, you can see in global and regional, all of them have hosting charges. But when you're deploying using the developer TM, it does not cost you anything. But then you can only deploy it for only 24 hours. So let's go back and see how you can deploy using developer t. Once you've looked at the options, we'll go back to our model and click on deploy. For this, we'll deploy using the developer tier because we're just experimenting. [snorts] And then we can also try and see how many tokens we want. So depending on now the amount of work it does, you can click on deploy. Once you found the right token limit for you, right, it will take a while for deployment to take place. And you can see it's already creating. Once it's done, you'll be able to use a model.