Managing ML experiments - AWS Machine Learning in 15

Key Takeaways

foreign [Music] my name is Prana Murthy and I am a senior AIML specialist solution architect at AWS today we'll be going over how to create manage analyze compare and track machine learning experiments at scale using Amazon sagemaker experiments so let's get started before we dive into sagemaker experiments let's take a few minutes to understand what is machine learning experimentation and why is it important machine learning experimentation is a process of trying different approaches to a machine learning problem in order to find the best one this involves testing different algorithms hyper parameters and data sets to see which combination yields the best results it is an iterative process that requires keeping track of each experiment and its results Amazon sagemaker experiments is a capability of Amazon sagemaker which lets you create manage analyze and compare your machine learning experiments at scale with ease sagemaker experiment is now a part of the core sagemaker SDK which means you no longer have to install any additional packages to log and track your machine learning experimentations today sagemaker experiments allows you to log and track your experiments from anywhere you are able to log and track your experiments from sagemaker training jobs from notebooks your favorite Ides from studio notebooks from sagemaker autopilot and even sagemaker Pipelines with the experiments you will be able to track who logged an experiment and when what hyper parameters were used to train the models which definitely helps with reproducibility run a qualitative analysis of different experiments in a single View and select the best model to either integrate it with the sagemaker model registry so you can perform some actions Downstream or deploy the model to an inference endpoint to run inference today we have simplified a user's interaction with sagemaker experiments by asking our users to Define just two components at runtime an experiment and a run a run is a collection of inputs parameters configurations and results for one iteration of model training an experiment is just a collection of runs you can have multiple runs within an experiment each with its own unique combination of variables now you may be thinking how can I leverage the capabilities of sagemaker experiments to log and track my current machine learning experimentations on sagemaker the answer is easy with just a few lines of code you can leverage the capabilities of sagemaker experiments in your existing code base let's see how so in this example notebook we have defined an experiment name let's call it python support and experiment today experiments SDK is a part of sagemaker python SDK so you can import run and load run which are two main components of experiment from your native sagemaker SDK now in this example we create a run called local pytos run and we just log a couple of parameters into it we log how many training samples we have and how many test samples we have along with that we also log our training and test images and files and what run.log file will do is it creates a key value pair of your images and labels and uploads the data to an S3 path and Associates that within your sagemaker experiment run so if we do this it creates a new run and it creates key value pairs for your data set now you may ask how do I find this one in your sagemaker studio if you click on your home there is a section called experiments so if you go to your experiment you'll be able to see your python support and experiment and the run that I just created if I hit refresh this is the run that I just created and you can see parameters such as when it was created when it was modified Etc now if I open up this experiment I can go down to input artifacts which shows me my training images and labels and the S3 patch to it and if I go to my parameters you can see the parameter that I just logged earlier which is the number of test samples number of training samples and the value as well now to this experiment run I want to be able to attach my training metrics such as loss and accuracy so how do I do that let's take a quick look at the the actual model training function in this model training function I've defined a simple CNN backbone and in my model training function I have a function defined as log performance which is exclusively to log model performance metrics such as loss and accuracy and with sagemaker experiment run I can Define I can log my metrics with a run.lot metric and I can choose a name I can provide a value and I can provide the epoch value for the step and I can do this with any number of metrics I would want to now how is where is this run coming from and how is this run being defined so if I go down to my actual training function um I Define a new load run and a run decorator and in my load run I provide the experiment frame and the Run name that I just defined earlier provide a sagemaker session and provide the run to my training model function and I can use this throughout my code to log metrics and parameters that I want to I'm also going to log a few additional things such as what device it was trained on what where the hyper parameters for training all of that under a log parameter scope so here log dot run.logparameter is going to have the device that I train the model on and other key value metrics such as Optimizer type that I chose the epochs that I chose and the number of hidden channels I can also use sagemaker experiment run to log confusion Matrix at the end of my training session and I can also log final metrics such as final Precision final recall and final left one score and I should be able to view all of that in my experiment so if I go back to this I already have an experiment that I had defined so let's open that up so I had to find an experiment earlier so if I open this so and one thing you may be able to see is the metrics that that I defined during my training session are presented to me in a tabular view with Statistics Define so you can see that during my training session the test loss minimum achieved was 0.07 maximum was around 0.14 my final F1 score you know the minimum it had reach was 97.68 maximum was 97.68 so it may just be a flat training response I can see the minimum maximum value for final Precision recall obviously it's just one parameter my accuracy metric at minimum was 95.47 maximum was 97.7 so from the single tabular view I can say the model is performing pretty well just looking at the metrics now what if I want to visualize some of these metrics so I can go into charts and in my charts it presents me with this nice looking confusion Matrix now along with that I'd like to see how my training loss was performing so I see a nice decreasing loss which is good now I can switch between compact View and comfortable View to get a more detailed look of my chart going back to the experiments now let's expand the scope a little more now I want to do a basic hyper parameter search for different uh ranges of hyper parameters and I want to be able to log all of these hyper parameter all of these training sessions to my sagemaker experiment run this is a very simple parameter search where I've defined number of hidden channels the optimizers vary between atom and stochastic gradient descent and my learning rate varies between 1e minus 3 and 1e minus four and I just Loop over each of these combinations of hyper parameters and lot the runs into my experiments so if I go to my experiments and go to experiments the top view now obviously we don't you know we don't want to see all of those runs being run in the session but I had already generated some of these runs before so here's one run from that session where this run had a learning rate of one e minus 3 with an item Optimizer and number of channels defined as five so I have a whole bunch of hyper parameter runs now I can go into each of these runs inspect how my metrics we're doing analyze the chart and go into another run and do the same thing but that's not efficient and you know it doesn't give me a high level view of how these different runs are performing so one thing that sagemaker experiments also allows me to do is select multiple runs together here I've selected all my hyper parameter runs and analyzed them at the same time so here it gives me a tabular view of all of the runs and from this I can quickly look at what the final F1 scores are and determine the best performing and the worst performing models so from this one I can say uh probably with a learning rate of this app Optimizer and channel size I've gotten the best performing model at 98 F1 score my worst performing model for some reason was a stochastic gradient descent and with a much and with a much lower earning learning rate it only performed at 16. now I will if I'm a more visual person so I'd like to see this as a chart so I can go in select a bar chart and I would like to visualize the final reference course for all of my runs now I can set a comfortable view so I get a more expanded View and I can look at the metrics right here so the best performing model is a 98.63 and I can quickly look at the Run name and determine what the hyper parameters were or I can open the run and check the parameters to see what hyper parameters were defined at runtime and I can also analyze the loss values as well so I can either select step wall time or relative time but in this example I'm going to select steps and I'm going to select the training glass and see how the training loss has been performing now the last question you might ask is okay so you may not be running sagemaker experiments in local mode maybe you're going to kick off a training job how does that work in a the short answer is in a very similar way when you create a new sagemaker training estimator you need to decorate that piece of code with the Run decorator and you define a new run name I'm calling this train job value train job with a unique date time value as apply my hyper parameters and I call fit so this is going to kick off a new training job for me on MLP to extra large instance that I've chosen and if I go back to experiments it is going to log all those values for me here in my training job so my training job right here I do have one defined already so I can open this up it is logging those metrics in real time for me so I can generate the charts just like I did before I can look at how at step my let's say training loss has been performing sagemaker experiments also integrates with clarify which allows you to generate bias report using a clarified clarify job which is also viewable in Siege maker experiments as well as get model explainability metrics through get a holistic view of how your model is learning and how it's performing with that I'd like to conclude this presentation thank you [Music] thank you [Music]