Hyperparameter Tuning with W&B Sweeps

Key Takeaways

hello again let's take care of the original task improving the model performance how can we make them all better and increase the intersection of a union metric in our previous video we refactored the Baseline notebook now we'll want to export everything to a train.buy script file there are multiple ways of doing this you can export every single cell by hand copy pasting them on the train.bi file or you can use a semi-automatic way like nbdev or nbconvert keep it simple I went the manual way I merge every single cell and copy paste them on the train.pi file at the end of the file you have the train function that depends on the config the only thing I have added it's a pause args functionality this enables you to override arguments on the Fly it converts your Python program into an interactive command line interface there are multiple tools to add this functionality I'm using default python Arc paths as we are using fast AI you can use the built-in call pass decorator the transform your script into a command line interface inside the environment to just call this file with python python trend.pi this way we'll run the file in the same way as you have presented on The Notebook this is not very interesting as it will run with default arcs and we already have a bunch of runs like that let's just cancel the Run using Ctrl C let's go to the workspace and see the new run that we logged there it is we can delete this partial run by clicking on the three dots as we said before we have an interactive Python program now so let's make use of that you can access the help menu by calling pythontrain.help and it will print you out all the parameters you are capable of overriding or with a different batch size let's try batch size 16 for a change you can override batch size by passing the argument name and the new value this will create a new run with batch size equals 16 overriding the default 8 let's confirm this on the workspace there it is let's click on the run and on the overview tab and scroll down to the config we can confirm that the batch size is 16 now let's also kill this run where we actually want to do is explore the hyper parameter space but we don't want to do it manually like that we want to define a way to orchestrate our hyper parameter optimization Here Comes Waits and biases sweeps our Apple parameter optimization tool with just a few lines of code on our already instrumented training script you will be running massive hyper parameter tuning in no time but how do we actually do this how do we tell weights and devices to run this code automatically this is done through a yaml configuration file first you define what script you want to run in our case train.pi second you have to define a method of exploration of the hyper parameter space we provide grid random unbiation optimization search you can refer to the weights and biases sweeps documentation to get more information about the algorithms then you have to Define which project your sweep will live in sometimes you want to use a different project to put your sweeps in to not pollute your main workspace in our case we'll use the same project as before then you define a metric to monitor in our case we want to maximize mean intersection of a union and finally you want to Define your hyper parameter space you can use these to overwrite default arguments like log predictions it is equivalent to changing the default values on the train.pi file you can use a distribution to sample continuous parameters in our case we'll sample the learning rate between the log of the mean and Max values you can also pass a list of values for discrete parameters we'll try a smaller batch size so we get more Optimizer updates as our data set is really small I had good results in the past using this trick with small data sets we'll also increase the image size a little bit this should improve the model performance on segmentation of small objects finally we'll try different image backbones these are my four favorite backbone from torch vision feel free to go to dodge Vision malls and try all the backbones there are plenty of them depending on your task and your data set you may want to try bigger malls torch Vision malls are trained regularly with state-of-the-art techniques our sweep configuration file is ready let's switch to a terminal now and start the sweep you can launch the sweep using 1db sweep and the sweep configuration file The Sweep has been created you can click the link and you will be redirected to the sweep workspace still empty but you can click on the overview Tab and see the configuration file that was used to create the sweep and you see the proposed sweep command to launch an agent this is the same command suggested on the terminal let's run this command before doing that let's check the options available for the 1db agent we see that we have a count parameter so we can limit the max number of runs per agent as we are doing random search if you don't pass any count parameter it will run forever so you'll have to kill it manually let's start by 50 runs running this command will launch the agent and you will start populating your sweep workspace with runs you can see the selected hyper parameters at the beginning of the script now we can switch to the workspace and we should see the incoming run that is you will see the plot updating automatically as more runs come in but I have an extra surprise for you I have switch machines I'm not using the same machine as we were using before this machine is equipped with two gpus you can check the available gpus on your machine using the Nvidia SMI command we see that the first GPU is being used and the second one is just sitting idle let's fix that let's open a new terminal we can override the Cuda visible device environment variable and force the code to run on the second GPU this command will create a new agent on the second GPU we will also pass a quota of 50 runs this is really powerful when you have access to large compute centers like a cluster or machines equipped with multiple gpus as you expect you can launch agents in parallel and greatly reduce the time spent performing The Sweep is we go to the workspace we see that two runs are coming in parallel great now we have two agents contributing in parallel to finish the sweep this should reduce in half the time needed to complete our hyper parameter exploration in the next video we'll explore the results of the finished sweep