🧹 Tune Hyperparameters Easily with W&B Sweeps

Key Takeaways

i am your host charles fry we'll be talking about how you can use the weights and biases platform to organize hyper parameter sweeps in the pi torch library so weights of biases if you look at the screen that i'm sharing you'll see a list of modular tools we've got experiment tracking and data set versioning we've got collaboration and reproducibility and then you we've got what we're going to be talking about today we've got our tool for model optimization which is sweet so the web page that i'm sharing here is our github repo for our examples this repository has just a whole ton of example scripts example notebooks and example colabs that demonstrate how to use our tools so to run this demo this morning i'm actually just going to go in this page here of all of our examples we've got examples for specific frameworks specific tools i'm going to go to this one for optimizing hyper parameters in sweeps and open it up so you can do the same thing there i guess first let's clarify what i mean by hyper parameter sweeps in general when we have a machine learning model especially deep learning models there are a lot of knobs to twiddle that we aren't going to optimize programmatically we are going to use gradient descent to determine the values of those knobs the things that we do use gradient descents to determine the values of would be the literal weights and biases say of our neural network model things we don't tend to use gradient descent to set are things like our learning rate or the amount of weight decay the amount of dropouts some of these are just prohibitively expensive to train others are fundamentally discreet and so not something that we can train with gradient-based methods the alternative to setting them programmatically is well one is to set them directly ourselves so to just say oh a good value for all of these parameters i guess you can see a list of parameters on the bottom right side of the screen like num layers batch size learning rate i just know good values these parameters i can set them but when you're working on a new problem it's often unclear what the right values for those parameters might be and so the alternative to setting them programmatically or knowing what they should be a priori is to figure it out yourself and the only way to figure it out that people have found is to try a bunch of them and to sweep through a bunch of the values so that is what a hyper parameter sleep is each of the dots on this chart right here represents a single run on the y-axis is how well that run did higher being worse and then over time we try a whole bunch of different values of the parameters and we can see that the best value that we've seen is getting better over time so what i'm going to show you how to do today is how to set up and execute a hyper parameter sweep in weights and pisces and log it to our web platform and end up with super cool really useful charts like these that not only are these just the static charts these images here are already pretty useful relative to what i at least can put together with matpot women in just a few minutes but they're also interactive and do things that are extremely difficult to do in raw python and we'll do all this for you for academic and personal projects the product is free and subject only to reasonable limits like no uploading 10 terabytes of data that actually represents say the lord of the rings trilogy in 4k that you can't get away with all right so to jump in how does a hyper parameter sweep weights and biases work there's basically three steps define the sweep what's the metric that we're trying to get down what's the program that we're going to execute in order to run our neural network training or our machine learning training what are all these parameters that we're using so that definition step that's the sort of biggest one and then the steps two and three are then just a single line of code we initialize the sweep we say hey i'm ready to run a sweep that has this definition and then we start running agents we say all right now let's start doing this sweep all right i want this computer to run a sweep i want this other computer to also join this same sweep and then you're ready to go and you just need to sit and wait for those cpus and gpus to turn those floats and get your final values we will begin at the beginning this is starting from essentially scratch you'll need to once you've got your python environment your deep learning environment set up on your computer you got to get the wnb library installed it looks like i already installed this you might see a little pip install going on here in your version of this cell uh and then at the bottom there's wandy.login we know who you are and so that we can organize your projects and make sure no one else logs things to your projects so if you don't have a way to buy these accounts you'll get a link to create an account there and so just give us your email password and under a minute later you'll be logged in and ready to go so moving on let's talk the three steps for defining the sweep fundamentally a sweep is a combination of a strategy for trying out a bunch of hyper parameter values and code that evaluates them those are the sort of two pieces that make up a sweep and that strategy could be as simple as trying every option or it could be something really complicated and optimal like combining bayesian approaches with early stopping whatever that strategy is you just need to define it in the form of a configuration and this is the most complex part of the sweep it's still fairly simple relative to writing all this stuff your own we're going to go through that in great detail to make sure it's clear how this configuration works so when you're setting up sweep in a notebook like this one then that config object is a nested dictionary but when you're running a sweep via the command line the config object will be a yaml file the core idea though is that it's a collection a nested data structure the keys are strings and they say okay this is the thing whose value i want to set and then the values in that dictionary then set what are each of these configuration settings this is a pretty standard way of structuring a configuration a lot of people who are writing code like this for themselves use this sort of nested dictionary structure like this at least i did that when i was setting up the experiments that were part of my pieces if i had done them as a way to biasy sweep i probably would have saved myself like two three weeks of engineering time of building that sort of stuff for myself so what goes in that nested dictionary the first thing you need to do is you need to pick the method of your suite the strategy for choosing new parameter values we have three search methods grid search random search and bayesian search grid search lays down a grid over the set of all possible hyper parameters and checks out all those values literally every possible value it's very effective but it can be computationally very costly the other sort of dumb and easy strategy for sweeps is random search just pick a random value for every hyper parameter and though that sounds very dumb it's not how we pick the weights for a neural network we'd do something much smarter random search is actually extremely effective and it scales very well to large problems the other alternative and a lot of people like is bayesian search which essentially tries to create a probabilistic model okay if i i've tried hyperparameter values over here and they were all really bad so i'm actually going to try my hyper parameter values over there those are probably going to be better than the ones over here so intuitively it seems like this should work really well but our intuition works best in low dimensional spaces when you have a ton of hyper parameters bayesian search scales poorly so you have to be pretty careful about using bayesian search often you'll end up expending a bunch of effort and not getting anything better than you would have gotten by just using a random search so we'll stick with random for this one though you're welcome to try and adjust this code to use the bayesian search instead if you're interested in that for those bayesian sweeps you need to tell us what the metric is that you are trying to improve and you need to tell us whether your goal is to make this metric low or metric high loss is a metric that we want to minimize accuracy would be a metric that we'd have the goal of maximizing you only need this for beijing sweeps but i like to include it in all of my sweeps because it's just good documentation it says okay what is the metric for this experiment and what do i want out of that just in case six months later you come back some variable name like valgey batch and you don't really know whether that was supposed to be high or low in six months or a year down the line a question in the q a is there a rule of thumb to use one hyper parameter optimization over the other i would say the first thing i i almost always start with random search and then random search we usually reveal that there are some values that are just not very good and i'll use that to shrink down the space of possible hyper parameters i'm looking over until it's small enough that i can do a grid search that works well where i can just check every possibility that's left i can check all of them to find the best one but i think most people tend to pick either random search or page inserts and learn some sort of tricks of the trade to make one of those really good and then specialize in it i think that's something that's generally true about architectural choices in deep learning the most important thing is that you know how to work with the tools that you're using not that you're using the absolute best tool whether it's non-linearities data augmentation strategies whatever just pick some things to learn how to use really well and you'll be able to get good performance doesn't matter which things you pick uh great question now we're at the the meat of the sweet definition which is to name those hyper parameters what are those parameters and what are the legal values for them what are the possible choices that i can make to set a value of the fully connected layer size the size of the hidden layer of the network or the amount of dropout that i'm going to apply the simplest way to do that is just list all the legal values and that makes a ton of sense with some of our hyper parameters like maybe i'm comparing the possible choices of optimizer and there's really only a finite number of options and most people just pick between two of them so for even for a really broad hyper parameter optimization i would really only need two values for choice of optimizer atom and sgd and similarly there's a lot of things where even though in principle there's an infinite number of choices like i can have a hidden layer that has between one and infinity nodes in it there's usually a pretty small range in which you want to look and checking every value 128 nodes in the hidden layer 129 nodes in the hidden layer 130 nodes in the hidden layer it's just too much so you want to specify down maybe these are the values that are probably good they're nicely spaced from each other so you can explore a little bit you don't need as much flexibility necessarily as every natural number could be a possible hidden layer size even though that's in principle true as i said the sweep config is a nested dictionary so the parameters go in as a dictionary under the key parameters in that overall sweep config there's also sometimes where we have hyper parameters that we do have the ability to set in our code but that we don't want to change so the example here is that i'm just going to run my training for one epoch to see how things do in their first epoch of training and i don't necessarily want to try and optimize this hyper parameter but i do want to set it so just setting the a single legal value for hyper parameter means that we just only ever try one value and this is also good you know as little as possible relying on things like defaults magic numbers things like that in our code and writing down okay this is the number of epochs that went into this code at this time this is the number of epochs that were run and that's all logged to weights and biases much better for reproducibility than say changing the default value on your script as i have done sometimes when i was crunched for a deadline and i really needed to change some experiments around for grid search actually that's all you're ever going to need and grid search is a great choice it's actually one of the more popular choices for our users who use our sweep tool but for uh random search all the values of a parameter are equally likely to be chosen on a given run and sometimes you know that just won't do maybe we want uh normally distributed values for a parameter we actually do want to try out every value between 0 and 0.1 any value could be a good choice and we don't want to a priori say oh it has to be one of these specific values ooh a question are there cases when grid is not an ideal choice yes grid is not a good choice if you've got a lot of hyper parameters with a lot of potential values if i've got 10 parameters that i'm optimizing over and they've each got two possible choices that's 2 to the 10 or 1024 options a grid search will take 1024 steps to try out all of those but we'll do them in a very specific order there's a decent chance that you might not see any good values of the parameters until the very end if you do a random search on those exact same values of the parameters you're less likely to spend a whole bunch of time only working on bad values of the hyper parameters so random can be a better choice than grid if you have a lot of values it converges a little bit faster a question do you have guidelines on picking hyper parameter ranges and distributions for random search so i'll answer this one but what i want to first do is just on the front of how do you set up these distributions for sweeps when you're doing these random searches what you need to be able to do is you need to specify the probability distribution for the values that are coming out rather than just saying it's uniform over these specific values and the way that's done is again we're going to nest another dictionary in there it's dictionaries all the way down and the way it works is the name of the parameter is your key the value there is a dictionary that describes the distribution you want to use if you check out our docs you'll be able to see all the distributions that we make available in the end uniform ends up being a really common choice it's just now instead of being uniform over a fixed set of values it's over an infinite set of values whether that's number between 0 and 0.1 or a normal distribution which goes over any possible floating point number we have a lot of sort of flexibility here we've got log distributed things which are very common in hyper parameter searches i want to try every order of magnitude and i want to try those uniformly but i don't necessarily want to try uniformly across every value between one and one thousand i want to try something like one ten a hundred a thousand something like that so we've got these log versions we've got quantized versions that you can get say integers out of these distributions details all on our docs there so to return to do you have guidelines on picking hyper parameter ranges and distributions for random search what i would say is the biggest mistake i see people doing is they use uniform distributions instead of log uniform distributions log uniform distributions like this one have evenly distributed logarithms which is a mathy way of saying that they try out orders of magnitude with equal probability that is very commonly what you want to do it's definitely the case for batch size and learning rate i think that's probably also the case for things like hidden layer size so i would if you're not using log transform distributions i would use log transformed distributions and then the other tip is you're probably going to run more than one hyper-parameter sweep so start off broad and then narrow down only once you've started to try out values great question okay we've added in these values that are random over these broad ranges and then when we're finished we've got our nice nested dictionary here that's our sweep config and you can see if you've worked with yaml files you'll see the similarity is maybe you can see it here that we've got keys and we've got values some of those are dictionaries some of those are things like lists we also also some things like early termination but i wanted to keep this relatively simple if you want to know more check out our docs or check out this big collection of examples these examples in the yaml format we've got a whole bunch of them at that link there all right now it is time to initialize the sweep so what we did in that first step was we defined our sweep we said okay here's how i want to try things and here are the things that i want to try now it's time to get started so the basic way that this is set up is that there is a sweep controller that runs on our machines and then on your machines the work is executed that actually trains your machine learning algorithms this is a division of labor between deciding what to do next and actually doing it right this is we're splitting those two parts of the sweep from each other you can set this up differently if you want to locally control your sweeps that is an option that we provide but the sort of the typical sweep is we do that sort of central decision making according to your configuration about what to try next and then we just are ready to send that to any machine that asks okay what's the next step in this sweep and this makes it really easy to scale up sweeps by just getting a whole bunch of machines and on those separate machines maybe those are things you're spinning up in the cloud with something like google kubernetes engine or aws something like that or maybe it's your laptop and your desktop and your friend's laptop uh your roommates out of town for the weekend and you can run jobs on their machine so long as it's got the code then it can participate in the sweep and we handle all this communication stuff for you so that's one of the trickiest parts about setting up a really parallel hyper parameter optimization yourself we're taking that taking on that burden for you and so the only thing that you need to do is to run this wb.sweep command with that config there's some things that you can set here configuring this sweep giving us a username or an entity so maybe you're working on a team setting a project name but fundamentally you just need to pass this thing this sweep config and it will start up a sweep on our website so let me actually click this sweep url so this here is now the web page for the sweep that we're about to run it's got some information about this sort of the config is already showing up here that config that we just built is now here this guy is now sitting here waiting for somebody to ask hey i want to run part of this sweep what are the hyper parameter values i should use that is step three running the sweep agent that is how you start querying that controller for hyper parameter values to use in order to run a sweep agent we have to define the actual training process up to this point we've just been talking about the hyper parameter optimization we haven't really talked about like what those hyper parameters actually get used for right they're going to be used to train a neural network we have to write the code for that so this section here first does that sort of basic pi torch pipeline if you want to hear a little bit more about how weights biases is instrumented in pi torch there's a collab link there it's just three four lines just these three pieces of our library here init config and log but most of this there's a lot of code there but that's pretty much just boilerplate high torch so now we're ready to start sweeping the sweet controller's sitting there waiting it's now time for somebody to talk to that sweet controller and say hey give me a collection of hyper parameters and i will go and check how those did and i'll tell you what the results were and that's this one b agent command so actually i got a question that's relevant here is there a way to set the maximum number of runs of the sweep or do they run forever with a random sweep there's no stopping criterion right with a grid sweep there is a stopping criterion once you've tried everything you can stop but with a random search there's no way to stop so you actually have to say okay this is a maximum number of times that i want to run that is part of a one b agent command so you can say this agent will run this many times this agent will run this many times so right here i've set it to five this is a random search we could have run infinitely but we're going to run just five let me make sure all the cells before have been run and then we can execute this guy so this is starting to run the agent got that config up there at the top of the screen highlighted right there from the controller from online agent says all right cool time to start training so the training starts we download the mnist data set because this is our first training run and then we do our run now the results of the run comes out and these are quick runs the training is already done one epoch is already done and we print some of the quick summary of that to the command line the details of it are on our web app we'll see that in just a second and then it's time for another run to start so then the agent says hey controller you've got another run for me controller says yes actually i do try these try batch size 163 dropout 0.5 fc layer size 512 learning rate 0.078 yada so it gets a new config from the controller and then runs again it finishes and this process repeats until we hit that count number in this case if it were a grid search it would be until we tried all possibilities in any case this repeats until we hit a stopping point and the controller says hey agent i'm actually i don't have anything more for you i'm done that's it and the agent stops so it's now run five different let's run five different hyper parameter settings and that's all been logged this is what the suite page then looks like let's focus on these charts here and let me zoom out a bit we've got the loss over time for each executed run what was the loss at the end of training we've got this parameter important spot so this we train a random forest and we do a quick linear fit to determine which of your parameters were most important and then we've also got this nice parallel coordinates plot so actually instead of going in greater detail about these these plots are most useful once you've really got a nice long sweep going right when you've got lots and lots of runs that you can compare so i'm gonna pull a martha stewart move and say i've got this sweep that i put in the oven a couple hours ago and pull that out so this is the exact same code but it's 100 runs instead of five and it's got those same plots here but now they've got a little bit more meat to them again with each run we log a loss and we keep track of that over time so you can see how did this sweep go one fun thing i've actually noticed a lot about sweeps is you can see that the the returns are usually pretty small for hyper parameter optimization that basically in your first couple runs you'll see one that's pretty good and then it will only get better slowly after that point so it's often not that important to try more than like 10 or 20 runs rather than 100. you'll usually find that you've gotten something close to optimal this is why i recommend random search that just empirically you see pots like this a lot so then now our parameter importance and correlation plots here are are maybe a little bit more informative it says okay the learning rate is really important to final performance maybe that's not so surprising we're only in a single epic the higher your learning rate the more you'll probably be able to learn within that one epic it looks like the atom optimizer has a positive correlation with the value of the loss so that means oh it looks like actually i think i am reading this the other way around so the correlation is positive between the learning rate and the loss and that means that when the learning rate goes up the loss actually goes up so that suggests maybe there's some instability in training and it looks like if our optimizer is atom then the loss goes up so you can see that it's nice to combine actually these parameter importance plots with these parallel coordinates plots so let me walk you through what this parallel coordinates plot is this guy says all right here are all of your runs each run is a single line here you can see which run it was it's pious sweep and sweepy sweep and all the various randomly named runs in this sweep each one is a line they're colored according to your metric so here it was loss and then they pass through on each of these vertical bars here they pass through the value of the hyper parameter so this is for all the ones that had a fully connected layer size of 512 they all passed through this point the ones that had sg is the optimizer all pass through this point then the learning rates those were random and uniform so they're all over the place here so how you read this parallel coordinates plot is basically you look for patterns where you see splotches of color and i see a splotch of bright yellow color indicating a high loss first here with the fully connected layer size being up at 512. the bigger networks we're actually struggling a little bit more seems like we might have some optimization issues and then you can also see it looks like they are primarily ones that had the atom optimizer so you can see these all these brightly colored these purple and light purple and yellow lines are all passing through the atom point here we can also these charts are interactive so we can do things like filter down to just these couple of runs and we can see them really clearly on top of a background of all the other runs in just transparent gray and that can be useful for picking out these patterns you'll notice that it also changes all these other plots here we can now see the losses of those their loss on every batch so you can see that there's this big zoom in the loss like jumped really high and then they never recovered the interactive nature of these charts is really great for debugging and also learning from your experiments really quickly so i love using these charts in the workspace and in the reports to explore the results from hyper parameter sweeps especially a hyper parameter sweep like this produces a really rich data set and without an equally rich visualization scheme you're going to really struggle to get everything out of that that you could there's still plenty of other stuff to talk about with sweeps we talked about all these things there's some links here for a couple of different examples that you can try if you want to play around with them it also has examples that can let you try out some of the more advanced features like bayesian hyperband for combining bayesian search with early stopping hyperopt also early stopping but with random search these features that can really soup up your hyperparameter sweeps and get really good configurations with really small amounts of compute so i recommend you check those out and start getting your hands dirty with the weight spicy sweep tool to select and optimize your hyper parameters hey friends charles here thanks for watching my video if you enjoyed it give it a like if you want more ways to buy these tutorial and demo content subscribe to our channel and if you got any questions comments ideas for future videos leave a comment below we'd love to hear from you [Music] [Music]