Hyperparameter Tuning Tips that 99% of Data Scientists Overlook

Key Takeaways

Hyperparameter tuning sounds fancy, but is it important? For most data scientists, it's a crucial skill that you need to build up to make sure your models are performing to their full potential. And when tuning models like XG Boost, which have a lot of parameters that you can work with, it does take a lot of time. So, in this video, I'm going to walk through hyperparameter tuning and XG Boost model, and I'm going to show you how with XG Boost 3.0 0 and GPU acceleration, you can speed up this process by 5 to 15 times. So, let's dive in. So, here we are in a Kaggle notebook. If this is one of your first time watching my videos, I like putting the link to this notebook in the description so that you can go and run this code yourself by just forking the notebook that I have. So like I said before, we're going to be creating a machine learning model with XG Boost and we're going to be tuning the hyperparameters for that model using Optuna. XG Boost is one of the most powerful and commonly used models for tabular data because of its versatility and how well it really does out of the box. But hyperparameter tuning is something that actually can give it a lot more performance if done well. And the nice thing which I'm going to show at the end of this video is you can actually speed up this process a lot by using GPU acceleration when you're training your XG Boost models. But before I get too far, I do have some notes. Consider this a disclaimer about hyperparameter tuning because I think this is missed on people so often when they're doing hyperparameter tuning. The main thing is doing this tuning without having a good cross validation setup with your data is going to be worthless. It's actually going to make your models worse because you're going to overfit that model to whatever data set you have. So in this example, I'm going to be doing cross validation. But if that's new to you, you should check out my video that I have on setting up a correct cross validation scheme when training models like this. The other thing to keep in mind, and I've been a victim of this so many times before, is to avoid this endless cycle of hyperparameter tuning when you're creating models. Typically, what I like to do is hyperparameter tune the model at the beginning with your data set. So, you have a good baseline parameters to start working with. Then, you're going to create all your features and engineer ways to improve your model with the data set. Then at the end when you think you have all the features that you're going to be using in the model, you do one final hyperparameter tuning session at the end to get that last little boost. It's really easy to forget this and to start hyperparameter tuning every time you make a small tweak to your model. And trust me, that's just going to be a big waste of time and it's better spent really focusing on improving your model before you tune that final time at the end. And the last thing I'll say about hyperparameter tuning is it's kind of a paradox where it's super important, but it's also not that important. So you definitely can have a really bad model just because you have bad parameters. But at the same time, once you get pretty good parameters, the tuning is just going to give you a small bit of improvement above that. So a lot of times I hear people talking about just parameter tuning their way to a perfect model, and that really isn't usually the case. Parameter tuning is kind of like the icing on the top. Once you have those baseline parameters, you can squeeze a little bit more out of it. And in some applications, having a little bit of improvement can actually mean a big deal. Like it could save your company millions of dollars if you have a little bit better of a model. So, not to say that that's not important. Parameter tuning isn't about making huge gains. Most of the time you're going to see that when you're building features for the model or you're improving the data set that you're providing to the model. Let's get to the setup. So, we're going to install XG Boost 3.0 here using pip because the Kaggle notebook doesn't have that out of the box. And then I'm also going to install Jupyter Black for formatting the notebook. So, the code looks a little bit nicer. Now, just to show you, if I go over here to the side, I do have preloaded this credit card fraud data set that we're going to be using as our example to train a model. It's a binary classification data set where we're trying to predict if an entry is fraud or not. Another thing to keep in mind is I do have an accelerator turned on on this notebook. So, we have a P100 GPU here that we're going to be using to speed up the process here in a little bit. without that GPU, we won't get that huge speed up. But luckily, Kaggle gives us that for free. Now that I've installed that, let's just load in the packages that we're going to be using. And I'm printing out here the versions of each of these. I also want to just show you and confirm that we do have a GPU running on this machine. So, if I run Nvidia Smi, you can see that we do have a P100 GPU running. What version? A CUDA, all that good stuff. This is a good sign that you have a GPU running on your system. Now I just want to answer a question of why do we even want to do hyperparameter tuning? Well, you could think about these machine learning models as having a bunch of different knobs that you can set to affect how it performs. So with an XG boost model, you're building a bunch of small tree classifiers and these are being created with a gradient boosted algorithm. But how does the model decide how to create those trees? Does it create really deep trees? How does it decide to split those trees when it's making its branches? Are there rules about how deep each branch could be? How long do we train the model? All these sort of things are parameters that we can set in the model. So if I I just show you here an XG boost classifier, not all of these are necessarily hyperparameters, but when you're creating this model, things like the learning rate, the maximum depth of these trees, these are all parameters or knobs that you can tune in order to make the model perform the best on your specific data set. But also turning the knobs don't really work in isolation. So if you change one parameter, it might affect the way another parameter works. So you really need to tune these all at once or tune them in a way that accounts for other changing parameters as you're tuning. Now, I went through here and I took each of the parameters that I think are kind of the most important ones to focus on if you're training an XG boost model, but I would recommend taking a little bit of time and reading about each of these parameters. If anything, just to get a general understanding of how they affect the model, and that will really help you in having that intuition for why a parameter change might be impacting your model in a certain way. Okay, now it's time to load in this data set. As I mentioned before, it is a credit card fraud detection data set. So, if we just look at the first few rows in this data set, we could see all these features that start with a V are going to be used to predict the class in this data set. And the class column tells us if that row is a fraudulent charge or not. Now one thing to keep in mind about this data set. If we do a value counts on the class is that it is an imbalanced data set. So you can see there are very few fraud cases for the size of the data set we have. This is not uncommon. But just know that for an imbalanced data set like this, just using something like accuracy isn't going to be super great because of how few examples we have to work with. A metric like area under the curve just gives us a little bit more granular understanding of how well our model is doing. Okay. So next I'm going to take this data set and we're going to subset into our features and into our target. So class is going to be our target. We're going to call that Y and then X is going to be our features. So let's take the columns of this data set and take it for when when the column starts with V that's going to be our feature columns and we'll just subset our data set to those. So now if we look at X which are features we have all the V columns Y our class that's our target. I did drop this time feature and the amount. Actually, let's add back amount to the features because I think that might be helpful for our model for predicting. Now, let's just do a basic example of what parameter search might look like for a single feature. Now, this is not exactly how we're going to do it when we train our final model, but it's good as an example. So I am going to import here that score metric that I mentioned before and scr stratified kffold which is um how we're going to split our model for cross validation. Then we're just going to create a list of max depth. These are the different values for the parameter max depth that we're going to try out and then we'll evaluate how they change how the model performs. And then we're just going to loop over these. So for max depth in these max depths and we're going to do train a three-fold cross validation then we're going to print the average score at this parameter. So we're going to print that max depth so we know where we are in the loop. We're going to create this stratified kf fold and we're going to create a list where we'll store our scores in. Then we're going to loop through these three folds and we're going to train XG Boost classifier. Now note we're setting this max depth value for each of these models as we train them. Then we're going to evaluate how well they perform using our area under the curve scoring metric. Finally, I'm just going to print out for that max depth what the average score area under the curve score is for that value. So let's go ahead and run this. And now that that's done running, you could see that our score is actually a little bit different depending on which value that we've set. So we could see here that the max depth of five actually gave us our highest score averaged across these three folds. But as I mentioned before, this parameter is going to be dependent on other parameters that we might be setting at the same time. So we need to evaluate a bunch of these at the same time and kind of do a smart way of arriving at the best parameters for the model in general. But we're going to use the library Optuna which is built for this type of hyperparameter optimization. Under the hood, Optuna has smarter algorithms that it's using to search this space of all possible different hyperparameters and helps us arrive at it a little bit faster than we would just doing a greedy search. To set up our optimization flow, we first need to start by creating this objective function. This objective function takes uh Optune a trial, which you don't need to worry about yet. Uh then you give it a parameter space based on this trial and we'll set that up next. Then we'll um define our model and do our cross validation and we'll return our score from that validation very similar to what we did before. So it's sort of wrapping what we did before in a function. So this is what it looks like all put together. So when we set our parameters, we don't actually give exact values for what we want. But we let Optuna pick these by setting a range of values that it can choose from. So you can see here for max depth, we're giving it option to choose max depth between 3 and 10. For parameters that are float values like the learning rate, we'll just use the suggest float method. And you can see it's from between 01 and 1. So you can really change the range of parameters that can be chosen from. But in the end, Optuna is going to search this space of parameters that we've defined and help us arrive at the best parameters. Then here you can see that we're defining our XG boost classifier with these parameters that it will choose. I'm using this built built-in method to really do that cross validation scoring that we did before. This is just a cleaner way of writing what we we did in that example loop above and we return the score. Okay, now that we have our objective function, it's pretty simple to call this with Optuna. We're basically creating a study that we'll name here. So, we're going to call this XG boost study and we're running this on CPU first. We're setting the direction to maximize because we want to find the parameters that give us the highest score in this example. Higher is better. And then we just run study.optimize. We give it the objective function. We tell it how many trials or how many different uh parameter settings that it will test out. Let's go ahead and run this. And we can see as it's trying the different parameters, it's going to print them out here and also tell us the final score that it arrived at. Now, one thing to keep in mind here is we're running on a CPU and this is taking a while. If I click in the resource utilization, we can see that the CPU is being completely maxed out and we're not even using our GPU. And I'm going to show you how later we going to speed up this process a lot by using the GPU. All right, so it's done running. Now, for 10 different trials, it took over 2 minutes to run. So, if you can think if we're running hundreds, thousands of different trials, this will take a good bit of time to run on our CPU. But we did arrive at our best parameters that it found out of these 10. And now that this study is over, there's a few things we could also run. The best value, which is the best area under the curve value we had, or the best trial, which will give us. And that's it. Now we have these best parameters that we can use when we train our final model on all the data or to use when we start to create features. And then we'll tune again at the very end. But I told you we're going to make it faster. So, it's super simple to run this now on a GPU instead. And we're actually see what the speed up looks like. So, for 10 trials, that took about 2 minutes. All we have to do to change this to run on our GPU now with XG Boost 3 is to change this device parameter in our model that we defined and make it run on CUDA. So all I've done here is created that same trial as before, but now you can see I've added the devices CUDA. So now I'm going to run Optuna on this objective function that uses GPU. Call the study name something different. So we remember that this is running on CUDA and let's see how fast this runs. Now we do get this warning which I'm going to show how we can fix this in a bit. But if we do go up to the GPU utilization now, we can see that a bunch of the GPU resources are being used as it's training and it's just a ton faster. So less than 30 seconds it looks like to run the 10 trials when before it was taking over 2 minutes. So running on the GPU for parameter search isn't necessarily going to give us better parameters, but it's going to speed up the amount of time it takes to run this search. So we can run so many more trials and potentially end up at better parameters than we would have otherwise. Now you might be wondering why this warning came up. It is because while we're training our model on a GPU, we're doing this cross validation and we're doing that on the CPU. So that's not ideal because you have the data being transferred from one device to your CPU to score and then back. And it's actually better if we could do the validation itself on the GPU as well. So this just takes a little bit more work and essentially we have to take our data set and use coupi to put it onto our GPU. So we do that by taking the x and y data that we had before and putting on the GPU and our objective function ends up being a little bit more complicated because we have to manually do this cross validation. The idea is pretty similar to what we did before. The only difference is we're running this inference now on GPU. So, it's going to be a little bit faster. Because it is so fast, I'm going to boost up the number of trials that we have from 10 to 100. And let's give this a run. So, we can see it's going pretty fast going through our 100 trials. Also to show you here, the GPU is getting a good bit of work. All right, so the study is complete. It's run a 100 different trials and here it took about 4 minutes. I've run this a handful of times and it is non-deterministic. So, it sort of depends on what parameters it ends up choosing. I've had it run in about two minutes for 100 trials or three minutes. Um, but regardless, we have a huge speed up over what we would have seen with the CPU where we're only able to run 10 trials in about 2 minutes. So, you can expect to see on average about a 5 to 15 time speed up just by switching that one parameter to run your stuff on GPU. highly recommend it if you're running XG Boost and you have a GPU available to you. Now, the last thing I want to do is show you these Optuna visualizations that they give out of the box. It lets you just kind of pass in your study here and see a few important things that might be helpful. The hyperparameter importances are essentially which parameters it found the most impacting your final score. You might expect this that learning rate has a big importance. And then this plot kind of shows you over time how the scores changed and how we arrived at this best score. So here we didn't find this best score until we actually were at um over 60 different trials. So if we were running just 10 of these on a CPU, we would have never found uh this better score. If you're still watching, thanks so much. Drop a comment down below and let me know what you want to see me make a video about in the future. And happy tuning.