The Ultimate Guide to Hyperparameter Tuning | Grid Search vs. Randomized Search

Key Takeaways

hyperparameters are the parameters of the model that are not learned during the training process but are set by the user before the training they are configuration settings that control the learning process and Model Behavior for example the maximum depth to allow the decision tree to grow number of hidden layers and nodes in neural networks or the number of nearest neighbors to consider in the K nearest neighbors algorithm the choice of hyperparameters can significantly impact model performance as they help balance the tradeoff between overfitting and underfitting let's see an example in the knnn algorithm the class of an observation is predicted by looking at the classes of its K nearest neighbors and performing a majority vote the value of K is a hyperparameter that can be tuned and it is often chosen to be an odd number to avoid ties intuitively increasing K means considering more Neighbors which can lead to underfitting conversely decreasing K may lead to overfitting I trained three models with different K to show how important the K hyperparameter is in training the algorithm often you won't just have one hyperparameter to tune but many so how do you find the best set of hyperparameters for your data set while hyperparameters are typically configured based on the data set's characteristics it's impossible to know the optimal set by simply examining the data a straightforward approach to finding a good set of hyperparameters is to choose several potential values for each hyperparameter and train separate models for every combination then you can use the validation data to determine which combination results in the highest score sometimes you'll notice that the best scores tend to be near the highest or lowest value of a given hyperparameter range in these cases you can refine the search by adjusting the range slightly while keeping other hyperparameters fixed this process is called grid search you create a grid of hyperparameters and search for the combination that produces the highest validation score however one major downside of grid search is that it can be computationally expensive especially when training complex models on large data sets having five hyperparameters with six options you will need to train the model six to the power of 5times which is equal to this number imagine the computational effort acquired to be more efficient we can do a randomized search instead of evaluating all possible combinations of hyperparameters you define a range for each hyperparameter and specify how many models you want to train the method then randomly samples from each hyperparameter range this approach is more efficient but it introduces randomization which can either help you find the optimal hyperparameter set more easily or take longer the effectiveness depends on the luck of the sampling and whether whether the specified intervals include the best set as a rule of thumb regular grid search is more suitable when you have a small number of hyperparameters with narrow ranges and when you are training a relatively simple model randomized search is used with many hyperparameters broad ranges and complex models for large neural network models where training each model takes several days the process becomes more complex different methods are designed to find a better set of hyperparameters in such scenarios [Music] if you want to learn more about artificial intelligence subscribe to our channel to be aware of the new videos press the like button and let's discuss AI in the comments section