Regularization in a Neural Network explained

Key Takeaways

[Music] in this video we'll discuss what regularization is and when and why it may be helpful to add it to our model in our previous video on overfitting we briefly introduce the concept of dropout and stated that it was a regularization technique we haven't yet discussed what regularization is so let's do that now in general regularization is a technique that helps reduce overfitting or reduce variance in our network by penalizing for complexity the idea is that certain complexities in our model may make our model unlikely to generalize well even though it fits the training data so given this if we add regularization to our model we're essentially trading in some of the ability of our model to fit the training data well for the ability to have the model generalize better to data it hasn't seen before to implement regularization is to simply add a term to our loss function that penalizes for large weights we'll expand on this idea in just a moment the most common regularization technique is called l2 regularization now we just said that regularization basically involves adding a term to our loss function that penalizes for large weights with l2 regularization the term that we're adding to the loss is the sum of the squared norms of the weight matrices multiplied by a small constant now if you're not familiar with norms in general understand that a norm is just a function that assigns a strictly positive length or size to each vector in a vector space the vector space we're working with here would just depend on the size of our weight matrices rather than going on a linear algebra tangent about norms in this moment we'll continue on with the general idea about regularization but given that norms are a fundamental concept of linear algebra there's a lot of information available on the web that explains norms in detail if you need to get a better grasp but to oversimplify know that for now the norm of each of our weight matrices is just going to be a positive number so let's break down l2 regularization we have our loss function here now we add the sum of the squared norms from our weight matrices and multiply this by a constant this constant here is going to be denoted by lambda divided by 2 M where m is the number of inputs now this lambda here is called the regularization parameter and this is another hyper parameter that we'll have to choose and then test in tune in order to assign the correct number for our specific model so to summarize we know that regularization is just a technique that penalizes for relatively large weights in our model and behind the scenes the implementation of regularization is just the addition of a term to our existing loss function with l2 regularization we saw that this term is the sum of the squared norms of our weight matrices that is then added to the regularization parameter lambda divided by 2 times our input so why does regularization help well using l2 regularization as an example if we were to set lambda to be a relatively large number then it would incentivize the model to set the weights close to 0 because the objective of SGD is to minimize the loss function and remember our original loss function is now being summed with the sum of the squared matrix norms which is then being multiplied by lambda over 2m so if lambda was large then this term here would continue to stay relatively large and if we're multiplying that by this sum then that product may be relatively large depending on how large our weights are so then our model is incentivized to make these weights small so that the value of the overall function stays relatively small in order to meet the objective of minimizing the loss intuitively we could think that maybe this technique will set the weights so close to zero that it could basically zero out or reduce the impact of some of our layers in that case it would conceptually simplify our model making our model less complex which may in turn reduce variance and overfitting so now that we have a general idea about regularization let's see how we can add it to our model and caris so I'm here on my jupiter notebook and in this first cell i've just imported the libraries and classes that we'll be working with the only one that should be new to us is this last one here i'm importing regularizer x' from Charis now one thing dementia here is that regularizer Xin Karis allow us to apply penalties on layer parameters or layer activity during optimisation so we're essentially setting regularization for specific layers rather than for the entire model in this next cell I have a model we've worked with in previous videos for all intensive purposes this model is pretty arbitrary we've got two dense layers with 16 and 32 nodes respectively with both using r lu and an output layer with two nodes for classification using the sigmoid activation function now there's only one difference in this model from how it was in other videos that we've seen it in the difference here is within our second hidden layer that has 32 nodes I've added this kernel underscore regularizer parameter here I'm specifying that I want to use l2 regularization by setting the parameter equal to regularize errs l2 I'm then specifying point 0 1 to be the regularization parameter which we denoted earlier as lambda and that really is all there is to it for adding regularization to one of our layers and caris so hopefully now you have a general idea about what regularization is and how it can help reduce overfitting additionally you should now know how to apply it in Kerris as well so I hope you found this video helpful if you did please like the video subscribe suggest and comment and thanks for watching you