Why Do We Need Activation Functions in Neural Networks?

Key Takeaways

what is going on guys welcome back in this video today we're going to try to understand why we need activation functions in neural networks I'm going to explain the math behind it and I'm going to show you a code example that illustrates how important activation functions are so let us get right into [Music] it all right so the goal of this video is to understand why we need activation functions in neural networks and why it's often times enough to have a very simple activation function like the relu function uh to make a huge difference when it comes to what a neural network is capable of so I want to give you here first of all the mathematical explanation and then I want to show you a code example that shows that what I'm saying here is actually true and makes a huge difference so this video is not going to be very complicated you don't need to understand a lot about neural networks you don't need to understand what back propagation is you don't need to understand what gradient descent is the only thing that you need to understand here is how a simple forward pass works in a sequential feed forward neural network so how we get from inputs to outputs given the weights and biases that's the only thing that you need to understand and for those of you who don't know what I'm talking about I'm going to give you a one or two minute crash course here now on neural networks very very superficial high level and basic uh the basic idea is you have some inputs whatever they are X1 X2 X3 you have some neurons and then these neurons are connected to other neurons in different layers these are called The Hidden layers in here uh they can have a different number of neurons whatever and in the end we have some output and the output neurons can be a single neuron it can be two neurons three neurons it doesn't matter and if these are all now just simple fully connected dense layers so to say uh we have all the inputs attached to all the neurons of the first layer we have the output of these neurons attached to all the neurons of the next layer and so on and all these connections this is now important are weighted connections so they have weights every single connection here that you see has a weight to it and this weight is a trainable parameter which means that you don't need to understand how it happens but there is this back propagation and optimization process where these weights are adjusted so that the neural network learns how to do a certain task and every single neuron also has a so-called bias to it so uh a constant that is added um or subtracted from the uh from the weighted sum so maybe to explain this in a more simple way I'm going to use a very very basic example here which is going to be just uh X is going to be an input a single input and then we have two neurons here and then we have one output neuron and that's our neuron Network so I feed X into this neuron and into this neuron then we have the output of this neuron and this neuron fed into the output neuron and then we get our output let's call it y uh that is a very simple neural network and now what we have is we have weight for every connection here so we would have a weight one here we would have a weight two here and now this is not a professional notation usually you use some superscript for the layer but I don't want to confuse now anyone who hears about this for the first time but you have basically uh different weights here let's call them for the sake of Simplicity now weight one weight two weight three and weight four and what we can do now is or what happens now is and of course we also have here the biases so we have bias one bias two bias 3 um in a simple forward forward pass what happens is the input X is now combined with all these weights and biases to produce y it's a very simple calculation so yes it's machine learning yes it's neural networks but at the end of the day it's just a simple multiplication and addition calculation so what happens here is X is multiplied by weight 1 x is Multiplied also by weight two the result of that is fed into these two neurons then a bias is added then we have the result of this whole thing multiplied with weight four or with weight three if we go the upper route then the results here are added together a bias is then uh added again and then we get our input uh our output y so what basically happens is to calculate y I have to say the following I am taking the buyas three let me use the colors here I'm taking the buyas three plus something what is this something this something is weight three multiplied with and I basically just have here weight 1 * X plus some bias so plus by is 1 that is what happens in the upper um in the upper Branch here I take x times weight 1 plus bias one and the whole thing times uh W3 and then plus the bias and also I do the same thing on the uh lower end here I do wait for time and then I have in this case weight 2 * X Plus and then bias 2 that is how we calculate the output y here now this is already enough to understand if you look at this closely why an activation function is extremely necessary here because what happens here if you look closely is I have two neurons and I could also have three neurons now this would just add uh an additional um yeah an additional term here and I can have also multiple layers and it will get more complicated and more nested but at the end of the day what happens here is I have a linear combination so what happens here is I can consider this to be if I if I expand this I have now I'm not going to use the colors I have B3 + weight 3 * weight 1 * X this is just when I multiply this part here plus weight 3 uh * B1 plus weight 4 * weight we 2 * X plus weight 4 * bius 2 that is what I'm calculating here and if you look at this you can spot we have two kinds of terms here we have the terms that don't contain X and we have the terms that do contain X for this I'm going to use now green and what I can do here now is I can extract X so I can factor out X I can say that I want to have this weight three weight one from here and I want to have this weight 4 weight 2 * X that is basically the green part and in addition to that I have a buys three a weight three * bias 1 plus a weight 4 * bias 2 um so basically I have I have this part here the green part and I have the red part from before 4 and if you think about this now what we can do is we can consider this to be just a number you know weight 3 * weight 1 plus weight 4 * weight 2 this is just a number so let's say I call this now weight five and I say weight 5 * X I can do that I can just rename this and then I can say all this here is also just a number right it's B3 + W3 * B1 + W4 * B2 at the end of the day it's just another bias so I can call it B4 and what happens now is from this complicated or a little bit more complicated calculation here I end up with a very simple linear combination I have uh basically just a weight and a bias so the equivalent of the network up here would be a simple Network that I can train by just having a single neuron and having a weight five here and a bi four here and then that's the output so my network up here no matter how complicated I make it I can add five more neurons and three more layers at the end of the day it's not more effective than just having a simple single neuron this is because a linear combination of linear combinations of linear combinations is a linear combination I have an input I have a weight and I have a bias and of course I can have more inputs and then it would be a more uh highly dimensional linear combination but at the end of the day this is no matter how complicated I make this no matter how many neurons I add and how many layers I add at the end of the day I have a linear model I can only solve linear problem so for example if I have let's say I have a coordinate system and I'm trying to fit some data some very simple uh regression task and I have the points something like this which would be let's say some X squ function no matter how many neurons and no matter how many um layers I add at the end of the day I will always just get a linear model I will never be able able to model a curve or a quadratic function or anything like that I will always have a linear model so what we need to do in order to prevent this is we need to break linearity we need to somehow introduce something into this calculation here which breaks the linearity of the linear combination and for this any function um works that breaks the linearity now there are some more constraints of course but theoretically every function that breaks linearity works and we have functions like the sigmoid activation function which I'm not going to uh write down now as a formula but it's basically something okay this is very very ugly let me shift this uh it's basically a function that returns a value between zero and uh one and it's kind of yeah somewhat like this um and we have also the t h function which is the same function just between negative 1 and one uh but we also have some very simple function like uh the relue function and the relue function is basically just give me zero for any value that is not um zero or above and then give me just the identity function afterwards so the Rue function is um rectified linear unit is the name uh is a function that basically gives you the maximum of either zero or the input value so if if that is X here um yeah it's a very simple function but you can see it's not a linear function because it breaks linearity here it's not a linear function it's not just continuing down here it has a breaking linearity and it's very efficient it's very easy to work with uh mathematically speaking or computationally speaking um but it's enough to break linearity and if I have because what we do with an activation function uh let me just remove all this here uh what we do with an activation function is basically instead of just saying we apply the weights we add a relue call so to say on the intermediate result so I multiply x with weight one and um then basically I call the ru function onto the intermediate result so in this case instead of just saying W3 * W1 * x + B1 I would sneak in Let Me Maybe make this a little bit smaller here so I have more room I would say W3 times and then Ru called on this term and then I would do the same thing here I would say W4 time ru so W4 time r on this result and then of course the whole thing here would also again usually I mean it depends on the output layer often times we don't have an activation function often times we have a different one uh but theoretically if you have multiple layers the whole thing here would be also again Rue this and so on and then you cannot just go ahead and extract um or expand and then factor out X it doesn't work like that because it breaks linearity and then the whole model is no longer a linear model uh but it can model also some more complex tasks so what I want to show you now is how much of a difference this actually makes even if we use more neurons and more layers and for this I have this pytorch example here what we do is we generate some X data between -10 and 10 and then we say that the Y data so the uh data that we want to predict and model is just the input data to the power of three so we have a cubic function that we want to model with a neural network and for this we have two classes we have the net no activation class and the net with relue class and they are exactly the same the only difference is here we use an activation function so here we get an input one value we have 50 neurons 50 neurons 50 neurons 50 neurons and then one output neuron and the exact same thing is happening here but we call the Rue activation function so the simple rectified linear unit activation function that I just showed you to break linearity here and the rest is exactly the same and now if I take this I generate the data I Define the two models I train both models and I also plot the results what we get here is we get the following result the blue thing here is the cubic function that we're trying to model that we're uh trying to make predictions for with the with the neural network and you can see here that the neural network without activation function uh without activation functions just ends up being a linear model it's just a line it's just a linear regression now this is probably the most optimal line but it's still just a line it's a linear model and it will never be able no matter no no matter how many neurons and layers I add it will not be able to come close to this cubic function here whereas if I just look at the results with the relu activation functions you can see that this is basically almost the same so it modeled it almost perfectly we have the model and um the actual X cubed function here which are almost the same and this is how much of a difference it makes to use even very simple activation functions like so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting a like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you in the next video and bye