Why Non-linear Activation Functions (C1W3L07)

Key Takeaways

why does your neural network need a nonlinear activation function turns out that for your neural network to compute interesting functions you do need to take a nonlinear activation function less you want so just the for prop equations for the neural network why don't we just get rid of this get rid of the function G and set a1 equals Z 1 or alternatively you could say that G of Z is equal to Z right sometimes this is called the linear activation function maybe a better name for it would be the identity activation function because it was just outputs whatever was input for the purpose of this what if a2 was just equal to z2 it turns out if you do this then this model is just computing Y or Y hat as a linear function of your input features x2 take the first two equations if you have that a1 is equal to z1 is equal to w1 X plus B and if then a2 is equal to z2 is equal to W 2 a1 plus B then if you take the definition of a 1 and plug it in there you find that a 2 is equal to W 2 times W 1 X plus B 1 a bit right so this is on a 1 plus B 2 and so this simplifies to W 2 W 1 X plus W 2 B 1 plus B 2 so this is just let's call this w prime B prime so this is just equal to W prime X plus B Prime if you were to use linear activation functions or we go to call them identity activation functions then the new network is just outputting a linear function of the input and we'll talk about deep networks later neural networks with many many layers many many hidden layers and it turns out that if you use a linear activation function or alternatively if you don't have an activation function then no matter how many layers your neural network has always doing is just computing a linear activation function so you might as well not have any hidden layers some of the cases that briefly mentioned it turns out that if you have a linear activation function here and a sigmoid function here then this model is no more expressive than standard logistic regression without any hidden layer so I won't bother to prove that but you could try to do so if you want but to take home is that a linear hidden layer is more or less useless because on the composition of two linear functions is a sailfin linear function so unless you throw a non-linearity in there then you're not computing more interesting functions even as you go deeper in the network there is just one place where you might use a linear activation function G of Z equals Z and that's if you are doing machine learning on a regression problem so if y is a real number so for example if you're trying to predict housing prices so why is a there's not 0 1 but is a real number you know anywhere from zero dollars is a price of homes up to however expensive right how's the scale I guess maybe houses can be you know potentially millions of dollars so however however much houses cost in your data set but if Y takes on these real values then it might be ok to have a linear activation function here so that your output Y hat is also a real number going anywhere from minus infinity to plus infinity but then the hidden units should not use them your activation functions they could use value or 10 H or Li Q value or maybe something else so the one place you might use a linear activation function is usually in the output layer but other than that using a linear activation function in a hidden layer except for some very special circumstances relating to compression that won't want to talk about using the linear activation function is extremely rare oh and of course they're actually predicting housing prices as you saw in the week 1 video because housing prices are all non-negative perhaps even then you can use a rare loop activation function so that your outputs Y hat are all greater than or equal to 0 so I hope that gives you a sense of why having a nonlinear activation function is a critical part of neural networks next we're going to start to talk about gradient descent and to do that to set up discussion for gradient descent in the next video I want to show you how to estimate how to compute the slope of the derivative of individual activation functions so let's go on to the next video