Logistic Regression Cost Function (C1W2L03)

Key Takeaways

in the previous video you saw the logistic regression model to train the parameters W and B of your logistic regression model you need to define a cost function let's take a look at a cost function you can use to Train logistic regression to recap this is what we had defined from the previous slide so you output Y hat is sigmoid of W transpose X plus B where sigmoid of z is as defined here so to learn parameters for your model you're given a training set of M training examples and it seems natural that you want to find parameters W and B so that at least on the training set the outputs you have the predictions you have on the training set we're going to write this Y hat I that that would be close to the ground truth labels Y either you got in the training set so to fill in a little bit more detail for the equation on top we have said that Y has as defined at the top for a training example X and of course for each training example we're using these super scripts with round brackets with parentheses to index into different training examples your prediction on training example I which is y hat I is going to be obtained by taking the sigmoid function and applying it to W transpose X I the input for the training example plus B and you can also define Z I as follows where Zi is equal to you know W transpose X I plus B so throughout this course we're going to use this notational convention that the superscript parentheses I refers to data via X or Y or Z or something else associated with the i'f training example associated with the i'f example okay that's what the superscript I in parenthesis means now let's see what loss function or an error function we can use to measure how well our algorithm is doing one thing you could do is define the loss when your algorithm outputs Y hat and the true label is y to be maybe the squared error or one hover square it turns out that you could do this but in logistic regression people don't usually do this because when you come to learn the parameters you find that the optimization problem which we'll talk about later becomes non convex so you end up with ostentation problem it would not talk local optima so gradient descent may not find a global optimum if you didn't understand the last couple of comments don't worry about it we'll get to it in the later video but the intuition to take away is that this function L called the loss function is a function we need to define to measure how good our output Y hat is when the true label is y and squared error seems like it might be a reasonable choice except that it makes gradient descent not work well so in logistic regression we will actually define a different loss function that plays a similar role as squared error that will give us an optimization problem that is convex and so will see in a later video becomes much easier to optimize so what we use in logistic regression is actually the following loss function which ones with my rent out here is negative y log y hat plus 1 minus y log 1 minus y hat here's some intuition for why this loss function exists keep in mind that if we're using squared error then you want this squared error to be as small as possible and with this logistic regression loss function we'll also want this to be as small as possible to understand why this makes sense let's look at the two cases in the first case let's say Y is equal to 1 then the loss function Y has comma Y is just the stress term right under the negative sign so it's negative log Y hat if y is equal to 1 because if y equals 1 then the second term 1 minus y is equal to 0 so this says if y equals 1 you want negative Y hat to be as big as possible so that means you want log y hat to be large to be as big as possible and that means you want y hat to be large but because y hat is you know the sigmoid function it can never be bigger than one so this is saying that if Y is equal to one you once Y has to be as big as possible but it can't ever be bigger than one so saying you want Y hat to be close to one as well the other case is if y equals zero if y equals zero then this first term in the loss function is equal to zero because y 0 and in the second term defines the loss function so the loss becomes negative log 1 minus y hat and so if is a learning procedure you try to make the last function small what this means is that you want log 1 minus y hat to be large and because it's a negative sign there and then through a similar piece of reasoning you can conclude that this loss function is trying to make y hat as small as possible and again because Y hat has to be between 0 & 1 this is saying that if Y is equal to 0 then your loss function will push the parameters so make Y hat as close to zero as possible now there are a lot of functions with roughly this effect that if Y is equal to 1 you try to make Y hat knowledge and Y is equal to 0 or try to make Y hat small we just gave here in green a somewhat informal justification for this particular loss function will provide an optional video later to give a more formal justification for Y in logistic regression we like to use the loss function with this particular form finally the loss function was defined with respect to a single training example it measures how well you're doing on a single training example are now going to define something called the cost function which measures how are you doing on the entire training set so the cost function J which is applied to your parameter W NP is going to be the average ly 1 over m of the sum of the last function applied to each of the training examples in turn when here Y has is of course the prediction output by your logistic regression algorithm using you know a particular set of parameters W and B and so just to expand this out this is equal to negative 1 over m sum from I equals 1 through m of the definition of the last function above so this is why I log Y hat I plus 1 minus y I log 1 minus y hat I because it can put square brackets here so the minus sign goes outside everything else so the terminology I'm going to use is that the loss function is applied to just a single training example like so and the cost function is the cost of your parameters so in training your logistic regression model we're going to try to find parameters W and B that minimize the overall cost function J written at the bottom so you've just seen the set up for the logistic regression algorithm the loss function for a training example and the overall cost function for the parameters of your algorithm it turns out that logistic regression can be viewed as a very very small neural network in the next video we'll go over that so you can start gaining intuition about what neural networks do so that let's go on to the next video about how to view logistic regression as a very small neural network