Logistic Regression from Scratch - Machine Learning Python

Key Takeaways

[Music] how's it going guys I hope you're doing awesome so in this video we're gonna do another machine learning algorithm from scratch and this one is very dear to my heart if you watch the introduction to my videos and so I'm talking about the the famous logistic regression and in this video I'll show you how to implement it and try to show the relevant derivations but I'll also link to to another resource if you really want to understand more about its mathematical derivation and this is more of a implementation video but with that said let's get started so we're going to start with import numpy as NP and then we're gonna do from SK learn that assets' we're going to import make blobs so this is just for an example data set so that we can make sure that it's working so we're going to do is we're going to class logistic regression and we can start with doing the skeleton code of what we're gonna have so we're gonna have in it and I guess this is going to be quite short we're gonna have fun in it and we're gonna have defined train and we're gonna send in x and y and yeah we're gonna have two more so we're gonna have predict then we're gonna have defined sigmoid and that's pretty much it and then we're gonna do if name equals main we're gonna call the the function from here so we let's do pass on this for now and let's start with the init function so just briefly we're gonna have the important sort of we're gonna define learning rate etc here number of iterations to train here we're gonna do the actual training step and the prediction predict will be after training we're gonna use that function to predict what what our class labels will be and and the sigmoid is just the nonlinear function that we're gonna use so let's do the inner self that learning rate is learning rate and we're gonna send in those here as well so we're gonna send in the training set X I'm gonna send you learn right and just set a default of 0.1 and then we're gonna have number of iterations to be another 10,000 quite arbitrary but and then self the number of hitters we're gonna do number of eaters and from the data set it's gonna be structured in a way that we have the training examples of the rows and the the number of features along the columns so we could do self dot M self dot n is X shape where m is the number of training examples so we can write that to be clear M for training examples and for features and I guess for a number of training examples and number of features and that's all we need to define and in it so I guess we can start with doing the this sigmoid so the sigmoid is just I'll show the mathematical sort of the formula for it but all we're going to do is return 1 divided by 1 plus and px of minus Z and that in this case just gonna be W transpose plus B W transpose times X plus P we're gonna come to that a little bit later and yeah so let's do the joint in the training now we can do self dot weights we're gonna define wait to start with MP numpy is zeros and it's gonna be soft of n comma 1 so it's gonna be a column vector with the dimensionality will just be the number of features that we have to to match and then the input X and then we have subt of bias we're just going to find a 0 after initializing the weights we're gonna do the training loop so for iteration in range of self num hitters ya plus 1 then we're gonna start with calculating the hypothesis hypothesis and we're gonna do y predict is gonna be self dot sigmoid right and we're going to MP dot of X self dot weights plus F Tobias so as I said before Zed is gonna be W so the formula is w transpose X plus B but remember here that it's that's for a single training example and what we're doing here is we're doing a vectorized variant where we're sending in X to be a matrix for all of the training examples at the same time so we sort of have to adapt the formula to to fit to this more vectorized implementation but it's exactly you can do it with loops and it's gonna follow the formula that I showed previously but doing it vectorize that's how you want to do it and then we're gonna do the cost which is just going to be following the formula I guess we're going to do minus 1 over self that m and then NP some of Y element wise multiplied with n P log of the prediction and also show the the formula for it on the screen so you can compare and then we're gonna 1 minus y times NP log of 1 minus y predict so there should be nothing strange here I think so we can write here calculate cost and after that I guess the most difficult part is the the back prop or the gradient the the derivatives so all we're doing here is we're just differentiating the cost with respect to the weights and the bias and these formulas if you do them and I'll link the resource for it and you can go through it yourself but they come out to be these formulas right here and there's gonna be two differences to what we're doing so what we see the formats that we see on the screen are for a specific feature of the weight vector okay but we're gonna do it in a vectorized variant so that we get the vector instantly without using a loop iterating through all of the features so J would be 1 up to n we're gonna skip that part and we're just gonna do it vector eyes instantly but we're also gonna add another thing which is that we're doing it for all of the training examples at the same time so we're doing it for altering examples and then we're taking an average of that gradient to get a to get sort of a better gradient in gradient direction so again I'll link it below if you want to go more in-depth on your formulas this isn't this is more of the implementation so I'm sort of assuming that you know some of the theory so we're going to back prop DW is gonna be 1 over self dot M times MP dot of X transpose and then y predict minus y so that's that's the formula for the cost with respect to W in a vectorized way and then DB is going to be 1 over self dot M times MP sum of y predict - why yeah so you might have to spend some time really understanding these formulas but with those now written it's gonna be quite easy so what we're gonna do is we're just gonna update the step set up weights to be minus equals set up learning rate times this this gradient so we're using gradient descent here and similarly for the bias and then we can do something like if iteration is 1,000 so modulus 1000 rather we can do something like print with F string cost after iteration iteration and then it's cost and all we want to return from this at the end after the training loop is just the weight and that's all we care about and after the training we're gonna do a prediction probably on a separate set so the test set or something and that's where the the predict function comes in so this is going to be we're going to I predict to be self that sigmoid of MP dot of X comma self that weights plus SEP device so following this right here right this is just copy pasted from this and then we're gonna do y predict labels is y predict we're gonna do greater than 0.5 so if it's strictly greater than 0.5 it's gonna guess that it class is 1 if it's below or equal or point 5 it's gonna guess the class is 0 and again remember logistic regression is for two classes so you would use the logistic regression if you have two classes and you can of course expand this using one one versus all method or expanding it using a multinomial logistic regression but we're just doing sort of the basic one here return y predict labels and yeah so now the the it's uh we're done with the logistic regression and we're just gonna make sure that it works and how we can do that is because that I random seed first of all so that if you try to follow the video you you should get the same results hopefully and X comma Y will be make blobs of number of samples we're gonna just send in a thousand and centers to be two again we use centers to be too because logistic regression is for two classes it's a binary classification algorithm and then we're just gonna do Y and then NP new axis just to make the the dimensionality work for what we want in this algorithm so I guess this would be instead of having a thousand comma we want it to be an actual vector of a thousand comma one and yeah just as a small detail I think it's because if I remember correctly it's because we were doing this element wise multiplication and the dimensionality needs to to be correct in in those multiplications well yeah so let's do log distribution to be logistic regression of McKesson in X and W B will be log Gregg dot train and we're gonna send in X&Y again then we do y predict log reg predict of X and I guess in this case we're just using the same data set you would normally have a training and as I said of course but we just want to make sure that it works and then again we'll do print accuracy and we can do NP dot some of Y is equal to Y predict and then we can divide by X shape of 0 which will be the the number of training examples and if we run this now this should probably be working hopefully if we don't get any errors see so we can see that step it's the loss is decreasing quite nicely and the accuracy is 100% so yeah it's probably overfitting but at least we can see that the the algorithm is doing something that seems to work you you could also implement a a regularization term included and that could be one step to add on this algorithm but hopefully you were able to follow this this tutorial industry progression yeah thank you so much for watching video and if you have any questions then leave them in the in the comment section below [Music]