How to implement Perceptron from scratch with Python

Key Takeaways

welcome to another video of the machine learning from scratch course presented by assembly ai in this series we implement popular machine learning algorithms using only built in python functions and numpy in this lesson we learn about the perceptron algorithm so we start with a short theory section and then we jump to the code so let's get started so the perceptron is an algorithm for supervised learning of binary classifiers it can be seen as a single unit of an artificial neural network and is also known as the prototype for neural nets so if we use only a single unit then we say this is a single layer perceptron and this can learn only linearly separable patterns on the other hand if you use a multi-layer perceptron then it can learn more complex patterns but in this lesson we focus on the single layer perceptron and this is inspired by neurons so it's a simplified model of a biological neuron and it simulates the behavior of one cell so let's say here we have our neuron and then it gets an input signal and then the signal travels along the way and if it reaches a certain threshold then this cell will activate so we say that the cell fires and then it gives an output signal so if we transfer this to our mathematical representation then we say it's a single layer neural network with the unit step function as activation function so we have the inputs and then we multiply this with some weights then we sum this up and then we have the actuation function and then we have the output and in this case the activation function is either one if the cell fires or zero if it doesn't fire and in our case one is the class label one and zero is the class label zero so now let's put this into a mathematical model so here we have the inputs times the weights so we have a simple linear model we approximate f of x with w times x plus a bias speed so these are the weights times x plus the bias b and then we also want the actuation function and in this case this is the unit step function which is very simple so this is one if the input reaches a third and threshold and zero otherwise and this is the whole concept so with this we approximate the class label with the linear model w times x plus b and then apply the actuation function the unit step function and now we need to learn the weights w and the bias b and we do this with the perceptron update rule and this is a super cool intuitive rule i will show you what i mean in a moment so for each training sample x i we say we update the weights we say the weights plus delta w and for the buyers the same the bias plus delta bias so now what is the delta so for the w it's alpha times y i minus y hat i times x i and for the bias it's the same except this part so and alpha is the learning rate so this is between 0 and 1 and this basically decides how far we go into this direction and then y i is the actual label and y hat i is the approximation um with this formula so let's explain what this part here means let's look at the different cases so we have the class labels one and zero so our binary classifier so if both y and the approximation y hat are one then this is correct right and then y minus y hat is zero so there is no change and the same if the um actual label is zero and the prediction is zero so again we have the correct prediction then we have y minus y hat is zero so again no change and if our actual class is one but the prediction is zero then the difference is one so this means that the um the prediction is too low so we need to increase the weights and the other way around if the label is zero and the prediction is one then this means that the prediction is too high so that the weights are too high and we need to decrease this so basically the weights are pushed towards the positive or negative target class in case of a misclassification and if the classification is correct then we don't need to update the weights and this is the beautiful perceptron update rule a very intuitive model and this is all that we need so let's summarize the steps so in the training part we want to learn the weight so we initialize the weights and then for each sample we calculate the approximation with the linear model and the unit step actuation function then we apply the update rule so the delta w is alpha times y minus y hat times x and delta bias is alpha times y minus y hat and then we learn the weights and then we in the prediction part with the test data we simply calculate y hat as again this linear model with the actuation function with which then gets one or zero so this is all that we have to do so let's jump to the code so first let's import numpy s and p and then let's create our perceptron class this gets an init function and here we give itself and then the parameters are the learning rate and let's give it a default of point zero and then we also give it an iter so the number of iterations for the optimization algorithm and let's say this is 000 in the beginning then let's store this and say self lr equals the learning rate and self dot n iters equals n headers then we also want to store the activation function and for this let's say this is the unit step function and for this we create a global function for example this could also be in a utility module so we say define unit step func and this gets the input x and then we can do this in one line so we say numpy where x is greater than zero we return one and otherwise zero then we also want to get the weights so in the beginning these are none and also self dot bias and this is also none in the beginning then we want to implement the to fit and predict functions so define fit with self and x and y so the training data and then we do define predict with self and also x so this is now the test data so let's start with the fit function so in here first we get the number of samples and the number of features from the training data and this is x dot shape so here we assume that x is a numpy and d array and then the first thing to do is to init on the parameters so for this let's say self.weights equals numpy zeroes with the shape and features so this is the simplest way to do this this is actually not the best way a better way to do this is to randomly initialize them so i challenge you to do this on your own instead of nump zeros try to use a random initialization but um in our case in our test example this still works pretty well so we can also do this and for the bias self the bias this is zero um this is also zero in the beginning and now we want to make sure that the class labels are one and zero and that for not for example one and minus one so we say y underscore equals and then we can again use this numpy where so we say this is numpy where y is greater than 0 then this is one and otherwise zero and then we do the optimization or let's say learn weights and now we say for underscore because we don't need this in range self dot n iters and then we iterate over all the samples so we say for index and x i in and numerate and then here x so enumerate gives us both the index and the sample and then we approximate this so we say linear output equals numpy dot of x i and self dot weight so w times x and then plus self dot bias then we say y predicted or y hat equals self dot actuation function of this linear output and now we want to apply the perceptron update rule and for this let's have a look at the formula again so the update rule is the delta w is alpha times y minus y hat times x and the delta bias is alpha times y minus y hat so this part is the same so we call this let's call this update equals self.lr times and then we say self and not self y underscore of the current index minus y predicted so this is the update part and then we say self dot weights plus equals the update times x i for the dot weights and self dot bias simply plus equals the update part and this is all that we need so now we are done with the fit method and now for the predict method we simply again do the linear output so we can now actually copy this so we say linear output equals numpy dot and here we can put in the whole x and then we do the actuation function so again y predicted equals self actuation function and then we return y predicted and now this is all that we need in order to implement the perceptron so now let's test this so i already prepared the code for the testing you can find this on github i put the link in the description so let's go over this very quickly so we import matpotlib and then train test split and data sets from sklearn then we create a helper function for the accuracy then we create a test set by saying data sets make blobs with 150 samples and two features then we split this into training and testing then we create our perceptron with a learning rate and the number of iterations and call perceptron fit with the training data and then predict with the test data and these are our predictions so then we print the accuracy by calling accuracy with the actual labels and the predictions and now i also want to plot the decision boundary here so for this you can use this code which uses the weights and the bias to create at the decision boundary so if we run this then we should see the plots so yeah these are our two blobs and this is the decision boundary so in this case it perfectly learned a decision boundary and now we can see the accuracy is one 100 so everything was correct so our code is working and this is how we implement the perceptron i hope you enjoyed this lesson and then i hope to see you in the next one [Music]