Gaussian Naive Bayes From Scratch in Python (Mathematical)

Key Takeaways

What is going on guys? Welcome back to another from scratch tutorial. We're going to build we're going to implement a gshian naive base classifier in Python from scratch meaning using only core python and numpy and we're also going to discuss the theory and the mathematics behind it. So let us get right into it. [Music] >> All right. So we're going to talk about gshian naive base in this video today. And we're going to implement the classifier from scratch in python which means we're not going to use anything except for core python and numpy. We're not going to rely on scikitlearn or any other machine learning package. We're just going to use numpy to work with arrays efficiently to do some basic linear algebra without having to implement broadcasting and so on ourselves. But besides that we're going to implement the pure math in python and as I said numpy. So we're going to talk as I said about gshian naive base and this is what I always like to do with these from scratch tutorials. I like to start with a section where we go through the theory the mathematics so we understand actually what is happening behind the scenes and then once we understand that we can put it into code which I would recommend as an exercise here. So if you watch the first part here, the mathematics, the theory, take that knowledge, maybe also go to Wikipedia, read up on the math again, and then try to put it into Python code and if you fail or also if you succeed, then watch the second part to see how I did it here. But I think this is a very good exercise for intermediate programmers. Now with the name here, Gshin naive base, we have three terms that are very important. We have gshian which points to a normal distribution. A gshian distribution which doesn't look quite like that but you get what I'm uh what I'm talking about here. We have a gshian distribution behind the scenes a normal distribution. We have naive a naive assumption and then we have base because of the base theorem. So for this algorithm to understand it, you need to have some basic understanding of probability theory and statistics. You need to know what base theorem is. You need to know what probabilities are and how they're written, what the individual symbols mean. You need to know what it means for um features or variables to be independent. And you need to understand what a normal distribution is and what a standard deviation and the variance and a mean is. If you don't know that, I recommend reading up on the basics. But we're going to require these things as a basis here for this uh gshian naive base algorithm and classifier. So the base theorem is actually quite simple. We have a probability. Let's say the probability here is that we have some uh class that we try to predict. We want to know what is y. What is the class that I'm trying to uh what is the class of the data that I provide you with here. And the probability that this class is equal to some class K given our input data X is equal according to base theorem to the following thing. It's equal to the probability that Y is K regardless of the data. So this is a prior probability just in general. How likely is it for Y to be equal to K without knowing anything about the data yet? that times the probability of seeing the data X if we know that Y is equal to K. So that is like the reverse of that and that whole thing divided now by the probability to see the data X in general. Now this is ordinary base theorem. Nothing too fancy here. This is not naive base. This is just a base theorem. But based on that now we can proceed to build this gosh naive base algorithm here. Now one thing that we're going to ignore for the most part is this lower thing here the px. Why? Because what we're interested in here is comparing probabilities. Gosh naive base works with computing two probabilities and then just siding with the greater one. So if you have for example uh let's visualize this in a simple way. If I have here some data and with gshian naive base we have continuous data. So there's also multinnomial naive base and different distributions that you can use but this one allows us to use continuous data. And with this data what we can do now is we can do a classification. For example, is my data yes or no? This could be something like will the person pass the exam? Yes or no? or is this data pointing to a dog or a cat? Maybe in the case of an image or in our case what we're going to do is we're going to use a breast cancer data set. So we're going to classify uh tumors to be malignant or benign. And how we do that is we compute a probability for this class. We compute a probability for this class and then we compare and the greater one wins and is our prediction. That is our basic u idea here. And since we are comparing these probabilities, this lower part doesn't really matter because the data is always going to be the same and we're always going to divide uh by the same data. So or by the same probability in this case. So we can just for the most part ignore this. Now in addition to all of this, in addition to ignoring that, we also make a naive assumption. Now I need one more color here. The naive assumption is that this thing here in reality is going to be quite complicated because if I know the class y is equal to k then the probabilities of the individual features are not independent. A very very simple example of this would be if I know the height of a person it's not independent from the weight. So the probability that a person has a certain weight does change when I know that the person has a certain height. So just multiplying the um probabilities together only works if they're completely independent features. But height and weight are arguably related. So if someone is very tall, then probably they're also going to have a little bit higher weight. And there's also of course some other stuff. Some other basic example would be to go to the extreme. Now someone likes sugar. This is not independent from whether they like cake or not. Now maybe it's not the same, but there is some relationship there. But we do make a naive assumption here for naive base. And this naive assumption is that we can just say the probability of x occurring when y is equal to k when we know the class we just say that's the same as the product j starting at one going up to d which is the number of features. So that is the number of features um the product of the probability of a specific feature occurring under y equals k. So we just take the product of all the probabilities which only works if we have independent features. Um so we're multiplying them together. That's naive assumption that this is true. And with this naive assumption now we can take the formula and rewrite it as follows. We can say the probability of Y being equal to K given the data which is of course what we're comparing right we're getting this here for one K and then for the other K and then we're just looking at which one is larger but that thing we just say is proportional now this is proportionality here that this thing here is proportional to P Y is equal to K so that is just a prior probability here Y being equal to K and that thing times this product here. This uh J1 up until D product uh like this J1 up until D um of the probability that XJ the individual feature is what it is when Y equals K. Of course again in reality this doesn't make a lot of sense because the probability changes if I know that a person is very tall. The probability for the weight feature is not unaffected by this. And now what we also do since this is a gshian base is we take this probability here for the individual features and we model these according to a gshian distribution or you could also say a normal distribution and there is a fancy n that you need to draw for this something like this. I'm not sure if that's correct. Uh but it's just the probability that XJ occurs as the feature value given the mean of this particular feature for the class. So K for the class, J for the feature and the uh variance for that feature K J and then here squared because this is the standard deviation standard deviation squared is variance. So we are saying that the probability of a feature occurring given that the class is K is the same as taking this feature or calculating the probability of this feature value for the mean and variance in a normal distribution giving the mean and variance of the respective class. So we all base this uh we base all of this on existing data. So think about it that way. We have our data set and what we do is we just say okay the prior probability is how often does the class occur out of all the examples. So if I have 100 examples and 30 of them are class K, I have 30% probability that this is going to be the class without data. And for this normal distribution here, I just use the mean from the data, the mean of feature J within all samples of or all instances I should say probably of class K. And the same here for the variance. Right? That is the idea. Now to put this now into mathematical terms even further, this is the same as 1 /<unk> of 2 pi uh variance for KJ * e to the power of xj minus the mean of j k squared divided by come on I need the line tool here divided by 2 * the variance KJ. That is the exact same thing just written as a formula. This is what we have here on the left just written in exact mathematical terms. So here now I only need to plug in the variance the variance again the mean the actual value that was observed and there you go we have the probability. So we can put this into Python code. Uh for this here for the prior probability it's very very simple. All we need to do here is we need to say we take the number of instances with class K divided by the number of instances. And maybe to finally define this uh correctly, let me take the yellow color for this. Um what is the mean? So maybe we can get an arrow here for the mean. What is the mean? The mean for KJ is just the same thing as 1 / N K since we're getting the mean. So we need to divide by the number of instances times from I where the class is equal to K. So all the instances where the class is equal to K X I J just sum up all the values divide them by the number of features of this class. And for the variance here we just say k J^ squar it's just the same thing 1 over N K * that I is again where the class is equal to K. And here now we just do the standard deviation. So minus minus the mean of KJ squared for the variance. So we basically have all the ingredients and this already makes sense. We have of course the features. We have of course the prior probability because we have data. This is all based on the data. We can also calculate the mean based on the data and the variance based on the data. So actually we're done here. However, we're not quite done because we also need to consider something called underflow. Multiplying very small probabilities together leads to an underflow which means the values get so small that computationally speaking in computers they don't make a lot of sense and the changes don't work. There's numerical instability here which of course mathematically is equivalent. So in terms of pure math it doesn't change anything but in terms of computation on our computer it does change something. So we need to work around this. We don't want to multiply small probabilities together because they get super small and meaningless. So what we're going to do is we're going to apply the logarithm to turn multiplication to addition and thus resolve the problem. So we take this beautiful part here. this yellow part or this white part I should say and we rewrite it with the logarithm. So we just take log of this. So we say log of p y = k given x the logarithm of that is equal now to the following or proportional I should say to the following the logarithm of the prior probability. So y is equal to k plus the sum. Now the product turns into a sum because we're working with logarithms. j = 1 up until d. And of course we take the logarithm of the probabilities. So xj oh sorry I forgot a p xj y = k. There you go. That is now our new formula. And of course we can now take that and turn it into actual uh numbers and values by using these formulas here. Well I would say that the interesting part here is just this thing on the right. So let's just ignore this because this is just a logarithm of the probability. But maybe as a context here in general what are we doing here? We're predicting we're predicting the correct class. This is a classification algorithm remember. So what we're trying to do is we're trying to arc max. We're trying to find something so that um we maximize the following term here. So we try to find a K the class K to maximize uh the term here to maximize this probability here to maximize the lock props. That is what we're doing here just as a context. So this part here is very simple. We just have the logarithm of the prior probability. We can ignore that. The interesting thing here is uh the normal distribution part. So what we do is we have here the sum. So actually we're only interested even in that part. So in the small logarithm part because that is the same as the logarithm of so what is this probability? It is the normal distribution. Right? So we have our fancy normal distribution with XJ here as the value with the uh mean for KJ and the variance KJ that is what we're trying to uh to calculate here and what this means is it's the same as this huge term that we had before this little thing here. So again 1 /<unk> 2<unk>i variance 2 * e ^ of -xj uh so the value minus the mean squared and then divided by 2 * the variance and the interesting thing is we can now calculate this even further. First of all this whole thing here can become a sum. So we can actually actually do it like this and turn this into a sum here. That is possible. This is just logarithms and then we can treat them individually. So what do we have here? We have the logarithm of um and then we have 1 over so that's the same thing as saying 2 pi variance kj and then that thing to the power of 1 /2. So that is the same thing as 1 over square<unk> right. And because we have a logarithm what this means is we can extract this out as a factor. Uh but let's do the the right side first here. What do we have here? We're working with a logarithm base e. So this gets cancelled and we have basically just the uh term negative. So actually we can remove the plus here minus and then xj minus mean kj squared divided by 2 * the variance. So that would be kj here again. So what we have here is we have -1 /2* logarithm of 2<unk>i variance for kj minus and then basically that's also 1 /2 but we can just keep it like this. So XJ minus mean KJ^ squar / 2 times the variance that is the final thing. This entire thing here plus the prior that is what we're looking for. This is what we're trying to maximize by choosing the correct K. So what I'm going to do now I hope you understand this. Um maybe let's briefly recap just as a context here. What is our goal? We're trying to figure out what class is correct uh the correct classification for the data. We get some data. In our case here for the breast cancer data set, it's going to be lengths and widths and uh depths or something like just continuous value describing the shape of a of a tumor, I think. And our goal is to figure out what the correct class K is to label this data as. And we base that on the data that we already have. So we look at the existing data. What we do with this existing data is we calculate the mean for each class. We calculate and for each feature as well. So for each class and for each feature we calculate the mean value and the variance. We also calculate the prior probability for each class. So how often does it occur in the data set in general? And then what we do is we use a normal distribution. the formula here for a normal distribution to estimate all the probabilities. We multiply them together here with the prior probabilities but due to numerical stability or instability concerns we do everything here as a logarithmic um as a logarithmic probability. So we add them together and that is the thing that we get as a result here. This plus the priors and this is now what we're going to put into code. We're going to see it works with the actual data set. We're going to get a high accuracy rate. All right. So now let us get into the practical part. We're going to take the mathematics we just discussed and we're going to put them into code. For this I'm going to go to my tutorial directory and in here I'm going to create a main py file. Now before that what you want to do is you want to install numpy onto your system. You also do want to install scikitlearn but not because we're going to use it for actual machine learning stuff. We're just going to use it to get the data set so we can evaluate our model on the data set and we're going to use it for basic stuff like train test splitting. So not for any machine learning stuff but for the pre-processing and for the data set itself and also for the evaluation. So what you want to do is you want to use pip or pip 3 install and you want to install numpy. You want to install uh scikit-learn. These are the two packages. In my case what I'm going to do is I'm going to set up a uv project. So, UV init UV add numpy and scikitlearn just because I don't use my um system Python installation. It's just not considered a good practice on my operating system which is of course Arch by the way. And now what I'm going to do is I'm going to go into the main file and I'm going to start with the imports. We're going to just import numpy SNP and the rest is going to be imported once we're done with the implementation. So I'm going to implement this as a class. I'm going to call this class gshian nb in the same way it's called in scikitlearn. And in this class now we're not even going to define a constructor because we don't need to pass any arguments. Theoretically you could pass a value for variance smoothing like a very small value for numerical stability. For the sake of simplicity I'm just not going to do that here just to keep the uh code concise. I'm going to start right away with the fit method. So this is going to take self x and y. So x is the matrix with the data. Y is the actual uh correct classification. And we're now going to train our model based on the data that is provided. So we're going to first turn the data into numpy arrays. So npas array x whatever the data is. So np as array y as well. Then we want to know what is the set of classes. So what possible classes can I have? And for this we're going to say self.classes underscore is going to be equal to np unique. So get the unique values from y. And then we want to know the dimensions. So how many classes do we have and how many features do we have? Because based on that we're going to create our arrays with the uh means and variances. So I'm going to say here n classes n features is going to be equal to length of self.classes classes and X shape index one. What this means is Xshape index zero would be the number of instances but we're interested in per instance how many features. So that's the second dimension index one that gives us the number of features. Now we're going to prepare empty numpire race with or for the priors for the means and for the variances. So self dot meanans underscore are going to be equal to np.zeros. Now for each class and for each feature we want to have means. So we're going to say n classes n features. We're going to have the same thing also for the variances. And now for the priors we don't need that dimension because for the prior we only need to have one probability per class. So, np zeros and classes. And now we iterate through all the classes. I'm going to say for index and uh class k. So for index and class k in enumerate self.classes. So we iterate over the classes and we also have an index. I'm going to create a matrix x k which is all the data. So all the input data where the correct classification is k. So where y is equal to k. And now very simple I'm just going to say self do meanans is going to be at this position. So self means index is going to be equal to the mean using axis zero of this particular set of data. And I'm going to do the same thing here for the variances underscore is going to be the var function. And for the priors, it's even simpler. We just need to say the uh number of instances that we have uh for this class compared to the total number. So x k dotshape this time index zero because we're talking about number of instances not number of features divided by x k or sorry xshape zero. X k is only the instances with a class k and x is all the instances. Once we have this done, we just return the instance. So, self. All right. And now comes the mathematics. This is now the only thing that we need to do uh based on our great sketch here. So, we need to put this whole thing here into the formula. But basically, we now need to take these uh lock formulas here and put them into Python code. So, I'm going to do this here. I'm going to say there is um a function here called underscore log gshion self given some data x and we're now going to apply the formulas. So let's maybe start with the right side here with this xj minus the mean squar divided by that. So what we have here basically is let's call this num is equal to0.5 which is 1 /2. Here we have an imaginary one and here we have the two. So 0.5 times and then we have this uh thing. However, we need to do this now uh in the numpy way. So we need to take x but we need to say give me the first dimension. So all the samples give me none. So no classes here or not considering the second dimension here. And then give me everything again for the features. And we take that and we take that minus the means. So self means actually I also need to put that in parentheses here. Self means underscore squared. That is the top part. We now also need to divide by the variance. So we take all of this divided by self dot variances. That is that. Now let me center this. And the second part now is this on the left here. this again 0.5g0.5 times logarithm for this uh we're going to use numpy we don't need math so I'm going to say here for the log prop I'm going to say0.5 or actually we should consider num already so we have one part already it doesn't matter if I take the right one minus the left one or the left one minus the right one because they both have a a negative sign in front so I'm going to say num minus 0.5 time np log of 2 * np.pi pi I hope this exists. I I think so. Uh times self dot variances and very important we also need to specify axis is equal to one. And since we have all these log probabilities of course in the beginning the probabilities had to be multiplied. So we had a product but here what we do is we just return log prop dot sumaxis equals 2. So let's get rid of the graph here. That is basically the math that we discussed that we derived. This is the lock gsh in here. And the only thing left to do now is the prediction logic. So def predict is going to be using that. We're going to get some x's input. We're going to say x is going to be turned into an array if it isn't an array already. And then we're going to say log likelyhood is equal to self.log gshian x. And then we say the log prior probability. We didn't have that yet. Remember we just implemented this bottom part here. We didn't talk yet about the prior probability which is nk / n. So the log prior probability is equal to np log self. Priors we calculated that as well. And now we have these two and of course we need to add them as well. So we need to say self or return self.classes. Now we can do that in one go. So we can choose the class where this probability is the highest by using arc max. I can just say np arc max. The following I want to maximize the lock likelihood plus the lock prior probability on axis one. And that is basically the thing. Which class do I have to choose to maximize this sum here? this probability uh sum of the likelihoods of the lock likelihoods and the prior probability that is the whole logic. This is a gshian naive base implementation. Definitely there's a mistake in there because I don't record anything without having one typo or more in my code. So I'm sure this is not going to work first try. But regardless what we're going to do now is we're going to import from scikitlearn the data set and also some evaluation mechanisms just to see if this works. So I'm going to say from scikitlearn data sets we're going to import the load breast cancer function from sklearn dot uh was it model selection we're going to import train test split and then from scikitlearn metrics we're going to import the accuracy score I think right Yeah. All right. And then we can use this just like any other classifier. So we load the data XY is equal to load breast cancer. And we're going to say return X Y is equal to true to not get this as a data frame. And then we're going to do a train test split. X- train X test Y train Y test is going to be equal to a train test split performed on X and Y with a test size of 0.2. I need to set up autocomp completion here in my new vim. Um been too lazy to do that. And then we're going to say here classifier is equal to gshian naive base and we're going to fit that directly on x-ra and y train. And then we make some predictions on the test data. The Y predictions are the CLF predict on X test. And then we compute the accuracy score which is going to be uh which one was the first one? I'm not sure. Was it first prediction then test or was it test and then prediction? Let's just do it like this. I'm not sure if this is the correct order but it should not matter too much. Now I can just run this in my case by doing UV run in your case just by doing Python or Python 3 run uh not run Python or Python 3 main py in my case UV run main py and what I get here is a problem of course because we have priors and um I didn't include the underscore as it seems and we also have no access for the logarithm where did I do that oh yeah obviously we don't need an access here besides that they Go. We get 93 96% accuracy. 92% accuracy. 92 89 91. There you go. We implemented gosh naive base from scratch in Python. So that's it for today's video. If you want to see more of these from scratch tutorials, let me know in the comment section down below. Let me know which algorithms you're interested in the most. And also don't forget to hit the like button to subscribe to this channel and hit the notification bell to not miss a single future video for free. Other than that, thanks much for watching. See you in the next video and bye.