Naive Bayes Classifier Explained | Machine Learning | Community Webinar

Key Takeaways

all right so i think now kind of seems like a good time to start mostly because i'm all settled in um so thank you everyone for joining today my name is nathan i am one of the marketing managers here at data science dojo um and and robert it's an hour early all data science does webinars play one hour too early in arizona i'm sorry robert um i think that's just a time zone thing right maybe we can do one one hour later like at one o'clock pacific sometime and then it will be the perfect time um anyways uh thank you all for joining today um we're really excited to be having this crash course on bay's classification with with our our presenter kevin and kevin is the chief statistician data scientist and senior research fellow at the heritage foundation center for data analysis which is a mouthful to say kevin uh so just a couple quick things um for q a today um we're going to be using the q a tab like we normally do so if you have a question and you're on zoom go ahead and post that question in the q a tab if it's kind of if it requires kind of a shorter quick response kevin's going to answer it kind of as we go um but if it's something that's going to take a little longer we're going to save those and answer them at the end or um kevin's final slide he has some ways to get in touch with him um you can get get in touch with him with your question he'll do his best to get back to you this webinar is going to be slightly longer than our normal ones normally we do these for about an hour this one can may go up to an hour and 30 minutes so if you have to leave early and drop off because of work or some other prior engagement feel free to do so we will have a recording of this and we'll have this be able to share the slides and everything as well so stick with us if you can if not you know we're happy to have you while we have you and um feel free to jump off if you're on one of our live streams um i have posted the zoom link if you want to join us on zoom i'll post it again here in a minute uh but if you have questions feel free to post them in the chat i'm gonna do my best to bring them on to zoom with us so that uh you can have your questions answered as well and with that so kevin why don't you go ahead and get started share your screen and then um we'll get going here okay um thank you nathan uh my name is kevin diarotna i am the the chief statistician data scientist and senior research fellow at the uh the heritage foundation washington dc so i'm gonna talk and i will share my screen let's see here let's do this about a crash course on naive bayes classification there are various types of problems that we might be interested in as statisticians and data scientists namely making new product recommendations predicting political affiliations predicting medical conditions uh among many others um in the realm of predictive modeling these are a variety of questions as well as many more that can be answered using the techniques that we're going to talk about today and uh we're going to get to specific examples of these things um later on in today's webinar so first i'm going to do a very very brief probability refresher um and if you don't have any probability or stats background that's perfectly fine i'm going to talk about the basics that we need for this and then i'm going to get into an introduction to naive bayes modeling both from a theoretical perspective and a coding perspective the coding that we're going to do today will be in the r programming language um so yeah feel free to the codes will be download available to you so you'll be more than welcome to uh to utilize them so i'm first going to talk about a probability refresher so probability is a concept that enables us to quantify how likely certain events are to occur such as and like in terms of a random experiment where you have different potential outcomes for something such as for example tossing a coin rolling a die tossing coin heads or tails rolling a die what the side will land on one through six there are variety of outcomes those are the outcomes say for example rolling a die one through six that you can end up with and the idea of probability is to quantify how likely some of these outcomes or all these outcomes are to occur so the basics so i've taught probability and statistics for many years the idea is when i when i when students come in they typically quantify before we have our course how likely certain events are to occur between with percentages between zero and a hundred probability is the same idea but the the end points are zero and one zero is no chance that the event will occur point five is the event is just as likely as it is unlikely and one is the ev that the event will occur with certainty or 100 chance so probabilities quantify the likelihood of something occurring between 0 and 1. now there are excuse me there are a variety of ways we can assign probabilities we typically speak about the probability of an event say event a occurring for example going back to the example about rolling a die one through six the probability of the die landing on six occurring or one occurring i should say or six for that matter is defined as the number of ways that event can occur over the total number of outcomes so in that case the probability would simply be one out of six there are a variety there are two main ways that people conceive of probabilities one is through a relative frequency and the latter is subject subjectively let me talk about the relative frequency concept because that's basically what we're going to be using here although the concept naive bays touches on subjectivity as well suppose you have a coin and you toss it ten thousand times roughly speaking five thousand of those will be heads and the idea is is that in this sense probabilities would quantify a long run frequency based on a random experiment subjective probability is like the term says more subjective or opinion based sometimes you don't have the luxury of engaging in these long-run experiments to generate a long-run frequency for example what is the chance that it will rain tomorrow and nathan i were just talking about rain in say seattle the the probability of that you you can't have a million tomorrow tomorrow's a sample from so you would use other techniques to subjectively assign that probability when you turn on say the weather channel you will hear me there is maybe a 30 chance of it rating tomorrow so for any event a p of a right here is denoted as the probability that the event a occurs and that's between zero and one s is known as the sample space or the set of all possible outcomes it is also sometimes denoted by omega the greek letter so here's an example you have an example of here's an example of a bag with ten balls one is labeled five two are labeled six three labeled seven four labeled eight and let's let a be the event that a chosen ball is labeled less than seven so what i'm to do is i will tell you how we would compute this because this is just a very basic example that i want us to get a get a hold of so uh so p of a equals so you go to the slide here one is labeled five two label six three are labeled seven and four label eight so you have ten possible balls but one two one and two are five and sixties are both less than seven so the probability of now in the concept of probability we have an event a and the sample space consists of all possible events so and this comes up in naive bayes modeling the probability of the sample the sample space is one so the probability of a bar or the complement of an event the opposite of that event occurring for example not rolling a six having choosing a ball that is greater than seven or greater than or equal to seven would be the opposite of event occurring and that's just one minus that associated probability now the we have the probably we talk about the events event probabilities of events occurring in terms of these set theoretic notations here the probability of a union b is the probability of a or b occurring that's how this is read and you could visualize it this way and this is what we get involved in with naive bayes mounting the probability of two events occurring p of a and b and that is the intersection of two events now the joint probabilities is one way that we define the probability of a and b uh that the probability of two events occurring together or multiple events occurring together is known as a joint probability a marginal probability is the probability of a or b or a single probability hey kevin yes sorry are you meaning to be sharing your slides i'm not oh i'm sorry i thought i started sharing it again no thank you let's do that sorry okay we'll go back here so let's go over this real quick this is these are the slides i went over so the complement of an event this is the visual representation of what i was referring to earlier that's basically everything that is not the event the sample space consists of the event and everything outside of it so if we're interested in the complement p of a bar that would be 1 minus the probability of a now we're interested in events probabilities of different events occurring a union b that is the probability of a and b occurring and you could visualize it this way the probability of a intersect b that is or probability of a and b that is the probability that two events are occurring together and that is also known as a joint probability probably of a and b the marginal probability is the probability of a single uh probability occurring like probably a single vine occurring i should say probably of a or the probability of b now here is some hypothetical data about stem ages finding employment and the reason i include this is just purely for illustrative purposes the probability of a stem major who so this is this is survey data quote unquote data i should say amongst respondents and you can see here that three percent of the students surveyed were stem majors who did not find employment within six months after graduating seventeen percent of them were non-state majors who didn't find influence six months after graduating fifty-six percent were still majors who found employment within six months after graduating and twenty-four percent were non-stem majors who found employment within six months after graduating and using the content of relative frequency probability one could interpret this as saying assuming the data is representative of the general student population the probability of a randomly chosen person being a stem major and not finding employment within six months after graduating is .03 or .56 for being a stem major and finding employment within six months after graduating and so forth so in a nutshell these are joint probabilities and you can see when you add them across the columns they are marginal probabilities 0.2.8 for example of those who don't find employment within six months after graduating that that probability is 0.2 0.8 is the probability of finding six employ finding employment within six months after graduating and you have these probabilities of being a stem major and not being a stem major amongst randomly chosen individuals this concept is something we directly use in naive bayes that is called conditional probability that is the probability that a occurs given that b has already occurred for example assuming somebody is a stem major what is the probability that they find employment within six months after graduating well this is the definition the probability of a and b this joint probability divided by this marginal probability and in this case it is 0.56 the probability of finding employment and being a stem major divided by 0.59 the marginal probability of being a stem major and that comes out to 0.95 so what does that tell us there is a 95 chance that given someone as a stem major they find employment within six months after graduating now two events are statistically independent this com this concept comes into naive bayes if and only if the probability of a and b decomposes into the products of the individual probabilities and another way of saying this is the probability of a given b does not depend on b and is purely the probability of a and same for probability of b given a and if you have dependent events then you can decompose the definition of joint probabilities this way in terms of the product of the conditional probability and the marginal probability to get this joint probability so now that is all the probability theory that i wanted to share before getting to naive bay so before we go on does anybody have any questions nope okay so so suppose we are shopping at the grocery store and we you know with technology these days we can equip uh sophisticated equipment to grocery carts that could have information about where you're from and provide direct product recommendations so here's a simple example can conditional probabilities shed light on how likely a customer is to purchase a grocery product given known information so here are some data this is from this book i believe it's titled the ancient art of numerati it is available online and shoot me an email afterwards my email address is in the slides if you'd like a link to it here is some hypothetical data about customer purchase behavior and you have the zip code whether they bought organic produce and whether they bought central green tea or amongst 10 customers here are some good questions suppose what is the probability that a particular event d will be observed that is what is the probability that customers who showed up belong to the zip code 88005 that is what is the probability of the event a 8805 occurring and what is the probability that a particular event d holds the given hypothesis that is say for example what is the probability that a given given a particular customer purchased sencha t that they were from zip code 8805. the first one is very easy to do 88005 so there are 10 customers how many are from the zip code one two three four five so we have i will um here five out of ten all right p of 8805 right now this question is a more interesting question at least to me what is the probability that given that they purchase census t that they will be from zip code 88005 so we can compute that using a definition of conditional probability that is that is done on the following slide the probability of being in zip code 8805 and purchasing century t all over the probability of purchasing central t so what is this probability being in 8805 and purchasing center tape well let me go back to the slide here from current slide so how many purchase entity and or from the zip code we got one right here this doesn't qualify this does and so does this so we have customers one three and nine i'm not i'm sorry one six and nine it's a three out of ten so as you can see here the probability is three out of ten divided by five out of ten which is three fifths or point six okay does that make sense so we would be interested in conditional probabilities of purchase intent that is what is the probability that a customer buys sensor t given that they are from a zip code and one's probably that they don't buy center t given there from a zip code for example you walk into a grocery store you have your shopping cart and you can it is a sophisticated shopping cart where you can actually enter in your locality where you're from you put that in then you could think about how to target ads so i'm not going to go into the details here but you can compute these using the techniques we just talked about the probability of purchasing center t and belonging to zip code 88005 over the probability of a of belonging 8805 is this conditional probability of purchasing energy given that you're from the zip code and that comes out to also be 0.6 the probability of not purchasing central t given here from the zip code comes out to be 0.4 so you compare the two the probability of purchasing century t is given that your zip code is 0.6 the probability of not is that purchasing this center taking being from the zip code is .4 so they are more likely to buy than not so therefore it's worth considering posting ads to people from the zip code or in the shopping cart for example providing specific ads to guide into particular areas of the store now i'm actually going to backtrack a little bit here because there is a very very useful way to consider conditional probabilities and this is very useful if you um when you consider the definition and the context of conditional probabilities and this is discussed in more advanced probability textbooks suppose you want the probability of purchasing center t given that you belong to zip code 8805 we showed how this can be computed right here another way to do this is to just look at the zip code 88005 how many are there here one two three four five we counted this before amongst these how many purchase center g excuse me one two um excuse me two three so that is another way you can get to this answer three out of five and i'll leave it as an exercise to you to do the same thing here but you will be able to do the same thing with that probability okay so now suppose we'd like to know how likely it is that a purchase that a person from this zip code 8805 has bought again organic produce and will purchase tea then not this involves a new technique known as bayes there this work was published by the reverend thomas bayes a couple hundred years ago and it was done posthumously i believe this is based there essentially the idea in date with bayes theorem is that it enables us to flip conditional probabilities we may have the condition of the probability of b given a but we need a given b bayes theorem is as follows the probability of b given a equals the probability of a given b times the probability of b all of the probability of a you can derive this using definitions we talked about this is done in most standard probability and statistics textbooks excuse me the probability of a and the probability of b are referred to as prior probabilities and this probability is probably a b given a is a posterior probability now suppose we have multiple statistically independent events a1 through a n and we like to compute the probability of given b all such events of b excuse me the probability of b given all such events a one through a n occur then this tweak to bayes rule would be as follows where you take the product of all these conditional probabilities multiply it by this marginal probability of b and divided by the joint probability of a1 through am and this is what we utilize for bayes for naive bayes modeling and if we want to compare the probabilities of different hypotheses h1 through hm we would do that as follows by computing these different posterior probabilities but you notice something with these equations notice that the denominator is the same so why should we bother to compute it if we keep dividing by this exact same thing what we can do instead probability is proportional to this product times the prior probability of the hypothesis in question i know that's a lot of statistical jargon but we can get to it we can do an example of that here in terms of what i had just talked about suppose for example we'd like to know how likely it is a person from the 8805 zip code who has bought organic produce will purchase tea than not probability of h1 given d is the probability of purchasing central t given they're from this zip code and they have purchased organic produce and we the competing probability is not doing so given the same information so given what we have discussed in the previous slides this is just the product of belonging to the zip code given that they purchased entity purchasing organic products given they purchased center t times the marginal probability of purchasing essentia t and you can compute these i'm not sure we want to spend time going into calculating all of these but we did the first one already 0.6 0.8 is the probability of purchasing organic products given that you purchased central t and 0.5 which we calculated is the marginal probability approaching essential t this comes out to be 0.24 now this posterior probability of not purchasing central t is proportional to the following and it comes out to be 0.05 so you can cut what does this tell us well you can compare the two and it is more likely that somebody from the zip code who has purchased organic produce will buy the sencha tea than not so let's post the ad for the customers okay yes there will be there are no links yet but there will be and the recording will be shared yes so don't worry about that welcome okay so now okay um before i go on i'm gonna stop this let me i'm just gonna stop sharing so okay see i just wanted to touch base because now we're gonna get into some naive bayes modeling and coding are there any questions okay it does what i will do is i might as well just share it right now i have it right here let's see sorry resets dojo i will give you all a link to the r codes that we'll be using and you can run them in rstudio or in our studio this zip file should have everything and we'll get to them in a bit um are there okay so i assume they're not questions okay so let's go on uh i have to i made this mistake last time we're not gonna do it again share the screen hey kevin before you get started it looks like uh steve did ask a question oh i'm sorry if that's a it came out right as you were right as you were getting started i just missed that i don't think i see the question is in the chat in the q a tab oh okay i was getting the chat sorry ah good question what is the probability that they will buy the second time when they return um i would have to think about that because i think you would want data on multiple visits to do that but that is a great question i have done some work on this type of thing when i was in graduate school in the past in particular looking at multiple purchase occasions and modeling things like price sensitivity taking into consideration past prices um there's a concept called a reference price where you have in your head for example when you go to the grocery store and you buy orange juice you'll have the price you paid say two three weeks ago when you purchased your orange juice before naive bayes so gabriel's question i basically calculated only considering two facts the probability for i'm sorry or is it possible to have two i'm sorry i'm not quite understanding the question gabriel can you clarify okay sure oh well it's steve's response gabriel if you want to clarify that uh please do so and then we can um we can talk so parso i asked a question can you do this in python the answer is naive based modeling can of course be done in python but my codes are r codes so they cannot be done in python but as an exercise i do encourage you to translate them to python so yeah i i will need a clarification for your question gabriel but we can um we get that and i'm happy to talk after the afterwards so oh so no yeah okay i i see i see the question here okay so naive bayes can it be calculated only considering two facts the probability for a third one there yes there is a concept called multinomial naive bayes and given the time we have i do not want to spend time getting into it but there are ways to use these methods for multiple outcomes this can these this can be tweaked this is only the beginning again this is a crash course on the topic but yes there this can be extended in a variety of ways to handle more than one outcome does that answer your question okay so is that it okay so let's um move on to the actual knife maze molly and i hope i've given you a good sense of the techniques uh the techniques involved okay here we go okay i'm gonna share the screen again so okay we went through all this let's see here okay here's a simple application this is one of the applications i had alluded to earlier in the talk can naive bayes be used to provide new product recommendations for fitbits this is an example from the ancient art of numerati eye health 100 and ihealth 500 these are different types of devices you could put on your wrist that measure um stuff about your circulation in your heart and so forth so here's the task would you like to devise what if we want to devise a recommendation system for customers and you ask them to fill out a questionnaire what is your main reason for participating in an exercise program health appearance or both what is your exercise exercise level do you sit at home do you have engaged in moderate exercise or you do engage in active exercise what is your motivational level modern or aggressive i guess they don't bother asking people who aren't motivated because they're not going to do anything and are you comfortable with tech devices yes or no so here is this data regarding um what these hypothetical customers filled out i-100 for the i-100 and i-500 and this can be used as the basis for constructing a naive bayes model so we got a coding example in r and we could develop a recommendation to decide which fitbits to suggest to customers let me first go into the theory here so how would we use naive bayes to recommend a model for a person whose main interest is health their current exercise level is moderate they are moderately motivated and are comfortable with tech devices so these are the two competing probabilities that you would need to compute i 100 given that information and i 500 given that information and you can compute these two probabilities accordingly you know what let me say this i've actually mentioned this when i've taught this at universities i'm not the biggest fan of how these probabilities are actually constructed because like you could see here i don't use the term equals i use proportional and the reason is you go back here we have dropped the denominator because the denominator is the same so they are proportional so you are not really computing a probability per se but it is a heuristic for a probability the value you are using computing is proportional to the probability and at least tells you something because then you can compare the different numbers that you are getting so anyway you have this here and we can compute it and this is how you would compute it you compute the probability of health being the priority given that they've chosen this excuse me this conditional probability this posterior probability i 100 given this information the probability of health being their priority given that they've chosen the i-100 that they engage in moderate exercise the probability that they engage in moderate exercise given they chosen the i-100 motivation being comfortable tech devices and so forth of course times the marginal probability and analogously for the i-500 so we'll you know what i don't want to leave everything to you guys to compute but i will go through you know one or two of them what is the probability that given that they chose i 100 health is their priority well we can go back so one two three four five six there are six i 100 let's double check that one two three four five six there are six customers who have chosen the i 100 amongst these which is health their priority this is the alternative way i suggest that you compute conditional probabilities and it's easier there is boom just one so that probability is 1 6. and you can compute these other probabilities as well so you multiply them together and your posterior probability is proportional to 0.00309 analogously you can do the same for the i 500 and your posterior probability is proportional to 0.01975 so the i 500 posterior probability is greater than the i-100 posterior probability so therefore you would recommend the i-500 to people whose main interest is health whose current exercise is moderate they have moderate motivation and are comfortable with tech devices any questions because now i'm going to open up the r code that does this okay so let's do that i will share the screen now this time we want to share our studio so these are the codes for the examples let's go to this one here excuse me so let's do this so i will walk through this with you this is the classifier you this is a function in r that takes into account the interest exercise motivation comfort level and it loads the data which i provided you to you via csv file i will load this for you so you could see it yep let's do this right now so here is the data this is what i want to show you you can see i just put this in into a csv file which can be developed which can be constructed in excel and then you can load it into r using the read.csv function sorry this is the wrong one where is it and we tagged on the titles here so you can see this let's bring this here actually so it's easier to see so then what do we do we find the unique model numbers there are two here i fi 100 i 500 this extracts them and what we do is we compute a matrix of probabilities that pertain to the various conditional probabilities that need to go in our naive bayes model this initializes it and we fill this matrix one by one going through vowel which is the number of unique models and we subset the data based on each model and then we compute the probabilities here that is done right here and this goes on for up to one less than the dimension of the matrix because at the very end of this we tag on the marginal probabilities which we need and boom we compute the product of those probabilities which we did in the slides and output the results and i encourage you to play with these codes uh tweak them do what you need to with them and here is an example of this in action so i will run this again and here we have we load the data and the classifier and then i have an example here here is a person whose interest is health they are moderately motivated this is exactly what we did in the example excuse me they engage in moderate exercise they are moderately motivated that comfort level is yes i can run that right here and as i run that you can see the probabilities associated with the i 100 and the i 500 the final vectors are compute excuse me this final vect this final vector is the choices of the two probabilities and then this line right here in my code which dot max computed the maximum value from this vector and that was i 500. well it could be it it finds a maximum value and then finds the associated device which is i 500 which is what we found in the slides here is are two more examples suppose somebody's interest excuse me is both health and appearance they actively engage in exercise their motivation is aggressive and they are comfortable with devices we compute that control enter x2 and i can look at the results here here are those probabilities and again this person is also suggested to have the i-500 device and again right here here is another example this person has denotes interest is there uh excuse me appearance is their interest they are more sedentary they are moderately motivated but they are also comfortable tech devices and you can see here you run it and in this case you have the probabilities here the final probabilities and this time the i 100 wins and that is a recommended product okay i see some questions here let's see what they are ah okay we will um okay so robert had a question i am currently working in python i have no experience in our just yet at your convenience can you recommend a resource for teaching and demonstrating such models in python yes um i will type my email in the chat and i will send you a book so here's the thing though with this robert i mean these models can be written in any language that you want so i just wrote these from scratch in r they can be done in python but you would typically i mean you one would have to write the models from scratch there so but i think maybe what you're getting at is are there books that have the that basically provide the presentation in python and there are some let me look into it shoot me an email and then i'm happy to answer that um and koshik had a question how does naive bayes take care of noise and the answer to that is that is a very good question and we will get back to questions along those lines at the very end of the talk so hold that thought but yes you you touch on a very good point that we are making a pretty big assumption about statistical independence and that's why they call it naive days but it's not necessarily that naive let's put it that way okay so all right um are there any okay so then parse out another question i get figured but so i'm sorry so i mean you could just um run this in r studio and i hit when i just ran each of these i'll share my screen again alright so the question how did i run these in r what i did was you can just hit ctrl enter in r studio and it will run things line by line and that's what i did and this eye health classifier code that i first presented to you uh that is invoked in eye health underscore nb which you should have and that is the classifier model okay i'm happy to share uh the slides as well but you should have the codes already okay okay so another question with this scissor question would the same technique be appropriate for teasing out what is the primary motivator for purchases of the i-100 um yes that's a good question quite possibly i never thought of that but um one thing you could do robert is play with this data and now you have it and uh look at that but yeah you could train that you can you can use those those potential motivating variables as your outcome variable and see what the model tells you uh any other questions there are more questions coming in okay yes chris and thank you by the way um yes the answer is yes you can classify each customer individually um and we you will see more of that in the next this is again was a very simple example we are going to ramp up the complexity pretty quickly the next example is not going to be regarding a customer purchase behavior though it is going to be regarding um uh political science but that's okay because it's useful to see a variety of different applications here so let me share again we're gonna go back to the slides so we need to okay so here is another simple application congressional voting can these models be used to predict how lawmakers may vote and i think this i thought about this this could be very useful for example at um not just you know nationally but at state and local levels as well as well as in other countries so here is an example suppose we have prior data on votes by members of congress and compute percentages accordingly the cyber intelligence sharing and protection act reader privacy act internet sales tax internet sleeping bill snooping bill this is an example from that book i was referring to and here is that data you got 100 members per party and these are the number of people that voted yes in each party for cispa 99 republicans voted yes for the reader privacy act one did internet sales tax 99 didn't assume bill 50 did you could see correspondingly for the democrats so let's convert these to probabilities which you could do here just by dividing by a hundred and now the question is take suppose you take somebody else and they voted no on cispa yes on the reader privacy act no on the internet sales tax and no on the internet snooping bill what party they belong to well what you can do is you can compute the probability of voting no given that they belong to each party for each bill for for cispa the probability of voting for the reader privacy act um given that they belong to each party the probability of voting now on this one calling out the internet sales excuse me given they belong each party and the marginal probability of each party and then this is the naive bayes probably estimate the product of all these together and as you can see in this case 0.48 is greater than 5 5 times 10 to the negative 7 and as a result in this case the democrat wins so this is a very simple example and note that it wasn't so simple as to just multiply these probabilities together you had to take into account how they voted this per this the person voted no and you have the probability of yes here so this is not 0.99 you're multiplying you multiply 0.01 and in this case you're not multiplying by 0.01 you're multiplying the 0.99 now here's the thing though this is starting to look like a very very useful tool but things don't always pan out so nicely in the real world when you are engaging in statistical modeling and data science let's bring into the picture another bill the internet snipping bill and you notice that in this case the person voted if the person voted no on it you want to include that in your naive bayes modeling if no democrats have voted no on it then you have a zero probability here that knocks the previous probability from 0.48 down to zero excuse me so in this case the republican wins so this is the problem here right because probabilities are estimates this is like the relative frequency concept that i was alluding to at the beginning of the talk just because something has been some way in the past doesn't mean it's always going to remain that way these probabilities estimate but they don't necessarily reflect reality and in many cases it's impractical to sample an entire population how for example are you going to sample i mean in congress that's one thing but how for example you're going to sample the entire american population or for example even sample every single customer that goes to a particular store you have nice records actually for people who use you know credit cards and stuff like that or loyalty cards but what if they use cash for example and you in many cases even if you can you can't do so indefinite so there may come a time when a democrat actually votes yes for the internet bill so what do we do in situations like this the answer is we move away from the standard frequency probability definition we tweak it slightly slightly with what we refer to as a laplacian correction and there are oh some questions coming in ah thank you anyway about those questions but i'm happy to chat further about these things so anyway you got the standard frequency probability and then you got your laplacian correction which we've tweaked it's like we're so what what are we adding to this equation m is known as the equivalent sample size and in this case it's the number of attributes a value may take which in this case is just two p is an a priori specified estimate of the probability many people chose one half just to be even-handed and you'll notice that the probabilities if you do this for example for cisco what you'll do is you'll just basically can be the problem the same way but in the numerator you'll add 2 times 0.5 and the denominator will just add 2. so instead of 0.01 and 0.99 you'll have these quantities instead so they change slightly that's not a big deal but what is a big deal is you knock out the zero here with the internet sleeping bill where it becomes a small very small but albeit non-zero probability and in this case with this correction the predicted member is a democrat okay so now we're going to do another coding example and this involves much more than the last one did but i have the code written for you um and so we will develop an algorithm well we've already developed it but i'm going to show it to you to predict what party of congress a member belongs to based on their voting record and there is this concept so i've stayed away from this so far in lecture but we're gonna have to talk about it just because i've noticed this all the time especially with models that are um you know in the real world or used by the government just because you have a model doesn't necessarily mean it reflects reality it could be trash because it could have no predictive validity so what we do is and i'll type this in the chat for everybody we have training and test samples and the idea is that we may take say 70 percent of our data and use that as our sub data to develop our probabilities and then see how accurately does the model actually predict any 30 of the data so the 70 of our data would be our training sample and the 30 would be your test sample and then you could actually compute like a percentage and i've done that for you in the codes of um of correctness of accuracy okay so i see some questions coming in here let me see all right yes so okay gabriel very good question i've only used classification features in my explanation is it possible use numeric features hold that thought we're going to get to that in a bit and that's why i wanted to go till 3 30. uh varus uh varus seath had a question um can we quantify the uncertainty in the predictions we're making yes and that is precisely what i'm getting at with the training and test samples you can classify you could quantify the accuracy of your model with the with testing the validity on your test sample and we're going to do that in a second so let me share my screen where is it yes here's the code okay so now trying to share this here we go voting model and okay we will bring this over here to this side so we have all these codes together okay so this is the model for this data now let me um how should i do this you know what before we get into this let me actually show you all the data itself codes voting so this voting data is much more expensive than this simple example we had just talked about let's let it load if not i can show it to you in r but it might be easier to see in excel i don't know what why that's taking so long all right i'll get another okay now we got it no okay here let's just do it here i don't want to waste time okay so oh is it loading no it's still not loaded or maybe it is here i'll show you here okay here is the data the voting data see right here you can see in your data browser in r but it's also open in excel hopefully your excel loads faster than mine did and it is right here okay only have to wait for that now that we have this okay here we go all right we'll just go through this so what you have here is vote on a variety of bills you have the political party votes in a variety of bills regarding say handicapped infants water project cost sharing the adoption of the budget resolution physician fee freezes and so forth you have all these bills here and whether or not they voted yes or no and you have 233 observations so let's do this in my r code we have denoted sixty percent of the data to be the training sample and we're gonna comp compare the accuracy for the remaining data and this this line here extracts your training sample from your data set and here we subset the data by party here the democrats we look at them first and you compute this probability matrix again like i did earlier to allocate your probability your conditional probabilities and here we do so for the democrats this loop goes through the data i encourage you to look at this every time i don't spend too much time on this in the interest of time and that computes the probabilities of yes votes engaging in this laplacian correction you can see m times p all over in the numerator and m in the denominator do the same for the republican party we again tag on the marginal probabilities the probability the number of democrats over the dimension the training data number of republicans over the size of the training data and then we compute our test data which is from the training sample onwards the end of the training sample onwards i should say and we have a function that we invoke to actually do this in particular to assign to make the assignments and that is right here where it takes into its realm the voting of a particular line of the data set test back and it takes in takes the naive bayes probabilities in and then it computes a probability test matrix if the person voted yes you use that probability if they didn't vote yes you take one minus that probability for both parties you multiply them together and again you extract out which one wins and that is returned here so i will run through this right now and at the very end of this let me tell you we compute and this is i think what the question was the percent accuracy right here percent accuracy is what we're doing is we take the assigned votes which are coming from i'm sorry the um the model right here at the end you are outputting the probabilities and the assigned party from the classifier and you take the assigned votes and you com you line them up against the actual votes and this percent accuracy compares the two at the very end it prints out this percent accuracy so we can run this right now and based on a 60 training sample that's what i had right we have a 90 accuracy so this shows that this model is pretty damn accurate it's reasonable it's not 100 but it's reasonable with 60 percent of the data what happens if you ramp that up to 80 95 so now that's fine that tells us the validity of the model but what if we want to see for example how this model engages in predictions here is an example i have this other example votingexample.r and you have the codes as well here is a hypothetical member and i just made up this data they voted yes on this bill handicapped infants no on the water project cost sharing bill and so forth what is what party do we does this model things that they belong to well we'll this takes my classifier function voting model function and puts in this example data and the probability matrix that was already computed and we can see these are the two probabilities and this is the signed party the democrats so this voting behavior is predicted by the model to be a democrat and again i encourage you to play with this this is an arbitrary member.csv again that data is right here and it's in the csv file which you can play with in excel and you can see how things change so that is discrete naive bayes classification with a training sample um are there any questions so i saw some in the chat let me get to that again okay all right q a okay yes please go ahead parson and if you have to go and you have still questions please feel free to email me this will be available online okay okay so yeah in the interest of time i will ask because we're going to get through this next exam pretty quickly in a couple of minutes for q a so what i'll do is in the interest of time i will just go into this next example and we'll move on so there was a question about actual numeric data well we're going to get to that right now let me share my screen okay so here is another application diabetes prediction can these models be used to say predict diabetes in patients if we are provided with medical data so what if numbers come into the picture such as medical measurements so i have this diabetes data set this is also from that from that online book the number of times a person was pregnant this is a data set of um of women plasma glucose concentration blood pressure these other metrics as well and we have finally the very end last column whether or not the patient developed diabetes so this is what the data set looks like and i have it in in a csv file we're going to load an r in a second but this gives you a sense about what we're looking at now unlike what we had before we are dealing with numbers so what do we do so we're going to have to utilize this concept called continuous probability distributions the most common probability distribution is the normal distribution this is known as also known as the gaussian distribution and it's a classic bell-shaped curve that occurs so naturally in nature it is described by two parameters a mean and a standard deviation uh or the ladder is sometimes referred to as the variance when it's squared and as you can see here you have a variety of choices of means and standard deviations for plotting the normal distribution and larger variances and standard deviations mean the data is more spread out smaller ones mean it's closer together the normal distribution has this probability density function right here and what people do is this and i will tell you i am not the biggest fan of this but i will say that it works in many settings so it's not i guess it's not that unreasonable to do but as a statistician i have problems because it's technically not correct from a theoretical perspective but what people do is they will take just like we conditioned on the outcome variables for uh the discrete settings that we had talked about earlier they will compute the mean standard deviation for the outcome variables in this case whether or not the person had diabetes and compute the normal distribution right like this for example what is the mean and standard deviation of plasma glucose concentration for patients without diabetes you can go through here and calculate the mean for these variables because these are the people who didn't have diabetes ii 108 all the way to 105 and the standard deviation and well here i've computed the variance and the square root is a standard deviation you can compute these and then you can this normal distribution what this mean in this variance will be what you use in your naive bayes model for prediction regarding glucose concentration this would be your equivalent of your conditional probability and we'll get you we're going to get to that in the example and the idea is that ins what you will do is suppose in your test data set you get plasma glucose glucose concentration to be 130. what you would do is you would plug into this probability density function 130 along with your mean and standard deviation and that would be what you plug into your naive bayes model i am not the biggest fan of this from a theoretical perspective because that is actually technically incorrect use of the nor