Implementing a Spam classifier in python| Natural Language Processing

hello all today we'll be continuing the discussion with respect to natural language processing from the previous videos of various concepts that we have discussed like stop key words limit ization stemming than me we saw something from something of count vectorizer we saw something called as bag of food how we can implement bag of words by using count vectorizer and then we also saw tf-idf model now we will try to implement the project and the project is basically called as how basically will will try to create a model which would be like a spam classifier this kind of class Pam classifier will actually classify your spam messages that you usually get in your email or suppose if you're getting a text message which are kind of spam this spam classifier can actually detect them so we will try to see that and today this is a whole machine learning problem statement other now and I will be discussing about this machine learning problem statement we will be starting from reading the data set I will just explaining of the data set from where I've taken this particular data set and then will follow all the processes of stop key words limit ization stemming and then try to apply a machine learning model and then try to find out whether this is whether this message is a spam or not so basically in this particular this particular data set I have I have taken from UCI website okay there is a machine learning repository over here and you just search in google for SMS spam collection data and you will be getting the complete data sets i+ so you can download this particular data set as soon as you download indicating two folders and your two folders will be looking something like this BAM file collection and there will be something called as SMS spam collection okay now if you just go this one is the readme file readme file has some information about this semi spam collection okay and when you try to open this particular you know this particular file and you can just do edit you can see that here is your whole data just zoom in here is your whole data the first column basically specifies that whether it is a spam or whether it is a half Pam is just a category it has taken indicating that this message is not a spam the second column will basically be having your message okay your this will be your spam message okay and the first column basically specifies whether this message is just spam or a ham so this is our data set based on this particular data set what we will do is that we will try to create a model and then we try to predict whether any messages are ham or spam okay so that is how we are going to use them now the first thing that I want to discuss about this particular you know this particular file is that how do we read this fine now in this file you can see that it is tab separated right so here you can see that this column and the whole message is basically tab separated it is not same is space separated instead it is tab separated the reason why I'm telling you tab separated because when you try to read this particular data set if by using read and understood CSV file will try to put a delimiter that is /t / T indicates tab now let us just continue and try to see how we can read this particular Tina set now you know that I have a folder called as SMS spam collection and this is SMS spam collection fine okay so what I will do is that first of all I am going to import my pandas okay under because I am using pandas for reading this particular together okay the next thing is that I will use T dot read underscore CSB and I will give the whole part of this particular file and I know that it is present inside SMS from collection okay and then I am going to use the separator which is called as slash T okay so slash T basically says that you know the first and the second column you know oh here the columns are actually tab separated okay so this this slash T will actually help you to divide this into two columns so this will be your dependent feature and remaining all the content with respect to the message will be your independent feature and when you see this all spaces these are missing is spaces you know just blank space and these are slash T because if you just try to open this file in notepad plus plus at that time you can see that there will be a slash T over here recognized otherwise all the other spaces are the space for blind people now what I do is that I'll use a separate of /team and since I don't have any I don't have any column names over here I'm going to specify my column names saying that the names is equal to and I'll provide two labels that is for the first column I'll specify labor and this label will basically be having time and half okay and the second label will basically be my message so these are my labels of the two columns that I am going to use now as soon as I read this particular dataset you see that how our dataset will be created so once I execute this there is more variable Explorer I'll just go and see my data it is something like a label and message now you can see that my data frame is basically having two columns one is label and one is message in the label I have ham spam okay and in the messages I have normal messages okay and this particularly label indicates whether this message is the ham or a spam okay now the next thing is that as I told you as we discussed in the previous NLP playlist right if you have not seen the playlist edge please go and see those plates because I'm going to use those concepts and I'm going to implement this full use kits okay because I'm going to use top keywords I'm going to use limit Isis sorry I am going to use stemming I'm going to use I'm going to create a bag of words so all those concepts has been discussed in the flavors if you have not gone through that first of all go through that then come and try to understand this particular you skip how I've integrated all these concepts now the next thing is that I am going to clean do some data cleaning and data pre-processing okay now in this messages if I go and see my messages column okay now in this messages column we have a lot of things like common full stop and many letters are there in capital letters and we also have question mark and we also have some of the words like the two and we do actually this this words are not that much responsible to indicate whether that message is a spam classifier on one you just have to focus on some of the words and first of all we have to first lower up all these words that are present in this particular sentence because there will be scenario that you may have this if you don't lower it up then you will be there will be a problem that you'll be having duplicate words with the print different some words will be in capital letters the same were to be in small letters so that kind of scenario makeup so what you have to do is that in data pre-processing first of all what I'm going to do I am going to import hari hari is basically a library which is used for regular expressions the next library that I am going to use is analytic it I hope you all know by analytic is used because everything when you do when you're using stop words when you are doing limit ization when you have to in stemming whenever you're doing bag of what all the libraries are present is identity okay so you don't have to worry about that so with respect to bag of words will be using sky Hitler but most of the libraries like stock words you know and some more language like stemming limitation will be present inside America to make sure that you import this ml TK and I hope you have downloaded then indicate from my three videos if you see that if you have not teen let's just see that how to download all the niceties so you can just execute this to download the stop words but in my case the shop words are already downloaded okay so what I am going to do is that next from analytic a dot corpus I am going to import stock words because stock words as I told you that there are some words in these messages like though if off then okay it small small words which are of no use like in a - okay all these words you know I have to remove it from you so I have to input stock word and stock words have all this kind of words over here ok so what I'm going to do is that first of all I'm going to import stock words then I'm going to import porter stemmer and you know porter stemmer is basically used for stemming purpose and stemming is basically a or technique which will help you to find out the base route form of the world ok so here and importing porter stemmer and even i am utilizing the variable of photo statement that is easy now my next line is basically I'm creating an empty corpus because after data pre-processing all the messages that are present inside this message data think I'm going to put that up in this part first list ok so now let me just show you what how the data clearing process is done as I told you first step is that I will remove all the unnecessary characters like numbers . comma okay we'll stop comma dot dot dots or explanation heart ocean marks I try to remove everything from here so for that only I have imported this regular expression so what I do is that I will run a for loop on all the messages that is five by seven to record you can see over here there are five five seven to record if I just scroll down there five five seven two records okay so for all this I'll just put a for loop and on each and every message each and every message you know I will try to remove I will try to remove all the character except except a to Z and capital e to them basically except small A to Z and capital A to Z I will remove all the other characters and I will replace it with blank that is what this re dotsub is doing okay and here you can see that I'm giving messages of message messages of message right messages won't mess it off I so this I will basically help you to indicate or basically whether you are in the in the first loop you'll be in the first sentence then in the second loop you'll be in the second sentence so this I've al you will be changing based on will sentences URL okay so that is the first step you are removing unnecessary characters like . comma punctuation dot dot dot or if you're putting any special character everything will get removed apart from these characters yep now in my next sentence i am going to lower up all these messages I'm going to lower up I'm going to lower all the words in this messages so that I do not have any duplicate words the next thing is that I am going to split I am going to split each and every sentences where I will be getting you know lists of words because I have to apply now stop keyword French stemming on each and every word right standing will help you to get the base root form of the word so what I will do is that first of all I'll just split this this particular sentence I will get a list of wording review then what I do is that I will put a condition for word in reviews since I have lists of words and good going to put a for loop where I say that forward in reviews if that word is not present in stock words not words of English type if that word is not present in it in this stop what is not worth I'm going to take that word and I am going to apply the stemming process okay I'm going to apply the stemming process so as soon as I apply this stemming process I will be getting the face form of that particular word okay suppose if you are if it is written I intelligently you may be getting something called as intelligent okay or if there is a word like going it will become cool okay so that will actually be done by the studying process then in this review I'll be getting again the list of words which will gain the base note form and remember all the stop words will be removed from that whole list of words then I am going to join that words okay I'm going to join that words sorry I'm just going to join that all the list of words into a sentence and then I am appending that in my new list that I created that is for this purpose so that is all the steps that I am actually doing so first of all let me just execute this whole statement at once sorry is saying import re is not defined so just go and import re and then a ticket some time error is good for her so I have executed it gonna take some time so they are around five five thousand five seventy two records I think that you should be happening within five to ten seconds let's see always remember guys remember the steps away what I have done first of all I've done the cleaning part then I have done the lowering plot then I have done the splitting part then for each and every words I have found out whether that word is present in the stores or not if it is not present in the store part uh-huh actually you know move to the stemming part and have I have found out the stemming or the base root form of that particular word finally I have joined all all the words and appended to one another list which is for this purpose now it is recordings Akutan now let us see corpus over here yes so this is my corpus and this was my real message is it a thing okay now you can see that they are lot of words like see this go until has got removed since it is in stop words write your own point crazy see a are cra-z why has become a CRA that I do - stemming okay what you can do is that you can also apply limit ization rights okay it is not compulsory that you only have to apply stemming because usually stemming takes less than amount of time but let's see if the currency is bad definitely I'm going moves towards the limit ization but initially I'll just try with Stefan okay so so this is my this is my new text that is my corpus now my my data is actually ready now now this data after cleaning I'm going to convert into bag of words and bag of words if I say bag of foot you should remember that in the previous class we have discussed about bag of wood it is nothing but a document matrix document matrix with respect to the words again the explanation is given in the previous videos about how to create a bag of wood types please see that I will provide the link in the description box about NLP play this ok and please go and see that also now next thing is that what I do is that I will just create a bag of words or the document - for that what I'm going to do is that I am going to I'm going to import a library which is smallest round vectorizer and it is present inside SK learn dot feature extraction okay SK learn not each extraction or text I am going to import this count vectorizer now when I am initializing the count vectorizer I just remove this maxi just for right now okay I will not have max features right now now just see this what will happen okay as soon as I will first of all initialize this count vectorizer and my object is CB and then I look fit transform on this night on this new list that I created that is my purpose okay and I'll convert into an array now you see this as soon as I executed okay now you can see that my ex my ex data which is my training data okay how many how many unique words I have how many total number of columns I have over here you see this how many total number of columns I have I have somewhere around you know six to nine six columns okay now when I have six two nine six columns I told you that I have to take the most frequent elements because they we'll be some characters where there will be somewhere like words like some some of the names which which may be a coming just once or twice right and they may be some of the words which will be just it will not be that frequent when compared to the other words right so because of that we should not take all these columns that is 6 to 9 6 instead we should just select some frequent columns some frequent columns or some frequent words right now this 6 to 9 6 basically indicates that in currently in my X in that X beta in my X data they are six two nine six words separate words that is indicated by caller if I move over to the right right each and every represents one word okay and with respect to this words I will be having ones or zeros that will be present over here right now because of that what I do is that now I don't need 6 to 9 see first the reason is that I know that only some of the words may be frequently present sorry some of the words will not be more frequently present it may be either one time or two times present so what I do is that I will just take you know I'm just selecting some random random max features so I will just say that instead of six thousand two hundred and ninety six columns I want to just take five thousand columns from okay top frequent columns basically I am just taking it as five thousand you can change it so you can play with it okay you can just take little bit more higher and see how they kill it system so what I do is that instead of taking six two nine six words I will just take 5,000 words now when I execute this to see that now you can see that my ex has now got converted into five welcome now I this means that I'm just taken the top 5,000 most frequent words from the when comparing to all the features right in that particular whole document or the sentences that I had what I am doing is that I am taking the most 5000 elements most pigment words okay and now you can see that this is my data set this is my old training data this is my whole training dataset sorry this is my whole data set power to the law now you can see that something or the other way ones will be present there are many words I can just should show you exactly where if you just explore it okay just let me just show you one at least one one I'll try to show you some more the other way we'll be presenting okay you can see over here one one has present so this basically means that this column which is indicated by that word it is present in this 491 sentence okay in 491 sentence this word is present that is what it is indicating so it's count is one okay so this is how your whole document matrix looks like okay and this is actually clear this is also pardon bag of bones okay now this is my whole data I have my whole data but what about my output data you know my output data is basically my label column where you have where you have ham and spam now what I will do is that I hate this okay now this currently my X is basically my independent features right this is my independent features now if I want my dependence which I have to take this label right so for that what I will do is that and one more thing guys here I have two categories one is ham and spam obviously I can't just pass Hammond's and directly to my model my model will never be able to understand the string or a text right it is not a human being so we will try to convert this into dummy variable okay and in order to convert this into dummy variable we will be using something for less or nows dot get dummies now as soon as I execute this you see this and I'm just passing the label column so as soon as I execute this now you can see that in my Y column I'll get converted into two categories two columns so ham will actually get specified with one camel actually gets specified with zero so in all these columns wherever there is spam okay it will get classified with feel sorry wherever there is ham that is what otherwise if it is not ham this will become zero okay similarly when spam is there to be 0 when spam is not there to d1 okay so this basically indicates that whenever okay sorry sorry it is not like that guys just a small small change in the explanation oh you are after converting into catechol bail use it basically indicates that in this time on ham craft column we're there is one that basically indicates that it is a ham category and in the spam column wherever it is one it basically indicates the spam category okay now I have two categorical variables over here right but instead of having two categorical variables if I had if I just have one categorical variable okay if I just have one column I can represent that I can represent the ham part by this column itself you know because you can see that when the spam is zero this basically indicates that this column will be ham right so ham value I am specifying it as zero and spam value and space specifying little one so what you can do is then you can remove one column from that so in order to remove this column I what I am doing is that and writing this Y dot I law : one God values so as soon as I execute this and see my Y value you can see that I am just taking one column that is the spam or the hand column you can take any of the color okay because one column will actually specify both the information if it is zero it basically specifies that it is ham if it is one it basically specifies that it is Sam something like that so you can you don't have to use two catechol features and you can just use one category filter and this is also called a dummy variable trap okay you should not get into this particular trap and there are some problems with this which I will be discussing in the data size interview question what it what does it mean okay so now my X which is my independent feature is ready okay my Y which is my dependent feature is ready that basically means that with respect to this sentence my first sentence I am having value at zero that basically indicates that this is not fair the third sentence that I have that value indicates one which two indicates that this sentence is basically spam so this is my data this is my independent feature this is my dependent feature now what I will do is that I just do a train test plate I do a train displays because I need to train my model for doing the Train test split I will just import the train to split library from a scale or not model selection and I will make a test size of 20% you know 20% of the whole data so let me just execute this so here it is my extreme extends everything is getting executed and I all the data oh yeah now you can see that my ex test is having triple one pipe but as my exchange is having four four five seven if I add up this it will be equal to X I okay now in order to solve this problem and basically using an a biased problem because neighbors work very well with respect to n NP natural language processing but even though will not be getting much good accuracy if you get also good accuracy we can find uni again the theoretical part of name bias I'll be taking another video station I'll be uploading another video where I'll be explaining the theoretical concept of name bias but for just now you understand that name bias is a classification technique okay this is a classification technique and it works completely on probability I just show you the probability formula I think you will be very very familiar with it because in the high school days we have work with that for your particular formula okay so that Nate Byars will be taken in the later of videos where I'll be explaining about the theoretical part but now I'm trying to implement a name bias okay and this is the classification technique that this player this is basically a classification table so for that I'm going to import from Escalon Dorne bias I'm going to import something called an multinomial MD so this is my name bias library that I'm going to use and which is which and this basically works for multiple classes or single classes or two classes three classes and in a whole class is fatima okay so after importing this what i am doing is i am initializing it and i'm going to fit this with my train and vibrator so after i initialize this and i do the fit basically what happens is that this will my model will get created so this is my spam detective model okay my span dated model then what I am going to do is that I am going to predict with respect to my technique that is my XLS protection so after I predict you can see that this is my wife red my wife red variable is created now this is my wife red and this is my white that's okay now what I'm going to do is that I'm going to compare this both now you can see that most of the values are being compared and it is looking almost equal right this is my wife red and this is my whiteness but they so many more values how do we compare it so for that I'll be using something called as confusion matrix so I can write from a scaler dot matrix I am going to import confusion matrix okay confusion matrix if you don't know about confusion matrix it is please go to my playlist I have created a video and confusion matrix which is just like a data function to the cushion okay and I explained completely about confusion matrix over a very detailing it is more than like you know 20 minute videos about that so for confusion matrix it basically gives you a 2+2 dimension matrix which will actually say you that how many number of elements are correctly predicted okay so for that I'll just write confusion underscore M is equal to L initialize my confusion matrix create a variable and then I initialize and inside this I just have to give my Y test comma Y Fred okay as soon as I give this you can see that my conclusion matrix will give me a 2 cross 2 matrix so it will give me a 2 cross 2 so here I am opening it it will give me a 2 cross 2 matrix and the diagonal elements where your value is 0 and the upper value is also 0 see this is your actual output this is your predicted out so 0 0 is getting matched so you have 9:46 correct predicted and you have 1 and 1 this block is also correctly predicted so you can basically say that 946 plus 1 phi2 is correctly predicted out of the total outcome now you need to check that how you can check the accuracy so for checking the accuracy score you can also use from a scalar dot matrix and I'll be importing another library which is called as which is called as accuracy score okay accuracy is 4 it will actually help you to find out what is the accuracy so accuracy underscores poor or this I write accuracy is equal to accuracy under scope or and I here also you just have to give you a white test comma boyfriend that's it and once it is execute that what you have to do you just execute it fine now you can see that your accuracy ninety-eight percent days see the accuracy it is wonderful accuracy of 98% and this is how you have basically implemented a spam classifier here how I have implemented it I've just used concepts on basic NLP consoles like stemming limit ization and not use limit ization of here over here but I see if you want you please go and try to use limitation aware try to see in this particular sentence you can use limit ization and you can increase the performance but here the accuracy is almost 98% it as we have done actually we have done a great job and created a good model again there will be some some dependencies like if my spam and ham are not in equal proportion okay at that and if I have an imbalance data set at that time my accuracy we may not be good okay so please check if the what what else we can do with respect to the how we can improve the accuracy for that what you have to do is that instead use limit ization over here and try to do with limit ization try to check and instead of count vectorizer of creating bag-of-words use tf-idf model if you want and tf-idf code everything is given in my previous videos in NLP okay you just have to go and replace this code over here that's it and automatically all the steps will be same and you know simple so make sure that here i have you name by and i told you that name bias will work very good because it works completely on probability and it does work very many nicely for NLT problems so i will just let you know about the theoretical concern on name patch in the later stages okay and why it works very good so I hope you liked this video guys if you have not subscribed the channel please do subscribe it this will really give me a motivation to meet more and more interesting content and videos for you I hope you like this video share with all your friends who is in need of this kind of problems who wants to become a data scientist and you know they have think they want some materials to study please do let me know and please do let me like if you want me to create any other videos I'll definitely do that for you and I'll meet you all in the next video thank you one and all have a great day I you