Stop Words: NLP Tutorial For Beginners - S2 E4

Key Takeaways

today we will discuss an important concept called stop words in nlp let's look at our same example that we discussed in last video which is you have news articles and you want to tag or auto tag the company name and how do you do that well when you are reading the article you notice certain words such as elon musk gigafactory model 3 if you encounter those words you know this is tesla if you encounter apple iphone those kind of words you know that the news article is about apple and this is a classic problem that they saw in nlp world where they try to auto extract entity or they try to tag the topics to a news article and this approach is called back of words which is something we already discussed in the last video where let's say you have three articles you don't know which company the article is from and if you can build a simple bag of words model where back of words is a jargon it's it's very simple thing actually you have a word and you count number of words so in first article tesla appears 14 times model 3 appears nine times and it's obvious this is a tesla company right and based on see the the second article iphone appeared 13 times ipad appears six times and remaining words are zero etc you know it is about apple but if you read the article carefully you will find a lot of other words as well such as wants time to prepare so see i represented a very simple model here you know very convenient model in reality you will have lot of noise you will have too far over and these words are general english words which can appear in any article without equal frequency based on these word count you can't determine if the article is from tesla or apple wouldn't it be nice if you can filter these words so let's say i'm looking at this article and some common words which appear in pretty much all the articles like two from the a and over if you can remove that you can make your bag of words model little less sparse so sparse means in this model you have a lot of noise the vocabulary is very big computation time is high it's a sparse model whereas if you remove these words your model will get simpler and you will be focusing on things that you care about okay so if you have this kind of model it's better so these words are called stop words and it's a common practice that during pre-processing stage in nlp pipeline majority of the time they remove top words now there are certain occasions where they don't remove it and we'll look into that for example let's say you have you're doing sentiment classification and you have this is a good movie this is not a good movie but when you remove stop words what you will get as a result is good movie good movie this might be a little you know incorrect because not not is a stop word so if you remove that and if you process remaining sentence you can run into issue however uh for sentiment classification you will see things like this is the worst movie okay there are some people who will say this is not a good movie but majority of the time this problem doesn't arise but i want to just tell you so that you keep this thing in mind just in case you're using stop words and you're not getting expected results for your nlp application when you're doing language translation let's say how are you doing the world i translate that into telugu if i remove top words in the pre-processing stage all i will get is daval and you can't translate double to telugu by keeping the meaning of original sentence same so you have to use common sense and look at a given nlp problem it's a machine translation it's a sentiment classification what kind of problem it is and based on that you use stopwatch removal technique during pre-processing stage stop for removal is actually common in many applications but applications like this machine translation another one is uh chatbot right someone is asking a question i don't find a yoga mat on your website can you help if you remove stoppers in pre-processing phase all you get is fine yoga mat website help what does that mean you know it doesn't capture the essence of the questions in a right way so these are some of the examples where you have to be careful about not removing the stop words but in majority of the other cases removing stop words during pre-processing stage is a common practice let's do some coding now as usual i have created a new notebook imported specie and by doing this from spacey dot lang dot dot stop words import stopwatch you can you can look at all the stop words which are there in this english model so you see that a all alone back cannot these are commonly used terms in english and removing them prior to building your actual model will give you a lot of benefit and there are 326 of them in total now let me create a new dock in our usual style and see how this thing works so i will just create simple document here and then you can go token by token so from for token in dock and there is this method called ease stopped so this will tell you if the given token is a stock word or not and if you print that token see these are all the stop words and what people do is they define uh some function called let's say preprocess where you give our text as an input and this is a common practice in nlp application and you want to do all preprocessing here so pre-processing means removing stop words stemming limitations and things like that so here i will say doc is equal to nlp text for token in dock if token dot is stop you know what i'll do i will remove it so if you want to remove it you'll say if not open so that's a reverse if it is not a stop word then maybe store this in some kind of array okay and the better way of doing this in python is using list comprehension so i will say here i want to use this comprehension and i will say no stop words and that is equal to see you get your for loop here first and before the for loop you will say token.txt so token.txt is what you want in this list as a final thing in your if condition you specify here this is how you write the list comprehension okay and you can return no stop words and then i can do pre-processing of this text and it will print me all the tokens just by skipping this thing uh the stopwatch now see you can have things like comma so you can remove that as well by saying and not token dot is there is a punctuation mark right this is a punctuation mark i don't want to care about punctuation mark and it will remove that see there was a comma punctuation mark which got removed so we can try a few other statements for example i my most favorite one is this you see the other is not these were all stoppers and it removed that checking our article on tesla right you you can have this big sentence but when you remove stop words you want only the terms that you care about which is mask wants time prepared i know some of these are not even useful wants etc but that's okay at least we removed the most common ones now how do we apply this preprocessing function on a pandas data frame let's say one of the common use case in machine learning is you load your data in pandas data frame and then you want to build nlp model now you want to do pre-processing on top of it so what i'm going to do here is i have taken this particular data set so let me show you so these are all the cases in department of justice usa from 2009 to 2018 these are all the press releases okay and the database is in json format so see there is some code case so it's basically press release of some code case the the verdict that code gave and so on and these are the content of the case this is a title things like that so i had taken this data set from kegel and i have placed a i have downloaded that json file and then that json file i am going to import in pandas by doing this if you don't know pandas in youtube search code basics pandas you will find my tutorials so here when you say lines true there is one line for one json object okay when you load this what you will find is 13 000 records basically and let me print first five records so see these are all the cases and what i have is case id the title the actual content big press release date and the topics now some of the topics are empty so let's say i'm building a model to auto extract topics that's my nlp application that i'm building i can filter all the articles which don't have any topic okay and the way you do that is see the type of this df.topics if you look at df.topics0 for example right the type of this thing is list okay so df.info if you do df.info [Music] it will also tell you a few things df dot these are all the exploratory methods that you use to kind of go over it but anyways we know that the topics type is list and you want to filter it so how do you filter it so you might have seen this in my pandas tutorial that you can use your filtering condition here in this this square bracket so you can say df is equal to filtering condition that you specify in this bracket and what is my filtering condition so you can say df.topics okay so let's see i want to say whenever df.topics is an empty list okay filter that and one way of doing this could be okay you write maybe some lambda function you check the length of the list or there is an easier way which is you convert this to str and you check the length this is a ready-made function that pandas provide and whenever this length is not zero which means list has some content you get that so now when you do df dot here first five see you have filtered all those rows which had empty list in our topics and then uh your pandas data frame is only four thousand arrows initially it was 13 000 rows so by the way by doing this exercise uh i i'm just showing you how would you apply stop word removal when you're dealing with a real database okay so this preprocessing function is kind of a simple function that works fine on one string but usually when you are doing any building any nlp application many cases you will have pandas data frame so i just want to demonstrate we are not going to build any machine learning model in this tutorial i just want to demonstrate how you apply pre-processing function on a pandas column and to keep things simple and fast i will just take first 100 rows of my data frame only first hundred because i want to show pre-processing and i can do it on a smaller data set right and now when i do df shape only hundred rows okay and now on the content see i have this contents right this contents if you look at this contents it is this and this is the idea so let's say 4 is the id so if you do i log 4 you'll be able to print see this big paragraph and this big paragraph look at the length and what i want to do is i want to remove all the stop words from this so that this length can be reduced and i can build effective simple nlp application out of this okay so let's remove stop words from this column from this entire column stop words i want to remove right so like for the see all these toppers in pandas so i will do let's say i will create i want to maintain that column and i want to create a new column called contents new and you all know if you have an existing column you can use apply function to create a new column and on this existing column i want to apply my pre-process function because preprocess function is already defined you see and by the way i don't want array to be back i want the string to be back so if you want to convert this list or array into string in python you just do simply join so when you do this join it will take all the list elements and create a string out of it right now you're getting a list you will get now a string see so that's the effect of doing this so my preprocess function now takes input string remove all the stop words returns another string and when i do that that's what is gonna happen and i can print the first five elements based on your computer and the size data size it may take time we have only 100 elements so it came back pretty fast and you see this new contents new and you will see things like see here it says 130 count criminal instrument was uc was released so see here was is removed and if you want to see it a little further what you can do is see this length was 5504 but if you look at new content see four two one seven so five five zero four minus four two one seven where stopwatch it remove all of them and if you want to print these things let's print it so condense new is after removing your stopper so now see what you have is 21st let me just print you know first 300 character or so so that is readable so see 21st century end so see end is removed off off is gone it is gone agreed affiliates all these things are gone hooray you just wrote a function to remove stop words and also you did this thing in pandas data frame so that's all i have for this video keep on checking video description folks because many times i add exercise after one or two week of publishing the video sometimes i add it later also this is a community of four for example see let me just show you core basics nlp when i was doing this video i did not have the exercise okay but later on kiran marala he's a community contributor he helped me build this exercise so now there is this exercise for bag of words and there is exercise solution also so you can look at the solution but i know you all are sincere students you will not look at the solution unless you have tried it on your own so for today's video also uh if you want to contribute just uh for the github repo and make changes and give me a pull request i will look into it but yes my idea is to have exercises available for all the videos at some point of time if not at the time of publishing i wish you all the best thank you for watching [Music]