Stemming and Lemmatization: NLP Tutorial For Beginners - S1 E10

Key Takeaways

today we are talking about stemming and limitization and these are the essential steps that you need to perform in pre-processing stage while building nlp application you might have noticed when you search in google with some word it might search for some other words as well which are matching with that particular word which is here i'm talking about talking and then it will match a word talk so what we are doing here is essentially reducing the word to its base word and not only like google search is just one primary example that i gave you for your understanding but even in text classification let's say you are doing sentiment classification and you are using some words such as talking during the classification review product review then if someone has used the base word which is talk you know talking talk talked in past tense they all mean the same thing so you want to map all those words to its its base word there is a value in reducing a word to its base word i am giving some other examples here eating to eat it is a past tense but if you can map that to eat which is the present tense the base word then it can have a value in your nlp application similarly adjustable to adjust you'll notice that to get the base word of a given word sometimes you can use simple rule such as if the word has ing as a suffix just remove ing talking walking running if you remove ing you get the base word eating there are other suffixes as well such as adjustable mi cable remarkable so if you remove able then you get the base word so the point i want to make here is by using simple rules without using any language knowledge just by using simple rules that if the word has ing remove ing or removable you can have a list of these suffixes and you can use the rule that remove these suffixes from a given word to get a base word since we are using fixed rules or heuristic this process is called stemming it's like stemming you're just stamming the given word and getting the base word but on some other occasions applying just the fixed rule is not going to work it to eat you can't apply this rule you have to have that knowledge of that language so whenever you want to use knowledge of the language or a linguistic knowledge jargon alert linguistic knowledge is nothing but a knowledge of a language when you want to use that knowledge of a language and you get the base word that base word by the way is also called lemma so eat is a lemma for eight and this process is called lemmatization so stemming is just to summarize using fixed set of rules dumb rules that remove this x number of prefixes or suffixes to get a base word whereas limitation is little more sophisticated where you need need to know the rules of that particular language you need to have linguistic knowledge you might be guessing already that with stemming you can get into issues sometimes for example if your word ability if you just apply the dumb rules it will remove iity and you will get this base word abil which doesn't mean anything whereas if you apply limitation you will get the same word ability but as you learn more about nlp application you will realize that there is a value of both so stemming and limitation both are used and we are going to write some code by the way for stemming we will use nltk because spacey doesn't have support for stemming there is an article where spacey authors have mentioned why they don't support stemming because limitation as you can already see is more sophisticated more correct so they just have limitation support but nldk has both stemming and limitation both they support both so i'm going to show you anal decay and i will demonstrate stemming i want to take a moment and talk about first language dot in which makes building nlp applications easier for you you can perform various nlp tasks which are shown here via this platform if you're using a spacey type of library you download it you run your code locally if you are training a heavy model you need higher compute resources such as gpus whereas here everything is in cloud you just make an http call things run on the cloud you don't need a local machine with lot of computer resources there is a demo which you can see by clicking on this link and i have bought this bulb recently and you see there is a negative review on amazon and when i copy paste this here and i select text classification and when i say submit it immediately classifies that as negative and if i have a positive review i just copy paste it here [Music] and if i submit it it will classify this as a positive see it's pretty powerful so it makes uh things much easier for you you don't need to have like too much like detailed nano pin knowledge you can just call apis and get things done quickly for yourself the free tire is something you can try today you can just sign in and you can get an api access once you sign in if you go to dashboard you will get your own api key and you can use that api key to make the calls they have sdks available both in python and typescript go check it out the link is in video description below and thanks first language dot in for sponsoring this video so let's go to a notebook so i imported analytics pc and by the way if you don't have analytic install you can in your command prompt where is my command prompt in your command prompt you can just run pip install nltk very basic right to install it i have already imported both of these libraries and now i'm going to [Music] import porter stammer from nltk so from nlt [Music] and ltk dot stem import porter you can just hit tab to auto complete and stammer this is a class so i'm creating an object of this class in python when you do this this is a class and you're creating an object of this class storing it here there is other uh stammer as well called snowball stemmer but i'm just going to show you one and i have a couple of words here and you can say for word in words print so i'll print a word and then sum separation okay and stemmer.stem that particular word and it prints the the it does the stemming and it prints the base word so you you see it applied fixed set of rules so from it removed ing got it then when it has eats it will remove s and it if you have a base word it will not do anything so it is it but not is this it it did not do anything because stammer doesn't have a knowledge of a language it is just using fixed rules adjustable it is saying adjust drafting rat look at this abilityable apple doesn't mean anything but still you will notice that many applications even production applications they use nltk and using stammer could be efficient because it is faster it just applies bunch of rules it doesn't require any language knowledge and it can have a value in your nlp application so although it is dumb it can have a value now let's do limit adjacent in spaces so this was a quick demo of stemming in nltk so as usual so i'm going to now create nlp object and say spacey dot load en core web sm is small you can use medium i'm going to use small version of english language processing pipeline if you don't know what i'm talking about you have to watch previous video and then i will create some document so let's use the same word here and see in space it's pretty straightforward you can just say for word or for token in a doc print token some separator and you can say token dot lemma underscore notice this ate it reduces to eat eating eat see it worked fine adjustable it did not say just because the limitation the rules which are set by default english model says that the base word for adjustable is adjustable ability is the ability but look at this when you have better it reduced that word too well so it has this mapping rules in the trained model that you just loaded so based on the language based on what kind of model you're loading you might be loading a different english model which will have a different rule so there is this is a quick demo of limited edition in spacey if you by the way let me show you one more thing which is this this thing called token dot lemma so see there is lemma and there is lemon underscore so lemma is just printing a hash so it's a unique hash of your lemma which is a base word because this is a train english model with a fixed vocabulary it has few thousand words and for each word it will have a unique identifier short of and that is this hash and you'll see that for eat the hash is same this is just for your knowledge by the way i just wanted to quickly show you different uh sentence so i'll just do copy paste and i will try i don't know if you all watch mandalorian mandarin is my favorite series on disney so mendo doesn't talk that much but mendo talked for three hours hooray mendo is becoming talkative mendo for three hours although talking isn't his thing he became talkative and i will remove so sit talk it reduces to talk talking is also reduced to talk but talkative is still talkative because this is this is how as i said this is how this particular model behaves sometimes you might want to modify a behavior right you want to customize it so how do you do that so if you remember our pipe names so we have for this particular model the pipeline that we have is this there's a talk to actor tagger parser attribute ruler and then there is a lemmatizer now attribute ruler assigns attribute to a particular token and you can customize it so let me explain what what i mean here so let's say you have this sentence now bro all these are the [Music] you know you mean brother basically when you say this but by default the language model doesn't understand this is slang right so it doesn't understand so it will say bro is bro bra is bra right and if you look at the first token itself so like the first token is this correct bro and if you look at the lemma of that it is bro because it doesn't understand the slang but let's say you want to customize this model you know that bro and bra means brother you can customize it by doing this so what you do is you go here you get this particular attribute ruler by doing this so you say dot get pipe attribute ruler so it will give you that particular component from the pipeline and then you can customize it by adding that custom rule okay so see here i am adding a custom rule where i am saying this so what i want to say is i want lemma to be brother for few words so for bro and bra i want their lemar to be brother okay so here you are assigning attribute and attribute is lemma so basically you're saying the lemma for the words that i'm going to specify here and the way you specify those words i mean that is this i'm just going to copy paste using for brother bro and bra make their base word to be brother so you notice previously bro was bro and bra was bra but when i run this again bro is brother and bra is also brother and for the bro see the lemma was broke but now the lemma became brother so i hope this gave you uh some understanding of stemming and limitization i don't have an exercise for this particular video but i might add exercises in the future as well so always check the video description below because sometimes if i don't find time for the exercise while recording this video and if i find time later on i add those exercises and i i talk about those exercises in the video description below so always when you are watching my videos on my channel any video make sure you check video description because i might have included more information after i publish the video if i find some correction if i want to add some exercises some additional information articles i always add them in the video description so make sure you check it out i hope you're liking this series if you do please give it a thumbs up and share it with your friends one of the way you can share this series is by posting on linkedin linkedin there are some serious folks so you can post about this series so that other people learning nlp can also benefit thank you very much for watching if you have any question post in a comment box below