Stemming - Natural Language Processing With Python and NLTK p.3

Key Takeaways

Hello everybody and welcome to part three of our natural language toolkit with Python for natural language processing tutorial video. If you're familiar with any of the machine learning video series that I have, you'll find that like I say most of your work when it comes to any sort of data analysis is usually organiz organization of data, cleaning up of data and all that and like analysis is like right at the very end. It's like the cherry on top that you get after like 99% of your work is just about like organizing and structuring and pre-processing your data. So, so most of what NLTK does for you actually is NLTK does not perform the analysis generally for you. It's good. You can use some stuff in NLTK to test things, stuff like that, but for the most part, it's a toolkit. Um, so most of what we we're going to cover at least in these first few videos is just the toolkit aspect of it. And then I'll show you guys how we can actually use it for analysis. But moving on, the next topic that we're going to be talking about is uh called stemming. So the idea of stemming is kind of like a it's a form of data prep-processing. And it's a form of kind of not really normalization, but it's the best word I can think of to compare it to. Uh and the idea of it is you take words and you take the root stem of the word. So for example, if you've got uh riding, the stem of riding would be rid basically. So you get rid of the ing and you have a stem of r and r is applicable to rid ro or or sorry ride riding ridden that kind of stuff. Okay. So uh what we're going to do now is well actually first why might you want to stem? Okay. So why are we even doing this? And the reason why is a lot of times you're going to have different variations of words based on their stems um or at least their affixes at the end. And but really the actual meaning of that word is unchanged. So for example, let's say uh we have two sentences. We have I was taking a ride in the car and then you got another one. I was riding in the car. Okay, these mean the exact same thing, but the word ride here is ride and the word rideish here is riding, but the sentences mean the exact same thing and the use of the word here is identical. So if you can imagine all the words in the English language plus their all their fixations at the end of them, you could imagine that you'd have a huge database here. And a lot of times you would have two words that basically mean the exact same thing taking up space in this database or table or whatever you're using to get values on words or meaning even from words. They would have the exact same definition. It would be very redundant, very inefficient. So we use stemming to kind of help this problem. So with uh natural language processing uh has had a stemming algorithm around for a really long time and it's called the porter stemer. This one's actually been around since like 1979. So if you think natural language processing is like a new thing, that's your problem. So anyway, um so to to use it with NLTK, we're going to go um from NLTK. stem import porter porter stemer stmer stemer um and just to just like everything else there's there are multiple stemmers you can train some stemmers yourself um porter stemer is actually pretty darn good so we'll just use it use it for now anyway next we're going to say from nltk tokenize import um we don't really need the scent tokenizer so we'll just do the word tokenize tokenize. Cool. Now we're going to say PS equals Porter stemer. And now let's make an example list of words. So example words and that's going to equal a list. And we'll have uh some stuff in here. Let's do five for now. Okay. So let's think of some example words. You've got uh Python. You might uh be a Python yourself. It might be the act of being a Python. Something like that. I don't know. Uh then you've got a Pythoner. This is someone who pythons. Uh think about pythoning. This is someone who is actively doing the Python. Then you have uh Python. This is generally what happens when you solve a problem with Python. That problem was Python. And then you've got Python Lee. This is just how you kind of handle yourself. He handles himself Pythonly. Okay. So those are example words. And now let us stem those words and see what we get. So, we're going to do 4 W in example words and we're going to do print ps. W. Cool. Now, uh let's go ahead and run that really quick. So, here are the stems. So, as you can see, the first one, two, three, four, uh are all the same, right? They all stemmed down to Python as the root stem except for the last one who was Python Lee. Okay. And so this one has its own little stem probably because of the meaning changes slightly. Uh but you could argue the same thing with all of these other ones. Um but anyway, we'll close that for now. And um so whenever you're stemming, like what would a sentence look like that was stemmed? Okay. So, let's see. Let me we'll comment this off and let's make a sentence with our words just to really drive the point home. So, new text equals um it is very important to be Pythonly while you are Pythoning with Python. All Pythoners have Pythoned poorly at least once. Okay, now we're going to say what happened. There we are. Uh, we're going to say words equals word tokenize new text. And then again we'll just use this exact same uh thing up here for w in example words only now it is words wads save and run that it is very import to be python le while you are python with python all python have python poorly at least not quite sure why once stemmed let's see is there any other iterate once. I can't think of one like I'm not sure why it stems down to C. Anyway, um fine. It's just not meaningful. So, as you can see, uh we can stem all this and important um importance important the meaning does not need to necessarily change that kind of stuff. So, anyways, uh that's just a quick example of stemming with NLTK and kind of why you might want to stem with NLTK. Um, it really depends on what you do and kind of like what your goal is cuz a lot of times as you'll see as we move forward, you won't actually have to stem uh you'll feed words through NLTK and they'll actually you'll use word net instead and word net will actually find you the synonym using sins set and yeah so really stemming is something you should know. you should know how to do it. But moving forward, you you may or may not actually ever utilize stemming because you just don't need to now that you've got word net and sins set and um nowadays even imageet. What? Anyway, uh so that's it for this tutorial. Uh if you have any questions or comments on stemming, please feel free to leave them below. Otherwise, as always, thanks for watching. Thanks for all the support and subscriptions. Until next time.