How to Lemmatize Text - Part 4 - Python Yelp Sentiment Analysis

Key Takeaways

what's happening guys welcome to part 4 of how to build a sentiment analyzer using Yelp business reviews now in the last video we went through how to clean up our data set in this video what we're gonna do is we're going to go through the process of limit ization now if you don't know what limit ization is what it really means is to take any word and transform it to the base version of that word so rather than having different word endings and different inflection points we've actually just reducing that word back to it down to what it truly means so for example you might have words including M R is and what will actually happen through the process of limit ization is that word will be translated back to just be because it's the idea of being so a limit is a ssin basically cuts down the number of additional words that you've got in your text and that helps to basically condense again really tone down on what the message is of that particular piece or body of text there's another process which is typically used when doing something like this or doing similar types of work and that's the process of stemming now stemming is pretty good as well but what a candor is take away the actual meaning or the context of that word so for example the word because might just be transferred translated into ca US which isn't really a word and doesn't really give you too much context let's go through it so we've up until this point we've got our data frame into a pretty good state so I've got the initial review we've got our metrics we've got our reviews at each stage of cleaning what we're gonna do now is add one additional column which gives us our clean data and eventually we're gonna drop all of this anyway so again keeping in the theme of keeping this nice and tidy we're just gonna add a little bit of markdown and we're going to call this limitation alrighty so to limit ties now in order to lemma ties what we're gonna be doing is using text blog great name if you ask me so text blog is a really great package and it's a pretty light way or high-level or highly abstracted language processing toolkit so but it does have a great limit ization functional method so for that reason we're going to use a text blob so to do that or the first step in doing that is to import it so we'll actually import text blob and we're going to import it so from text import boom successful now what we're going to do is going to loop through each one of the words and we're going to limit eyes each word so to do that we need to loop through each word obviously so we're going to grab a clean review column and we're going to go through each so we'll use the apply method and can never spell that right first up and we are going to go through each word word index dot split and rather than just returning word we're going to do a few extra things here so we're gonna first up we're going to convert it into a text blog word object so it's gonna seem weird but word to text blob word object and now we're gonna limit eyes it and we're gonna do that simply by using the limit eyes method as a lemma ties and what have we done xx is not defined where are we lambda X times and weeks is because we need to join it to join all right we so that's limit izing and okay that looks like it's limit eyes amazing it looks like we've got a slight issue there all right not to worry but that's run limit eyes function we're gonna store that and I think let's say limit eyes right okay so that gives us limit eyes review so we basically condense the words back down to their base format and now we're finally ready to get onto a sentiment analysis stick around step 5 we're going to knock this out