Text Classification - Natural Language Processing With Python and NLTK p.11

Key Takeaways

What is going on everybody? Welcome to the 11th NLTK with Python for natural language processing tutorial video. In this video, we're going to actually start creating our own algorithm or text classifier is what it's called. And now we're going to be doing a text classifier for sentiment analysis, but you can also use text classifiers for all kinds of stuff. Maybe you're trying to classify the text as uh stocks writing or politics writing or whatever you want, economics or anything. Um or another form of text classifier might be uh discerning whether or not something is spam or a legitimate email, that kind of thing. So uh our text classifier is going to classify something as either a positive connotation or a negative connotation basically or meaning or sentiment uh as a form of opinion mining basically. So let's go ahead and get started and and show how we might actually do this because this methodology uh our methodology here can be applied to any categories as long as they're tagged and they're two categories. So we just have two choices. There's not a degree of choices. It's just one or the other. So, this is spam or it's not spam, right? When you have a spam filter, you've got spam or not spam. You don't have like a bunch of little folders that are like maybe spam, highly likely spam, definitely spam. No, usually it's just your inbox and spam. Okay. Uh same thing here. We'll do sentiment analysis, positive or negative. Now, of course, with sentiment analysis, that's not necessarily the case. You might have something that's slightly positive, highly positive, extremely positive, and and the same for negatives. But anyway, this is just an example, but you can feel free to get your own list and your own tagged list even or create your own list. And as long as it is a document with some words in it and it's labeled as either one of two labels, you can use anything you want. It could be spam, it can be defining, you know, whether or not something is a text message versus a, you know, official document or something like that. It could be anything. So anyway, let's go ahead and get started. We're of course going to be using NLTK. We're also going to import random and we're going to use that to shuffle up uh the data set that we have because right now it's very highly ordered. It's all negative for the first thousand, all positive for the second thousand. That's no good. And then finally from nltk.corpus, we're going to import the movie reviews. So this is what I was just talking about. It's a thousand positive and a thousand negative movie reviews. So we can train against them and they're already labeled, you know. So that's the whole idea of training data. Now, when you read the NLTK documentation, you're going to see the following oneliner. This is kind of a confusing oneliner if you ask me. It's kind of a wasted oneliner because it's basically the same amount of lines. If you didn't make it a silly oneliner, but people are just in love with their oneliners. So, I will show it to you because this is how most people do it anyways. So, let's go. So, documents is going to equal and it's going to be a list and it's also going to be a it's going to be like a list of tupils basically. And the tupil itself will be the words. And we're going to call words as features. So if you followed along any sort of machine learning u tutorial, including some of my own machine learning tutorials, you'll know that you've got features. Uh and those features are what makes up, you know, the elements of something. And we use those features to train based on their categories and or tags or whatever you want to call them. So anyway, we're going to have a list of tupils. And the first ele or the zeroith element in that tupa will be the words uh the basically the presence or the nonpres absence there it is of a word. Uh and then the second part of that tupa will be the category. So this will be anyway the list of movie_reviews.words for a file ID. And this is why I hate um oneliners cuz file ID makes no sense at this point but we'll get there. And then category. So again it's it's a list of tupils. So you can even see this is a list of a tupil. Uh and then basically this is where or for the category in movie reviews.categories. So this is basically for category positive or negative and then again for file ID in movie reviews.file ids for the specific category category. Okay. So this is a oneliner. We could put all this on one line. We're just choosing not to cuz that would be absurd. And the oneliner works basically if you wrote this yourself. You would have, you know, documents documents equals an empty list. Then you would do this is the first part. So it's asking, you know, for category movie reviews. That's the first part. Then for file ID, whoops, let's get just this for file ID in the movie reviews colon without the L. And then we would have the tupil that the tupil equals that and we would do something like uh you know documents.append and we would append this tupil of course. So anyway this is actually easier to read. I don't know why everyone insists on oneliners. Sometimes they make a whole lot of sense. They really do make everything nice and compact but this one doesn't. So why anyway uh documents equals that. Good to go. Next uh what we want to do is we're going to do a random.shuffle shuffle of those documents because we're going to train and test and when you train and test you test on separate data that you did not train against because if you train and test on the same data that's extreme bias. So we want to shuffle the documents. So now uh let's go ahead and just print uh documents and uh whoops the first element in the document just to see where we are and make sure uh we're not doing anything too crazy at this point. But it should be documents one will be a list of words and um yes there it is. And then whether or not it's positive or negative. So this is the list of all the words in this document starting with uh Denzel Washington is among the many blah blah blah and then finally when we get to the end this is all the features and then the rating here is positive. So this will be part of our bag of words so to speak. Now, what we're going to end up having to do is take all of these words though, literally take every word in every review and compile them. And then basically what we do is we'll take that list of words and we'll find the most popular words used. And then we take um of those most popular words, which one appear in positive uh text and which one's in negative text. And then we simply just search for those words and whichever one has more negative words or more positive words, that's how we classify it. So, uh, hence the word naive for BA's algorithm. Anyway, uh, documents one. And so now what we're going to do is now that we have all of the documents loaded, we don't need to print documents one anymore. That's a waste. Uh, so I just wanted to show it to you. And now we're going to say all_words is going to be an empty list. And then we're just going to say for W in movie_reviews.words. Uh what do we want to do? W first of all is going to equal W.Lower. We just want to make sure everything's normalized. So we're not going to care about casing in words. So we're just going to convert everything to a lowercase. And then we're just going to say all um and in fact we can just do this all words. All words.append W. So now this adds all of the words to this list. And then later on I I sort of misspoke. The documents is basically what we'll do to create training and testing sets. Um, so we're actually not going to add all the words from documents. This this element here, I suppose. Uh, basically we have the reviews and then all of the words. So this is just just words from all the movie reviews. So this is actually how we're going to compile this massive list of all words. Then later on we use the features of documents which are words to compare. So anyway um so moving on now what we can do is we can actually find out really quickly who are the most uh common things. So first of all words is a list right now. So let's convert that to an NLTK frequency distribution. So we can do all words um equals NLTK frequencist uh of all words like this. And then we can do something like this. Print all words domost common. And let's do the 15 most common words. And then while we're doing that and waiting for that, we'll do uh we can also find out like how many words um are there. So this is still kind of coming up. Let me just pull it up. There we go. So these are the top 15 most common words. So, as you probably recall with NLTK, you have uh punctuation that is classified as being possibly words. So, you can change that if you want. We're not going to bother worrying too much about that right now, but if you wanted to, you can. And and then uh quite a few videos later, we're going to talk about um how to improve the algorithm if we wanted to. But anyway, you've got commas, these, uh period, a and of. like these are basically all words that there's nothing in this list that actually matters to positive or negative at all. So, uh just keep that in mind. Now, all words uh eventually we're going to kind of shrink all words, but all words is actually a huge uh amount of words. I mean, we've we've I think we have like, you know, 50,000 or maybe even more. So, don't worry that the top 15 are kind of useless words. That's just because that's the English language. But um just showing you that how to use this frequency distribution mostly. But you can also do something like this print and then all underscore words and then uh like stupid. Okay. So let's see how many times does the word stupid pop up um in our entire kind of corpus of movie reviews. So we'll kind of wait for this to uh show up. Let me just pull it up here. And um this will mark the end. Okay. Okay, so here we have uh 253 times the word stupid appears uh in these reviews. So keep in mind that we've got 2,000 movie reviews total. 253 times the word stupid appears. So I mean obviously someone might write a movie review and use the word stupid many times. That's totally possible. But you can see that quite a large number, almost a quarter possibly movie reviews have the word stupid in them, which is understandable because we do have, you know, a thousand negative reviews. Uh anyway, or not a quarter, my bad, an eighth because we have 2,000 reviews. Anyways, um that's it for now. We'll continue building on this in the uh next video. Now that we have all words and then we have all the words per document here and their category, now we can actually start to train. So we can train based on the words in each document and the category it has. We can use the naive bay algorithm to say okay well these words are generally positive these words are generally negative. Uh so we can do that and then we'll end up testing it and see how good this algorithm actually is. It's actually pretty basic algorithm. Anyways uh that's it for now. If you guys have any questions or comments up to this point please feel free to leave them below. Otherwise as always thanks for watching. Thanks for all the support of the subscriptions and until next time.