Sentiment Analysis in 4 Minutes

Key Takeaways

hello world welcome to sjy today we're going to be building sentiment analysis in 4 minutes let's get started sentiment analysis is the process of determining the opinion or feeling of a piece of text we humans are pretty good at this I can look at this tweet and immediately know that it's negative it feels like the writer sentiment is one of anger and disgust due to the negative wording companies across the world have implemented machine learning to do this automatically it's super useful for gaining insight into customer opinions once you understand how the customer feels after analyzing their comments or reviews you can identify what they like and dislike and build things like recommendation systems or more targeted marketing campaigns for them in this demo we're going to be building a sentiment analysis program in Python that will identify whether a movie review is positive or negative based on the text in the review we'll get our training and testing data a bunch of labeled reviews from a site called kaggle we'll start off by importing our dependencies we'll import the op operating system module to help us perform commandline functions then we'll want to import the scit learn module which is a machine learning library in Python with a fast learning curve then we'll import a helper class that will help us clean our data pandas helps us read our data CSV files and nltk will be used to remove unnecessary words from our data set all right so step one is to just read the data from our hard disk we'll import the label training data and the testing data then we'll print out the first review to the command line to ensure we read the data set Cor correctly once we've read in our data step two is to clean it that means ensure that we remove all the HTML non-letters and stop words stop wordss are words that are insignificant we can download them from the nltk or natural language toolkit Library words like the or to or as since it's hard to analyze emotion from them we'll iterate over every review in our training data set and fill our new clean review array with the cleaned reviews our helper class will do the cleaning for us step three is to create a bag of words the bag of words model is a simple numeric representation of a piece of text that is easy to classify we just count the frequency of each word in a piece of text and create a dictionary of them this is called tokenization in natural language processing we'll use the count vectorizer object in the pyit learn package to create it we'll set the Max features to 5,000 to keep things simple so our bag of words will contain at Max 5,000 words and their Associated frequencies then we use the fit transform method to fit model to the bag of words and create the feature vectors we can then store the feature vectors in an array step four is to create the classifier a classifier is a machine learning model that will be used to classify whether a piece of text is positive or negative in this example our classifier is a random Forest consisting of 100 trees a random Forest is a set of decision trees decision trees are graphs that model the possibilities of certain outcomes so let's say a piece of text has a word hate appear more than 20 times the probability that it's negative could be something like 80% then based on other word frequencies we increase or decrease that probability accordingly until we get to the leaf of the tree which will be a positive or negative rating this is different from a standard regression classifier where if a data point is on a certain side of the line of best fit we can easily classify it a random Forest tree is more like a series of lines one for every tree that segments are possibilities once we've mapped all the lines onto the graph and we plot a new data point or review based on its coordinates we can then classify based on whether it's in a positive or negative space it's time to test our classifier on our testing data so let's format the test data by cleaning the reviews and creating a bag of words once we have our feature vectors for test data we can move on to the last step the last step is for our program to correctly classify the reviews in the testing data set as positive or negative we use our random Forest to make a prediction we'll then take the result and write it to a new CSV file that's it let's run our program and see what happens okay it printed out the first review that means it's correctly reading our data set then it's going to clean and parse the training set create the bag of words train the classifier then predict the test labels awesome Let's test the first three predictions where one is positive and zero is negative let's see the first three are 1 zero and one so positive negative and positive let's skim these it's truly a masterpiece positive it's so awful that once you know Okay negative awesome looks like it's performing sentiment analysis like a charm sentiment analysis is still an evolving field of machine learning there's so many grammatical nuances and misspellings and slangs involved in human language that we haven't really taken into account but we can with more powerful algorithms so check out the links in the description below for more information and please please subscribe for more technology videos there's so much I want to make thanks for watching