Twitter Sentiment Analysis - Learn Python for Data Science #2

Key Takeaways

[Music] Uh, check out the Two-Minut Papers YouTube channel for some cool research videos. Hello world. It's Siraj. In this episode, we're going to learn how to write a Python script that uses Twitter to understand how people are feeling about a topic that we choose using the natural language library text blob in just 14 lines of code. People really confuse me. Like, how does Donald Trump have so many supporters in 2016? And why are beers manly and fruity drinks girly? Oh, and why do people think anything is beautiful? Like what even is beauty? The soundtrack is beauty. Human preferences are practically unpredictable. And we've invented the sciences of psychology and sociology to help us study these things. Both are the scientific study of people and all the emotion and relationships and behaviors that they display. Traditionally, psychologists would formulate a hypothesis. Then to test it, they would find a subset of people that fit their category, bring them into the lab, then ask them to do some task while recording the results. But with data freely available, we data scientists can do the same thing with the best psychology tool out there, Twitter. Twitter is a treasure trove of sentiment. People around the world output thousands of reactions and opinions on every topic under the sun every second of every day. It's like one big psychological database that's constantly being updated. And we can use it to analyze millions of text snippets in seconds with the power of machine learning. So, how does sentiment analysis work? Well, when we receive some input text, like a tweet, we'll first want to split it up into several words or sentences. This process is called tokenization because we're creating small tokens from big text. We can just count the number of times each word shows up once the text is tokenized. This is called the bag of words model. Then we could look up the sentiment value for each word from a sentiment lexicon that has it all pre-recorded to classify the total sentiment value of our tweet. Okay, so our process will be to register to access Twitter's API, install our dependencies, then write our sentiment analyzer script. Let's start by signing up to use Twitter's API. An API or application programming interface is a gateway that lets you access some servers internal functionality. In our case, Twitter. So, we'll be able to read and write tweets right from our app using Twitter's API. Let's sign up for it in the browser. First, we'll want to make sure that we have our own Twitter account set up with a verified phone number attached to it. Once we have that, we can create our first Twitter app. We'll select the create new app icon, then type in a name and description of our app, as well as a website. The website field is kind of like a liberal arts degree. It doesn't really matter. Then we can click the create button at the bottom, and our app is created. We can click on our app and go to the keys and access tokens tab. We're going to need to use these keys in our script to let us authenticate or verify our identity with Twitter. Okay, let's move on to installing our dependencies. Our first dependency is called Tweepy, which is our library for accessing the Twitter API. Our other dependency is called Text Blob, which will help us perform the actual sentiment analysis. Before we get to the code, let's do a little text blob demo right from the Python interpreter in terminal. We can access the interpreter by typing in Python. That's it. Now we can write Python directly into terminal. We'll start out by importing text blob. Now, let's create our first text blob. This is just a piece of text. We'll call our variable blob and assign it this value. We'll wrap a string using the text blob method and our string will be Siraj is angry. He never gets good matches on Tinder. Now that we have our blob variable, if we want to see parts of speech, we can just access the tags attribute. If we want to tokenize this text, we can just type in the words attribute. And we can see the sentiment of this blob easily by accessing the sentiment.polarity attribute. That's on a scale between negative 1 and 1. All right, first things first. We want to create four variables that authenticating with Twitter will require. We'll find these in the dashboard and we can copy and paste each of them as strings. Let's authenticate with Twitter which basically means login via code. To do that, we'll create a variable called O for authentication and use the OOTH handler method of Twei. This method takes two arguments, the consumer key and the consumer secret. This method was written inside of the Twe Pie library with a bunch of code that is hidden to us. We don't need to know the details for our use case. We can just name the method and all of its functionality is in our hands. The arguments are what the method uses to perform its internal calculation, whatever it is. We're halfway through authentication. The other half is to call the set access token method on the O variable, which takes two arguments, the access token and the access token secret. That's it. We've created our authentication variable. Now, we want to create our main variable from which we'll do all of our Twitter magic. We'll call it API and assign it a value from the Tweepi API method which takes a single authentication argument which we'll fill in right here. Now that we have our API variable, we can perform a whole bunch of different possible methods. But what we're going to do is search for tweets. For our use case, we want to collect tweets that contain a certain keyword. So to do that, let's create a public tweets variable that is going to store a list of tweets. To fill it, we'll call the search method of the API variable. The search method takes a single argument. Let's make it trump. This method is going to retrieve a bunch of tweets that contain the word Trump. From here, we can print them all out to terminal. To do that, we'll create a for loop. For tweet in public tweets colon a for loop increments through every value in a list. So, the question now is, what do we want to do with each [Music] tweet? Okay, let's print them out. Each tweet has a text attribute, which is the string version of it. So, we'll print tweet.ext. It's time to perform our sentiment analysis. We'll create an analysis variable that will store our analysis and call text blob with our tweet string as the only argument. Then we can print out the sentiment attribute of the analysis variable. We can see each tweet related to Trump as well as its sentiment analysis. Polarity measures how positive or negative some text is and subjectivity measures how much of an opinion it is versus how factual. So to break it down, sentiment analysis is the process of extracting and understanding human feelings from data. An API lets you access a server's internal functionality like Twitter right from your app. And the text block Python library lets you perform sentiment analysis in just a few lines of code. There's a lot more we can do with our sentiment analyzer. And the challenge for this week's video is to extend this code to make it create a labeled CSV file with the clean tweets of whatever topic selected. Details are in the code readme. Post your GitHub link in the comments and I'll announce the winner in next week's video. The winner for the gender classification challenge from the last episode is Naresh Nagubashan. I especially liked how he created a dictionary at the end to store the result from each of the three classifiers, then used the argmax function to print out the one with the best result rather than doing a bunch of if then statements. Badass of the week. Also, the runner up is Victor Matay. Very clean code. For more info, check out the links in the description and please subscribe for more programming videos. For now, I've got a Siraj, the president's on the line. You're our only hope. I've got to save the world. So thanks for watching.