Analyzing Twitter Accounts with Python and Personality Insights

Key Takeaways

have you ever gone through some people's twitter profiles and thought what the hell is going on through their head well in today's video we're going to take a look at how you can demystify those people by building a twitter personality analyzer yep that's right you're going to be able to input a twitter handle and pre-process an entire user's twitter feed to get a view of what their personality actually looks like so by the end of the video we'll actually take a look at a couple of key celebrities profiles and take a look at what makes them tick now let's dive a little bit deeper as how this is going to work so we're going to be covering a bunch of stuff today but specifically we're going to be covering how to set up a twitter dev account and an ibm cloud personality insights account we're then going to take a look at how we can query twitter using the twee pi library with python and then last but not least we're going to be able to visualize profiles using that twitter data and using matplotlib and seaborn so in terms of how we're going to be doing it we're going to be using a jupyter notebook and connecting to twitter using twepi we'll then take that tweet by data or that twitter data pre-process it and use the ibm personality insight service to generate a user's profile so this is going to allow us to look at things like their personality their needs as well as their values and then last but not least we're going to visualize it all using matplotlib and seaborn now one of the coolest things about the personality insight service is how rich the data is that you actually get back so in this case we're taking a look at a sunburst chart so you actually get a whole bunch of information you get information on users needs their beliefs their values again this is all based on their twitter data but it's still really interesting to see what you actually get that so in this case you're looking at a sunburst chart that was done for oprah in the past so you can see her level of openness conscientiousness how extroverted she is her level of harmony and a whole bunch of other stuff but you can see that you get really really rich information and it's actually quite cool so without further ado let's get right into it and get to building our twitter personality analyzer so the first step in building our twitter analyzer is getting twitter data so in order to do that we can use the twitter api now in order to get access to the twitter api you just need to go to developer.twitter.com and then hit apply so once you get to that apply page all you need to do is hit apply for a developer account and you're going to be prompted to log in and eventually you'll be able to get access to your developer account so in this case i've already got access to a developer account so we can actually kick that off and actually start building our twitter analyzer now the first thing that we want to do once we get into our twitter developer account is go over to our name and select apps if you haven't created an app before what you'll first need to do is hit create an app put in your app description so in this case i've said that our particular app is going to be to do with post curation so fill out all of your app details you can leave some of these blank and hit create once the twitter developer account team has given you access you'll be granted access to an app that looks a little bit like this so in this case my app name is called pi post curator if i actually step into this app and hit details i'm able to access my keys and tokens now these are going to be really important when we go to actually start connecting to our twitter data using python now the next thing that we're also going to need is a personality insight service so ibm has this amazing service called personality insights that allows you to analyze profiles just using text so in this case we're going to be using personality insights to pass through our twitter data to generate our personality insights so in order to generate a personality insights profile all we need to do is go to cloud.ibm.com forward slash catalog and select ai machine learning over the side here and then from there you're going to be able to access the personality insights service so if we select personality insights all you then need to do is select a light and so that's going to basically give you a free api so you can start to generate these personality insights so in this case i've already got a light profile set up so we don't need to go and do that again but assuming you're getting started just hit light and hit create and you'll be able to set that up so if i hit tiered i'll be able to do the same thing so once you've got your personality inside service set up you're basically going to be redirected to a page that looks a little bit like this so you'll basically have a getting started tutorial and a credential section so this is going to be what contains your api key as well as your url so these are going to be the two things that you need from a personality insights perspective to be able to generate your twitter profiles all right cool so now that we've got that set up we can get started and actually build our twitter analyzer so our twitter analyzer in this case is going to be built using python so we're going to be using a jupyter notebook so i'm working inside of cloud pack for data on ibm cloud but if you've got a jupiter notebook you can do that as well so once we've got our jupyter notebook set up the first thing that we want to do is basically give it a little bit of a layout so rather than just working with a blank canvas i like to try to lay out the code that we're actually going to populate so in terms of what we actually need to do here there's four key things so we first up need to install our dependencies connect to twitter and grab our twitter data set up personality insights and generate our personality profiles and then last but not least we're going to visualize it using seabourn and matplotlib so let's go ahead and lay out that profile first and then we'll start writing some code so in this case in order to layout our profile we're just going to lay out a little bit of markdown so let's do that first perfect so that's our markdown laid out now let's go ahead and install our dependencies so i've written import i actually meant install so let's go ahead and do that so we can do that really easily inside of a jupiter notebook all we need to do is include an exclamation mark then type in pip install and whatever we want to install so for our twitter analyzer we're going to be using twepi so that's just a really nice library that allows you to work with the twitter api a whole lot easier and we're also going to need the ibm watson dependency as well awesome so those are our dependencies installed so you can see we've installed twepi and ibm watson now let's go ahead and get some data from twitter so we've installed our dependencies but we actually need to import them into our jupyter notebook so let's go on ahead and do that so in this case we're going to import os to work with our operating system we're also going to import twee pi to work with twitter and we're also going to import pandas to work with data frames all right so those are our dependencies imported so we've got os we've got twee pi and we've got pandas now what we want to do is set up a couple of variables to hold our twitter credentials so in this case we need a consumer key or api key a secret and an access token as well as our secret token so let's go ahead and set up variables for that and copy over those details from our twitter developer account all right so we've set up our variables now we need to copy this data from our twitter developer account so remember when we set up our app we were able to go through the keys and tokens and generate these api keys as well as our secret keys so in this case we need to copy over our api key and our api secret key so these are going to represent our consumer key and our api secret key then we also need to get our access token and our secret access token so those are available down here so you're only able to generate these and see them once so in this case we need to regenerate them and then we'll be able to copy them perfect so those are our keys set up now what we need to do is just authenticate to our twitter api and then start grabbing our data so let's go ahead and do that all right cool so that's all done so basically what we're doing first up is we're setting our oauth handlers so in this case we use the twee pi api and we pass through our consumer key and our consumer secret then what we do is with our oauth handler we set our access tokens so these are our two variables that we've got here and then we instantiate our api so in this case we pass through our authenticator and we connect to our api so if we run that we can now start grabbing our data from twitter so in this case we can work with a number of handles so my personal favorite at the moment is trevor noah absolutely love him so let's go ahead and set up a variable for our handle then what we can actually do is start to grab that data so we can do this through our api so let's go ahead and set that up perfect so we've just gone ahead and grabbed our data from twitter so you didn't see too much happen there because we weren't actually outputting it let's quickly go through that code and then we'll actually take a look at what we've got so in this case we're using the twepi library which we set up here and we're accessing our user timeline we then pass through the handle that we want so in this case we're passing through trevor noah and we're limiting the number of tweets to 100 and we're also including our retweets so what we can then do is take a look at our response and you can see that this is all of our twitter data here now there's a lot of stuff here and it's a little bit messy but what we can actually do is clean this up and just extract the text from our tweets so in this case here you can see that we've actually got a text key and we've actually got the text from the tweet itself now what we can do is we can actually loop through each one of these tweets and just extract that text so let's go ahead and do that and that's going to make it a whole lot easier to work with it later on awesome so we've now gone and extracted our tweets so now if we take a look at that you can see that we've got all of our tweets coming out there so this allows us to grab all of our data from twitter so if we wanted to change this for example if we wanted data from oprah so really quickly with a couple of lines of code we're able to pull out our data from twitter and start working with it so let's leave it as trevor noah for now and then what we're going to do is we're going to append all of these strings together so that we can start to work with it with personality insights so basically what we're doing is we're joining all of these tweets together into one big block of text so let's go ahead and do that okay so that's now done now what we can do is actually take a look at the text so in this case it'll just be one big block of text and you can see that we've taken all of the individual tweets that we had up here and we've just combined it into one big string so this is going to make it a whole lot easier when we want to bring it back into our personality insight service so now what we want to do is actually go and set that up so in this case we've sorted out our twitter feed we've got our twitter data we're now up to step two setting up our personality insight service so there's a couple of steps to setting up our personality inside service so we just need to again import our dependencies connect to the personality inside service and then perform our scoring or sending our text data to that service so let's first up start out by importing our dependencies awesome so those are our dependencies imported so in this case we've just imported the personality insights sdk from ibm watson and we've also imported the iam authenticator so this is just going to allow us to authenticate so now what we want to go and do is grab our personality insights api key and url so remember we set it up when we were setting up our personality insights api so what we can do is copy our api key and paste that in and copy our url as well and then what we'll do is we'll store these in two separate variables so we'll store our api key in a variable called api key go figure and we'll also store our url in a variable called url now what we're basically going to do is authenticate to our personality insight service similar to what we did for twitter but now we're doing personality insights awesome so we've now authenticated to our personality inside service so let's take a quick look at what we wrote there so the first thing that we did is we set up an instance of our iam authenticator and to that we passed our api key so this basically allows us to authenticate against the personality insight service we then created a new instance of our personality insights sdk so basically we've now got personality insights available and we've set two keyword parameters so version as well as passing through our authenticator and then we've basically passed through our service url so this is basically where our personality insights api resides in the world wide web so that's pretty much all of that done now now what we can actually start doing is actually building our personality analyzer so because we've already got all of our text stored in a big text block here we can now send this to our personality insights api so let's go ahead and do that and start taking a look at our profile there you go we've now gone and profiled trevor noah's twitter account so you can see here that we're actually getting a whole bunch of stuff so it's just a big json string that's been returned so this isn't the best way to review and try to work out what's actually here because you can see that there's actually a fair bit of information included in our profile the best way to actually analyze this is to extract the stuff that you actually want to take a look at so in this case what we'll do is we'll loop through our personality our needs and our values so we'll be able to extract each of those components so when we actually generate a user profile that's what we're going to get back so we'll get traits to do with their personality we're also going to get a profile of their values as well as a profile of their needs so what we can do is go on ahead and extract those out and see what's most relevant okay so we've now gone and extracted our personality components so you can see that for openness trevor now has got a score of about 73.9 percent for conscientiousness 56.5 extraversion 49.54 and agreeableness 31.62 so again we can do this for the other components within our profile so rather than just having those we can also do it for our values and our needs as well so if we copy and paste that all we need to do is change this profile component to values and you can see that he's about getting a score of about 52.9 on self-enhancement uh 40 on hedonism and 63 on an openness to change we can also take a look at the needs and you can see that he definitely likes a challenge uh he's super curious um he's got a bit of idealism and he likes structure so again we can do this for another profile so say we wanted to do it for uh lebron let's grab his handle so if we went go to twitter and we search for lebron james so his handle is king james all we need to do is copy over that handle so in this case we just copy over king james and all we need to do is reset our handle here and go through all this again and now we should have a profile for lebron james so again you can see that he definitely likes a challenge um he's got a reasonable level of idealism love stability and love structure so really quickly we've now gone and generated personality insights for both lebron james as well as for trevor noah so the last person that i wanted to do it for is for tiffany haddish so again let's go and grab her twitter handle so in this case it's just tiffany haddish so again we're just going to change our handle and again run through this and you can see we're now extracting each of those different components out for each of those twitter profiles so i've done trevor noah lebron james and tiffany haddish but again we can do this for virtually any twitter profile that's out there now so far we've just been working with numeric results but what we want to do is we actually want to visualize these profiles as well so in order to do that we can use cborn and matplotlib to actually go and visualize these people's personalities so let's go on ahead and start doing that so again we're going to start out by importing our dependencies so in this case we're going to need matplotlib and we're going to need cborn so we've already imported pandas so that's going to allow us to work with data frames so let's go ahead and bring those in alrighty awesome so we've imported our dependencies so we've brought in matplotlib and we've brought in cborn now let's go ahead and actually visualize our profile so there's a little bit of data wrangling that we need to do but not a whole heap and then we'll immediately be able to visualize these profiles so let's go ahead and do that also drop a mention in the comments below for whose personality you'd like to see profile so let's go ahead and start visualizing these profiles okay so what i've gone and done is a little bit of data wrangling so we've basically extracted our needs out of our profile we've then gone and converted it into a dictionary to make it a little bit easier to work with and then we've loaded this data into a pandas data frame so then what we can actually do is view that data frame so you can see that we've actually gone and taken out our needs data and we've converted it into a table that we can really easily visualize now what we can do is pass this data to seaborn in order to visualize it as a bar plot or any other plot that we want so let's go ahead and do that last step awesome and there you go so we've now gone and taken our needs data and we've gone and converted it to a bar plot so in this case you're probably thinking well you've only got five values there when we use the head function we basically show the top five results within our pandas data frame so if we actually remove our head function or a head method you can see that we've got all of those values let's put that back on you can see that we can actually take a look at tiffany's profile and you can see she's super curious she's um got a lot of harmony lots of self-expression um stability and structure again is still pretty high pretty cool hey so in this case we're just visualizing the needs now if we actually take a look inside of the big five personality profile there's actually quite a fair bit of information there so let's go on ahead and visualize that in a similar manner to what we just did here so in order to do that i'm just going to create a function and we're just going to loop through each one of those different components of the big five personality profile okay so we've gone and created our plotting function now if you take a look what we've basically done is we've just taken a lot of the code that we had within our baseline visualization and paste it in in there so in terms of what we've actually got so again we're looping through our traits and we're converting it into a dictionary with the first key being the name of the trait and then the percentile we're then again converting it into a data frame and then last but not least we're visualizing it so let's go on ahead now and actually loop through each one of these traits and visualize them awesome so we've now gone and visualized our different profiles you can see that our titles are just showing up as needs so we should just change that quickly this should actually be name here then if we run that again so you can see each of the different components as part of the big five personality profile now so you can see the degree of openness conscientiousness extroversion agreeableness and emotional range so really quickly we're able to get a high level profile of each of our celebrities or virtually any profile that we've got within twitter so i do quite like that tiffany haddish is quite fiery she's awesome probably one of my favorite celebrities and favorite comedians out there so you can see that there's a whole heap of really really cool personality data that you can get simply by connecting to twitter using tweetpie and then visualizing it using personality insights again if we wanted to rerun this on any other twitter profile all we need to do is just go up to here and change the handle that we want to run it on so again let me know in the comments below whose personality you'd like to see analyzed on that note this about wraps up how to build a twitter analyzer using python and personality insights thanks so much for tuning in guys hopefully you found today's video useful if you did be sure to give it a thumbs up and hit subscribe and if you get stuck at all or if you need some help from a technical perspective be sure to drop a mention in the comments below and i'll get right back to you thanks again for tuning in peace you