Was Captain Marvel Bad? A Sentiment Analysis of Twitter Data

Key Takeaways

hello everyone can hear back with a new data science video today I wanted to do something special I saw that there was a lot of really negative reviews of Captain Marvel on YouTube and some other resources and I wanted to see what the crowd thought what the Twitterverse thought so I am doing a sentiment analysis on Captain Marvel I'm also comparing it to Avengers infinity war horse in context if this interests you if you liked the video please hit that like button and if you'd like to see more like it please subscribe I do this analysis in Python and a couple of the modules are very helpful and actually getting this doc so the first module that I'd like to give a shout out to is Twitter scraper so it's more simple than Twitter trust API you don't need any login or credentials or anything like that and it also gets through some of the limitations of that API where you can go back in time as far as you want which is really cool I also use the Bader sentiment tool which it takes a lexicon approach to sentiment analysis so it has a big dictionary of words and it attaches sentiments to each one of those and also based on the context that the word is used it gives it a sentiment score there are a couple different levels of score so they mark how many negative features used how many neutral features you use and how many positive features you use and that's all combined into one composite score bitter is one of the best practices in the industry it's not perfect some things do slip through the cracks but at a large scale it works very well and I'm excited to actually build this into my analysis I haven't had a ton of exposure to sentiment that's why one of the reasons why I'm actually doing this because it lets me fool around with some new tools and to explore a very interesting topic you can find three interesting links in the description below the first is a medium article that highlights well that was actually the inspiration for this video that talks about using sentiment analysis to analyze the anthem game lock the second is another medium article about the details of how they that works and the last is a link to my github where you can find the entire code base for this analysis so let's just jump into how I set up this analysis I'm not gonna go through all of the code but again if you want to see that that is in my personal github the first thing we do is import all of the relevant modules pandas were going to be using because we're converting a lot of this into a data frame to analyze it date time we're gonna be using to filter by the specific time of day and I talked about Twitter scraper and Vader sentiment analysis we're also using lang detects because we only want to evaluate english-language tweets Vader does not work as well in other languages from my experience and so that is something that we really need to do to get the most useful analysis possible we're also including some visualization tools mouth portlet and Seabourn to make it look pretty I created this detector function because for the language detection we need to make sure that it is a texturing that it's this big fit in if you feed in a number or something that is non text you will get an exception thrown so this is just saying that if there is an exception we put none in instead then we build this analyzer object and create some of the parameters for us to query the tweets I'm looking at you know three or four days before the premiere and until now basically after the premiere so we will right here actually end up queering the tweets this takes 10 or so minutes it takes a while we then put them into dictionaries so that we can put them into a data frame finally down here is where we actually sort for the English language now I would recommend saving these to CSV files your data frames now because it'll save you some rework in the future if you you know mess up Python crashes you will have that saved so you don't have to run the Twitter query again which is fairly time-consuming now let's jump into the actual sentiment analysis so right here this is pretty much all we have to do we apply our analyzer clarity scores to each of the text segments in the tweet and we get those four features that I've mentioned before the amount of negativity the amount of neutral nests the amount of positivity and the composite of those things which is your actual sentiment score we then append that to our data source and I looked at a lot of these tweets and it looks like there were a ton of duplicates so those were advertisements or things along those lines and I wanted to get rid of those so we get a true or feeling of what the people really thought so I dropped the duplicates here finally I was having some trouble filtering by date time so I switched our timestamp into day time and made sure we were only getting after premiere tweets that were actually after the premiere of midnight the day after the movie came out after looking at a histogram I saw the majority of the tweets had a sentiment score of zero which means that they weren't necessarily negative or positive for our analysis sake I removed all of those tweets because I wanted to understand the actual parity between before the movie and after the premiere had launched needless to say I was pretty surprised by the results the even before leading up to the movie it had a fairly positive sentiment score of 0.3 for and after the movie premiere the sentiment score actually went up to 0.42 that's almost a 20 percent increase which is fairly impressive it suggests to me that a lot of the Twitter community at least enjoyed the movie even in spite of what has been said on YouTube and some other mediums are in this same analysis on Avengers infinity war from last summer and I also got very interesting results Avengers before the premiere had a sentiment that was positive but lower than Captain Marvel of right around point one for after the movie had premiered the sentiment dropped to point one that doesn't surprise me a ton because of how controversial the ending of that movie was but it still surprises me that in general all of the sentiment was lower now I'm not qualified to make any claims about the quality of Captain Marvel but this analysis suggests to me that the general Twitter community didn't feel too badly about it that it was fairly well received in spite of what a lot of people were saying about it and the ratings that it got thank you so much for watching and please let me know what you thought of the movie in the comment section below if you have any questions about my analysis please leave them there as well until next time