Scrape Twitter Data in Python with Twitterscraper Module

Key Takeaways

hello everyone ken here today I'm showing you how to scrape Twitter data using Python and the Twitter scraper module so on my screen is the Twitter scraper documentation it is also in the description below as you can see you can do it from the command line but I prefer to do it from the notebook it allows me to read it in directly into a data frame and allows me to manipulate it from there to get started we install it pip install Twitter Ripper I've obviously already downloaded it so all the requirements are met here now I'm going to open up spider which is my ID of choice for data science so from Twitter scraper and for query weeks now we're also going to import date/time because we want to set a date range for the tweets and we're going to import pandas because we want to turn this into a data frame after without so the first thing we're going to do is query tweets takes a couple parameters the first one and a relevant one to us is begin date is equal to so all in a query is something recent I'm gonna do the notre-dame fire that happened very recently that'll be a good example to show us something that is relevant and a lot of the information associated with that on Twitter will also an end date we're gonna do tomorrow today is the 17th for me and that's because it just takes all of yesterday's yeah well today's data as well and that's relevant if you just want to do a time window for example in one of my previous videos I did the timing around a movie premiere we can also put in a limit so let's say this is a really really popular topic we can set a limit of a thousand or so so we don't go crazy we don't have millions and millions of tweets that would take forever to download now we can also set a language so lang equals English in this case if we're doing the Notre Dame fire there's gonna be a lot of French tweets I would expect so we'd want to filter those out so it's understandable for me mainly if we wanted to just look at one person tweets we could also just query them we would set the user is equal to let's just say Rio Trump if you wanted to here is wacky comments on the Notre Dame fire you'd be able to type that in and filter specifically for that now for the real magic here we just query tweets so and we put in our parameters right here we're asking for better than fire which is our key component that we're gonna query you can also put hashtags anything relevant that you look like there we do begin date equals begin date end date equals we're gonna set the limit and so we can load all of this in and this should take just a couple minutes if you're doing anything over ten hundred thousand it can take up to you know twenty thirty minutes actually wrong the way they do this is they just have from my understanding a bunch of threads hitting Twitter so it will it depends fairly heavily on the processing power of your computer if you want to increase the speed or not so looks like we got a thousand twenty which is pretty close to our limit so I can't complain too much it it isn't super exact just because of I guess the the structure of the back end there now we want to make this relevant to us as you can see the tweets here are just in tweet objects that don't really mean it down to us yet so let's transform this into a data frame so data frame is equal to all right so after doing that we have all of our Twitter data we have the username we have the number of likes replies etc you can filter by that if you want or you can order by it etc and we also have all of the tweets as you can see there's a ton of duplicates so that's something that I like to remove duplicates of a lot of them are news related most of the URL and the actual username of the person who posted these tweets you can do a lot of really interesting stuff with this I've done the sentiment analysis in the past on Captain Marvel the movie premiere and you can see that above you can also combine it all make a word cloud related to the event or the topic that you're covering and there's plenty of other text-based analysis that you can do with this information this is a great tool it's fast easy and free and I highly recommend that you use it thank you so much for watching my video if you enjoyed it please like if you want to see more content like this please subscribe have a great one