R Tutorial: Text cleaning basics

Key Takeaways

if you have ever heard the phrase garbage in garbage out when creating a model the same applies with text analysis we just learned how to tokenize which can really expose potential garbage in our text let's take the next step after tokenization and create better input text so we get better analysis before we look at some simple pre-processing steps to clean our data I'd like to introduce a second dataset we will be exploring 538 recently published a ton of public data one of these datasets consisted of almost three million Russian troll tweets these are tweets from bots that tweeted during the 2016 US election cycle we will explore the first 20,000 tweets as well as use some of the metadata such as the number of followers number following published date and account type to aid in some of our analysis this is a great data set for topic modeling classification task named entity recognition and others you can imagine tweets probably have a lot of garbage to show this look at the most common words in the troll tweet dataset first we tokenized by words and then we count how often these words occur the results are not that surprising tico is a shorthand for Twitter's web address and was probably picked up when these tweets were scraped from the web HTTP has a similar story but none of our top four occurring words are helpful we need to remove them removing stop words with the tidy text package takes just one additional command tidy text anti join function will remove a table of words from a column of text the typical entry in this table is the word you want to remove and the lexicon or source for where word came from anti join will return the original temple with all stop words removed from the text column note that stop words is a table of common words provided by the tidy Tex package let's look at the results a second time okay so Tico HTTP and HTTP protocol we finally have two interesting top words black lives matter and Trump we will not get political in this course but these are still interesting results we still need to work on those first common words we can add to our table of stop words or create our own here I am adding three stop words to the stop words Tibble HTTP HTTP and Tico we can run through the process of removing stop words and counting the word occurrences one last time we get some interesting results within the first twenty thousand tweets these seven words occurred the most often one additional step I want to cover is called stemming stemming is the process of transforming words into their route for example both enlisted and enlisting would be trimmed to their route enlist this is an important step when trying to really understand which words are being used we will use the word stem function from the snowball see package as it works extremely well with the tidy principles consider this example we want to take our tidy tweets and perform a mutation this mutation will stem the words using the word stem function notice here that matter was trimmed to Matt even though as part of a much larger word and cop was the seventh most common word before stemming but now it has jumped to second let's