R Tutorial: Tokenization

Want to learn more? Take the full course at https://learn.datacamp.com/courses/introduction-to-natural-language-processing-in-r at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Now that we have looked at a basic way to search text, let's move on to a fundamental component of text preprocessing, tokenization. Tokenization is the act of splitting text into individual tokens. Tokens can be as small as individual characters, or as large as the entire text document. The most common types of tokens are: characters words sentences documents and even separating text into tokens based on a regular expression. For example, splitting text every time you see a 3 digit or larger number. R has an abundance of ways to tokenize text, but we will use the tidytext package - which describes itself as "Text Mining using 'dplyr', 'ggplot2', and Other Tidy Tools" The tidytext package follows the tidy data format. Taking the introduction to the Tidyverse course may be helpful if you are new to the tidy concepts. Throughout this course, we are going to use a couple of different datasets. The first being the 10 chapters from the book Animal Farm. This is a great dataset for our course. Although our data is limited to just the text, and the chapter number, it has a rich character list, themes that repeat themselves, and simple vocabulary for us to explore. The tidytext function for tokenization is called unnest_tokens. This function takes our input tibble called animal_farm, and extracts tokens from the column specified by the input argument. We also specify what kind of tokens we want, and what the output column should be labeled. Our tokenization options include: sentences, lines, regex, for a user-specified regular expression, and many others. We can take this a step further, by quickly counting the top tokens by simply adding the count function to the end of our code. Not the most interesting out