Python Tutorial: Advanced tokenization with NLTK and regex

Want to learn more? Take the full course at https://learn.datacamp.com/courses/introduction-to-natural-language-processing-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- In this video, we'll take a look at doing more advanced tokenization with regex. One new regex pattern you will find useful for advanced tokenization is the ability to use the or method. In regex, OR is represented by the pipe character. To use the or, you can define a group using parenthesis. Groups can be either a pattern or a set of characters you want to match. You can also define explicit character classes using square brackets. We'll go a bit more into depth on groups and ranges soon. Let's take an example that we want to tokenize using regular expressions and we want to find all digits and words. We define our pattern using a group with the OR symbol and make them greedy so they catch the full word or digits. Then, we can call findall using Python's re library and return our tokens. Notice that our pattern does not match punctuation but properly matches the words and digits. Let's take a look at another more advanced topic, defining groups and character ranges. Here we have another chart of patterns, and this time we are using ranges or character classes marked by the square brackets and groups marked by the parentheses. We can see in this chart that we can use square brackets to define a new character class. For example, we can match all upper and lowercase english letters using Uppercase A hyphen Uppercase Z which will match all uppercase and then lowercase a hyphen lowercase z which will match all lowercase letters. We can also make ranges to match all digits 0 hyphen 9, or perhaps a more complex range like uppercase and lowercase English with the hyphen and period. Because the hyphen and period are special characters in regex, we must tell regex we mean an ACTUAL period or hyphen. To do so, we use what