Natural Language Processing: Using sequencing APIs in TensorFlow | Machine Learning Foundations

Hi, and welcome back to Machine Learning Foundations, where you can learn the basics of programming machine learning. I'm Laurence Moroney, your host and guide to this wonderful new world of developing software. Last time, in our introduction to natural language processing, we looked at how you can tokenize words with simple APIs. This allowed you to turn words into numbers or tokens, so that they can be more easily represented in a computer's memory. It's the first step in processing language. The next step, which we'll look at in this video, is to turn sentences into sequences of tokens, and we'll explore the tools that make this very simple to do in TensorFlow. So, let's get started. Here's the code that we were looking at last time, where our sentences are represented as elements in an array. We use a tokenizer to turn the words in those sentences into numeric tokens, and we can inspect those tokens by looking at the tokenizer's word index property. But, there's a few differences. First, I've added another sentence to the corpus, just so that we can get a variety of sentence lengths. All of the others were four words long, so I've added one that has seven words. Additionally, I've added this call, tokenizer.text_to_sequences, which does the hard work of turning the array of sentences into arrays of tokens. As the tokenizer already did the job of tokenizing the words, it's nice that it can handle the sequencing, too. But, this is just the first step in what you'll need to do when you're preparing text for NLP. The results of this, if we print out the word index and the sequences, will look like this. We've a lot more words now, and we can see how some of them have been added. Remember, the order of words is the frequency of them in the corpus. So, my is the most common word, and then love, etc. We can see that the new words that we've added, such as amazing, think, is, and do, are there, also. And the sentences have been encoded into numeric sequences. So, for example, 4213 is our first sentence. If you substitute these numbers for the associated words, you'll see I love my dog. So, think about this in machine learning terms for a moment. All along we've had data that we've trained neural networks on, and then we show those neural networks new data with a view to having the network predict what that data is. In the case of pictures, for example, we had images of horses and humans, and after the network was trained on lots of images of these, we'd show it a new one, and then it would tell us if it thought it was seeing a horse or a human. With NLP, we have a similar approach. We'll train a network with lots of sentences, and then those sentences will be labeled. Later, you'll see a data set of headlines, some of which are sarcastic and some of which are normal, and we'll train a network on those. But, think about what happens with a trained neural network when you show it new data. It tries to use what it knows, what it's been trained on, to understand that new data. In the case of words, the network is trained on the words in the corpus. But, what happens when you want to show it new data and have it predict from that? The new data will need to be encoded with the same tokens as the training data, and it will have to be sequenced in the same way with the same rules. So, let's go back to our code and see what this might look like. So, while we haven't trained a neural network yet, we have started to prepare imaginary training data. Using the tokenizer, we were able to get tokens for the words in our corpus and create sequences out of our text. So, imagine now that we've used this to train a network, and we want the network to understand this test text. I really love my dog, and my dog loves my manatee. We should use the same that we used on the training set because we want to have the same tokens. For example, the token used for dog should be the same when you train the network and then later on when you're using and testing it. So, if we were to encode these sentences into tokens and sequences, we'll get this result. I really love my dog would become 4 2 1 3, which if you do substitution would be recognized as I love my dog. Not bad. But, my dog loves my manatee would become 1 3 1, which is my dog my and it's really lost all meaning. The network will not be able to understand this sentence reasonably because words like loves and manatee just aren't in the corpus. You might think a simple solution to this would be to add the test data to the training data so that the training data would contain words like loves and manatee, but that isn't really feasible. You can't train a network with new words every time it sees a not previously seen word and you'd of course end up over fitting where the network is only able to parse data that it's previously seen. There's no simple solution to this, but there is a simple thing that you can do to begin solving it. We'll see that next. Here I've updated the code and I've added a new option on the tokenizer. The oov token parameter allows you to specify a token type for out of vocabulary. Choose a string you don't expect to see in the corpus like I've done here with oov and pass that to the tokenizer. Now, there'll be a new token list with oov being number one. It'll always be number one regardless of how many times it's used. So, the rest of the tokens from number two onwards are in order of frequency and the sequencing of the sentences will use it. So, now our two sentences are encoded as 5 1 3 2 4 and 2 4 If you swap the words back in, you'll get I o o v love my dog and my dog o o v my o o v, which is a small step in the right direction. One important thing when training neural networks is to get the input shape of your data uniform. With images, you saw that we resized all the images to be the same size so that we could have a network fed something of that size to produce a prediction. With language, we generally have to do the same thing. There are exceptions with something called ragged tensors, but I'm not going to be covering those here. Given that our sentences are different lengths, we can get them to uniform lengths using something called padding. We'll see that next. Here's the updated code to handle padding. Let's step through it piece by piece and we'll explore what's new. First, we'll import the pad_sequences API's from tf.keras like this. Now we can pass our sequences to pad_sequences to get back a padded list like this. If we print them out, we'll see the sequences have been padded for us, so they're now all the same length. They're also no longer lists of comma separated values, but real tensors in an array of tensors. So 5324 becomes a bunch of zeros at the beginning. And our longest sentence 869241011 gets into it without any padding of zeros. You saw the first sentence was padded with zeros at the beginning, and that's the default behavior. If you want them at the end instead, you can just say padding equals post. The other default behavior was that each padded sequence was the length of the longest sequence. So the longest sequence had no padding at all. You can change that to another length using the maxlen parameter, and here I've said it's a five. You might wonder what would happen to all of the characters in sentences longer than five. Well, they're going to be truncated. And with this parameter, truncating, you can specify if it's cut off at the end of the sentence using post or at the beginning using pre. You've now seen the steps involved not just in tokenizing all of the words in your sentence, but also in sequencing them into sentence arrays and using padding to get the arrays into the same shape and size. Now, let's take a look at how all of this will work in code. Okay, so here's the code for this Colab. You can try it out yourself in a few moments. What we're going to do is we're going to import the tokenizer, but we're going to extend it now to do sequences. So, we're importing pad_sequences. Here's our set of sentences, and we'll use these with the tokenizer to fit on texts with these sentences so that we can get a word index. The sequences will then call text_to_sequences with the sentences, and they'll be turned from words into sequences of numbers where the numbers are the tokens for the particular word. We'll print them out so we can see what they look like. But, there are also words that the tokenizer wasn't fit to, like manatee, for example. So, when we look at I really love my dog and my dog loves my manatee, we can also use the tokenizer to turn these into sequences to see what those sequences would look like. And here it is. So, first I'll see my word index, and my word index is generated from these sentences. After that, I'll see the sequences for these sentences. So, 5 3 2 4 is five is I, three is love, two is my, and four is dog. So, I love my dog gets turned into 5 3 2 4. Next, we can look at the padded sequences, and we can see that they've been sequenced into five tokens. So, when we look at the fit on text, did say max length equals five, and that's why. So, these three sentences of four tokens each get padded with a zero. This sentence of seven tokens gets truncated to just five. So, now if we look at our test sequence, our test sequences become 5 1 3 2 4, and that's because we're using an out of vocabulary token as one. So, 5 1 3 2 4 becomes I out of vocabulary, which was the word really, three is love, two is my, and then dog. And when it comes to my dog loves humanity, well, we have two, which is my, four, which is dog, out of vocabulary, my again, out of vocabulary. So, it's my dog out of vocabulary, my out of vocabulary, that type of thing. And then, of course, we can pad our test sequences, and in this case, we said the max length was 10, and as a result, we get these test sequences, which are much longer. Give it a try for yourself, experiment with the words, and see the kind of results that you get. All of that code is at this URL, where you can try it out for yourself. Pause the video, give it a try, and come back when you're done. Up to now, we've just been using hard-coded sentences to experiment with tokenizing and padding. Before you can train a neural network, you're going to need to read in text from a data source. In the next video, you'll see how to do that, reading thousands of news headlines, tokenizing and sequencing them. After that, you'll be ready to start building your first models that understand language. So, stay tuned, and don't forget to subscribe. Thank you.