Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Processing

Key Takeaways

hi and welcome to episode 8 of machine learning foundations I'm Laurence Moroney from the Google AI team and I'm here to be your guide through the basics of machine learning up to now you've learned how machine learning works and explored examples in computer vision by doing image classification including understanding concepts such as convolutional neural networks for feature identification an image augmentation to avoid overfitting making your networks that little bit smarter we're now going to switch gears and we'll take a look at natural language processing in this video we'll take a look at how a computer can represent language and that's words and sentences in a numeric format that can then later be used to train neural networks this process is called tokenization so let's get started consider this word it's the English word listen and it consists of six letters we're used to reading it based on the sounds and putting those sounds together to form a word but how can a computer understand this word well one way as computers deal better with numbers than they do with letters is to assign a number to each letter a common coding format is called ASCII where common letters and symbols are encoded into the values from 0 to 255 it's useful in that only one byte is needed to store the value for a letter but it has been superseded by later encodings in order to give access to characters and letters beyond 255 in particular international characters but for the purposes of illustration we can stick with ASCII where for example the letter L is 76 I is 73 and so on so we now have the word listen encoded into six bytes one for each letter now this is a perfectly valid encoding and often when you use neural networks you'll see character encoding or sub word encoding and stuff like that they lead to things being a little bit more complicated and in these tutorials I'm going to do word based encoding and not the letter based that we just saw now why would I do this one reason is that if we're taking a word as a set of numbers unless we take the sequence of those numbers into account we can have two words sometimes with opposite ish meanings like this and they can have the same letters thus if we want to use character based encoding a computer can't tell the difference between these two words unless we have a sequence model and that's a little bit more complicated than we need to look into right now so let's consider a different encoding and that's a word based one that way each of these words can be represented by a single number and each number will be different there's also a nice hidden advantage to this which we'll see in a moment so consider this sentence I love my dog it's pretty straightforward one if i encode based on words I can come up with an arbitrary encoding say the word I is number one and then love my dog become 2 3 & 4 respectively now if I were to encode another sentence for example I love my cats the words I love my already have numbers so I can just use 1 2 & 3 again for them and I can create a new number for cats which I'll say is number 5 so now my sentences are 1 2 3 4 and 1 2 3 5 what's interesting here is now that the words are gone and the tokens for the words are just used we can begin to tell that there's a similarity between the sentences so maybe we're beginning to get a glimpse at what it might look like to have sentences turned into numbers yet maintain some kind of meaning the process I just outlined is called tokenization and it's an inherent part of doing natural language processing or NLP tensorflow gives you AP eyes that help you to achieve this very simply we'll take a look at them next here's all the code that you would need to tokenize the sentence as I showed earlier we can break it down and go through it line by line the tokenizer tools are part of the tensor flow Karros libraries and they're in the pre-processing namespace so make sure you import these I'm going to hard-code the sentences into an array now while this is a super simple corpus there's just two sentences and five unique words this design pattern can work for much bigger sets of data you'll soon be working with tens of thousands of sentences with thousands of unique words and it's all pretty much the same code so don't worry right now if the looks a little bit too simplistic you can then create a tokenizer with a lowercase T by simply creating an instance of the tokenizer uppercase T and initializing that with parameters one of these is the num words parameter which specifies the maximum number of words that you want to care about there's only five unique words here so it doesn't really make a difference but with larger sets of text it can you'll commonly encounter bodies of text weren't many thousands of unique words in them and lots of these words may be only used once or twice by specifying the number of words that you care about in your tokenizer you get an easy way to filter those out the tokenizer is smart enough to assign tokens to words based on how commonly used they are in the corpus so the most common word will be at index 1 the next common word via index 2 etc etc to get the tokenizer to do its job you can fit it on texts and pass it your corpus of text in this case it's our simple array of sentences to see the word index that the tokenizer created you can just get the word index property this will give you a set of name value pairs where the name is the word and the value is the token for that word and then you can just print this to inspect it when you print it out it won't necessarily be in any order but keep an eye on the values like I said earlier the most common words will be the lowest index and in this set I love my appears twice while dog and cat both appear once so I love my are the lower index words one two and three and dog and cat are the higher indexed ones four and five so what if we expand our sentences and then add some more content like maybe you love my dog with an exclamation mark and note that exclamation the default behavior of the tokenizer is to strip punctuation out like this it can be overridden but we'll keep it in for now it also makes all of the words become lowercase so my capital deed dog will be treated in the same way as a lowercase D dog the tokens will now look like this notice that they've moved around a little love is now the number one token because it's the most used word and it's similar with my also notice that dog lost its exclamation and there's only one token in here for dog and it represents both usages of the word despite the exclamation being on the second one and we've added a new word u because that was first used in the new sentence that was added to the corpus so I hope you found that pretty straightforward despite the underlying power in the tokenizer next up I'm gonna step through a collab with the code for all of this and then you can try it out for yourself okay here's the URL of that collab pause the video to give it a try for yourself this is the code that we were looking at in the videos here we can see we're going to import the tokenizer and i have a number of sentences here i love my dog lowercase i i comma love my cat uppercase i and things like you love my dog with an exclamation will then use the tokenizer and we'll fit it on the text of these sentences and take a look at the word index so here if we take a look at the word index will see that the word love is indexed as number one it became token number one and that's because love was used the most it was also my it was used the most as well there are three of those just like love and you can see there at the top of the index so for example if I were to add another sentence here like hello hello hello hello I am in a place called vertigo and then I were to run it on this and then would see hello is now the top one because there are four of them where there are three loves and three minds that kind of thing oh I also now shows up because there are three eyes so this is basically how the tokenizer works hopefully this will be useful for you have a play with this code to try some examples yourself and see what you can come up with great I hope that was easy for you in this video you got your first taste of NLP using tokenization where you were able to take sentences and have no words encoded into tokens the next step on your journey is to replace your sentences with sequences of tokens you'll see that in the next video so don't forget to hit that subscribe button and I'll see you there [Music]