Build a Custom Transformer Tokenizer - Transformers From Scratch #2

Key Takeaways

hi welcome to the video we're going to have a look at how we can build our own tokenizer in transformers from scratch so this is the second video in our transformers from scratch series and what we're going to be covering is that the actual tokenizer itself so we've already got our data so we can cross off now onto the tokenizer so let's move over to our code so in the previous video we created all these files here so these are just a lot of text files that contain the italian subset from the oscar dataset now let's maybe open one ignore that and we just we get all this italian now each sample in this text file is separated by a new line character so let's go ahead and begin using that data to build our tokenizer so we first want to get a list of all the paths to our files so we are going to be using the path lib you could also use os lister as well it's it's you import so sorry import path so from pathology import path i'm using this one because i don't know i've noticed that people are using this a lot at the moment for machine learning stuff i'm not sure why you would do it over os list there but it's what people are using so let's you know give it a go see how it is so we have this and we just want to create a string from each path object that we get so for x in and then in here we need to write path and in here we just want to basically tell this where to look so we're using path here and we're just in the same directory so it's not we don't really need to do anything here that's fine and then at the end we are going to use glob here i think this is why people are using this and we just create like a wild card like we want all text files in this directory so we we just write that now let's do that i'll look at the first five and see that we have our our text files now so that's good and what we can now do is move on to actually training the tokenizer so the tokenizer that we're going to be using is a byte level by pair encoding tokenizer or bp tokenizer and essentially what that means is that it's going to break down our text into into bytes so with most tokenizers that we you probably use unless you've used this one before then you use it for we we tend to have like unknown tokens so like for birth we use sentence piece encodings and we have to have this unknown token for when we don't have a a token for a specific word like for some new word now with the bpe tokenizer we are breaking things down into bytes so essentially we don't actually need an unknown token anymore so that's i think pretty cool now to use that we need to do from tokenizers so this is a another hugging face package so you maybe you need to you might need to install that so pip install tokenizers and you want to do byte level bpe tokenizer like that okay now we take that and we're going to initialize our tokenizer so we just write that that's our tokenizer initialized we haven't trained it yet let's train it we need to write tokenizer train and then in here we need to include the files that we're training on so this is why we have that past variable up here so this is just a list of all of the the text files that we created which are all separated by new line characters each sample is separated by a new line character now the vocab size we're going to be using a roberta model here and i think the roberta model typical roberto model vocab size is 50k now i mean we can you can use that if you want this up to use but i'm going to stick with the typical bert size just because i don't think we need that much you know we're just figuring things out here so you know this is going to mean less training time and that's a good thing in my opinion we don't set the min frequency so this is saying what is the minimum number of times you want to see a word or a part of a word or a byte so it's kind of weird with this tokenizer before you add it into our vocabulary so that's all that is okay and then we also need to include our special tokens so we're using the roberta special tokens here so writes special tokens and then in here we have our start sequence token i'm going to put this on the new line so not not like that like this so we have this start sequence token the padding token end of sequence which is like this the unknown token which with it being a by-level encoding you'd hope it doesn't need to use this very much but it's there anyway and the masculine token so that's everything we need to train our model and one thing i i do remember is if you train on all of that all of those files it takes a really very very long time which is it's fine if you're training it overnight or something but that's not what we're doing here so i'm just going to shorten that to the first 100 tokens and maybe maybe i'll train it after this with with the full set let's see so i will leave that to train for a while and i'll be back when it's done okay so it's finished training our tokenizer and we can go ahead and actually save it so i'm going to import os i'm just soon so i can make a new directory to store the tokenizer files in and a typical italian name also i've been told is filiberto which fits really well but so this is this is our italian italian bert model name philiberto so that is our new directory and if we just come over to here we have this working directory which is what i'm in and then we have this new directory philiberto in here that's where we're going to save our tokenizer so we just write tokenizer save model and here we can can do we you can see here we can do save or save model save just saves a json file with our tokenizer data inside it but i don't think that's a standard way of doing i think this is the way that you want to be doing it and we're saying it's filiberto like that so we'll do that and we see that we get these two new files vocab.json and mergers.txt now if we look over here we see both of those and these are essentially like the two sets of tokenization for our tokenizer so when we feed text into our tokenizer it first goes to mergers.txt and in here we have characters words so on and they are translated into these tokens so these are characters on the right tokens on the left so we scroll down we can see different ones we can keep going so here we have zeone that's like although my challenge very bad that is like the english t ion so tion and we we would say stuff like attention right italians have the same but they have like attention so that's what we have there so it's part of a word and it's pretty common and that gets translated into this token here now after that our tokenizer moves into vocab jason and i don't know what side of the at the bottom there go to the top if i clean this up quickly we can see we have a json object it's like a dictionary in python and we have all of our tokens and the token ids that they will get translated into so we if we scroll down here we could we should be able to find was it va i think okay so va which is our zeone into this token here and then that eventually gets converted into this token id so that's our full tokenizer process let's open that file back up if we wanted to load that we would do that like we normally would with transformers so we start from transformers import roberta so we're using a roberta tokenizer here so we're about to turkenizer we can use either the robot tokenizer or the fast version it's up to you and we just initialize our tokenizer like that we from pre-trained and in here rather than putting a model name from the hooking face website we would put the path local path to our directory our model directory so it's philiberto for us and then we can use that to begin encoding text so go ciao coming back which is like hi how you if we write that we can see that we get these are the tokens here i wonder if we did a 10 um certain i'll do i'll try in a minute so we have the start sequence token here and the sequence token here so the the s and the [Music] s like that so we have those at the saw and end of each sequence and we can also add padding in there so padding equals max length and also max length needs to have a value as well so maximum 512 and then we get these padding tokens which are the ones so that's pretty cool and i just want to let's purely mark curiosity anything else so we have potentiona let's see if we if that if we recognize the number there so no we don't so i suppose this is probably the the full before word in fact it is so this is a the full token here if we if we just do this maybe we will get i can't remember what number it was just the three three two two maybe maybe that's right i'm not sure but anyway that's that's how everything works so that that's it for this video in the next video we will take a look at how we can use this tokenizer to build out our input pipeline for training our actual transformer model so say everything and i'll see you in the next one