Transformers in NLP | GeeksforGeeks

Key Takeaways

hello everyone welcome to the session so in this particular video we will talk about Transformers in NLP so this is quite vast topic and I believe that uh you know you have heard this term Transformers a lot many times but many of the students are not aware about the concept of Transformers or what are the applications where nowadays this Transformer architecture is being used let me tell you that uh you know it is nothing you know bigger it's just a variant of I would say machine learning or the Deep learning Concepts that we are uh usually covered up in our different different architectures but why there is an advancement in Transformers and nowadays you know apart from NLP it is widely used in so many other applications it is just because that the data set that we have used or the developers have used to train the model is something which is quite vast which is quite big and that's why you know the results that we're getting from the Transformers which is a state of the art model in NLP is quite good as comparable to the other models that we have right so I'll talk about a very uh you know basic application which we are using nowadays quite frequently and I hope that you must have used this language translation I hope you all know right what is language translation so in language translation If You observe what will happen for example I am the person who just knows Hindi language I want to translate that language from Hindi to English because um in a country maybe newest who and with the people who are not aware about Hindi language so usually that is the only alternative that I have where I can convert via a mobile phone or Google translator to Hindi to English for example I am saying in Hindi so what will happen is that whatever be the Hindi word sent or the sentence that you are saying here that you have so what will happen you will pass this Hindi word to something which I would say is the Transformer based model to something that I am saying is the Transformer based model now what does Transformer based model will do it will try to convert whatever in the word or the sentence that you said like to something that we are saying into a English word into a English word so when I am saying meh I don't know much Hindi because I am writing quite after a long time so suppose so it is converting that Transformer based model to a English word that is how are you and that kind of development that kind of translation happened just because of this Transformer based model so it is nothing but it's a encoder decoder based Network it's a encoder decoder based Network where you will observe that in an encoder we are inserting the input for example in my case in this present scenario the input is so what we are inserting we are inserting an input to the encoder encoder will process that input now usually the very first paper which is uh developed by Google people and tornado people is attention is all you need so it's developed in 2017 I believe there is a paper I I will suggest everyone after this session go and just have a look onto this paper by the name of attention is all you need I'm not sure why I am unable to write properly here attention is all you need just go and refer this paper this paper was developed in 2017 and in this paper for the first time the Transformer architecture was there and that Transformer architecture you will observe is the one which we are talking about in today's session so there is a complete Transformer model architecture which you will be easily able to see in this particular paper so what I am saying is that that Transformer architecture is nothing but it's a encoder decoder based architecture where in an encoder so it's a encoder decoder based architecture where in an encoder we are passing a input for example in this particular problem our input will be the Hindi sentence that we are we have passed so here we are passing the input then we will be having a layers of encoder and after that the final output of encoder will be passed to the all the decoders and further the English sentence will be given by the decoder which is responsible to generate the output this is how basically the encoder decoder architecture will be developed initially it's a six layer based architecture which you will see a stack of six layers of encoders and six layers of decoders encoder is taking the input processing it and onto all the six layers final output of encoder will be passed to all the decoders that we have and finally the output will be generated this is how the complete architecture of Transformer will look like when you go and see the research paper but now after some time a lot of variants of this Transformer came into picture people or researchers started you know working very keenly towards the architectures of Transformer what they did is that they have taken completely only encoder portion a stack of only encoders and now layers will vary so I am just writing a stack of encoders a stack of encoders now how many number of layers it depends because there are so many architectures of Transformers nowadays we have and that's where the concept of bird came into picture similarly there is another concept which again you have heard of so everything is just a version of you can say Transformer only where if I'll take a stack of encoders itself a different different kinds of architectures been developed by the name of bird similarly if I take only a stack of decoder if I'll take only a stack of decoder anyone any idea that what's that version of model is called as where I am taking a stack of decoders present in the Transformers and that is something that we called as GPT generative pre-trained Transformers now why it is called generative because here it is responsible to generate the context generate the text out of the given sentence for example you will see that there are a lot of you know examples here for example if I'll say Once Upon a Time what I want from my model is my model will automatically complete that story right my model will complete that story so in language translation which I just now talked about one application here somehow I would say a mapping is required mapping of words is required right but the second example which I was talking about is the language generation task and GPT is a language generation model so in language generation task where GPT plays a very important role here what we want is for example if I will say that I love reading novels so what I want is given this is this has the input text if I will pass this input text to A GPT model now again there are a lot of versions of GPT you will see gpt2 you will see gpt2 large you will see gpt2 medium depending upon what kind of architecture that you are using but fundamentals are same but you will observe that for every architecture you are using a decoder stack only what it will generate it will generate a complete you know story out of it because you will observe that these GPT models have been trained on a huge amount of data set and that's why we are calling it as a pre-trained models because the training is already been done on a huge amount of data set with the help of which it is just trying to fetch the records the data where this particular line resembles so now here you will be easily able to understand that somehow we will be having a Transformer which is we will be able to understand is the encoder decoder type of architecture where it is responsible for many of the real-time application tasks one more example which I can give you is the auto completion task where for example you will observe that in in the mails while writing if you're writing something automatically further the points or the sentence sentences it is trying to give you Auto suggestion there all these are the applications of the Transformers which are very widely used nowadays now the question that you can ask from me is that how it is going to work the very first important thing is that you will observe in the in the Transformer architecture there is a component called attention again that's a very important uh you know um concept but just I want to give you overview that what are the components that we have in Transformers one is feed forward which is nothing but is a multi-layer perceptron that we have in deep learning apart from that you will observe that we will be having something called as attention something called as attention what does that indicate what is the meaning of attention and why it is required it is required to understand the context because if my model will not be able to understand the context how it will be able to you know complete the sentence whatever applications I am talking about right or how it will be able to map and convert it into a proper English language from Hindi to English maybe from French to Hindi anything so understanding of context is the bigger problem in the initial cases in the initial part but then there is a concept of you know attention came into picture for example I can just give you one example I will say the I'll just write one sentence the chicken didn't cross the road the chicken didn't cross the road because now suppose this is the sentence because it I want the sentence to complete now I told you right this is the completion of the story which is again an application of Transformer which can be easily done by GPT to GPT 3 models now the question is but here we need to understand that how my model will get to know that this it will represent the chicken or it will represent the road how so for understanding the contextual part that okay this particular word represents the road or maybe the chicken then only my model will be able to generate or complete the story out of it right and that's where the concept of self-attention or the attention model came into picture and that is what it is trying to have in their encoder decoder architecture specifically in decoder architecture I would say attention is there so that it will be able to understand the context out of it out of the given sentence and then only it will be able to generate the output and that's where the biggest you know advantage or the results success will came into picture before that we do have the different different kind of techniques to do the same task but the major problem that we were facing is the contextual understanding in NLP specifically but afterwards that problem get resolved right so that is something that we are we want I want to talk about in this particular specific session where I just want to give you a Transformers architecture overview now uh specifically you can ask me one question that okay you told me that it will be automatically able to complete the sentence but how is that possible or it will be able to do the mapping how is that possible obviously when we are we are passing it to a Transformer model our model doesn't understand any of the character doesn't know I love reading novel if I'm writing what is the meaning of I love reading notebook what it it will do internally is the very first task is it will try to tokenize the sentences into the words tokenization of the words will happen and then for every word there is a mapping with the numeric IDs so you will be able to generate a numeric IDs out of that now after getting this numeric IDs my model will be easily able to understand the numeric numbers then we will pass those numeric IDs to the model and then my model will be able to understand the context out of it and that's where the concept of word to vector and all those things whatever with the word embedding techniques that we have came into picture because my model doesn't know what's the meaning of the character that we are using here what my model know is the numeric numbers and finally whichever numeric number is having a higher probability of having a next word so for example if I am saying I love reading now I don't know what is the next word so what is the next word is my why is my y this is the target value which I want to look for and this is something which will become the input feature that we have and this is where the concept of conditional probability came into picture where what we want is what is the probability of Y given x 1 x 2 and X3 as the input feature and what we want is we want that kind of word which is having maximum conditional probability here which is having maximum likelihood here and that's where the overall concept came into picture and which server is having a maxim whichever word is having a maximum probability that word will become the next and this is how complete text will be generated in a generative task right so this is overall idea behind how the generation will happen I can demonstrate you one very simple demo here so that you will be able to understand whatever I am talking about here but you know uh you have to understand the different different uh you know hugging face libraries to uh to get the overall understanding of the same just give me one second I'm just connecting to GPU what I can do is let me connect it now in order to you know use any of the Transformer based models or any of the pre-trained models in Transformers you need to first of all install the Transformers you need to first of all install the Transformers so that you can use any of the pre-trained models present in the Transformers I would suggest go today and see the list of the pre-trained models that we have in Transformers here if I'll just install it it will automatically able to install now from Transformers what kind of you know pre-trained models you want to use you can import that again this is a huge list for example I want to use gpt2 tokenizer to tokenize my sentence I want to use GPT to LM head model which is a pre-trained model that we have to generate the text so it will be automatically downloaded now now after that what will happen is we will be able to directly call the tokenizer here which is nothing but is equals to GPT to tokenizer Dot gpt2 tokenizer you need to download that obviously so here you need to Define what is that pre-trained model that you want to use here for example I am saying I want to use a GPT to large pre-trained models so this particular model is having a huge number of parameters which is already being trained on and a huge number of who huge size of data set which has already been trained on now here if you will observe I'll use my model my model is generative model is GPT to LM head model and inside this model what I will do is again I'll call this pre-trained model which is nothing but GPT to large and I'll mention the pad token ID which is nothing but is equals to tokenizer Dot end of sentence token ID why I am writing this because it represents the vocab size so usually if I'll just show you the encoding part of this this much huge is the vocabulary size that we have and for every particular word we will be having a numeric ID so if I'll just try to decode this I'll show you this is the indication of the end of text that we have so let it run it will take some time and after that I I'll show you that how we can tokenize the sentences so let me write one sentence here meanwhile for example I am saying I love watching motivational movies okay this is a sentence that I have and I want to generate a numeric ID out of this particular sentence which I am saying so what I will do I will first of all encode that the meaning of encode is to convert a character to a numeric ID so what I'll do I'll pass the sentence and I'll pass what is the return tensor which I want to return it I want it in the pie torch form so that is something that we have so see it is taking some time if you have a subscription of collab Pro so it will do much faster job as comparable to what it is doing right now so here you can see that we will be having a 50 000 approximate vocab size if I'll just decode it you will be easily able to understand that it is representing something called as end of text here if I just want to show you that how every particular word is been converted into a numeric ID that's the real objective which I want to show you here so if I just print what is the numeric IDs I am getting here then you can see that we will be having 40 one eight four two four nine so for every particular character I will be having a numeric ID how can I say that what I will do is I will say tokenizer just to show you demonstrate you dot decode and here I am saying numeric IDs zero one what it is indicating I think it is indicating love right one eight four two if I'll say what is indicating by indication by 0 1 2 3 four five movies right so you will observe that uh zero one two three four okay it is four so it will be movies so you can observe that starting from 0 1 2 3 4 so we will be having a little five words and for every five words we will be having corresponding numeric ID and that is what I was talking about for every particular word first of all is the conversion to a numeric ID and this numeric ID we will use to pass to the model to generate the further text now how it will do let me just show you how it will do okay so here you can you can see that what I am saying I am passing this numeric IDs here and I am passing what is the maximum length of the sentence which I want number of beams is just for the optimization purpose as of now just are not going in very much step towards that just I want to demonstrate you that how the model generation work and how we will be able to get an output using the Transformers because I have Illustrated the concept uh theoretically so here if you will see if I'll just show you the result here and then if I'll decode that tokenizer dot decode what is the result I am getting skipping the special tokens equals to true I'll just show you how the result is going to look like you will very soon see that we will be able to get a complete sentence out of it and how it is possible only with the help of the Transformers that's the amazing application that we really have in the Transformers that's the capability that we have and that's why it is widely used nowadays here by the way in the meanwhile let me talk about the parameters which I am using here numeric IDs I have shown you that whatever sentence we are picking up maximum length how much is the biggest sentence that you want number of beams is it will keep on checking the probability that whichever would be the next word having the higher probability that what it will pick up when I'm saying 5 at a time it will pick the topmost five probabilities and finally which server complete sentence sentence is having a higher probability that particular sentence will be the one which would be the final one no repeat engram size is something which is saying that there will be no 2 grams will be matching there is no duplicacy early stopping equals to True indicating that there should not be overfitting in the model and finally we will be able to get the result here you will observe that once you will be able to generate the result the result will be in the form of a numeric number or numeric IDs but again for every particular word we will be able to get the numeric IDs but we don't know the meaning of those numeric IDs right in order to get what the sentence will look like what we will do finally is we will try to decode it we will try to decode it so that we will be able to get the final output out of it so let it run see it it can take some time and you can see right uh it is using heavily the ram as well and the disk size as well so let it run so I hope that you will be meanwhile easily able to get an idea that what a Transformers in NLP is all about so I would say that it's really an amazing you know uh model a state of the art model and in the next upcoming videos I will for sure talk about birth in very much depth I haven't touched upon but in a depth part although in GPT I have touched up one and I have shown in the implementation as well how it will be able to generate the text out of it basically the Foundation is the conditional probability which I already have talked about and I hope that you already being aware about what is conditional probability and how basically it will be easily able to give us the probability of a Target given the input features you can ask me one more question that how will I decide how many input features should I take it should be sequential you can take three four five according to your own is but don't take too much and don't take too less depending upon the context size you can refer let's see whether it will be able to get or not so it is taking time maybe let me do one thing if I can maybe reduce it to suppose 50 let me just try to stop it and just do it again if I will be able to get a faster result I believe I should because now though length is only 5. otherwise I have to reduce it to further you are connected okay I am getting the output I believe so see I will be able to get 50 words here but these 50 words are the numeric IDs again I am getting further I need to decode that so if I'll just copy this and if I'll just paste it here uh not properly visible so what I can do maybe I can open a one word pad here and I can show you see I love watching motivational movies and this one is one of my favorite it's about a guy who is trying to lose weight but he can't seem to do it so he goes to a weight loss clinic where he meets a woman and so on so basically there is a proper story behind it and it seems like that someone you know human being is coming and telling to me and that's the power of Transformers That's The Power of training with a huge data set that we have on the web and this is something which I really want to explain to you all in this particular video if you still have any sort of Doubt regarding the understanding of how Transformers basically work and whatsoever context which I have given you here please try to go through the research paper attention is all you need I have written it down right there a complete full-fledged architecture is also there you can go and try to check it out and there is a complete example which I already have given you but you if you will read one more time you will be able to get a better understanding of the same if you still have any sort of Doubt do let me know I'll should try to resolve it as soon as possible with this happening to all bye bye everyone and I'll see you all in my next upcoming videos