How LLMs REALLY Understand Text: Positional Encoding & Attention Explained

Hello hello hello. So uh let's continue our LLM uh series and uh in previous video we talked about um our inputs and how we uh convert our input or statement into input tokens. We talked about by pair encoding. In this video we will talk about attention weights. We will talk about attention score. We will talk about positional encoding. uh why it is important and where we are going uh with this is uh the topic of this video. So um let me take a real life example to make you understand this concept in a similar way. So let's say if I say that we uh fought with each other u and uh I got hurt. So in this statement you the context of this statement is that we had some sort of argument and we forward with each other right and I got hurt in that process. But if we uh complete this statement and uh if I just add one more thing that he abused me then I fought with each other like then we fought with each other and uh in that process I got hurt. The context of uh the first statement and the context of the second statement is totally different because we have new information that the other person has abused or had said some nasty things to me and that's why the fighting was taking place. So in LLM also when we uh talk about anything when we give input uh to our LLM it makes a context matrix or context vector it associates every word with a context vector. But how we know that how how to make that context vector uh which word is more important with us uh in in in a in our input uh with respect to uh the words which we have given. Let's say I have um uh given a word or a sentence that uh I am u making you I am explaining LLM to you guys. So in this statement which uh input or which token or which word is more important uh in context of which word is the whole idea of making a context vector or context matrix. So to get to our context matrix, we will go through the series of concepts um uh first of which is positional uh encoding. Second one will be your uh attention uh weights. Then third one will be our attention score and after that we will go towards our uh context factor. So uh let's start from positional encoding. So um I'm trying to make this video as simple as possible and for people who are just interested and not from a STEM field. So there must be some u details or key words which are very basic to our STEM people from a STEM field. But I think I should also explain those uh uh words to people who are not from STEM field. So uh let me make this two three words very clear for for the people who are not from STEM field or from science field as we say in India is there are two types of uh properties or how we measure things one is scalar property and one is our dot property or vector properties right so uh let me give you an example to make it easy to understand so let's say if I say the weight is of 10 kg or 10 lb for US audience or international audience then we are just talking about the magnitude of that weight like it's a 10 kg weight or 10 lb weight but if I say that um uh the I I am accelerating uh my car in the direction of north with let's say 30 km per hour or 30 mph then it's a vector property so the difference between scalar and a vector property is that that scalar property only has magnitude 10 kgs 10 pounds. Uh but the vector uh quantities have a direction also associated with them like acceleration in the direction of north or let's say I have uh the current is flowing in the direction of south. So with some uh magnitude associated with that. So vector also has a magnitude plus direction. uh why have why I explained vector to you is because we will use the vector again and again while explaining the positional encoding. So that's the vector part of it. Okay. Uh there there is one more concept let's say let's talk about when we talk about higher dimensions because our LLM functions in a very higher dimensions but uh for simplicity of this video I will keep it to the uh two three dimension only X Y and Z. So to uh uh make you understand what is XYZ dimension. So the video which you are watching right now is in 2D because it's a it has length and width to it. So it has X-axis and a Y-axis to it. But when you watch a 3D uh movie in theaters and all you feel something coming towards you like it has length and width but something feels like coming something feels that it is popping out of screen towards you then that's your Z-axis which is third dimension I hope that is clear like our 3D dimension or vector now why I have explained vector in 3D dimension to you before I go on positional encoding is that when we uh take our input tokens uh which uh we got by embedding or by doing by pair encoding uh in our previous step uh we say let's take an example here also let's take an example of I am explaining you guys uh LLM or I am explaining uh uh you guys about LLM or something like that right so let's take If I take LLM as our token for which I need a vector drawn towards it. Let's say this is an LLM token from that statement. LLM is a LLM is our token which is u let's say which uh will uh guide the vector like it will give us this attention weight of every token associated to LLM. So if I take out that LLM token in our uh uh from our statement as you can see on plot then which another token in our statement is more important or have higher pro uh priority to make a context more clear is uh what we know as attention weight or attention score like which token I should give more attention to get the real context of LLM. I that was wrong. So to get that context or attention weight, which token should I put more emphasis on to get the context of this statement uh with associated to LLM token is the is the score which we get is known as attention weight. I think I got misdirected in that statement. So keep it simple. We have a statement. I am trying to explain you guys LLM. LLM is we picked out LLM as in token. So we got our LLM token. Then we go through the statement. We go through every token like every word. Then we see okay which word in our statement is most important to get the context or to get the meaning of our statement. like you guys or LLM in associated to LLM. So if I plot this in a graph, you can see that uh the closest one would be guys or you because that gives us the context of our uh this statement. So how we uh make our attention weights? So we go through every uh we go through every token or word uh and we plot or we plot a vector to our LLM token here like this this this because we are plotting only three because we are uh um using three dimension space only if you go in a higher dimension space uh you will uh like plot more vectors towards LLM token right so let's take some different example let's take uh u uh u start from a single step right. So if I have to plot this and I if I take u start start as our token in associated uh to that token I want to plot that vector with every token like you will have three vectors towards our uh start token then start from our sentence will have three vectors to our start token and then so on and so forth. So you will get a grid of three three three uh score like 0.1 0.4 0.8 is for you uh 0.3 0.8 0.1 is for uh uh start and so on and so forth and I'm I'm putting it as an example and as I can see like as I will put some graphic here to make this explain like more clearly. So that score which we got like how close a token is with another token is known as our attention score uh or attention weight. So we get that attention weight from there we will calculate our attention score which I will explain you guys in the next video what is an attention score and after getting the attention score we go towards our context vector. So to get our context vector we have to get the input tokens. We have to find a positional embedding. I'm uh let's say again if we take an example of I am teaching you guys an LLM. So if I want context around LM so I would take LLM out and I will plot a vector for uh towards LLM for every token. So I will have if I am using three dimension will have three uh vectors pointing towards LLM. So I will have different three different scores. Then uh U will have I am teaching teach M will have three vectors towards L&M. U will have three vectors towards L&M and so on and so forth. And that score which we get like how close uh the token is to another token is known as attention vector. I hope uh it I'm able to explain you guys that the attention weight and uh positional embedding. So yeah in next video uh we will go towards towards L attention score and context vector and how we get our context vector because uh to get to our context vector we make a different like weight matrix, query matrix and key matrix and uh then from there we calculate our context u vector or we get our context matrix there. So yeah I will see you guys in the next one. Till then take care. Peace. Byebye.