Code Your Own Llama 4 LLM from Scratch – Full Course

Key Takeaways

This course is a guide to understanding and implementing Llama 4. Vuk Roshik will teach you how to code Llama 4 from scratch. The course starts at the very beginning and teaches everything step by step for beginners. Llama 4 full open-source model just dropped and it's currently second on the Chatlm Arena blind testing website. In this video, I will presume you don't know anything about LLMs and I will teach you how to code lama 4 from scratch. So we will start with like basics, basic mathematics and basic Python code and code everything from scratch. If this stuff looked complex to you, it will be so simple now that I explain it. It's going to be so simple you will understand everything in one hour, two hours, I don't know. But once you understand this, every LLM is same. You will know how to code not only llama but quen deepseek gpt gemma gemini claude everything everything under the sun. In my previous course coding a deepseek from scratch I got some comments mathematics was way over my head. So right now I made mathematics so simple so intuitive so understandable that anybody can understand it. Just a quick heads up if I in the beginning of the video mention quen 3 instead of llama 4 accidentally. Long story ignore it. This video is about llama 4. You can find all of the uh materials, presentations, courses code in in this repository. So it's vauosich/courses on GitHub uh link below and you can support me on Patreon to help me bring free value to uh people to YouTube so I can do this full-time. Good to have intermediate Python and high school math. However, even if you are lacking in some of these areas, I will still try to explain everything. Also, there is no need for a high performance machine or laptop because in this video, we are going to learn the base mathematics and the coding how to code it. So, this is what you need as a base and after you learn this, you will be able to train any kind of model, make your own adjustments, train it on any machine. So we will focus on training in one of the future courses. But in this video we will learn basics of LLM large language models and how to code one from scratch. Realistic expectations. I don't want you to think that just in like a day or two you can learn all of this because this is really a PhD level mathematics that I will be explaining. So we are be we are going to be learning uh everything from high school maths to PhD maths and it will feel very dense and require months to understand but when you understand it once it's almost exactly the same for every LLM chlama deepseek everything almost exactly the same not only will you be able to create any LLM large model but you will also be create be able to create any AI that's built on the transformer architecture, it will be very easy for you as well. So these are all of the latest AIs. A lot of them are converging and using transformer architecture, almost all of the latest ones. So even though I try to include all of the information, everything in this video, this course, there are so many good courses and videos on YouTube explaining other or specific concepts. So there are thousands of hours of courses that you can watch besides this course. And I will also be posting more of these courses. So you can gradually grasp all of these concepts over a few months. So if I'm explaining a concept in this course, you can also search on YouTube for maybe different kinds of explanations to understand it even better. Let's start with large language models. This is going to feel very simple in the beginning for a few minutes, but I encourage you not to skip because it will escalate real quick. And by the way, you can find this presentation, this PDF below the video. So if I'm explaining to a beginner what is a large language model, I would say AIS you can chat with using text messages. So this is not a precise definition, but this is for somebody who who sees this for the first time. This is example chat conversation between human and AI. So I send a message and the AI responds. To be more precise, LLM large language models are AIS that generate text and they are usually adapted to work in a chat interface where you can chat to them. Famous LMS include GPT which powers Chad GPT website, Claude, Gemini, Deepseek, Quen that we are learning about today, Grock from XAI and Llama from Meta AI. So if you understand Quen, you will understand 90% of all of these. Sorry, 90% 90 of all of these how LLMs work. LLM takes some text as input, analyzes it, and predicts a next word that makes sense. It's important for you to understand the base, the original LLM has no concept of what a chat conversation is. This is just something that looks at text and predicts the next word and that's it. For example, the sun is shining and the sky is this would be the LLM input and the LLM output would could be blue. LLM predicts blue here because it makes sense. But as I said could be which means it doesn't have to be. So actually LLMs don't just predict a single word. They have a list of all possible words, letters, characters they can choose from. And for each word or letter or character, they calculate probability that this is the next word based on the context previous context. Now I will show you more realistic example. So LLM input is the same. the sun is shining and the sky is the model will would assign probabilities to every possible word character letter token whatever it knows it has in its vocabulary. So for example, blue is the next word 35% probability clear 25 cloudy 17 and for every single thing so apple doesn't really make sense. So probability 0.002 002 banana sim some letters some random characters some dots all of them will get low probability because they don't really make sense here and we will talk more later about what does it mean to make sense for the AI how does AI like what what what do I mean by make sense one important thing is that all of these probabilities will add up to 100% because if LLM says As blue has 35% chance, we need to know that it's 35% of all of the tokens, this entire vocabulary. And if the sum was not 100%. Then this 35% wouldn't have much meaning because we wouldn't know 35% of what. So when LLM has probability distribution, what are some ways to pick a word? Well, an obvious way is greedy sampling. We just pick the most probable word. But this would make LLM very repetitive. So maybe it's good for tasks like mathematics where there is only one correct solution but it's not good for tasks like writing an essay where you need more creativity although creativity is not a concept that exists in LLM. You can you can say randomness not creativity. So we can increase randomness of words of selected tokens. It's better for writing an essay. Random sampling. So we choose any word based on probability assigned to it. So word blue gets chosen 35% of the time, clear 25, etc. Top K sampling. We have a number K, for example, three, which means we choose from the three most likely words. And the way we choose from them can now also be random sampling or any other way to choose between these probabilities. These are just few examples but there are many ways to pick a word and in a lot of the times like four or five ways are used to sample. So LLMs will have for example four different uh methods all combined. So the first method would narrow down from which words we would pick. Second, third would also narrow down even further. And for example, the last one would just pick a word from there. Another thing we can do is modify these probabilities. For example, if we are doing a mathematics or coding problem where there is more of a single correct solution or only a few correct solutions, then we want to modify all of these probabilities such that higher ones get even higher and even sharper. So we want to increase higher and decrease lower. So the higher ones which are probably the correct solution to the math problems have even higher chance of getting picked or if we are writing some essay do doing some creative writing then we want to increase randomness. Now people will say creativity but what people mean when they say when they say creativity is randomness because this doesn't include concept of creativity. It's just randomness. So what we if we want to make it more random then we want to bring probabilities closer together. So high probabilities we want to lower them and low probabilities we want to increase them to bring them closer together. So this is called temperature sampling. If you've ever seen temperature in some API settings or something, this is what temperature does. it modifies these probabilities. So imagine it as some formula that you that that's applied through to this probability distribution that modifies probabilities. And the way this formula is set up is if you move temperature towards zero or at zero then it will uh these probabilities high will become even higher and low will become even lower. So we will sharpen this. So that's why low temperature is used for math and coding where we have only one correct solution or only a couple of correct solutions but higher temperature for example towards two in many APIs in many APIs you can set temperature between zero and two but numbers don't matter here uh it can be between zero and million it doesn't matter but higher temperature towards the higher end will make probabilities come closer together. So it will be more random as I explained late before. So we will use them to write essays for example. And by the way on my channel right here I also have a course on coding deep sequ 3 from scratch if you are interested. Let's go on to next concept. So here I have words but this does not have to only contain words. It can contain characters, symbols, different words, different languages, Chinese characters, Japanese, everything, emojis. So, one term that's used for all of those together is token. So, LLMs are now generating the next token, not the next word, not the next character, the next token. And token includes all of the possible things I just mentioned. Tokens, anything can be added as a token that AI can generate. That means predict based on preceding text. So let me explain it better. AI scientists and engineers in the beginning of the when they of the creation of AI LLM they are creating this vocabulary of tokens that LLM can choose from when predicting the next word the next whatever the next token and in that vocabulary they can add anything they want. Here are some examples. Words, letters, cars, Chinese characters, Japanese, whatever. Even whole words, sentences or phrases. It can be a single token that LLM learns what it is and when to use it. You see, we have emoji token, we have number token, we have Chinese character token, a complete sentence token, a complete Chinese sentence token, and arbitrary token. that's like random letters, characters and stuff. And during the training of the LLM, it will learn the meaning of each of these things. So let's go to the next slide. Let's talk about car token. It will learn that car is a vehicle. It has wheels. People drive it. it used in transformation transportation and many other concepts around the car. It will maybe even associate brands like BMW and Lamborghini and other brands with the car. If a token is for example B, model will learn that it's a second letter in English alphabet. It can start words like book or bird. It also learns where and how it's used to construct words and many other concepts around letter B. Going back to word car, we say that model learns semantic meaning of the word car. And by that I mean what do you think in your head when you say the word car? That's what model tries to learn. Now does it learn that exactly? It's a highly debatable right now. Everybody's trying to debate research that what's actually learning. But we know that it does it can use it as a vehicle. It can it knows that it has wheels. It knows all of this stuff. How exactly does it know it? We don't know 100%. So we can make it. We can build the model, but we don't know exactly how it works, how it understands the semantic meaning of the word car. We just have some ideas that I will also explain later. Let's look at this example token that's one one two three. So model learns that it can be used to represent a quantity. So when there is 123 of something, it can say there is 123 of this. Uh it knows that these are numbers that follow mathematical rules and it needs to use it. If it's using it in formulas, these numbers need to follow those rules. So it appears in context of counts, measurements, dates, etc. And also this can also look like a beginning of se of counting sequence when I'm counting 1 2 3 4. So maybe if model if user asks model to count numbers 1 to 10, it can use this token as the first token to start the count and then uh generate next tokens. Here we have an example where token is an emoji. Model will still learn in which context to use this token. For example, if it wants to maybe express some emotion or make the conversation more light-hearted. Also, it knows that this represents forest or tree or maybe can be used as like a representation of life or greenery or forest scenery. And I also want to show you how it handles other languages and especially for example Chinese characters. And I chose Chinese because I'm trying to learn it in my free time and so I'm also able to explain it better and understand what the model learns from it. So it will see this character that's pronounced ch and it will know that it's a Chinese character and it means car or vehicle and it can be used to create other compound words for example teacher or it's used in the context related to transportation and vehicles. It has a specific stroke order and writing p pattern and many other things associated with this as it's learning many other things associated with with every of the tokens. A token can be a whole sentence as well. The sun rises in the east. So model would learn that this describes a natural phenomenon that happens daily. It has subject sun verb rises and prepositional phrase in the east. It's also stating a fact that's universally understood. It can be used literally or metaphorically in different contexts and many other things associated with this phrase. We can also have entire sentence in Chinese or any arbitrary things. So, so it would learn that this is in Mandarin Chinese language. It means we are learning artificial intelligence. It mean it's it's going to learn that it's present continuous due to this chai. And uh it's it can be used for example if somebody asks people who are currently learning AI what they are doing it can be used to answer that. So we can also have an arbitrary token like this with these random characters. So it will learn that this token contains random mix of letters, numbers, characters from different languages possibly or symbols or emojis if it does have it. Uh it doesn't match any common language pattern. It will know that this is that this looks kind of random. It might be code, password or random string that generated and it lacks semantic meaning and many other uh concepts associated with this. The question is which way of tokenization is best? Is it putting a whole sentence as a token? Is it splitting it by words? Is it splitting it by letters? First I would like to say that each token requires same amount of compute to process. So if we have a sentence, the quick brown fox jumps over the lazy dog. If each letter is a token, then we need to spend 44 times more compute because there is 44 different letters or characters, then if it's just one sentence as a token. And if each word is a token, then we need to spend 9x more compute than if the whole sentence is a token because there is 1 2 3 4 5 6 7 8 9 words. And there is an argument to be made. If tokens do not require same compute, then letter tokens would require more compute because understanding how letter can be used in all of the possible situations in a language is harder and has more information than understanding how a single word or sentence can be used in a language because letter can be used in a lot more different situations. So it wouldn't be just 44 times more compute than sentence. It would be a lot more than this as well if these tokens are bigger. But there are some issues with word and sentence level tokens as well. Massive vocabulary, not only every word, but every version of every word needs to be in the vocabulary. So run runs ran running. and is there any other uh versions? So this means that we have probably tens of millions of possible words and versions in a vocabulary and we need to calculate probability distribution over each as I explained earlier which is a very high compute. Compare this to just vocabulary of 26 English letters or let's say 100 or 200 characters that are from other languages as well. And then for sentences, it would be even worse. Every version of every possible sentence not wouldn't work. If you forget to add any word, for example, running to the vocabulary, the AI will not learn what it means, it will see it later and have no idea what it means. Even though we humans we can understand we can even infer if we see this for the first time we would know that run what mean what what does it mean to run and in will convert it to present continuous so we can logically conclude AI cannot do that this is because we humans we can see this word from many perspectives we can see this as from each letter perspective we can see it as this word and this art we can see it as a whole word but AI does not do that AI just has something in the vocabulary word letter whatever it is it learns what it means what it means if this is not in the vocabulary then it doesn't know what it means word tokenization also can't handle misspellings what if somebody says rand like this it's an incorrect form and it doesn't exist in your vocabulary but We humans we can understand infer the meaning or for example aocado without a we can infer meaning but AI would treat this as out of vocabulary doesn't exist so it has no idea what it means and what AI would usually do is it would replace it with some kind of token for example a token that looks like this and it learned that whenever it sees So I need to I need to first explain that tokenization comes before AI processes text. Tokenization is a separate thing from the brain the AI brain. So first the text will be processed through tokenizer and all of the words that don't exist in the vocabulary will be replaced with this unknown token. And then when the AI is processing it, it will just see unknown token here and it will have no idea that it was word running. And even if it saw the word running, I don't know what will happen. Would there be a bug error? Maybe it wouldn't work. I'm not sure. But it definitely would not know what word uh running means. This is actually interesting now that I think about it. I don't really know what will happen if we just pass in token that doesn't exist in vocabulary. So I remember before maybe seeing some errors pop out that like token is out of vocabulary or depending on the architecture of the AI maybe you can program how to handle it because it will try to process this but it wouldn't exist in the vocabulary. So I guess like there would just be an error. I'll think about this. So maybe in one of the next courses I can tell you better. Also most words are rare and do not appear enough times in the training data. So while AI is training while AI is reading the text a lot of words will just appear a couple of times and it's not enough to uh for the AI to learn the words because unlike humans AI needs a lot a lot a lot more data to learn something. We humans can learn a lot faster. For example, we can learn to drive a car in 20 hours, but AI has probably I think millions and millions of hours and still cannot uh learn to drive. And one really important cannot construct new words like letters can or cannot modify existing words. So if we don't put running and we don't necessarily have this in as a separate token then it cannot take run and in another n and create running because as I said we are just using words as tokens. Now if we had separate tokens like run n and in then it would be able to construct these things these words. So I'm foreshadowing here a little bit. I'm showing you that we have these subwords tokens and they are going to prove to be the best way to tokenize where we don't do it letter by letter because that's too much computation. But we group letters but we also don't do full words because it's too rigid. We can't modify it. So we group common we create common groups of letters like run in and we also have letters individually as well. Why not? So the AI can construct words. Before we go onto subword tokens, I will say that for sentence level tokens, issues are even issues that we have with words are even more exaggerated. If you just read through them, you'll see. But a quick note, there might be breakthroughs in the future that make word or sentence level tokens best options. Research in this field is very fast and accelerating. Can we get best of both worlds? Low vocabulary size and ability to construct new words like letters tokens do and lower computation cost by creating tokens of multiple letters combined like we have in words. Introducing subword tokens. Instead of tokenizing as run, running, play, playing, stay, staying, we tokenize as run, play, stay, and in. And now we can we have a lot smaller vocabulary while also being able to construct these words and doing it with a lot lower computation cost than constructed them with each letter. So vocabulary size goes from tens of millions for words to just hundreds of thousands. I think around 150,000 in Quen and probably others very similar as well. And as a note, this is also better in understanding test text with spelling mistakes. Now if we go to this URL we will see the tokenizer of Quen and this is for quen 2.5 but I think it will probably say stay exactly the same or a bit different but it does not matter if Quen Quen 3 has a bit different tokenizer. What's important is to understand how these tokens what they look like and I will scroll past these special tokens. I will explain them later. Let's just scroll down here. Here, let me zoom in. I think I can. Okay. So, vocab vocabulary and we have a bunch of different characters on the left side and some numbers on the right side. So, for now we will ignore these numbers. Here we can see exclamation mark. Uh this is slash this is hashtag dollar sign. So we have all of these tokens and LLM will learn what each of them means. We have also numbers the digits. We have capital letters. We have uh small letters. We have a bunch of different signs here. You see a bunch of different signs. Let me scroll further. Okay. I see some from yeah different languages. And this is funny and this is going to be like this is going to be repeated pattern. Uh I would say this is this maybe it shouldn't be the way they constructed the tokenizer these kind of funny tokens emerged and they're going to become longer and longer. That's just funny. So maybe it shouldn't be so long, but you know, sometimes you can't make it perfect in the tokenizer. And we can occasionally find some English words like ace. Where was it? I don't know where it was. Or here. I lost the ah one. Okay. Name div. This is from programming er like error. We see about and this could be like a subword of error part of that you see here we have this uh these two characters combined name with a capital N app it's actually very interesting to see all of this data return it's a whole token uh you will find that the more more common the word is the more likely it is to be entire word as a token and besides having uh here self as a whole token we probably also we let me see if we have SE as just a token. So let me draw these quotes around it. Yeah, we have this capital SE. Do we have we have a capital one and then lower two and we also have small se. Okay. So do we have seel? Yeah, we do. So uh this these tokens are not chosen manually one by one they are constructed we will program our own tokenizer that will create that will take a bunch of text and create tokens in a way that's most optimal. So we will look to have most optimal tokens. Before I talk about optimal tokens, I want to explain these numbers next to the tokens. So, we see here tokens on the left paired with numbers from zero to 100, 151,664 on the right. Vocabulary contains that number of tokens actually contains one more token because the first token is indexed with zero. So this is the total number of tokens and each token has an index number corresponding from zero to the maximum number. To understand what we need these indices, these numbers for we first need to go back when I said that LLM understands a semantic meaning of a token and stores it. So how does LLM understand what a car is? What does it do in the computer? Introducing vector embeddings. Fancy words, but don't get scared. You will understand it. Vector embedding is just an array of numbers. This is what it looks like. And for simplicity reasons, I chose to only include numbers between zero and one. But it doesn't have to be. Each token in the vocabulary will have a corresponding vector embedding, a corresponding array of numbers. For example, token car can have some array of numbers. Token grass, sorry, can have some other array of numbers. Where was it? So, I think that's pretty easy to understand. Each token has this corresponding array of numbers. In reality, these vectors are thousands of numbers long, maybe even 7,000 for DeepS, I think, or maybe one to 2,000 for other models. And each number captures some characteristic of this token. So each number is explaining for example first number could be explaining quickness how quickly this thing is moving and for example for car it can move very quickly and I clamped these numbers between zero and one but they it doesn't have to be that way but it's easier for me maybe to explain. So car is moving very quickly and grass now grass could get zero here because it's not moving at all. So LLM is not perfect maybe. So it will but it will put a very low number at grass. But it also could put here 0.01 because grass is growing. So maybe it can also consider that as a movement like we wouldn't know that if LLM would do that or not. This is part of the things that we don't understand about LLMs. Now these numbers here, I made them up to uh show you easier. So they I didn't get them from anywhere specifically. So this is just an example. Uh I changed them a little bit to match my examples better. So second feature could be aliveness and car would get a pretty low number here because it's not alive and grass would get a pretty high number. And the third feature could for example be greenness. And I want to show you how abstract these features can become. It's not just quickness or something that we are familiar. It can be any kind of feature uh adjective description. So grass is very green. Car can also be green. So it got a bit lower number. Fluffiness. You see here as well. Playfulness. Now I put this thing vehiceness because like LLM could actually describe decide to describe like how much of a vehicle something is. I also added these random uh features just to show you how abstract it can get. Parallel universess, regular exerciseness and who knows what else we don't. So during training LLM will decide what each feature is and it will decide the number for each feature for each token. And so Q humans we don't decide what each feature means. We just try to understand what each feature means after which we struggle with. And uh these features will be decided relative on all of the other tokens. So you can look at LLM as it's trying to figure out what are the differences between all of the 150,000 tokens and how can I encode these differences so I know so I know like how to use them relative to each other these words. So it will try to find features that display differences between tokens. So it knows learns differences between tokens. If there is some feature that every token or most of the tokens would get the same number for very similar then it wouldn't be so helpful for the LLM to understand the differences between tokens and when to use which token when it's generating the language. So I just gave you a lot of information. It might be confusing. Maybe uh I always recommend maybe trying to explain it yourself, trying to draw it, maybe re-watch and this uh way you can make sense of all of it in your mind. Now you will notice that I deleted a sentence here. That's because I just realized it's not correct. So I was trying to explain it to you in a simple way to make you understand what features are. So I explained it in a way that first feature here and first feature here determines the same characteristic like quickness. But in reality that might not be the case. Each feature could be completely different for every token and it could be unique and it could be mixed up in order uh with relative to other tokens. So we don't know in reality and this is very hard to interpret understand from LLMs and there is a lot of work trying to understand right now what are different features how to read them how to understand them what are these numbers actually encoding AI learns by itself what each feature should be about and the numerical value of it for each token and those might be different features different characteristic for each position And we don't know what exactly those numbers represent. Uh research on this is actually accelerating and maybe in in within a year we will be able to know a lot better but not yet. I will show you another quick example tokens dog and cat here we will also presume for simplicity reasons that first feature encodes same character characteristic second feature encodes same characteristic etc. So we can we have uh first feature fluffiness amount dog 0.42 42 cat 0.73. It's more fluffy, for example. Second feature, willingness to save you. So if you are in a lake and can't swim, the dog will jump after you with heroic urgency. Risk life. Bring floaty. The cat is the one who pushed you into the water. And third feature, gratefulness. How much it appreciates everything you do for it. So dog thinks you hung the moon. worships your every move, uploads your shoe tying skills. Get you are butler void of any purpose beyond servitude. In reality, features at same positions across tokens don't necessarily encode same characteristic. Also, different tokens can have unique characteristics or multiple features could work together to encode one or multiple characteristics. So it's very messy in reality. Now get ready for the next concept. I put it in the summary. So these vector embeddings aren't manually created by human scientists. We I think we know that the AI learns them by reading the entire internet trillions of tokens. Trying to predict the next token as it's reading it. So it tries to predict and then checks if it's predicted well and adjusting these vector embeddings changing these numbers adjusting them and other parame so for each token and uh other numbers other parameters that we will talk about besides these vector embeddings there are other parameters that help the AI think let's say so its predicted token matches the actual token in the training data That's what it's trying to do. It's trying to adjust numbers. So next time it tries to predict this in in this situation, similar situation or same situation, the token that was in the training data is more likely because it's adjusting these numbers, vector embeddings, and other parameters. So it's adjusting them based on the actual token in the training data to make it more likely to be generated by the AI in that situation that was in. As it fails to predict the next token during training, it updates vector embeddings and other parameters pausing the correct token from the training data to be more likely next time. in this or similar situation with this or similar text and incorrect tokens less likely. Now how LLMs use vector embeddings? Let's say we have a sentence the car is fast and the car is red. We know that each of the tokens for example let's say that we tokenize it in this way and it would be tokenized with these tokens in reality. So each token has a vector embedding and also each all of these small character tokens have this space before. Uh this proved to be the most efficient way to tokenize and I explained that here. So this is in reality how tokens work. Just a reminder in reality these tokens will be thousands of numbers of long because just four numbers is not enough to encode the difference between the and car and is and fast. So LLM will replace each of the tokens with a corresponding vector. It obviously needs to do this because it doesn't know what these words are, but it knows what these vectors are. So this is the token for the and then this is the token for space car you know space car here space car and so on it's just replacing the tokens but here is the thing it's not good to tie vocabulary tokens to a specific vector because maybe you have different versions of the model. Maybe you will train your model a bit more so these vectors will change. Maybe you want to use this vocabulary not just for different versions of the model with different vectors but for different models as well. So you don't want to actually map this vector to this token. So instead there is a trick to keep the token vocabulary always the same while easily swapping in and out different vector embeddings. So for each token you will assign an index and it does not matter which index goes to which token. This is completely arbitrary and unimportant. Just go one by one and assign. And then you will have another matrix. So this is array of arrays such that whatever embedding is first, it corresponds to the first token. Whatever embedding is second. So this at this position will always correspond to this token with index zero because this position is zero in this big array. So here you will always place embedding corresponding to this token at index zero. So it's easy to swap embeddings here from different models, different versions of the model. It doesn't matter. So this is how you can just take this index of this token and just pluck out the vector embedding corresponding in this matrix corresponding to that index. So we will map each of the words. The car is fast and red onto index and then if we have a sentence the car is fast and the car is red then we have uh this indexes. Now I forgot to put comma here but it doesn't matter. It's okay. And by the way you will need to do search across all of the strings all of the tokens in the dictionary. So that could be a bit slower. But once you have indices, then it's very easy to just pluck out uh the vector embedding with this particular index from the matrix. The scaling is O1 if that means anything to you. So that part is fast. So this marks a big portion of our course important portion of our course finished. Now we are going to go and code our own tokenizer that will take a big text figure out which characters to group to make the most efficient tokenizer that trades this granularity versus computation. So let's code our own tokenizer building a bite pair encoding tokenizer from scratch. So I will explain what this bite pair encoding is as we go. Step one, prepare training data. So this is a bit misleading because there is no neural network that's being trained. We are just creating a list of tokens from data. So I don't know why people call this training data. So I guess I'll call it as well. The first step in building any tokenizer is to have a corpus of text to train it on to create a tokenizer from the tokenizer learns merge rules. I will explain based on the frequency of character pairs in the data. Okay, if that went over your head, this is an example. First, we take some text and we just split into every character. The tokenizer will still have every character in the tokens, but we will add more tokens, subword tokens that will make it more efficient. And those um subword tokens will take priority when tokenizing, when splitting over the characters. The characters will also exist. But as I said earlier, characters have weaknesses. For example, since each token is same size if you have token is it takes 2x more 2x less computation than if you computed both characters together or separately sorry separately. So this is what we will do. We will go through the text and uh count how many times each pair of characters is appearing. For example, look at this. We have is, this, his, so in each of these words there is pair and we can conclude that for example this pair is occurring very frequently. So if we merge if we create a new token here uh of this of this pair. So we have this token I this token S and we create a new token we assign new index to it the latest index. Now we have reduced computation by a lot because this is a very frequent pair. So we will replace it at a lot of places. And now now uh the token is here instead of being two tokens one two with index indices one two it's just one token with index three this will get replaced with one token I just noticed that you couldn't see the whole text so I moved rearranged the screen I can read this even though I and S are separate tokens we create a new token is by merging them as they frequently appear together is this his So we will just uh count each token in our whole corpus of text each pair and then merge them. And the thing is if we merge them here is and then we notice that this token is pumps frequently with this age is then we can also use this new token to merge it with another token and now we will have this his and his here. So these new tokens can also be used uh in new merges. This is how we will reduce computation and we will find the most frequent tokens and merge them and then reduce computation the most. Okay. Now let's go down below. So let's say we have some let's call it training corpus. This is the first document. This document is the second document. This is the third etc. Step two, initialize vocabulary and pre-tokenize. What does that mean? Let's uh read on. And I'll leave a link to these files in the description below. The bite pair encoding algorithm starts with a base vocabulary consisting of all unique characters present in the training data. So we take the training data, we get all of the unique characters into a list. You see here initial vocabulary has space dot question mark ait c. So these are all of the unique characters and we have this new character. Uh this will mark the end of the word. So then what we will do is we will split each word uh into characters. So you see here uh okay so we see this JSON object or dictionary and inside of that dictionary this is the first element and the key and if you have intermediate Python knowledge you should know what you can know what key is so the key is the word this but it's a tpple and it's split into letters, characters and there is this end of word uh token added and then uh this is the number of the times this word appears in the training corpus. If we go here this and this there is two times that word this appears. So this is what we mean by pre-tokenizing. we are splitting it into words. There are other ways to do this but uh this is maybe beginner simple way. So let's ignore this is all of the code here like all of this but let's ignore that for now. Let's just focus on this uh theoretical part. So uh we also need to pre-tokenize the corpus. This usually involves splitting the text into words or sub or word like units. and then representing each word as a sequence of its individual characters. We often add a special endow token like this one to mark word boundaries which helps the tokenizer learn subword units that align better with whole words. So this helps tokenizer to for example learn this unit and like a subword units as opposed to learning like this subword this unit or this unit. So we want to try to keep it keep the subword units within words. That's why we add this end of word token so it knows where the end of the word is. Let's go through the code now. uh first we will create this initial vocabulary of tokens that contains all of the unique characters. So unique characters is equal to set. So set will only have unique uh elements. So if you add another question mark to this set, it will not be added for doc in corpus. So we will go one by one through all of these documents. this list of this list of strings and then for character in document we will just add uh unique characters add character. So this will add character but because it's set it will only add it if it already doesn't exist. And then to create a vocabulary we will convert this set to a list. Uh we will later expand this list with new like end of word end of word token for example. And this is being sorted for uh consistency or for reproducibility or to find bugs later. And we are using set here just to make sure the characters are unique. And we are converting to a list later. So we can order it or sort it which you can't do natively in set because sets are unordered. And uh so we can access these elements with an index. So list zero element first element you cannot do that with a set. So here we have printing initial vocabulary. This is what we print. So this is a list of initial vocabulary. We added our and the word token and the length is 20. Pre-tokenize the corpus split into split into words and then characters. So now we are doing what they call a pre-tokenization. So this is a different thing from creating initial vocabulary all all of the characters. So now we are going to split this these documents into words. So word splits we initialize empty dictionary for document in corpus. So we go so now we get whole string uh words is doc.split on space. So we so here's what we will get now. We have this. So we will get this is and the space will not be included. So you just get words and then for word in words. So now we have list of separate words if word exists. So if it yeah if it exists I don't know why it wouldn't exist but let's just check if it exists. uh we will convert that word for example this into a list of characters. You see here later we will get this list of characters. For now we will create that by saying list word plus we add this end of token and end of word token. And if since we are appending we are appending this with this list then we need to put it also as a list. So this is a list of one element. So we have these square brackets and we will get uh this list at the end. Now here uh this list has these uh normal brackets instead of square brackets. That's because we are going to convert it into a taple. here. So word tpple and we will convert this character list into a tpple. Use tpple for immutability if storing counts later. I added explanation you want to create a tpple because list cannot be used as a dictionary key because it it's mutable. It can be changed reordered. You can add new elements, remove elements. But in tpple, once you create it, you cannot do any of that. You cannot change it at all. So tpple can be used as a key in the dictionary because uh it has a consistent hash value. So it will uh it will always be the same. So then we will check if this tpple already exists in our dictionary word splits. For example, it would exist if we had a word this already and then we are seeing it for the second time the same word it would already exist. But if it doesn't exist this word if not in word splits then we will put uh we will create a key with this tpple. So it would look like this the key the and this is the key the tpple and we will initialize it at zero we will set the count the amount of these words at zero if it doesn't exist and then after that regardless of if it exists or not doesn't if it existed or didn't exist now it exists and we will just increase the count by one. So this will count how many times each word appears. Is appears three times, the appears four times and then this word first appears two times etc. And it's counting through our entire core corpus of data everything all all of the text and uh the reason I will I want to just repeat the reason we are first splitting into words and then uh finding the most frequent pairs within words is because we don't want to have tokens that go like this that are across words. We want to find we want to merge most frequent pairs of letters or tokens within words only. So in the next step we are going to count how many times each pair appears. So this is one pair th and then hi and then I s and uh s and then the word that's also a pair and then we will also uh we will count single pair across all of the words. So we will count like this or pair a yes it appears once two times. Okay. Is there more maybe and notice that we will not count pairs where it has like and the word and I that's that we will not count that because those that's why we split this. So we don't uh create crossword tokens or this s i or s space i whatever uh whatever the way we would be able to do it we will not do it that way. So we will code this new function get pair stats and that function will take the entire dictionary of words that we just created and it will output each pair and the amount of times that pair appears is four times and then s and the word four times as well. th two times. Uh this is the function that we will code. Now we will import these collections and I will explain why. So the function name is get pair stats and we pass in splits and as I said splits is our dictionary that we just created with all of the words and counts. So we see here pair counts equals to collections dot default dictionary and pass in int. Okay. Uh I explained it here but there is a difference between this uh collections.default dictionary and the regular python dictionary. And the difference is that if you try to call uh a key within this dictionary that uh doesn't exist, the regular dictionary would throw an error. But this dictionary will create that key and the value it will assign to that new key is whatever we define here. And here we define this int factory. And it will return uh zero if since it will it so this in factory if it doesn't get any arguments which it will not here it will just return zero. So we will create a new key and assign zero. This will happen if we call this try to call this key within this dictionary that doesn't exist. Uh let's check the next line. So this line uh splits dot items split is the whole dictionary as I said and then when you say dot items then it will first take this item and this item contains of the key and the value and so we will take this key and put it into word tpple variable and it will take this value and put it into frequency variable. So this is what this line of code will do. It will go through the dictionary one uh pair by or this word by word pair element by element and take out frequency take out the word tpple. Okay. So once it's taken out these two then symbols equals we will convert the word tpple into a list and lists are uh mutable. So we can change them reorder them add new elements remove elements. So now we have a list of symbols that's going to be these elements going to be the same. My bad. My bad. My bad. This is just an example. This is not the executed code. This is just like hardwritten example. And in the end we will execute this code. But in the end here we we say pair counts. We return pair counts and that's it. So what we want to do in future steps is take this pair the most frequent pairs and within this dictionary of splits or words we want to replace this pair with a new merged token. We will replace most frequent pairs with the new token and we will also keep track in a separate variable of all of the merges we did. So later we can undo everything. So next we will create this function merge pair. Uh this function will take as an input the pair we want to merge. So the pair will come from this pair counts and it will also take this splits dictionary and what it will do is you see here merge new split so new splits uh new words has this merged token. So what's important to understand is uh we still have these I and S tokens in the dictionary they are still there but now we created a new token is and now that's a new token added to dictionary and we will use this token whenever we have these two tokens in sequence so we reduce computation now it's the computation is double like 2x less also it's easier for AI to understand what is means whether it's a part of the word this it's a part of different word but it has a lot less possible ways to be built into the language than I and separately. So we are killing two flies with one hit. So we say less computation because 2x less computation and this uh this is more targeted and more narrow meaning of the either of these separately. So it's easier to learn for the AI. Well, this code is going to be pretty similar. So I think we can go a bit faster here. New dictionary. We have pair to merge. So we have this pair and then we are going to split it. I and S. So first first will be I second will be S. Merge token we just now concatenate those two strings and then for word. So we again um take this dictionary of splits and we do the same thing. We extract this tpple and this frequency again. And then uh we also convert this tpple into list. We call that list symbols. It's going to be the same as like five minutes ago. And then a new symbols. We create a new list. Initialize this counter. This index counter at zero. And uh this is very simple. So here's what we will do here. We will go check this guy and then we will check one guy ahead of it and then we will check if these two are equal to the is I and S. And if they are not equal to the I and S, we will just um we will we will now create this new symbols array but we will just copy both of them. And if the guy and the next guy are equal to this pair. So is then we will not copy both of them separately but we will create a new uh is pair just is token and then we will just continue after checking each pair. So in short, we will make the exact same list, but if we stumble upon the pair that we want to merge, we will just merge them into the same and save save it. So uh while I is less than length of symbols, so I already said I is index. So the index is going to be one less than length. length is uh five but index will go 0 to four. Okay. If index is less than length minus one. So the first character it's going to be uh this guy, this guy, this guy and this guy. It will the first character cannot be the last here because there is no next character. So there is no pair. So the first character in in the pair can just be um less than length of symbols minus one. So even less is by one less. Okay. And then uh if character at this index current index for example at index two is equal to the first character that we want to merge which is I. So if this is I and next character next index I + 1 is equal to the second character that we want to merge S. So is then in the new symbols array we will append this merged token which is going to be just a string merge of those two characters. And because we merge two characters then um we will skip I by two not one because if we merge them then we cannot merge this guy and this guy because this guy is already merged within here. So we will not just increase i by one we will increase i by two to skip this guy. But else if we are at a point where this pair this does not contain this is not is or this pair this is not is then we will just add this exact like if we are at this pair we check the next one it's not okay so we will just add this guy to the new symbols here here append symbols I and Then after we add that guy, we check the next guy. I is equal to I + 1. So basically we go one by one. We go here. We check if this and in the next one are is no. Okay, just add this one. Okay, go here. Check if this and next one are is. No, just add this one. Then we go here. Check if this and next one are is. They are. Okay. Now we add is string. We merge them. Add is string to the list. And we don't go from here to here. We skip this one because it's already merged. So we increase i by two. And then as a bonus as I I is now equal to this instead of being less than that. It's not going to go this route. It will just add this token. So it will just go here else and add that last token and that's it. That's done. So I know this is a bit confusing. Uh I hope to improve my skill of explanation explaining over time. I think next courses will have maybe the same or similar thing because these tokenizers are also the same. So you can even watch uh different courses and try to understand these tokenizers better. And as I said watch different YouTube videos. there is thousands of hours. Now we will combine all of these functions that we just defined and we will create this uh bite pair encoding merging loop. So we will just use functions now. Now we perform the core BP training we'll loop for a fixed number of merges. Number of merges. So here we have number of merges 15. In each iteration we will calculate the frequencies of all adjacent pairs in the current word representations using get pair stats. This is the first one that we defined. It just counts how many how frequently this pair is is appearing throughout the whole text and other pairs. Find the pair with the highest frequency. It will be easy I think. Uh merge the best pair across all word representations using merge pair. This is the second function we defined. This will merge is into is pair. Add the newly formed token concatenation of best pair to our vocabulary. As I said, we will still have I and S as tokens. But now we will have another token is. Besides, we will of course also have all of the other characters as tokens. The initial vocabulary contains all of the characters as tokens. And now we will just keep expanding with these most frequent pairs. And the more frequent the pair the better because we will then reduce computation. Uh we will reduce a lot of computation by adding just a little bit of overhead of this extra token because when we add extra token now we need to calculate probability distribution across all of the tokens. So one more token. So more computation. But if this token happens very frequently then we will remove a lot of computation while processing tokens in the other part of the neural network. So now we process less tokens for a bit more computation. So the more frequent this part this token this merge this pair is the better because then we will replace a lot more computation. So besides adding the new token to the vocabulary we will also store this merge rule. So we will store that we mapped I and S in onto is in this new merges dictionary. So later we can undo. Yeah. Number of merges 15. So we will run this loop 15 times and we will 15 times we will merge the most frequent pair and uh one of the elements of that pair can be a merge. For example, it can be age plus is to create his. And then this merges, this will store all of the merges that we did step by step. So later we can undo them. For example, uh if we have T and H. So we can have TH. These are the merges that we to that we merged into th and we will store all of this into dictionary. So here uh current splits is word splits copy. So we will start with the initial word splits. So this is a dictionary that contains a word and number of times it appears. So here I have some print statements. Initial word splits it's actually quite long. So uh word number of times is number of times the number of times etc. Next let's see what we have at the uh this is just for printing this line. So we separate it. So for i in range number of merges. So 15 times. So merge iteration. Merge iteration one out of 15. Calculate pair frequencies get pair stats. This is just uh our function that will create these pairs for example s and endo word. So looks like the most frequent pair because here I'm printing I'm just ordering it by frequency. I want to show you like the most frequent pairs. So the most frequent is s end of the word and then is seven times th seven times a seven five times. So we are just counting pairs here and this key lambda item one. So it means we will sort them by this number and not like this whatever this sort sorting would be. So we will sort them by this number. So when you say key lambda uh it will create a new lambda function that will be used for sorting. And this item this determines what will be like which value will be used for sorting. For example uh because we have item of one that's going to be the second here. This guy the the this value. So this is item zero. This is item one. And so we will sort by this. So this will be passed into this temporary lambda function that it gets created for sorting and it will be sorted by this. So the first yeah but u in this code. So yeah but this is just to print most frequent pairs. We don't need this functionality in our code. the sorting by most by frequency. And then the second step, we will take all of the pairs that we just created and pick the one that has maximum value. So this one has eight. So we will pick that one in this step. The way we will do that is we will say best pair is equal to max and then pair stats. So that's just like all of the pairs with their frequencies. And we have some code here. Key is equal to pair stars.get. Now this is tough for me to understand as well. But the way I understand it, I also have like AI explaining it here. So this will just tell it not to look at the key. This I I know it's confusing. It says key but it tells it not to look at sort by this key but by this value that you get from calling the key. So we will not sort by this words but by this value and so it will pick the maximum tpple the best one best frequency. So let me close this for now it's more visible pair stats best pair. So the best pair will be the just this key and then frequency. We can get frequency by calling this key pair starts with this key and then we can say u found best pair with frequency. So found best pair s and end of word with frequency 8. It's confusing for me as well. And now in step three we will do our merge. And as you remember in merge pair merge pair takes the best pair and all of the current splits the current words. We can go back and see input best pair is like a tpple of the two tokens and current splits is just our dictionary of tpple as the key and the frequency and output is going to be this uh merged token within this uh dictionary splits dictionary. Now let's go down. You see merging best pair into new token. So merging s and end the word into this new token. Uh splits after merge. So splits after merge. This is the new dictionary. Thi and then s and this guy. Now you will notice that since we merged this, we will not be merging i and s anymore. Maybe it will. Maybe we will because maybe it could still be the most frequent but we will not count this pair anymore. So if it's most frequent it will be somewhere else where S is not merged with uh another token because we cannot now count just this pair. We need to count this entire pair if we want in the next iteration. So maybe it will still be as it seems like it will. Okay. So yeah. Um but before we go on to next iteration where we repeat this again let's see update vocabulary vocab append new token. So now I did not uh print that out but now our vocabulary besides having all of the all of these characters it also has new new token which is going to be this s and then the word token and merges best pair maybe I'm going to look to also print these two here it is updated vocabulary has all of our initial uh characters and the new one s and then the word and updated merges. So now we have a single merge that's going to be S and then the word into S and then the word and so we now repeat the process but with this new like merge. So now we repeat uh we have our documents as well but now this is going to be considered that as as a single token and it turns out that that still remains part of the most frequent token. So I and S end of the word so it's because it's in is end of the word and then this end of the word. So that's why it's most frequent and then we will uh found best pair which is going to be that pair merging and then creating a new token is and then the word splits after merge. So this is what now our splits our words look like th is and then this is the whole new token is and then the word new vocabulary. Now we have this token and the new token. Now we have both the tokens and the new splits S and then the word into S and the word and I and this new token into this latest token. Lastly, we can just examine what we have. So review final results. After the loop finishes, we can examine the final state. The learn merged rules, merges, the final representation of words after merges, current splits of the current tokens. What did we get from all of the words? Uh the complete vocabulary containing initial characters and learned subword tokens. Let's see. Final vocabulary size 35. Learn the merges. We had vocabulary size at 20. 20 characters and we did 15 merges. So now we get 35. So these are the merges. First the one that we merged, then the second one. It turns out the next most frequent one is going to be th and then th e. So this is the end in they. And then the next one, it looks like the whole word the. Yeah. And then do do. So you see how we now get this whole word as a token. This is because it's still very frequent and it makes sense to put this whole word as a new token because of its frequency. And then we get I and I and dot end the word and D and the word. So these will be our 15 merges and our final uh our final text. It's a bit weird. Wait, wait, wait, wait. So I'm I'm thinking like it's not weird because this is capital T and this is H. So we didn't merge capital T. We just merged the lower T. So capital th did not prove frequent enough to be included into our 15 merges. So we have is we have the whole word purse you see I document this yeah this proved to be very frequent. So uh this document okay so we have document that's ending with a dot and then end of word and we have document that's ending just with the end of word without dot and then second none of the merges here we have end yeah this is interesting to see document as well and final vocabulary So it's sorted. Uh maybe I shouldn't have sorted but it's okay. So we are just oh my god sorting it. So D and then D and the word document document document. We have all of these tokens. And in the language model if we see document we will use this token. We will not use this token. So we will use the highest the biggest possible group big longest possible token. So that's going to be it. The one difference in a real world language model is uh these characters will be replaced with their indices. You see here instead of using that character that particular character you would use its index here instead of using exclamation mark you would use zero. So you would form these words these all of this process would just be use these indices. Uh this is for computational efficiency. It's uh faster to search over numbers integers or than strings. It's also easier to store integers into memory while you are processing it. It's requires less memory than to storing strings. And so as you are creating the vocabulary in the real LLM, you would also be appending these indices to each of the tok each of the tokens. But we didn't do that for simplicity reasons. That's going to conclude our simple version of the tokenizer. I'm still not sure in these courses in how much detail how long I should make each part. For example, tokenizer. I can make it 30 minutes. I can make it one two, three, four hours. So, uh this is something I'm going to look to figure out in the future. Uh one advantage is that if you understand this bite pair encoding tokenizer, it's going to be very similar or same for other LLMs. So if you are interested in more details, maybe one of my next courses will include more details that build on top of this where I will also explain this as well because I will look in every course to explain everything from the beginning from scratch. But you can also find more details in my other courses, future courses I hope. So this is something that I will ask you in the comments below. What do you think? How much in details? how much time I should spend etc on all of these parts and look for your feedback. Now we've come to the dreaded point. So the next 30 minutes to 1 hour will be the toughest, the hardest, the most difficult to understand. But once you understand this, you will know how to make any AI LLM that exists in the world. GPT, Gemma, Claude, Gemini, Deep Seek, Quen, Llama, whatever, whatever. So, the next 30 to 60 minutes, it's the toughest, but it's also the only thing you need to understand to make any LLM. And once you understand this, even those AIs that are making changes, it will be very easy for you to understand. So different AIs, DeepS is right now inventing like crazy changes, crazy upgrades. Very, it's not difficult at all to understand them even though they are crazy changes because uh once I understood this, I can understand all of the changes, upgrades, everything. So you can take a pen and paper, try to draw this yourself, try to explain this yourself over today, over days, over weeks, over months, over years, decades, centuries, doesn't matter. I'm I'm kidding a little bit but hopefully over days weeks hopefully not months. So uh and I also explain this same thing same thing because it's the same in my deepseek for coding deepseek from scratch. So you can watch it there. Um it's going to be it's going to be like same thing maybe explained a bit differently. So or uh in different courses in the future this self attention mechanism always the same. This is the the heart of it. self attention and feed forward network all of that in the next uh 30 to 60 minutes. The way I was I learned all of this is uh I was over a couple weeks I was trying to draw this myself explain draw these numbers vectors do the conversations do the calculations uh and trying to understand it and trying to remember it let's begin um okay here in the beginning let me let me move this here so as I said we have some sequence of tokens and I expect you to know everything I've talked about until And each token has its vector embedding. And as I said, vector embeddings are a lot longer in real world, thousands of numbers of features long. Okay. Uh let's say this dog is represented by these three numbers in this vector. So these three numbers will encode the meaning, the semantic meaning of the word dog. If you don't know how, you can go back and you should go back and you should know my examples that I gave. So, um, this would for example encode fluffiness or fur, how much fur it something has. This would encode how much I don't know something is like an airplane. So, dog is not like an airplane. So, it's low number etc. So, these are features and we don't know what uh l what features these are in reality. We don't know. We cannot know because it's hard to interpret and scientists are trying to understand what these features that LLM comes up with what they are but we don't know the LLM comes up with the features and numbers itself we just tell it how long this array should be and that's it and nothing else. Okay. Uh let's extract let's just take the dog. Okay. So even though all every vector represents the meaning of that word, we want to modify these vectors such that they also show the entire context in which this token is present. For example, this ran shouldn't uh this vector in the beginning of AI, it just uh encodes the meaning like this is a verb running. it represents running in the past. That's all. But at the end of our LLM AI processing, we wanted to actually encode that ren is referring to the dog, the single dog, the dog. So the means like we know which dog and dog is running. Uh now the one thing is we don't want to actually get any context from from the later words into this run because we are trying we are building model that's going to predict words. So we don't want to get any context from the words that it's trying to predict. So we don't want to get any future word context. So to reiterate for word quickly this vector in the beginning of our LLM our AI it's just encoding like quickly means like speed like fast but we want to somehow do something in our AI our LLM such that it knows when it see such that this vector says not just not just encodes this meaning but also it the numbers here will change to also encode that it was running that was quick and the dog was running quickly and the there is like one dog the dog is running quickly. So we want to change this vector to encode all of the meaning of the previous tokens not just this token but not the meaning of the following tokens because we want to predict the following tokens. We don't know them. And so once we have this vector that encodes entire context not just the word this token then we can use then basically we have use this vector that has all the context to predict the next word like around and as I said before we are not predicting the next word we are predicting probability distribution over the entire vocabulary. So we are for every possible word that LLM has in a vocabulary. We are predicting probability that that is the next word. So our model should be designed in a way that first it will properly create this that is modify this vector with the context such that if later parts of the LLM that's doing the predicting it's called output head. If out output head needs to look into this new created newly created vector that's going to be going to have context and output head and both output head and this new vector should be well created or well learned by the AI such that word around gets a high probability token around gets high probability and token like airplane would get a low probability because it wouldn't make sense here. The dog ran quickly airplane wouldn't make sense. So, uh to repeat, our LLM needs to be designed well and made well such that it gives it adds changes these numbers to add context from these numbers up into this word quickly. And the output head that's converting that new vector with all of the context into probability distribution over entire vocabulary does it well such that word um around gets a high probability because it makes sense and words that don't make sense get lower probability. Okay. Until now, how much of what I explained can you repeat? Can you explain? Uh because it's important. So besides that I want to also say something else. We have this sentence the dog ran quickly around the and it looks like we are predicting the next word here. uh now there is training and during the training when we are training the LLM what we want to do actually is we have this one sentence but we want to predict every word this is just to train the LLM so what we want to do I I'll explain we can just start by giving it the and try to predict dog and then update entire model to be better at predicting uh the word dog. Now you would you will say like okay but it's just the there can be anything here and that's okay. That's okay. Uh that's why we design a model not to very be very biased towards predicting a dog here. But uh we also design a model that's not going to predict any word that's that's that wouldn't make sense here after the. Okay. So during training we in the first step we can have just the predicting dog. Then we can also have the dog and predicting rand predicting quickly. So during training we can take a sequence of tokens and try to predict every and then the model will see the truth the ground truth the real token and then update all of its parameters weights biases everything I will talk about later so that this word dog ran quickly is a little bit more likely next time not too much more likely but a little bit and then we do many many many iterations over many words, entire internet and bit by bit the model learns what kind of words can come after word the and what cannot come and the probabilities and stuff but during inference when we are using the model then we if we have a sentence we don't need to go and first process the try to predict dog and then the dog try to predict Ren no that's just during training So we can get more of our data during inference. We can just take this entire sequence and just predict the next word. So uh in my tutorial now I will be explaining how we would go about calculating context for each word. But remember to predict a word you just need the previous token to be in context. So if you want to predict quickly, you just need this ren to have context of every other word. So you don't need the and dog tokens. You don't need these tokens. You just need this token and infused with context with numbers of these other tokens to predict the next word. So in uh so because during the training we will be predicting every word based on so for example quickly based on the dog ran uh we will be calculating we will be infusing every word with context just to clear up that confusion. Why are we infusing every word with context if during inference we just predict the last word? So we would just need to infuse this word with context not every word but we will be infusing every word because during training we do that with context before it. So during training we predict every word during inference we just predict the last word. Here we are simulating training. So it's time. Okay we got this token quickly. Now somehow we need to get context from these three tokens ahead before it and add it to this to this number. So somehow we need to add these numbers or some parts or some way of adding some of these numbers somehow into this. Um we wouldn't just add them up. It wouldn't work. We need to somehow process them first. we need to somehow figure out how do we process them so that when we add them here it makes sense it works well. Uh definitely one way we can add context from these previous tokens is just to sum up. So 0.3 plus 0.5 plus 0.8 plus 0.2 and result written here. So that would be away but there are many issues with that. For example, a quickly word might actually be a lot more dependent on this word ran than the word the. So this word ran might be a lot have a lot more influence in determining what the context of word quickly because it's ran quickly than this word the. So maybe we need to add more of this context and less of this context because this word the it would maybe explain that there is the specific thing that was quick. But maybe this word ren is a lot more important or even if this word the is more important than this word ren. We it doesn't matter. We just need to have some way of measuring how much important context of each word is to this word quickly and the AI will learn learn how to measure the importance itself. So AI learns this. We just need to somehow enable AI to learn to judge how important each context of each each word is and how much of the context should be added to this word quickly to this token. To give another example before I continue, imagine this is a very long sequence of words, very long text and we have this word dog because the dog ran quickly. But let's say before there is like some mention long before in uh there is a mention of a cat. Now we know that this quickly is referring to the dog running quickly and not to some random word cat before in the text. So we need to enable LLM AI to figure out that this word dog is what's actually running quickly. So it should take a lot of context from this word dog, not from some random word cat before. And now we go into the unexplored confusing magical charts and parts and swamps of AI. So okay, this it's funny because it's I know it's going to be confusing, but it is what the best scientists understand by now. Okay, it's going to be such a dumb explanation. Okay, every token, every token, every vector embedding we will convert to some other vector. So for every vector we will create some other vector based on some rules and it doesn't matter which rules I will explain rules and how we create the vector but just imagine that every vector will be create converted into another vector called E and rules of conversion for every token are the name. So that corresponding vector key, let's say uh this dog has a another vector here next to it. Let's imagine it key and that key will say what this dog vector has. So this is confusing. So when uh tokens ran quickly around, they want to check if they are referring to that dog that's running quickly or some cat that's running quickly that was before. They will look into this new key vector. They will not look at this meaning vector. This vector describes the meaning of this word dog. But a new vector will describe what this token contains. And it's funny because it's confusing to me and nobody understands it. But just imagine that for this word quickly, if you want to check, if it wants to check, if it's referring that the dog ran quickly or the cat ran quickly before, let's say, it cannot look at this vector that describes the dog. It should look into a different new generated vector by some rules that's gonna say what this token contains and uh that new vector key it will be always same uh no matter it will be always same for this token. So no matter if this ren is looking at the key, if this quickly is looking at the key, if this around is looking, the key is never changes. It's it's calculated once and never changes. And it describes what it describes which context you can get from this dog. And so if we have this dog, its vector embedding and its key, and we have this cat somewhere else, and this vector embedding, and it's key, uh, this word quickly will be able to look at both of those keys of the dog and the cat, and it would figure out the key of the dog would tell it somehow magically. I don't know how. Nobody knows how. But it would tell it that it should look into this uh word dog because the dog is running quickly. Here I put a key. This is some example key. Let's say yeah this is key for the dog and it will tell the other tokens what context you can get from this token. So the difference between this vector is this vector encodes the semantic meaning of the dog and key encodes what context you can get from this vector if you want context. This is the extent to which I can explain it and quite frankly to which probably anybody understands. Now another thing let's say we are add the vector quickly and we are checking if uh we can get any interesting context from the word dog. So we we're going to check the key of the word dog to see if we can get any interesting context. Now how are we going to check it? So you can't actually check it with this vector. You need another vector query. Okay. Um, it's funny. So, so when you want to check, you need a query. You can't check with this guy. So, as I said, this guy encodes this guys encodes semantic meaning of the word quickly and query of quickly encodes what kind of context quickly is looking for. So what kind of context this yeah word token quickly with its whatever embedding is looking for. So you need to check what kind of context I'm looking for to against uh which kind of context I have. So you check query against key and now uh you have this word quickly and as I said you want to take information context from previous uh previous tokens and so you will take query of the word quickly and compare it against the key of each previous token. And uh if your model is good, what you should find out is that it's going to interact pretty well, pretty strongly with this ren and the dog. The result would be a pretty high number. And the result, let's just say the number between zero and one shows how related these words are. Ran quickly, dog quickly. So if we uh measure query against key of the word run, ran, it should get a high number. And for dog, it should have a higher number. But if there is a unrelated token cat somewhere before but this quickly is not referring to the cat ran quickly but dog ran quickly then query of the word quickly should have a really low score with the key of the uh word token cat and they should not that means that this will not take any context or will take on almost no context from the word cat because it's not related. And I know I'm getting repetitive sometimes, but I just want to say query says what context am I looking for? He says what context do I have? And these embeddings, they just describe the semantic meaning of the token. They don't say what context I'm looking for or which context I have. So if we are trying to predict word around we don't have word around so we want to predict it we will have the last word which is quickly we will take its query that says which context am I do I want to get enriched with do I want to take in and we will compare the query against key of every previous token and we will get some score and that score. High score will say I want to take a lot of context from this word but low score will say I don't want to take any or very little context from this other word because it's not relevant. Important thing in real world key and query are these vectors are a lot smaller than vector embeddings of the words. So usually these vectors are thousands of numbers long and these vectors are usually hundreds of numbers long not hundreds of thousands hundreds. Now I will explain if you have query and key how do you get a single score single number between zero and one showing how much these two match. Okay. Uh vector this is a vector. Vector is an array of numbers. Vector can also be a point in a coordinate system. So vector if this is your key this can be your key and it has two values which means this can be x value this can be y value. This can be y value in the on the coordinate system. Uh let's look at this vector. Its x value 0.8 let's say on x-axis and y value 0.4 on y- axis. Now if we have a third number then this would be a 3D vector. Usually these have like 500 numbers. So it's a 500 dimensional vector which can't be we can't imagine that. But anyways, if you have this vector, let's say this is the key and let's say this is sorry, sorry, sorry, sorry. Let's say this is the query. Query for quickly and let's say this is the key for dog and this is the E for cat. As I explained earlier and I made this example such that uh So quickly and dog vectors will match more than quickly and cat vector because quickly is referring to the dog in our text. And uh how do we calculate this? So it's very simple uh dot product. Now if you know what dot product is I don't need to I can just tell you you do a dot product between this guy and this guy and you get a score. If you don't know what dot product is, I'll draw it here. Wait, what is this? Oh my god. So, here's what you do. You draw a Let me get green. You draw a perpendicular line from the end of this vector to this vector that you want to dot product with. So, this is a 90° angle here. And let me get another color. This is their dotproduct. The length of this, how much they overlap. So when you project this vector onto this vector, what is the length of this? And there is um this will show similarity between these two vectors. And as I said, we want to calculate similarity between the query and the key to see how much those words, those tokens depend on each other. And so to get a better understanding of what's happening here, let's imagine this this green vector is here. This would be a more similar vector because now uh they would overlap like all the way like until here somewhere. If you see you you draw a parallel a perpendicular you see now the length of this what is the length of this so this is like 90% length of this query vector so 0.9 is the similarity if this was like here in the middle maybe this guy was like this I don't know some somewhere like this so now if you draw a perpendicular line it will get maybe somewhere here so Now the length from zero to this point is like 55% or 55% on the length of this whole vector. So now the dot product is 0.55. So it's a bit less similar than if the dot product was like you know this green was here at zero and then the dot product would be 0.92 or something. And as you can see, the dot product of the blue guy is smaller and it probably would be even smaller in because I I said I want to make this cat very irrelevant. So maybe it would be like this and the dot product will be very small here 0.2 or would be even smaller 0.0 something. And just a fun fact about dot products. If this is completely perpendicular, so there is 90 degrees between this vector and this pink vector, then the dot product is zero. Another important thing I didn't mention until now these queries and keys, we will add positional information where where the token is in the sequence. it will be aware of the positions. So the way I was explaining it until now is it just gets semantic meaning of the word quickly and matches it with like the query is made just from the semantic meaning of these words but it's we also want to add position. So it should also know that dog is just two words behind the query and the word cat here that we don't have is maybe 50 100 words behind 100 tokens behind the word quickly. So we will add that positional information within these keys and queries. So when this query is going against this key uh and or this cat key the cat key will be modified such that because of the position the query will get a really low dot product with it mainly because of the position because it's so far away. So the positional when we add position we will just modify these numbers directly we will just modify them directly and then dot product will give uh appropriate results. How do you calculate dot product by the way? So you take first number time first number plus second * 2 plus third time third and you get a single number at the end showing your score. The higher that score the more similar these vectors are. And uh I said that we look at similarity to figure out how much context we should take from that vector and add it to this vector that has the query. We're going to look at this article. I just want to credit it for a good example. Let's look at this right box. So this is these are the attention scores that I was explaining this whole time. So it looks confusing but it's simple. Uh we have a sentence life is short eat dessert first. So let's say uh short we have a query for the word short and uh first we do a dot product between the query for the word short and the key for life to see how m much they match. So 0.19 let's say then we do query of short versus key of is and query of short versus key of short itself. Uh usually those will be very high. uh this means that later when you are adding all of the values to enhance the context. So the most of the value will come from itself as it should. So the most of the value of this of the new enriched um embedding that's going to be this quickly that's going to represent this token quickly but juiced up with all of the context from before. it should still have the most context come come from itself describing itself and then a bit of context from each of these. So that's why in AI AI will predict that uh query of word quickly multiplied with key of world quickly will have a very high score. We do that as well by the way. We don't just compare to different previous words. We also compare it to itself and that will show us how much of the meaning context should come from itself which is going to be good proportion 50 60%. We do that to calculate the ratio between it keeping it its original meaning the word quickly and all of these new meanings because we don't want to add too much of the meanings of context of previous words such that we drown out the meaning of the word quickly itself but we don't want to add too little either so we want to calculate we want to let AI learn how much to keep, how much to add. Back to the example, you see that every time when you compare query of the word it with the with the key of the same word, it will have a very high value. Now here uh this value doesn't seem very high, but usually it will. I don't know if this is real example or madeup example, but usually it will. And what is this weird triangle? Uh this is just to say that if we have a query of a word eat, we don't want to measure it against dessert because dessert is the following word, the next word. We don't want to take any context from the following word because we are predicting the following word. So that's why this upper triangle is uh we will set all of them to zeros because they are keys of future tokens. So query of this token only can attend to its own key and previous keys and we can only take context from itself or previous words. This is called attention matrix and we will code this but to code this you need to understand it. I would suggest take pen paper try to explain everything I explained. If you don't know, you don't know. It is what it is. Couple days, couple weeks. Next step, once you create this matrix to mask all of the future tokens, uh all of the remaining. So for for this life, there is just one box and this box should be one. Exactly. Because you want to take 100% of the meaning of itself the life because there is no tokens before. So you cannot take context from any other token. So the 100% of the context will come from itself because this is the first token and for the token is uh we have we we see that uh this still does not add up to one. So you need to uh scale this such that it adds up to one. So maybe this will become 0.85 and this will become 0.15. I don't know maybe something like that. Um but then we will know that we should take 85% of the of this meaning of is and 15% of uh life for the new for the new meaning for the enhanced juiced up meaning of the word is. So the next step here would be to make these add up to one and make these three add up to one and these four add up to one. So we need to scale the numbers. So we can do that by applying a softmax function and uh it's simple and when we are code when we code it and I will explain but as they as they are adding up to one we will know how much of the percentage of this we should take of this of this of this and add them all up. So, we will just add uh all of them with a plus and that's going to be our new juiced up token with all of the context. Now that we have the scores that say how much of this word quickly wants to take from the word dog, we cannot just uh add part of this token to this token. Instead we will create another vector or both of them and call it value. Now values are what we are adding up. So we are not adding up this vector but we are adding another vector that's value. So these are the three vectors the key query value those are the three vectors in self attention mechanism. So why are we adding a separate back vector value instead of adding this vector? Um I I don't understand 100% again I don't know if anybody does but my understanding here is why why this works better is this factor encodes semantic meaning and this factor encodes uh what you should get from me to encode my context. So maybe maybe this vector directly encodes the semantic meaning of the dog but this encodes like the context that token dog would give. Sorry, this this value of dog encodes the context that uh token dog would give and this vector is a lot smaller is a lot smaller than this big vector that encodes the whole meaning of the dog. So this you can look at this value is just like uh part that just gives the context. So it doesn't have any other information but just like this context that it gives that gets appended to or that gets added into this value and as I said we have these probabilities um not uh percentages. So when we are creating a juiced up vector or quickly that has all of the context we're going to take percentage of each of the value based on the our score percentages. So if with itself it's going to have the highest scores. So as I said maybe 50%. So 50%. So it will be this value times 50%. Plus this value of dog times I don't know maybe 10% plus value of ren time 12% plus value of the time 8%. etc. And in the end those percentages should all add up to one which is like 100%. If one or 100% whatever. So basically what you will have in the end is if you are trying to predict word around you have these four words the last token will have value and that value will be juiced up with all of the values from previous tokens. It's just going to be added but added like the percentage based on the affinity score between key and query. And so you have a juiced up uh value vector for this quickly token and you can use that to predict the next token. So that's going to be the attention mechanism. After the attention mechanism creates these juiced up values for each of the tokens, then we put each individually through a separate neural network. It's called feed forward layer. It can be a simple neural network or it can be mixture of experts. It doesn't matter what it means. But basically what that part will do is it will take this value for this token quickly. for example and this is just other values have just been added with a plus. So plus is a linear transformation and this neural network will have activation function nonlinear transformation. So you can imagine it as changing these values uh in a nonlinear way such that important features get strengthened important characteristics for example if if dog ran dog ran was important for quickly. Now we have even better way of strengthening those meanings, those parts, those that context and we can weaken context that was if there is any context referenced from the cat from the the or other words that are not important. Now we have even better way because in the attention mechanism we could have just added some context that is irrelevant or added a little bit or none of it probably a little bit but now we have another way to even improve it further the context. So strengthen important and diminish less important context. And now let's talk about how these uh keys and queries and values get created. So basically you will have some token. So this is the dog token. It's same token for the dog dog vector embedding. And you have these matrices for query key and value. They are learned by the AI during training and they are always the same for each key query value. There is just one matrix for query for each query and it will transform this dog vector into the query. So again this matrix is learned this is calculated. This is learned during training. So once this is learned it doesn't change. This is calculated every time for every token and this is also different for every token. Okay. So you know how matrix multiplication works. You have this this is 1D. You can say this is matrix with one row. So this is a vector. So you multiply this row with this column. You do a dot product. So you do first one * first one here plus second one * second one plus third one * third one. So you add all three of the sums. That's going to be your first number here. Okay. the pos uh okay the second number is going to be this row dotproduct with the second column and the position of the result will be where these two intersect so it's f first row second column so it's the going to be the first row second column the the end of the the position of the dot product okay so that's how matrix multiplication works so in terms of like what does this matrix even do how does it transform you can make some logical sense of it like this you have the first feature that's describing something second third feature so this first column is made by the AI such that it takes this whole vector and creates the first feature of the key so as I said key key says what am I looking for so features of the key can be like okay I'm looking for something that's an animal I'm looking for something that's uh like showing speed of movement I'm looking for something that's different than than a truck so each of the features can say something what it's looking for so the first column will take the whole vector and generate the first feature like I'm looking for something that's an animal second column does the same to the whole vector generates second feature I'm looking for something that's a track record that was I don't know and we can the same way we generate key and value. So just to quickly say so again key is what context I have what context I can give you and then value is what you what it actually gives you or what it gives to the context to the token if when you calculate query and key dot product. So for value for example it will be the same. The first column takes the whole vector and just transforms it into the first feature. So second column also is gen for generating set second feature etc. In reality you will not process tokens one by one. You will take this whole sequence and do matrix multiplication with the whole uh w with the matrix for queries. So when you do matrix multiplication, it works the same way. Uh for example, for this the you multiply with this column and put so that's going to be the first feature and then you take the multiply with second column second feature the third column third feature and then you take the second vector which is dog and you will multiply the same and put put it here below. is going to be second row because uh the way you do matrix multiply is you multiply you multiply column or sorry you multiply row with let's say a column and then you write the result at the intersection. So it was second row and first column. So the result will go into second row here and first column and that's how in the end we will get this is going to be result this is going to be first token query second token query third token query and same for keys and values and when you have queries matrix and keys matrix you will do again matrix multiplication you won't do one by one but here is the thing it's important you will actually transpose key matrix. Look, look, look, look, look, look. As I said, when you are multiplying two matrices, you need to multiply row against column. But so we but since we want to multiply query against key then we need to put this uh key to be a column. So you will make this key actually a column and the second key actually another column and the third key another column. So that's why you you need to transpose the key matrix to turn rows into columns such that you can multiply because that's how matrix multiplication works. So you see here it's transposed. So you see 0.21 8365 0.21 8365. So now I've swapped a row. I've transposed it. So now when you do math mult matrix multiplication it's going to properly multiply query with a key query with a key and you will get some affinity scores. So you will just as I said you will just get some scores. So 0.81 0.88. So this is the product between first query and first key. And then this is the product between uh first quer no second query. So second token and the first key and then I artificially replaced these with zeros after I calculated them because this would be the first token looking at what second token has but we want to predict the second token so we don't want to allow that. This would be first token looking at what third token does has but we don't want uh to look at that because we don't know the f the we don't want to let AI to look at the future tokens it it hasn't predicted yet for example during training we want to mask that this during training so to simulate that it doesn't know what the future tokens are even though we have the future tokens during training we don't want it to look at that we want it to predict the future tokens And then it's going to upgrade its weights. It's going to uh change these numbers where are the like these things that they that it learns to better predict. So as I said now we need to make rows add up to one but also keep these at zeros because we don't want to take any value from the feature tokens any context. So we will first replace zeros with minus infinity because the way we will make these rows add up to one is we will apply softmax and to keep you'll see the formula you need to do these to minus infinity so they remain zero after softmax then we will apply softmax. So softmax is very simple. You put so for each row for for each number in the row you say the e to this number and e is oiler's constant. It's 2.7 something. So e to this number divided by e to this number plus c to this number plus e to this number. So e to this to minus infinity is zero. So 0 0. So it's going to be e to this number divided by e to this number because these others are zeros and you're going to get one here and 0 0. Then second row you're going to get e to this number divided by e to this number plus e to this number plus zero. So you will get these two and they add up to one. So the first row adds up to one, the second row adds up to one. And the same thing here e to this e to this. Yeah, you you know you know the same thing here. So e to this number divided by sum of e to these numbers you get this number and they all add up to one again. So now you have for each so let's say this is the third token. So third token should uh take 39% of the context of the first token. 32% of the context of the second token of the value and 29% of the value of the third token of itself. So as I said in reality this diagonal this will be uh higher usually than these two because usually it will want to take more value of itself um to keep the context but these are made up numbers that I came up with. I didn't know it will be lowest anyways uh this diagonal is where you have the token against itself. So this diagonal will all you will you will usually be high high numbers highest numbers in the each respective row. So this is the value. So now we have we have our attention weights that we calculated and we have value matrix first second third token. So the third token value will take 39% from the first and then it's going to be 32% time second plus 29%* third and we can see that here in the row three. So we see 39%* first token plus second token plus 29% time third token. So after some calculations we get and now this is the juiced up value for the vector three for the token three that contains information from previous tokens that are properly weighted based on affinity scores between queries and keys. How important the context for this query for this value for this token is. In the end we get another matrix and this is weighted matrix of values. This is juiced up context for each of the tokens. So you see this vector is put here. It's just shortened a little bit but those are three vectors with context all around them. And you'll see that this vector is going to be the same as as it without being added because it was just taking 100% context from itself and no other context because it was add it was the first token. So it's going to be the same remain the same. Usually each query key value will be divided into parts or heads. Here you see query for the first token and I made this space here to show you that actually we will not process entire query token against entire key token. We will divide it into parts what's called heads and then you will take this head and process it independently from this head. So you will take this head against this head of the key and then use this value head. It's like completely separate vectors and then uh this head of the first token against this head and this head and same for second token head head. So in the end you will get values head one output for token dog head two output for dog and talk. So the these are ju values. Uh the reason to divide into heads is the logic is each head will learn something different. So for example, this head will learn to just focus on context around animals or being red or vehicles or whatever. This head will learn to focus on different kind of context. Even though features within the head also have their kind of focus but they they their their focus might be intertwined and messy and but uh now we completely separate heads and process them separately. So now they can really manage completely separate context. So that's the idea just to separate key query value vectors so they have separate to to learn to accumulate and process separate context in the end you get value head one value head two and then you just concatenate them. You see this is the first head concatenated with the second head. They're just a big value vector now. And then regardless of if you had heads or not, as I said this value vector, this is a lot smaller than the original vector embedding for the token in the beginning. Our vector embedding in the beginning here it's going to be a lot smaller. So we need to convert this value vector back into the big vector token. So for that we will use output conversion matrix. This matrix is same as every other matrix. So it will convert this smaller value vector into the big. So you know how it works. It takes this whole vector and this whole column and creates the first output uh the first vector embedding token feature then whole vector second column second feature and so on and so we will get this vector embedding for the token dog that's now juiced up with all of the context around it as well. So in the end we get vector embedding for token dog with context around it. So now we are back to the original size. No more key query value size. Let's do some coding. So we will import we will code a llama attention mechanism. Llama for attention mechanism. Now this is where you need to have maybe a little bit of machine learn pietorch like knowledge. So um let's go hidden size. This is the dimensionality of the feed forward layer that we will talk about later. Hidden size of it. Number of attention heads. I said attention heads dividing quick key query value into separate heads. So 16 for each. So dividing key into 16 heads query and value as well. Uh number of key value heads. So llama is using for efficiency purposes. We are sharing key value across a group of queries, a group of heads. So this is important. Um for every head you would need to calculate query key value. However, it turns out that you can take a group of heads and share same key and value for each head and it will reduce computation without reducing performance a lot. So here we have 16 heads but only four keys and values. So it means we will share same key and value or four of the heads. So group of four heads has same key and value and each head will have a different query. So 16 queries. Okay. Uh this is also good because we uh as we are calculating keys and uh values for token token token token we are storing them into memory. So we can use up a lot of memory and we if we have a lot of tokens 100,000 million 10 million tokens as they have. So we if we just divide the number of keys and values by four then our memory requirement goes down by four. Guys I made a bit of a mistake uh here hidden hidden size. This is not the size within the feed forward network. This is the size of the embedding vector for the token that we are talking about all the time. This is the size of the embedding token vector. Okay. Head dimension. Dimension of a single head is you take embedding vector of the token and divide by number of attention heads. Okay, this is weird. So this means that the dimension of the key query and value will be same as dimension of the token embedding. But I was telling you that key query values dimension should be a lot lower smaller. So I don't know this is like a design choice they make in deepseek R1 or V3 this dimension was a lot smaller of the like key query by so I don't know they make the dimension same here anyways dimension of each head uh will be hidden size so the total token embedding dimension divided by number of heads. So max max sequence length maximum number of tokens 256. Uh so these these are some numbers for our simple version. These are not like the real numbers. It has the real llama for can take a lot more tokens. Now we have some rope and RMS norm. So now I'll explain explain the rope the rotary positional embeddings. So this is how AI knows LLM knows that the position relative position between tokens. So if uh dog is running dog is very close to the word running then it knows that it's referring to the dog is running and not some cat is running that was hundreds or thousands of tokens before. So I explained this in my understand the deep sequ 3 video here rope. So you can watch this explanation and my explanation now maybe both will help you link in the description below. So the thing with a rope is if you have uh dog is chasing the cat or cat is chasing the dog the context matters on the order of the words. So rope will also show the order of the words, not just like the distance between words. That's why you need it as well. So as I said in this video, if you have some long novel in the end to uncover like who did the crime, you need to attend to many parts previously far uh before in the novel. Rope will allow you to do this. Uh also rope is a method where we can encode positional position without additional learnable parameters. So LLM doesn't need to like have a bunch of other parameters that it should learn. Rope is just a simple mathematical trick that lets us encode the position. We can also extend the context window after the training which is what Meta Lama for did with scout to extend it to 10 million tokens. So you can and if in the previous 2017 version where they use like some sign cosign for rope you couldn't extend uh context length after the training. it would just become completely confused because it wouldn't know like what those positions are. But here positions are relative not absolute. So you can extend. So let's say we have some token cat uh it has one two dimensions in this embedding vector for as an example. So dimension will be two and this m is the position of this token in the context window in the sequence. So this is cat. So it's number three number three position. So let's say uh this is the vector embedding of this cat. Now let's say we have another vector embedding of some like running This is cat that's running and let's say this token should be like maybe they're running should be there. Now the whole trick with rope to encode their position. So this is semantic meanings. But if we want to also encode position, if this word running is a lot later, not just two words, if it's like 10,000 words later, then we would use rope to rotate this vector away from the cat. So as I said, uh you see here now the dot product is high dot product. they overlap when you project this there and then this would be the whole dot product. But if you rotate this vector then the dot product when you project here would be a lot lower. So the idea between rope is the further away these vectors are in the context window the more you rotate them. So you rotate them also based on like position in the context window. So if this is third position, you rotate a little bit. But if this is 10,000 position, then you rotate a lot more. And this is how you encode positions. Now they are further apart and so their dot product is lower. So they will attend less to each other. Now you can already see a problem. But wait, if we rotate this vector, don't we change its meaning? I mean that was the meaning for the cat. And as I said before, this vector literally the vector determines the meaning of the word. So I can't just like change this vector because then it it's going to become a different word, different meaning. That's why we use we make a bit more complex system that can handle both meaning and position. And it seems a bit more complex, but it's not. Trust me. The idea here is instead of rotating whole vector, you divide the cat vector into these pairs of numbers and then rotate each pair individually by a different amount. And so now we have two kinds of information. Um AI will learn to differentiate between rotations due to position and vector rotations due to the meaning because rotations due to position they're always going to be same relative to each other because we will rotate this by one amount this by uh mult multiple of that amount this by multiple of that amount. Uh I I'll again try to like simplify this. So now this I number is just saying which pair are we dealing with. So let's say we are dealing with this orange pair. So now we are rotating this. So the rule is the following. these lower dimensions, first dimensions are going to rotate by a lot smaller angle. So it's going to be angle times the position of the token. So if we have another let let's say this white line is first two dimensions of a different token like a dog or running or something and this pink line is first two dimensions of this cat. So uh this is rotating very slowly based on position. So the further a high that it is but it's rotating very slowly. So it will never go like all the way here. So this is going to show to the AI like if this is rotated by this chunk that means that these two words are very far apart because this is rotating so slowly for each position. So if it's rotated a lot it means these two words tokens are very far apart. However, higher dimensions are rotating a lot quicker. So the blue one, the blue dimension. So if this is like the if there is if let's say this is the third dimension for the last dimension for the dog token. This is the last dimension for the cat token. So if this is like one token away, it's going to go like this. Second, third, fourth. So it's going to start circling around. So this vector will show the local difference. So if this will be very sensitive to local difference. So if it's like one token away, two tokens away, three tokens away, four tokens away, but if it's 200 tokens away, it won't really be so useful because it will just you don't know 100% where it is. So this is just sensitive to local uh differences but this guy will be very good for like 200 500 uh position difference because then this guy will go like this. So, LLM will learn to take both long range dependency information and short range local dependency information and combine them to understand where precisely what's precisely the difference in the positions here and we will have this for each pair. So maybe this one will be like mid mid amount middle range mid amount of rotation. So it will not rotate as much as the blue but not as little as the pink. So it will rotate like this for each position. And LLM will just learn to check the relative rotation of each pair of both of the tokens and figure out the distance between them. And the rotations here will be different than the rotations when you fully change the meaning because when you fully change the meaning everything will kind of rotate at the same speed. But here um everything will rotate and the whole every part will rotate different speed. So LLM is able to figure out the difference between rotations for semantic meaning and rotations for position. I recommend you watch this explanation in my R1 video. It's actually maybe even better. So just to summarize, the beginning pairs are rotating a lot slower. So if they rotate like this, that means like uh if they rotate like this, it's a huge distance between these two tokens in the context window. But if they rotate like from here to here, the distance is very little, but it can still mean hundreds of tokens. So you don't know 100% what's the distance because this is such a small rotation. It's not granular enough. You don't know could could be like 100 100 2003. That's why you have a more sensitive rotation. So this guy will rotate a lot by a lot more chunks. So now if you look at uh where this guy is and where this guy is, this will give you more general longrange dependencies but not like exactly and this guy will give you more precise dependencies in the local area. So you can combine them to figure out the exact position difference. So it could be like this means like this it's about 100 tokens away. So then this guy says like okay 101 1023 104 105 106 107 108. Now this is the problem is this 108 or 101 because it's same kind of but that's why you have these as well. So this should go like by five. So maybe this guy will say like okay it's about 102 it's about 107 it's about 101. So if you go here about 107 and about 100 and about and like about nine so it's like 100 109 um rope theta this is one of the hyperparameters for the rope that we will understand better later when we see the code theta is the angle of rotation but it's used in some ways in the code uh normalization RMS norm This is for normal normal normalizing the vector embeddings and we will learn more about this later. There is no bias in attention. We won't use that in the matrix projection matrices. So these are projection matrices and we will just use matrices not bias and uh attention dropout we will not use dropout either. uh this is to prevent overfitting but we will not use that and then use QK normalization so whether to apply L2 normalization to queries and keys before attention we're going to use that and we're going to say batch size two that means like two independent context windows of text two independent sequences of text sequence length number of tokens it's a context window hidden state. Uh this is the well okay so hidden states will be one hidden state is one token embedding and we will have a bunch of token embeddings for each token in the context window. So we will have 10 tokens which means 10 hidden state vectors and two of those context windows 2 * 10. So two independent context windows. So we will just initialize these to be random. Uh the dimension of this tensor will be batch size, sequence length, hidden state and position ids. Then we will create position ids for each token in the sequence repeated for each batch. The shape will be batch size time sequence length. So first we're going to create a 1D 1D tensor just in a vector with values from zero to sequence length minus one. These are going to be ids for this sequence length. But I said we want to have not just for this sequence length but for every sequence length that is every batch because one batch is one sequence length. So unsqueeze zero adds an extra dimension at the zero position. So instead of just having sequence length which is one vector now we have one sequence length which is vector and imagine another square brackets around and now in those square brackets around this vector you can put multiple vectors inside which I should say so so yeah so this inner vector is just an array of numbers showing position for each token in the sequence. So just from zero to length minus one. And now with the outer vector you can fit in multiple sequences. So you can maybe read this once more here to understand. We're just expanding. Now we create the attention mask. And if you remember attention mask is sequence length times sequence length. It's a square and upper right triang triangle are zeros to make sure that queries of every token cannot attend to keys of future tokens. And then after we have that triangular matrix we will unsqueeze that is add two dimensions in the beginning and then we will so this dimension this will actually be batch size. So we will have attention matrix for every every sequence every context window in the batch. So I think what this number one here means is This is for the attention heads and it means that this same attention mask is applied to every attention head. I think it makes sense. So below that we can print these values but we don't need to. Let's go on to the next cell. Llama 4 uses group query attention. This means there are fewer keys and values than Q heads. The number key value groups tells us how many Q's queries heads share the same key and value head. This reduces computation and memory requirements. Okay. So these are projection matrices. These are these these W's. So Q projection that's going to be this hidden size it's projecting from hidden size and hidden size is this dimension. So this should be the number of u columns or rows number of rows. Yeah this is the number of rows. So one, two, three rows for this is a bit confusing. Actually this number should be same as this number. Okay. Uh so that's hidden size and the output or the other dimension is going to be number of attention heads times head dimension which is actually same as hidden size because I said that this size of this query vector they made it same as the size of the vector embedding. So it's going to be the same. So actually in my example here it's also the same. It's three * three. Yeah. But that's going to be the same input output. Okay. If you know how m matrix multiplication works, you will know what I mean by this is input. This is this that is this same dimension. And then output it's going to be this same as this. I mean same dimension we're not you really using bias if you remember bias was false or something so and then key uh value and output projection and output projection if you remember what output projection is if I find it here here. So after we concatenate all of the heads then we multiply the concatenated heads with the output projection and it will turn the concatenated heads of values into the token embedding. So this is just one. So this is for the token dog output dog embedding. So if we have running output matrix running embedding the output head is going to be reverse. So the output head will go from the number of attraction heads times head dimension two hidden out size output token. So it's going to be reverse um calculate projections. So query states is equal to uh query states. So now we want to calculate these queries. So we will take these vector embeddings multiply this matrix with the query mat uh query yeah weights matrix to get these queries for each of the this is query for first token query for second token query for third token etc. That's what we do here. We pass query hidden state through the query projection matrix to get query states query list of the things and uh I did not mention that here it's using number of attention heads but here it's using number of key value heads which is four in our example and this is there is 16 queries and four keys and values and then four four queries are sharing one key and value for efficiency reasons. So four query heads and then you have same key and value for each of the four heads. Next we want to reshape our so this is this gives us one big query uh vector or matrix of big query vectors. Now we want for each of the query vectors we want to divide it into heads separate it and then process each head independently with other key value heads so they learn different things different extract different information. So reshape query key value from multi head attention target shape batch size number of heads sequence length head dimensions. So it's a bit confusing why is number of heads here. This will just make sure that for the first head, we take all of the tokens, put it just a first head of all of the tokens and group them into a sequence like a context window sequence of tokens. And then all of the heads, all of the second heads of every token, group them together into a sequence. And then we basically process each of them as if it was a standalone context window sequence just with this head at the at that position. So second heads all together in a sequence like as if it was independent sequence in the end we will take result of each of the heads and concatenate. So here we are doing this transformation where we are transposing sequence length and number of attention heads. So transpose one and two dimensions. So this will help us separate it. And then same for the keys and values. We want to also separate them into heads. If you go ahead and uh print if you want the shapes, you can see that it's going to have this uh shape. And here below you can print u number of key value groups. So you say number of attention heads which is number of basically queries uh and then number of key value heads and you're going to get four because it's 16 divided by four and you're going to say it's going to say number of key value groups that means query heads per key value head is going to be four. So four query heads per one key value head in the calculation. The way we will process it is we will have still 16 heads for the queries and 16 heads for the keys and values but four of those will be the same of the keys and values. So we will just maybe take one same and multiply it with every of the four different query heads. Then uh rotary positional embeddings rope I was explaining that so instead of adding absolute positional embeddings which was done in 2017 llama models use rotary positional embeddings. This is used by every model right now I think rope applies rotations to the query and key vectors based on their position injecting a relative positional information directly into the query and key representations before the dot product. So before the calculating attention score of key against query this is often found to improve performance especially on long sequences. You know this part rotary this could be like an hour explanation probably which I hope to be able to do but I need to have like I need to figure out how to explain it better in the next video. So maybe even next courses uh you will be able to watch this in my future videos if I have already published it by the time you watch this. The core idea is to represent the embeddings in complex number space and rotate them by an angle proportional to their position. Actually I explained this so maybe I don't need to uh read this now. In order to explain this how we calculate these inverse frequencies let's say dimension of the vector is eight and base is 10,000. So we first do torture range 02 dimension to 8 and going by two. So we get uh 0246 that's what this will do. uh put the to the GPU and uh float just like the the we're going to use float numbers doesn't matter then we divide by dimension to normalize so we will put this array between 0 and one instead of zero and dimension so we normalize so here we're going to get this array 0.025 0 0.25 0.5 0.75 and then base power this array. So let's see for the first number base power 0 we're going to get one base power 0.25 25. This is like base power 1/4 which is fourth root of 10,000 which is 10. And then second root of 10,000 which is 100 and then third root root of 10,000 which is th00and wait what? Okay. Okay. I get it. I get it. This is 10,000 to 34 10,000 power 3 over 4. So that's going to be same as for example you say fourth root of 10,000 and then cube that result or you can first cube 10,000 and then take fourth root of the result. It's going to be same. So it's going to be thousand. So we will have uh in the final array 1 1000,000 and then when we say one over this then we're going to actually get reciprocal values. So the final inverse frequencies will be 1 0.1.0 1 0 0. You see here the second row. So here we will just create uh a vector that shows position of each token. So this will just have like zero for first token, one for second token in the sequence etc. And then we will multiply this uh tokens positions and inverse frequencies. Here is what we will get going to get this matrix here. So for position uh for the first token we don't want to change any angles. We don't want to rotate anything. For the second token we want to rotate first pair by the full amount of the angle that we will know later. Uh then second pair for 0.1 of the angle. Then third pair or 0.01 of the angle. Okay. Okay. So for the third token we want to rotate first pair by two of the angles, second pair by 0.2 of the angle etc. So these are like how much to rotate each pair in each token. And so I was explaining the third row what it does. Now let's check this one. So as I said we are rotating the pair of dimensions. So we want to duplicate. So if we rotate like this shows how to how much to rotate this dimension but because there are two of them the pair then we will just duplicate it to rotate both of them. That's why we will create this concatenate these two matrices. H you can read more here. That's what I explained. So now column 0 and four hold the angle for the first pair. So this column and it's going to be this column because they are just concatenated. So this is first pair first value first pair second value. I explained better or some parts of the rope in this video as well. Rotary positional embeddings. So here I explain mathematically. Maybe you should watch this because it really shows like how you use how you rotate entire what do coordinate system space in order to rotate vectors. It's actually I think it's pretty good. So you can watch it. So there is some mathematics behind this rotating u vector. I recommend you watch that video. It will actually help you a lot to understand it. So this video by three blue one brown uh linear transformations and matrices. This is like this is what we are doing right now. I think it's pretty well explained here. So we will just create the s cosine and put it into a complex variable because we are using s and cosine for the linear transformation. We are basically using s and cosine to measure how we can how much we should rotate the vector. It's explaining those two videos. It's hard for me to explain like this. And um this function that we here that we see here this is what we used to calculate this line which was if you remember we were calculating for each position each token how much each of its pair pairs should rotate. So for third token first pair rotates by two second by 0.2 two three. So that was that function that calculated it. Let's check this. Apply rotary embedding storage. So XQ is query tensor shape batch number of heads sequence like hand dimension. Uh yeah we know this. So key tensor is going to be batch number of head sequence head dimension. uh we know that key tensor will have different number of heads here. It's going to be number of key heads and these frequencies premputed complex rotations shape max sequence length and head dimension for each for each head. So we will apply them to each head and uh you see here we premputed them here. They are precomputed. They don't depend on the sequence length of the anything. They just yeah they just depend on the sequence length of the number of tokens. That is we can just take this maximum sequence length that we have in here somewhere here and then just premp compute for all of them and that's it in the beginning and never again. And then whatever the batch the sequence length we can because we have maximum possible so we can always apply to any shorter sequence length. Okay. Insure frequencies. Yeah. Put them to the device if you're using GPU. So you remember in the beginning that we had these position ids. So that's going to be our ids positions for each of the token. So we will actually use that right now as soon as I here for each of the tokens we will pick the frequencies. So now instead of having the frequencies for every possible place every in the whole context window the full length we will just have frequencies for whatever actually tokens we have right now in all of the batches and then add a dimension for broadcasting across attention heads. We want the same rotation apply to all heads for a given token position. So we will just now in you see here we don't have attention heads here. We have one head dimension and that's it. But now we want to add this positional embeddings encoding to every head. So we create an extra dimension here for heads. Then we will reshape is and query vectors such that here so that pairs of numbers are treated as complex numbers. This is related to how we are going to do this linear transformation. So we're going to use this method of complex numbers. There is I think we use different method in deep sequ 3. So you can check that method as well. But this is just like containing maths. So we start with batch size num heads sequence length head dimension and uh because as I said what we want to do is we have the full head dimension now we want to separate into pairs within that head dimension. So we will reshape two the same thing until here but it's divided into two and then this is the dimension that has two. So I guess we have head dimension and uh we will have actually instead of one head dimension we will have two two two two. So the number of pairs will be half the head dimension. That's the the number of pairs because we are getting pair pairs out. So if you remember uh we duplicated the matrix. So for each pair initially we had just one rotation but then we duplicated the matrix. So for each pair we had two rotations. Here we will undo that. So now we want our one rotation back. This is confusing even for me. I I swear. Oh, let's ignore this. I don't know how to even explain this. I don't know how if I even understand it 100% but I think watching those videos will help me and you as well. Uh let's now go here. So here we are just we just have like the query which is batch num heads sequence length head dimension divided by two it's multiply by the frequencies and number one here in the frequencies this will broadcast across the number of head dimensions so it will be same for every head. So the division by two is because we are treating pairs of values, pairs of the coordinates as complex numbers. We did that here view as complex for query and key. Maybe it can make more sense and then after those rotations we are casting them back into the original data type for example load 16 and returning both of them. So now both of these have also the the query and the key full batch uh have also encoded positional encodings of each token into the all of the heads of that token. So now we will just uh use like just call all of these functions that we defined. I think like from now on it's simpler. This this was like the most stuff part. Yeah. So we just do uh frequencies. So this is the premputed frequencies for each token that's not dependent on the token just on the position. So we do this once in the beginning of our LLM and then applying rope apply rotary embedding torch. So that's this one. Apply rotary embedding torch. And so we get keys and queries uh with all of the information rot positional information. Okay, I think that was like the toughest part of the video, the rotary embeddings. Maybe I need to like make some animations, drawings. I'll I'll figure out what to do in the next course. So we will define simple L2 normalization. So we have some small epsilon value just to make sure we are not dividing by zero. Uh this is a classic normalization. After we apply this normalization uh then step four uh since we have fewer keys and values heads than Q heads. grouped query attention. We need to repeat the uh key and value heads so that each Q head has a corresponding key and value to attend to. The repeat uh key value function handles this. Here it is. Repeat uh gets hidden states and number of uh repetitions. So just here where we say have number key value heads times n repetition. So that's going to be our new dimension and then sequence length head dimension. So we just expand we call that function to repeat to get repeated keys and values. Here we calculate attention scores. So we have batch number attention head sequence like head dimension and we transpose this key. So that's going to be it attention weight uh scaling factor. So we're just going to downscale. So you see here you are dividing by square root of the number of dimensions and then let's apply our mask. So the query of our current token cannot look at the future tokens keys just add causal mask and then we will skip dropout because we don't have it. Calculate output weighted sum of values. So we will so each of the tokens is going to have attention score of how much of the value of other tokens beforehand to get. Yeah. This is what I explained earlier. Yeah. And now once we concatenate all of the values we want to project it back into the uh vector. So we have this like out projection matrix that we apply to this attention output and actually here we just put all of that into like code. So we have hidden size uh number of attention heads. So everything we've talked about these are the linear layers. So this is the simplified llama for attention. It's going to be exactly the same just grouped up all of the code. Apply a rope optional QK and normalization then repeat them. Attention calculation we just talked about that drop out if we need it. And that's it. And um this config dictionary is like what we used here or is it? Yeah, it is uh simplified attention module and then it returns final output and final weights simplified. Yeah, attention weights shape from simplified module. So this is attention weights. We're also I guess returning this for some reason here or inspection and that's going to be the llama for attention. So all of this time since tokenizer until now we're doing llama for attention mechanism. Uh here we can see the architecture of the transformer from 2017 and this is what we are actually making. So we are done with the masked self multi head attention or grouped head attention in our case and we are skipping this because this doesn't exist anymore and we just have this feed forward that we this is all simpler and then depending on the layers number of layers this is going to be repeated multiple times. So but that's easy to implement. So you have m and then feed forward mass feed forward mask and in the end we got this linear that's like creating probabilities. So now let's just just implement this feed forward and um this normalization can go before here and before here. So uh the FFN feed forward network is applied independently to each token position after the attention mechanism and residual connection. So after you it goes through attention mechanism, you actually also want to have a residual uh connection. That means that this output of the attention will also get added to the original token. So necessary AI learns to preserve necessary information from the original token as well. And so that whatever we get there gets passed this token embedding will get passed through the feed forward and each token will be passed independently and then feed forward will look at the token and right now until now we have just been adding attention context from other tokens by plus but feed forward has nonlinearity so maybe it can create more uh like process it in a nonlinear way a bit more complex Okay. So it can strengthen important features, diminish less important features and context and that's that's what it do what it does. So we will just apply uh import this good stuff. So first hidden dimension hidden size this is the token embedding length. So intermediate ratio. So you know the this feed forward will have uh input size of this token. it will expand to the input size that's to the size that's bigger and then contract back to the size of the token. So inside of this expansion there will be like processing and stuff and that's the magic there. And usually the expansion ratio is like 8 over3. So 2.7 67 that's going to be the inter the middle. Okay. uh common multiple for FFN intermediate size. So uh this line of code this code will just adjust intermediate size to be multiple of 32 and so it explains here the math but that's what it does hidden activation. So this is going to be selu rms normalization again the same thing we have some small epsilon and no bias and then sample input we're going to have two b two uh and sequence length 10 so we will have two independent sequences of tokens that is two independent context windows. Uh this is the okay so we create some random inputs. So batch size sequence length hidden size. This is like the beginning and then we will apply before pro uh passing the hidden state through the FFN. Llama applies a layer normalization step unlike standard transformer that often uses layer norm after the FFN. Yeah, I already uh said this but this is what uh actually all of the LLMs are doing now. doing normalization before attention and before FFN. So that's going to be the same attention as we did earlier. It's just uh RMS norm attention and uh so we pass that we get normalized hidden states. Here we have some trickery gated projection. So this is linear layer that has input. So token size and intermediate size okay up projection is same and we'll see why and then down projection will go input intermediate to hidden size. So we go like that and then go back. Okay. Uh if activation is celu then we will uh apply activation function to be celu or not implemented error. So normalized hidden states now we have them we apply the linear layers so we project them up but also we do the same with just app output. So we will use this to apply silo here. Yeah activation function. So right now these two are the same but then we will apply activation function here. And then we will perform element wise multiplication. We will just multiply element by element at the same position and we get this gated result with both linearity and nonlinearity and then we squish it down with with the down projection. Okay. Residual connection. So we just take the out the output that and add the tokens just as they were after passing after exiting the attention. So we process them and we also add them as well without processing. I mean that's that's it then. So this is the same thing but put together. Yeah, that's the same thing but put together. Final output module. Yeah, that's that's what we just did. And that's it. I think that's it. So this is a simplified version of the llama 4. Uh there are some things that were even too complex for me. But check this out. I'm going to keep making these courses. As I said, you understand this. You understand how to make 90% of all of the LLMs and u I'll look to add figure out how to make animations better, explanations, drawings, graphs, vectors to explain better. So you keep watching these courses. uh and you know over a couple months you will also be able to code any LLM. Now what my goal is I also want to learn to be able to eventually have a course where we together go and code like 1 billion parameter model. We implement all of the best parts of deep sea lama whatever all of the models and just train it from scratch and code it train it from scratch and then see what we can go further. Can we make like 10 million 10 10 billion sorry not not 10 million 10 billion etc. You can check my channel here uh there is a bunch of like AI I'm going to be focusing on these courses now you can watch this deepseek V3 and I told you you can watch also the rope embeddings video. Thank you for watching. Drop a like, share, subscribe, comment and see you in the next video. Also you can give feedback in the comments how I can improve my courses. See you next time.