Decoder-Only Transformers, ChatGPTs specific Transformer, Clearly Explained!!!

Key Takeaways

Decoding is all that you need. StatQuest. Hello, I'm Josh Starmer and welcome to StatQuest. Today, we're going to talk about decoder-only transformers and they're going to be clearly explained. Trust me, whatever transformer you want to use, it's better with lightning. Bam! Right now, people are going totally bananas about ChatGPT. For example, StatSquatch might type something into ChatGPT like, "What is StatQuest?" StatQuest is awesome. Bam! Now, in order to learn more about how ChatGPT works, let's learn about decoder-only transformers, which is the specific type of transformer used for ChatGPT. Note, if you're not already familiar with basic transformers, don't worry. We'll go through each concept in a decoder-only transformer one step at a time. That said, if you are familiar with basic transformers, then you might want to just skip to the last chapter, normal transformers versus decoder-only transformers. Anyway, in this StatQuest, we're going to show how a decoder-only transformer can take a simple input prompt, "What is StatQuest?" and generate a simple response. Awesome. Now, since a transformer is a type of neural network and neural networks usually only have numbers for input values, the first thing we need to do is find a way to turn the input and output words into numbers. There are a lot of ways to convert words into numbers, but for neural networks, one of the most commonly used methods is called word embedding. The main idea of word embedding is to use a relatively simple neural network that has one input for every word and symbol in the vocabulary that we want to use. In this case, we have a super simple vocabulary that allows us to input a short phrase like "What is StatQuest?" And we also have an input for a potential response, "Awesome." Lastly, we have an input for the EOS symbol, which stands for end of sentence or end of sequence. Because the vocabulary can be a mix of words, word fragments, and symbols, we call each input a token. The inputs are then connected to something called an activation function. And in this example, we have two activation functions. And each connection multiplies the input value by something called a weight. Hey Josh, where do these numbers come from? Great question, Squatch, and we'll answer it in just a bit. For now, let's just see how we convert the word what into numbers. First, we put a one into the input for what. And then zeros into all of the other inputs. Now, we multiply the inputs by the weights on the connections to the activation functions. For example, the input for what is one, so we multiply -2.38 by one to get -2.38 going to the activation function on the left. And we multiply 0.10 by one to get 0.10 going to the activation function on the right. In contrast, if the input value for the word is is zero, then we multiply 0.61 by zero to get zero going to the activation function on the left. And we multiply 0.17 by zero to get zero going to the activation function on the right. In other words, when an input value is zero, then it only sends zeros to the activation functions. And that means is Stack Quest, awesome, and the EOS symbol all just send zeros to the activation functions. And only the weight values for what end up at the activation functions because its input value is one. So, in this case, -2.38 goes to the activation function on the left and 0.10 goes to the activation function on the right. In this example, the activation functions themselves are just identity functions, meaning the output values are the same as the input values. In other words, if the input value or x-axis coordinate for the activation function on the left is -2.38, then the output value, the y-axis coordinate, will also be -2.38. Likewise, because the input to the activation function on the right is 0.10, the output is also 0.10. Thus, these output values, -2.38 and 0.10, are the numbers that represent the word what. Now, before we move on, I want to mention that all of these weights and all of the other weights we're going to talk about in this quest are determined using something called backpropagation. To get a sense of what backpropagation does, let's imagine we had this data and we wanted to fit a line through it. Backpropagation would start with a line that has a random value for the y-axis intercept and a random value for the slope. And then, using an iterative process, backpropagation would change the y-axis intercept and slope one step at a time until it found the optimal values. Likewise, in the context of neural networks, each weight starts out as a random number. But, when we train the transformer by getting it to predict words in known documents, backpropagation optimizes these values one step at a time and results in these final weights. Also, just to be clear, the process of optimizing the weights is also called training. Bam. Note, there's a lot more to be said about training and backpropagation. So, if you're interested, check out the quests. Anyway, we now know how to calculate the word embedding values for the first word in our phrase, what. So, we'll keep track of those values with this diagram in the upper right-hand corner. Now, we reuse the exact same word embedding network to convert the remaining words in the prompt, what is StatQuest, into numbers. Thus, the word embedding values for is are 0.61 and 0.17, and the word embedding values for StatQuest are -2.38 and 0.10. And now, we have converted all of the words in the prompt, what is StatQuest, into word embedding values. Note, reusing the exact same word embedding network for each word in the prompt allows the decoder-only transformer to handle prompts that have different lengths. Because we can just copy the network as many times as we need to convert all the words in the prompt into numbers. Also note, there's a lot more to say about word embedding. So, if you're interested, check out the quest. Anyway, now that we know how to convert words into numbers, let's talk about word order. For example, if Norm said, Squash eats pizza. Then, Squash might say, Yum. In contrast, if Norm said, Pizza eats squash." Then squash might say, "Yikes." So, these two phrases, "Squash eats pizza." and "Pizza eats squash." use the exact same words, but have very different meanings. So, keeping track of word order is super important. So, let's talk about positional encoding, which is a technique that transformers use to keep track of word order. Although there are several ways to keep track of word order with positional encoding, one of the most commonly used methods uses a sequence of alternating sine and cosine squiggles. Each squiggle gives a specific position values for each word's embeddings. So, now let's see how we can add positional encoding values to the word embeddings we created for the prompt, "What is StatQuest?" The first word, which in this case is what, has an x-axis coordinate all the way to the left on the green squiggle. And the position value for its first embedding is the y-axis coordinate zero. The position value for the second embedding comes from the orange squiggle. And the y-axis coordinate on the orange squiggle that corresponds to the first word is one. Now, to get the position values for the second word, "is", we simply use the y-axis coordinates on the squiggles that correspond to the x-axis coordinate for the second word. Likewise, we use the corresponding y-axis coordinates for the third word, "StatQuest". Note, because the sine and cosine squiggles are repetitive, it's possible that two words might get the same position or y-axis values. However, because the squiggles get wider for larger embedding positions, and the more embedding values we have, then the wider the squiggles get, then even with a repeat value here and there, we end up with a unique sequence of position values for each word. Anyway, now we just do the math to get the positional encoding for all three input words. And we end up with the word embeddings plus positional encoding for the prompt. What is StatQuest? Now, because we're going to need all the space we can get, let's consolidate the math in the diagrams. And let the sign and cosine and plus symbols represent the positional encoding. Now that we know how to keep track of each word's position, let's talk about how a decoder-only transformer keeps track of the relationships among words. For example, if the prompt was, "The pizza came out of the oven and it tasted good." Then this word, "it", could refer to pizza, or potentially, it could refer to the word "oven". Josh, I've heard of good-tasting pizza, but never a good-tasting oven. I know Squatch, that's why it's super important that the decoder-only transformer correctly associates the word "it" with pizza. The good news is that decoder-only transformers have something called masked self-attention, which can help correctly associate the word "it" with the word pizza. In general terms, masked self-attention works by seeing how similar each word is to itself and all of the preceding words in the sentence. For example, masked self-attention starts by calculating the similarity between the first word, "the", and itself. Then masked self-attention calculates the similarity between pizza and itself and the preceding word, "the". And then masked self-attention just keeps calculating similarities like this, allowing each word to look at itself and the words that came before it, but not after, until it gets to the end of the input. Once the similarities are calculated, they are used to determine how the decoder-only transformer encodes each word. For example, if you looked at a lot of sentences about pizza and the word it was more commonly associated with pizza than oven, then the similarity score for pizza will cause it to have a larger impact on how the word it is encoded by the decoder-only transformer. Oh, no, it's the dreaded terminology alert. Because masked self-attention only allows access to the words that come before it and not the words that come after, it is sometimes called an autoregressive method. Anyway, now that we know the main ideas of how masked self-attention works, let's look at the details. So, let's go back to our simple example where we had just added positional encoding to the prompt, "What is StatQuest?" Now, since what is the first word in the prompt, its masked self-attention values will only reflect its similarity to itself and ignore everything else. In contrast, the masked self-attention values for the second word is reflect the similarity to itself as well as the similarity with the first word, "What?" And the last word, "StatQuest," takes into account its similarity with itself and everything that came before it. Now, I know that "What?" is the first word in the input, but it will make it easier to understand how masked self-attention works if we start with the second word, "is." So, let's move the position encoded "is" over a little bit to give us some room. The first thing we do to calculate the masked self-attention for the word "is" is multiply its position encoded values by a pair of weights and then add those products together to get -2.4. Then we do the same thing with a different pair of weights to get 2.6. We do this twice because we started out with two position encoded values that represent the word is. And after doing the math two times, we still have two values representing the word is. Josh, I don't get it. If we want two values to represent is, why don't we just use the two values we started with? That's a great question, Squatch, and we'll answer it in a little bit. Grr. Anyway, for now, just know that we have two new values to represent the word is, and in transformer terminology, we call them query numbers. And now we're going to use the query numbers for is to calculate the similarities with itself and the first word what. And we do that by creating two new numbers, like we did before, to represent the word is, and creating two new numbers to represent the word what. In transformer terminology, both sets of new numbers are called key values, and we use them to calculate similarities with the query for is. One way to calculate similarities between the query and the keys is to calculate something called a dot product. For example, in order to calculate the dot product similarity between the query and key for is, we simply multiply each pair of numbers together and add the products to get 5.9. Likewise, we can calculate the dot product similarity between the query for is and the key for what by multiplying the pairs of numbers together and adding the products to get -25.7. The small similarity value for what relative to is, a negative 25.7 compared to the large similarity value for is relative to itself, 5.9, tells us that is is much more similar to itself than it is to the word what. That said, if you remember the example where the word it could relate to pizza or oven, then the word it should have a relatively large similarity value with respect to the word pizza, since it refers to pizza and not oven. Note, there's a lot to be said about calculating similarities in this context and the dot product. So, if you're interested, check out the quests. Anyway, since is is much more similar to itself than it is to the word what, then we want is to have more influence on its encoding than the word what. And we do this by first running the similarity scores through something called a softmax function. The main idea of a softmax function is that it preserves the order of the input values from low to high and translates them into numbers between 0 and 1 that add up to 1. So, we can think of the output of the softmax function as a way to determine what percentage of each input word we should use to encode the word is. In this case, because is is so much more similar to itself than the word what, we'll use 100% of the word is to encode is and 0% of the word what to encode the word is. Note, there's a lot more to be said about the softmax function. So, if you're interested, check out the quest. Anyway, because we want 100% of the word is to encode is and 0% of the word what to encode is, we create two more numbers that we'll cleverly call values to represent the word what and scale them by 0.0. Then we create two value numbers to represent the word is and scale them by 1.0. Lastly, we add the scaled values together and these sums, which combine separate encodings for both input words, what and is, relative to their similarity to is, are the masked self-attention values for is. Bam! Now that we have the masked self-attention values for is, we can go back and calculate them for the first word, what. Remember, because what is the first word, it only needs to know how similar it is to itself. So, we can get rid of most of what we did for the word is, but keep the key and value numbers that we calculated for the word what. Now, so we have a little more room to work, let's move everything over a bit. And, just like we did before, we create a query for the word what. Now, we use the query and key for what to calculate the similarity with itself and we get -12.4. So, we plug -12.4 into the softmax function and we get 1.0. Lastly, we scale the value numbers for what by 1.0 and -2.9 and -1.3 are the masked self-attention values for the first word, what. It might seem a little silly to do all this math just to end up using the value numbers for what as the masked self-attention numbers for what. However, doing it this way gives us a unified method for calculating masked self-attention. So, so far we've calculated the masked self-attention values for what, which only required taking its own value numbers into account. And we calculated the masked self-attention for is, which required taking the value numbers from what and is into account. Now, we need to calculate the masked self-attention for the third word, StatQuest. And that means we need to take the value numbers for StatQuest into account, which we calculate like before. And we need to take the value numbers for what and is into account. So, the first thing we do is calculate the query numbers for the word StatQuest. And then calculate its key numbers. And then we bring back the keys for what and is and calculate the similarities between the query for StatQuest and the keys for StatQuest, is, and what. Now, we run all three similarities into the softmax function. And the output from the softmax tells us what percentage of each word's value numbers to use when calculating the masked self-attention for StatQuest. Now, we bring back the value numbers that we calculated earlier for what, is, and StatQuest and scale them based on the similarity scores. Lastly, we add the pairs of scaled values together to get the masked self-attention numbers for the word StatQuest. Bam! Note, before we move on, I want to point out that we reuse one set of weights to create query numbers for each word. In other words, the set of weights we used to create the query numbers for what is the same set of weights we use for the query numbers for is and StatQuest. Likewise, the key numbers are calculated with a different set of weights that are shared for each word. And the value numbers are also calculated with another set of weights that are reused for each word. Reusing the sets of weights for the query, key, and value numbers lets the decoder only transformer handle prompts that have different lengths because we can just keep reusing the weights as many times as we need. Now that we understand the details of how masked self-attention works, let's shrink the diagram so that we can keep building our decoder-only transformer. Bam! Josh, you forgot something. If we want two values to represent what, why don't we just use the two position encoded values we started with? First, the new masked self-attention values for each word contain input from all of the other words that came earlier, and this helps give each word context. And this can help establish how each word in the input prompt is related to the others. Also, if we can think of this unit with its three sets of weights for calculating queries, keys, and values as a masked self-attention cell, then, in order to correctly establish how words are related in complicated sentences and paragraphs, we can create a stack of masked self-attention cells, each with its own sets of weights that we apply to the position encoded values for each word to capture different relationships among the words. In the manuscript that first described the original GPT, they stacked 12 masked self-attention cells. Bam! Okay, going back to our simple example with only one masked self-attention cell, there are a few more things we need to do before we start generating a response to the prompt, "What is Stack Quest?" First, we take the position encoded values and add them to the self-attention values. These bypasses are called residual connections, and they make it easier to train complex neural networks by allowing the masked self-attention layer to establish relationships among the input words without having to also preserve the word embedding and position encoding information. Bam! Lastly, we need a way to use the encodings we have for each word in the prompt to generate the word that follows it and then generate a response. In other words, we want to use these two numbers that represent the word what to generate the word that comes after what. And we want these two numbers that represent the word is to generate the word that comes after is. Lastly, we want these two numbers that represent the word StatQuest to generate the word that comes after StatQuest. Hey Josh, I don't get it. Why do we want to generate the word that comes after what when we already know the next word is? Because this is a decoder-only transformer, we need one thing that can both encode the prompt and generate the output. Thus, even though we are not yet generating a response, we need to include the parts that will do it. Also, we can compare the known input to what the model generates when we train the model. So, and now let's see what the model generates given the first word in the prompt, what. And that means taking these two values, -5.28 and -0.2, and plugging them into something called a fully connected layer. This fully connected layer has one input for each value that represents the current token. So, in this case, we have two inputs and one output for each token in the vocabulary, which in this case means five outputs. Note, a fully connected layer is just a simple neural network with weights, numbers we multiply the inputs by, and biases, numbers we add to the sums of the products. Also note, in the original GPT manuscript, instead of a new fully connected layer, they use the word embedding network that we started with, but in reverse. In other words, they just reuse the same set of weights that we used to encode the words into numbers, and then flip them to help decode the numbers. However, not all decoder-only transformers do it this way. And a very common alternative is a fully connected layer. Small bam. Now, when we do the math, we get five output values, which we run through a final softmax function to generate the next word, is. And since the prompt was what is StatQuest, the model generated the correct word. Bam. The second word in the prompt is is, and it generates the word what, which is not correct. Womp womp. However, the correct word, StatQuest, was almost correctly generated, so we hope our model doesn't feel too much shame. Note, if we were training the decoder-only transformer, then we would use the fact that we made a mistake to modify the weights and biases. In contrast, when we are just using the model to generate responses, then it really doesn't matter what words come out right now. So, in this case, we'll just note that we made a mistake and move on. Lastly, the word StatQuest generates the EOS token, which is correct since we are at the end of the prompt. Bam. Now, let's review what we've done so far. We started with an input prompt, What is StatQuest, and we used word embedding to convert each word into numbers. And we used positional encoding to keep track of word order in the prompt. Then we added masked self-attention to determine the relationships among the words in the prompt. Then we added residual connections to make it easier to train the model. Lastly, we added a fully connected layer and a softmax to generate the next word. Each part of the decoder only transformer is reused so that it can handle prompts of different lengths. And the encoding for each word in the prompt can happen at the same time rather than sequentially and thus can be done quickly when multiple computing cores are available. Lastly, generating the output uses the exact same steps that we used to encode the prompt. Thus, generating the output starts with word embedding. And it uses the exact same word embedding network that we used to encode the prompt. And because we just finished encoding the prompt, we start generating the output from the EOS token. In this case, we're using the EOS token to start generating the output because that is a common way to initialize this process. However, sometimes you'll see people use SOS for start of sentence or start of sequence to initialize the process. Josh, starting with SOS makes more sense to me. Then you can do it that way, Squatch. I'm just saying that a lot of people start with EOS. Also note, we will start with the EOS token regardless of whether or not it was the last token generated when we encoded the prompt. Anyway, we plug in one for EOS and zero for everything else. And we end up with the numbers that represent the EOS token. Now, let's shrink the word embedding down to make more space so that we can add positional encoding. Note, the EOS token comes after the three tokens that represent the input. So, it's in the fourth position. And since the EOS token is in the fourth position with two embeddings, we just add those two position values. And before we move on, we need to say a few more words about masked self-attention. So far, we've talked about how masked self-attention helps the decoder-only transformer keep track of how words are related within the input. However, it's also important to keep track of the relationships between the input sentence and the output. For example, if the input sentence was "Don't eat the delicious-looking and smelling pizza." Then, when generating new output, it is super important to keep track of the very first word, "Don't." If we focus on other parts of the sentence and omit the "Don't," then we'll end up with "Eat the delicious-looking and smelling pizza." And these two sentences have completely opposite meanings. So it's super important that, when generating the output, we keep track of the significant words in the input. The nice thing is that all we have to do to add this ability to our decoder-only transformer is just include the prompt when we do masked self-attention while generating the output. So the first thing we do is calculate the query numbers for the EOS token. And then we calculate the key numbers. Using the same sets of weights we used for the prompt. And then we bring back the key values for the prompt, "What is StatQuest?" Then we calculate the similarities between the query and the keys. And run everything through the softmax function. Now we calculate the value numbers for the EOS token. And bring back all of the value numbers we calculated earlier. And scale all of them based on the similarity scores. Lastly, we add the pairs of scaled values together to get the masked self-attention values for the EOS token. Bam. Anyway, and now that we have the masked self-attention values for the EOS token, We add the residual connections. Now we move the diagram to the left and run the numbers that represent the EOS token through the same fully connected layer we used earlier. And the same softmax function we used before. And the first word generated by our decoder only transformer is awesome, which is totally awesome. However, even though the output is awesome, we're not done yet because the decoder only transformer will keep generating output until it generates the EOS token. So we plug the word we just generated, awesome, into another copy of the word embedding layer. Then we add the positional coding using the values for the fifth position because now the sequence is what is Stack Quest EOS awesome. Now we calculate the masked self-attention values. Bam! Now that we have the masked self-attention values for awesome, we add the residual connections. Now we move the diagram to the left and run the numbers that represent awesome through the same fully connected layer we used earlier. And the same softmax function we used before. And our decoder only transformer generates the EOS token, which means we are done generating output. Double bam! Now let's talk about the differences between the decoder only transformer that we just learned about and a basic transformer. As we just saw, a decoder only transformer uses the exact same components to encode the prompt that it uses to generate the output. And it uses masked self-attention, which is calculated with only the current word and everything that preceded it. And the masked self-attention is applied equally to the input prompt and to the output that is generated. Masked self-attention allows a decoder-only transformer to determine how words in the prompt are related and make sure that it keeps track of important input words when generating the output. In contrast, a regular transformer uses one type of unit called the encoder to encode the prompt and a different type of unit called the decoder to generate the output. When encoding the input prompt, instead of using masked self-attention, a normal transformer uses self-attention, which includes all of the words in the input, not just the ones that came before, to determine how the words are related to each other. And a normal transformer uses encoder-decoder attention to let the decoder keep track of important words in the input. Encoder-decoder attention uses queries from the decoder, but only keys and values from the encoder. Now, so far we've only talked about how attention is used in a normal transformer during inference, when it encodes the input and generates new output. However, during training, a normal transformer uses masked self-attention in the decoder. For example, during training, we know the output should be awesome EOS, which means we don't have to decode the initial EOS before we decode awesome like we do when we are generating new output. Instead, since we know we will be decoding awesome, we can do its math the same time we do the math for the EOS token. Doing the math at the same time means we can train faster. And thus, during training, a normal transformer will use masked self-attention on the tokens in the known output. This allows the transformer to learn how to generate the correct output without cheating and looking ahead. Note, when we are training a normal transformer, the masked self-attention only includes the output tokens. In contrast, a decoder-only transformer uses masked self-attention all of the time, not just during training, and it includes the input and the output. Bam! So, the three big differences between a normal transformer and a decoder-only transformer are a normal transformer uses one unit to encode the input, called the encoder, and a separate unit to generate the output, called the decoder. And a normal transformer uses two types of attention during inference, self-attention and encoder-decoder attention. Lastly, during training, a normal transformer uses masked self-attention, but only on the output. In contrast, a decoder-only transformer has a single unit for both encoding the input and generating the output. And a decoder-only transformer uses a single type of attention, masked self-attention. And a decoder-only transformer uses masked self-attention all the time on everything, the input and the output. Triple bam! Note, there's a lot more to say about normal encoder-decoder transformers. So, if you're interested, check out the quest. Now it's time for some shameless self-promotion. If you want to review statistics and machine learning offline, check out the StatQuest PDF study guides and my book, The StatQuest Illustrated Guide to Machine Learning, at statquest.org. There's something for everyone. Hooray! We've made it to the end of another exciting StatQuest. If you like this StatQuest and want to see more, please subscribe. And if you want to support StatQuest, consider contributing to my Patreon campaign, becoming a channel member, buying one or two of my original songs, or a t-shirt, or a hoodie, or just donate. The links are in the description below. All right, until next time, quest on.