8: Deep Learning for Natural Language – Transformers, Self-Supervised Learning

Key Takeaways

Okay. Uh, all right. So, we'll continue with transformers today. Part two. Uh, we're going to do the second pass. Uh, this is going to be a deeper pass through the transformer stack. Um and I think maybe the next 30 minutes it's potentially the most demanding 30 minutes of the entire course. Okay, with that motivational speech, let's get going. Okay, so quick review. Why do we want transformers? Because we want u we want an architecture that can generate output that has the same length as the input. Same length. Oh, there it is. Uh number two, we want to take the context into account and we want to take the order into account. And as you saw last time, the transformer architecture delivers on those three requirements. And so uh just a quick review, if you have a phrase like the train liftation, we have all these little arrows which stand for the the standalone or uncontextual embeddings. Uh and then sometimes this works. So I'm going to put it close to me here. Okay. All right. So um so if you here if you we start with either standalone embeddings i.e. the contextual embeddings uh which have been pre-trained or random doesn't really matter. If you look at the collab we did uh the other day we actually just start with random weights for the embeddings and then we add positional embeddings to them. And so you know each embedding each word here we take it standalone we take its positional embedding we just literally just add them up element by element then we get a total embedding and that's called the positional embedding of each word. Okay. And then uh that's what we have position input embeddings. So this whole thing goes into this transformer encoder stack and what pops out the other end is contextual embeddings. Okay. So that's the overall flow. Now we applied this uh the transformer stack to the word to slot classification problem where we basically took every incoming natural language query that comes in. We calculate its positional embeddings and then we run it through the transformer stack. uh and then we get contextual embeddings and then at this point uh since each word that comes out each embedding that comes out needs to be classified into one of 125 possibilities we run it through a ReLU and then we and when we attach a softmax to each embedding right this is basically what we did last class um so this is the transformer encoder okay now actually any questions on this before I continue I was wondering why when how do you decide where to add more self attention and where to add transformer layers? You mentioned that chart has 96 of them. >> Yeah. So right so GPD3 has 90 96 transformer blocks. Each one is a block. Um, so I think the question goes to do you add more attention heads within a single block or do you add lots of blocks? And both are good things to do. Um, what increasing the number of attention heads in a block does for you, it allows you to pick up more patterns at that level of abstraction. But if you add more blocks, much like later convolutional filters can build on earlier convolutional filters, you're going up the levels of abstraction. So to go to vision for instance you have the notion of lines and so on in the beginning and then you have a notion of edges which are two lines then you have you know nose eyes face and so on and so forth. So both are worth doing. So typically that's what you you typically find that people typically have you know maybe a dozen heads or you know five six a dozen heads. We'll see examples of how many heads in a couple of architectures later on today. And you can the more you go up the more uh more capable the model becomes. as long as you have enough data to train it well. So the perennial question of do we have enough data to train this large model because if you don't have enough data we might run into overfitting problems and so on. That's always the trade-off. So okay so here I just want to quickly switch to the collab because we didn't get have a chance to finish it. I'm not going to run it because it's going to take some time. So where we left off last time. Okay. So here we we basically took this architecture that we just saw on the slide and then we essentially wrote it as a keras model and I went through this model in the last class so I'm not going to go through it all over again. What we did not do last class was to actually run it. Um and so uh so if you actually run it right you can just run it for 10 epochs just like we normally do. Give it data give it a bunch of epochs choose a particular batch size. I just arbitrarily chose 64. You run it for 10 epochs and then you evaluate it on the test set. You get a 99% accuracy on this problem. One transformer stack. That's it. One one block rather. One block. That's it. And uh of course here there's a little trickiness going on here because a naive model can literally say every word that comes in is other. O. And since the O's are the majority of the words, it's not going to do badly, right? It's like having a classification problem in which one class is very predominant. So the naive way to actually do well is to just say every time something comes in, oh it's that majority class. The same thing happens. But if you then adjust for that, it turns out that the accuracy on the nono slots, which is really what you care about, is actually 93%. Which is actually pretty good. Okay. Uh and then I had some examples of, you know, lots of fun queries you can do, including queries where I try to break stuff like cheapest flight to fly from MIT to Mars and see what happens, you know, things like that. So have fun with it. Okay. Um, all right, back to PowerPoint. So, this is what we had. Now, what we're going to do in today's class, we are actually going to take the encoder we built last time and introduce three new complications into it. And when we finish introducing these three complications, we will actually have the actual transformer that was invented in the 2017 paper. Okay. All right. Um, the first tweak is the hardest tweak. So we'll slowly work our way to it. U so the thing to remember is let's review self attention. What is self attention? You have a bunch of words and we further said that for any particular word like station we want to take its positional embedding and then make it contextual. And the way we do that is by taking each word's embedding and then calculating these dot productducts between all the other words. And then since these dot products can be positive or negative we want to make them all positive and normalize them so that they nicely add up to one. So we then exponentiate them and then divide with the total, right? Which is basically soft max. And when you do that, you have nice fractions that add up to one. And then we said, well, the contextual embedding for W6 is just all these weights S1, S2 all the way to S6 multiplied by the original W's and then you get the context for W6. So this is the basic logic we covered last time. Now it is obviously the case that we explained it only for one word but we have to do the same exact operation for every one of the other words too so that we could calculate W5 hat, W4 hat, W3 hat and so on and so forth right so there's a lot of computations that are going on and they all look kind of similar where you got to do a bunch of dot products you got to like you know do some soft maxing on it and stuff like that so the natural question is is there a way to organize it very efficiently And the short answer is yes. In fact, if you could not do that, there wouldn't be any transformer revolution. Okay, because there is that ability to package it up into a very interesting and efficient operation that allows you to put the whole thing on GPUs. Okay, so now I'm going to switch to iPad uh and give you some iPad scribblings of mine which were concocted last night because I was very unhappy with the slides that follow. So, we're going to do iPad. Okay. U All right. So if it works, you folks are lucky. If it doesn't work, last year's huddle class is luckier. So let's shift to that. All right. So we're going to go here. So let's assume we have a simple thing like uh oops. Okay, instead of you know train left the station which is a long sentence, let's just say you have a simple sentence like I love hodddle. Okay, and so I love hodddle is what you have and then you have these standalone embeddings W1 W2 W3. Okay, so it comes into the self attention layer and let's assume that these W1's, W2, W3, they're already positionally encoded, right? We have already added up the position encoding, all that stuff also. It's all behind us. That all happens outside the transformer. So you you you get it here. Now what you do is you actually make three copies of this thing. Okay? And let's call this whole thing as just X. Okay? I'm just giving it the name X. It's a matrix of these three vectors. And so the first copy goes up here, the second copy goes straight, and the third copy goes down. And don't worry about the third copy just yet. So if you look at the the first two copies, here is the key thing to focus on. Okay, this whole thing here. Remember that we want to calculate dotproducts between all these vectors. And basically we want to calculate the dot product of every pair of vectors, every pair of words. The whole point of self attention is that every pair of words we figure out how attracted or related they are. Right? Which means that we have to calculate all pairs of dot products. And so what you do is you take this vector right there W1 WW3. You take this other copy that went up. Okay? And then you transpose it. So when you transpose it, it all becomes nice and vertical like that. Right? All the vectors come in came like this. When you transfer, it becomes vertical. And now what you do is you take each one you take W1 and then you multiply it by W1. Here you take W1 W2 W1 W3. You calculate all those dot products like that. And when you do that you have these nice cells where every pair of words their dot products have been calculated in this grid. Okay. And the key thing to see here and folks with a matrix algebra background will see this immediately. All we are doing is we are taking this x which is the matrix that came in and then xrpose which is the matrix that we went sent up and then brought back down. We are basically doing a matrix multiplication of x * xrpose. That's all we doing. And when we do that we're getting this nice uh grid of where in which every pair of words their dot products have been calculated for you with one matrix multiplication. Boom. Done. Okay. Okay, so if you have three words, there are nine multiplications, right? So if you have a million words, that's a lot of multiplications, right? One trillion multiplications on the order of all trillion. And the reason to say order is because you know W1 * W3 is the same as W3 * W1. So there's some duplication here. So you get this grid, okay, in one shot is one multi multiplication. And then we because each of these numbers is just a dot product which can be negative or positive, we need to softmax it. And so what we do is we take all these numbers and we put it into a softmax function where for each row it calculates a soft max. And what do I mean by that? It takes each number here does e raised to the top e ra to the number. It does it for each of these numbers and then divides by the sum of those numbers for each row. And when you do that okay you can think of this operation as soft max applied to x * xrpose you get this nice little table of numbers. This table of numbers basically says that for the first word right W1 for the first word take 0.1 of the of the first one 7 of the second.3 of the 2 of the third and add them up. We do a weighted average. So we have this table here. We have now the third copy shows up here. Okay is right there. So we do this times that which is just a matrix multiplication again. And when we do that we get the final contextual embeddings. So this for example is just 0.1 * w12 * w2 point sorry 7 * w2 and then2 * w3 right there. And you can see the same logic here as well. Okay. And you can read it later on. I will post this thing uh to make sure you understand exactly how it flowed. But the larger point I want you to focus on is that the entire sol self attention operation we just looked at here basically is this this beautifully little compact matrix formula. Okay X comes in you do XRpose you do a matrix multiplication you do a softmax on top of it and then multiply by X again and boom you're done. So that is the magic of taking the transformer stack and representing it using matrix operations because then lightning fast on GPUs. Okay. All right. That was the warm-up. Now let's crank it up a notch. So recall that in the last class um I talked about the fact the self attention operation the W's are coming in and we're doing all this stuff with the W's right and then we're getting some W hats out but there are no parameters there's nothing to be learned inside the transformer self attention layer right there are no there are no weights there are no biases there are no coefficients so well okay What are we learning then? Right? So what we now do is we going to make the self attention layer tunable. We're going to inject some weights into it so that when we train it on an actual system, it'll the weights will keep changing to adapt itself to the particularities of whatever problem you're working on. Right? So that takes us to the tunable self attention layer. Okay? Tunable self attention layer. So this is the key thing to keep in mind. U any questions on this before I continue with the tunability thing. Okay. Is this picture working out by the way? Okay. Uh all right. So what we now do is we have the same exact logic as before where we have this thing that comes in. Okay. We have this input that comes in the same we call it X again. this whole this matrix of embeddings and then before we just send three copies instead of doing that what we're going to do is we'll take each copy X and then we will actually multiply it by a matrix okay this matrix is called the key matrix okay and this matrix this matrix of numbers are weights that will be learned by Brack prop so basically what we're saying is that when this thing comes in let's see if there's a way to transform this X into some other set of embeddings which may be useful for your task. We don't know if they're going to be useful, but surely giving it a bit more ability to have weights which can be learned means that it giving it more expressive power, more modeling capacity. And whether it actually uses the capacity will depend on how much data you have and how well you train it. And maybe if it's not useful, it won't use it. In what I mean is if transforming X actually doesn't really help at all, then this matrix A is going to be what? it's going to be the identity matrix because you take basically one and multiply by X you'll get one X again. So in the worst case maybe it just says I have nothing to learn here but maybe there is something you can learn. So so that's what we do. So we multiplied by this matrix A K and then we come up with the same you know some embeddings transformed embeddings and we call these things K okay K. Now this KQV as you will see has its origins in the in this field of information retrieval but I personally find that that interpretation is not super helpful because transformers are used for lots of applications outside information retrieval. So I'm not going to go with that kind of interpretation. I'm going to go with interpretation of let's make each of these things tunable. Okay. And tunability means we need to give it weights. All right. So that's what we have here. Now the second copy we did this with the first copy. Well, let's do the same thing with the second copy. We'll take the second copy and multiply it by some other matrix called AQ. And when we are done with that, we get these embeddings. And we will call these embeddings as Q. Okay. Now, just like before, we will take this this thing here and we'll transpose it. So, it all becomes nice and vertical like that. And then we'll do exactly the same as before. We'll calculate all these pair-wise dot productducts using one one shot one matrix multiplication. And because we are calling this Q and we are calling this whole thing as K. This thing just becomes Q * KT. Okay. At the end of it you come up with a grid of numbers just like before. Okay. And these numbers could be negative or positive. So we need to do the softmax on them to make sure they are well behaved fractions that add up to one. So we take this Q KT business and then we do we just run a we put it through a softmax function for each row and when we do that we we'll get basically the the like a table like the ones we saw before by the way the numbers here are the same just because I duplicated it because I'm lazy in reality given it has gone through all these transformations the numbers are not going to be the same right uh you have these numbers and then you take the final copy which is x * av Right? Each copy is getting multiplied by its own matrix. Right? And this copy is being multiplied by AV. And let's call this X A. Okay? Which is here as just V. And so what you have here is this soft max QT * V is exactly the same kind of dot product as we saw before matrix multiplication. So we have these contextual embeddings and that's what's coming out of the of the transformer block. So now the whole thing we did here the whole thing can be represented as soft max of Q KT * V. Okay. So if we zoom in a bit. Come on. Okay. Okay. So X came in. Three tracks went here. The first track X * A K X * AQ X * A V. And this thing is called K. This thing is called Q. This thing is called V. And then we do the same transpose as before. We do the dotproduct thing to calculate the pair-wise dot products for everything which is just Q KT. We run it through a soft max. We get soft max of Q KT. We multiply it by one to do the final waiting and then boom the output comes and that's this function. That's it. Okay. So what we have done is we have introduced three matrices learnable matrices into the self attention layer. Okay. Now, okay. Let me just stop there for a sec. Questions. Yeah. [clears throat] >> Is there a relationship between AK, AQ, and A >> independent independent matrices? >> Yes. >> Like we have >> could you use the microphone please? >> Here we have three set of parameters K, Q and P. If there are let's say if there were 100 the total length was let's say the number of total totals were let's say 50. So you would have uh 50 for a set of parameters like you'll have to >> so if you have a 50 if the dimension is 50 long what is coming in the W's are 50 long then the key the what comes out of it if you want it to be 50 as well so this matrix needs to be 50 * 50 2500 >> U Luna >> what are the different things the three the three matrices are trying to Sorry, >> what are the different things that the matrices are trying to learn? >> We don't know. All we are saying is that we have a self attention layer which can pay attention to every pair of words. But we need to give it some ways to transform what is coming in into potentially useful things. Right? As to their actual usefulness, we'll have to figure out if if it actually helps or not. And of course, as you know, the the punch line is that yeah, it helps massively. That's why we do it. In general, what you will find in the deep learning literature is that whenever you want to increase the capacity, the modeling capacity of a particular model, you just take a small piece and inject a little matrix multiplication into it. You take a vector that's showing up in the middle and then you make it run through a matrix to get another vector and then further after you run it through a matrix, you run it through a little ReLU as well. Even better. So that's how you inject modeling capacity into the middle of these networks. Okay? And that's what these people are doing here. Yeah. >> In the last step, you had the matrix V. So on the previous example, you had used the original matrix X. So could you just say for why is it not using X? What does that mean? >> So what we're saying is that the in the initial version we had three copies and we treated them all identical. Now we said well there are are there ways to transform each copy into some other representation which could be useful. So we may as well use three different matrices for it. Why stop with two? There are three opportunities to make them more expressive. We'll use all of them. >> Yeah. >> You mentioned that these are kind of you're tuning it. You're kind of fine-tuning it. Is there any risk? >> We're not fine-tuning it. Uh just to be clear on the on the vocabulary here. So we have added more weights to make them tunable. What that means is that we when we finally train this entire model, remember all the weights are going to be updated using back propagation, right? In particular, these matrices will also get updated using back propagation. >> So there's no risk of is there a risk of >> there's always the risk of overfitting when you add more parameters to a model >> which means that you have to look at the validation set and all that good stuff. We are basically adding more parameters in a very interesting way because we want to add more capacity to the self attention layer. We want to give it a more of an ability to learn things from the data. Before it could not learn anything. It could only do dot products. So we we want to solve that problem. All right, I'm going to continue and we'll come back to this. Okay. Um so uh all right, let's just just for fun, I'm going to do this. Um the the original paper is called attention is all you need. This is a transformer paper. You folks should read it at some point. Just want to show you something. Uh You see that? So that is the famous transformer formula. Okay. And the only thing we ignored is this root of DK business in the back under it. I wouldn't worry about it. The reason they have it is because these soft maxes when you have lots of numbers and some numbers really really big what's going to happen is that all the other numbers are going to get squashed to zero. Okay. And so to make sure the gradient flows properly, they just divide it by a particular number to make sure no number is too big. Okay, that's a small technical important but bit of a technical detail which is why I ignored it in my iPad. But the rest of it you can see this is exactly the formula we derived qt * v softmax. Okay, so this is the famous transformer formula and congratulations now you understand it. You seem less than fully convinced. Okay. Yes. Hi iPad. Now I have a bunch of slides which I had but actually I'll come back to this. I had a bunch of other slides. This is from last year uh which actually explains what I did in the iPad in a very different way without using any matrices and so on. I was looking at it last evening and I was getting very annoyed by these slides for some reason because I felt that it wasn't really conveying the core matrix sort of the matrix uh the ability of using matrix algebra to to actually do this so efficiently and compactly which is why I decided to like handdraw this thing on the iPad. Okay, but you should read it afterwards to make sure that whatever you saw on the iPad actually matches this. Okay, because two different ways of understanding something always helps. Um okay so this what we have here now to just to recall the by making self attention tunable we get a very interesting benefit which is that when you have these different attention heads before you could have two attention heads but because there were no parameters inside their outputs would have been identical because the inputs are the same for both therefore the outputs would be identical but now by since each attention head will have its own aq matrix the outputs are going to be different. That's why it makes sense to do the tunability thing because that's what actually makes multiple attention it's actually useful. Um is is there actually any relationship between AK AQ and AV or is the A just for like a notation standpoint? >> Just notation. The thing is we want to use QV for the resulting matrix and so I had to find something else to use for the first one and I was like okay aqaq and we at MIT we do subscript super subcripts right so yeah >> what what is the the size of the matrices are there like square matrices or >> yeah so typically what happens is that um there's a whole bunch you can think of it as a hyperparameter in some ways um typically what people do in most implementations is that they will actually just preserve the size so if the incoming embedding is and they'll make sure the the thing coming out of thing is also 10. So you just do a 10x10 matrix to transform it. Uh but the the the value v av matrix on the other hand there's a bit more technical stuff going on where it often tends to be smaller. Um so for example let's say that your incoming is 100 you do 100 to 100 for the key 100 to 100 for the query. But if you have say five attention heads, you may do 100 to 20 for the W's because ultimately all the V's are going to get concatenated into another 100 again. So I can tell you more offline but fun broadly speaking these things tend to get transformed. They don't they preserve the dimension 10 and 10 out. Yeah. >> So this uh aq uh these numbers are random when you start with it and then allow it to back. >> Exactly. Exactly. So all right um yeah so the values in these matrices are weights learned through optimization using SGD. Uh and then what that means is that each of these attention now has its own copy of these matrices. It has its own matrices and over the course of back propagation these matrices will look very different. Okay. So important each attention head will have its own mat set of three matrices. So if you have 10 attention heads 30 matrices will be learned. So by the math it seems like it's creating essentially a relationship between all of the content being ingested and if you're creating if you're ingesting all the content for each attention head are there different categories of attention head type that you're trying to go after? >> Yeah. So basically what we're trying to do is to say a particular attention head. So in any particular sentence it may turn out to be the case that one pattern could be about the meanings of these words right like the word bank and what it means the word station train things like that. That's what really we've been talking about. But there is a whole other pattern to do with grammar and tense and things like that. There could be another one in terms of tone. All those things are very important. And a priority we don't know how many such patterns exist. Much like in a convolutional network, we don't when we're designing how many filters to have, we don't know how many kinds of little things we have to detect, you know, vertical line, horizontal line, semicircle, quarter circle, stuff like that. So, you just give it a lot of capacity so that it can learn whatever it wants. All right. So, um so that that is the transformer encoder. So, we have done one the first of the three complications needed to make it like industrial strength and legit. Uh the second thing we do is something called the residual connection. So what we do is that whatever comes out here right W1 through W6 goes in and comes out as W1 hat W2 and so on and so forth right actually sorry what comes out here is the hats but what comes out here is some intermediate W's right that is what the selfident is going to give you some intermediate W's what we do is and because what's coming out here these vectors are the same length as what goes in we can just add them element by element So we take the input and we actually add it to what comes out. So why would we want to do that? Why would we want to you know go to a lot of trouble to process this thing and then when it comes out we like literally add up the original input? What's like what do you think the intuition is? So turns out, think of it this way. You have a bunch of inputs. You send it to a neural network. It transforms it and gives you something else. Right? At that point, you might be thinking, well, everything that go everything that happens in the network from that point onward can no longer see your original input. It can only work with the transformed input. Right? But what if your transformations are not great? So as an insurance policy what you can do is you can take the the transform stuff and you can take the original stuff and send both in. Right? And this whole thing is and you can Google it. It's called like a wide and deep network and things like that. But the whole point is that let's not lose the original input anywhere. Let's also send it along. But if you keep adding the original input to every intermediate layer, it's going to get longer and longer and longer and bigger, which you don't want because you want it all to be the same size. So the simplest alternative is to just add them up. You take the transform stuff and you add the original input. You get the same thing again. The the what came out what came in W1 was a 100 long vector and the transformed version is also 100 long. So just literally 100 100 add them up. That's it. You get another 100 long vector. So that is what's called a residual connection. Okay. And as it turns out, residual connections make it m improve the gradient flow during back propagation dramatically and that's why they are very heavily used. And in fact, RestNet, which we looked at for computer vision, it stands for residual net because it was the first network to actually figure this out. It's not this this is not just a transformer thing by the way. It's widely used in you know lots of new architectures. The notion of a residual connection that's what it means. Okay, so we do a residual connection and then we come to the final tweak which is called layer normalization. So once we add the residual connection, we are going to do something else here to these vectors before they continue flowing. And what layer normalation does is it basically says that I you will recall from the very beginning of the semester I've been saying that whatever comes into a neural network the inputs let's just really make sure that they are all in some sort of a narrow well- definfined range they can't be in a big range right so for pictures for images we divided every number by 255 so that every little pixel value is between zero and one okay for continuous things like the heart disease example we standardized by calculating the mean and the standard deviation and doing subtracting the mean and dividing by the standard deviation. So when you do that all the numbers are going to roughly be in the minus1 to +1 range. So in neural networks it's for backrop to work really well you have to make sure that no numbers get too big that all the numbers are always in some sort of a narrow range. So what layer normalization does is to say you know what whatever is coming out here I want to make sure none of these numbers are too big. I want to make sure they're all well behaved in a small range because if I don't do that back prop is not going to work very well and so is this what we do to ensure we don't problem of vanishing right >> so um so the there technically there are there could be two problems there's an exploding gradient and vanishing gradient both are bad this is a way to address it so you will find a whole bunch of dash normalization techniques layer normalization batch normalization and so on and so forth all these are methods to make that these numbers stay in a small range so it doesn't cause gradient issues later. All right. So in particular what we do is or what happens inside this layer layer normalization is we just calculate the mean and standard deviation of every one of these embeddings. Okay? Right? If you have let's say six embeddings here, we'll have six means and six standard deviations, right? For each one across the rows and then we standardize it. Meaning subtract the mean divide by the standard deviation. And when you do that, all these things are going to be nice and small. And then we do this a little other thing where we we have introduced two new parameters to rescale it and move it around a little bit just because adding more weights always helps make these things better. So we add them and this gets slightly complicated because of the way the dimensions work. So I'm not going to spend much time on it. Uh and then what comes out the other end is a very well- behaved set of numbers in a nice and small and narrow range. Okay, so this is called layer normalization. Um, you can see this link to understand it a bit better. Um, and we do that as well. So to put it all together, so this is a transformer encoder where we have this multi head attention layer where each attention head in the inside of it is tunable with those a matrices and then we have a residual connection. We do that and then we do layer norm and then we do the same thing in the next feed forward layer as well. And then boom out pops the output >> by that definition in the multi head attention layer when I'm doing tone and everything theoretically I can add even the biases or the hate speech aspects which come in to take care of it right so the model can account for the fact that something is biased or something is not >> um the thing is it's not so much the model is accounting for it is capturing whatever patterns happen to be inherent in the data it's capturing Right now what you do with that capture is up to you. It depends on the actual problem you're trying to solve. In particular, it is going to capture all the bad stuff too because if your training header has a lot of biased stuff in it, toxic things in it, dangerous things in it, it doesn't it doesn't have a sense of values as to what it's good or bad. It's just going to pick it up. >> Yes. >> On that then how do you actually make it angle on those or how do you mitigate the effect of those? That's a whole course unto itself, but I'm happy to give you pointers offline. All right, so this is what we have and remember what I said that this is just a single transformer block and since what comes in and what goes out are the same dimensions, we can just stack them one after the other, right? It's very stackable. You can do it, you can multiply, you can you can stack it vertically as much as you want. And as I mentioned, I think GPD3 has 96 of these things stacked one on top of the other. Um and so yeah that brings us to that is it that is the transformer encoder and this exactly maps to that. So basically the input embeddings come in you add positional embeddings and then you send it to say these many attention blocks and they all get added up and then it comes over the attention block you add the add and nom here means add means residual connection because you're adding the input which is why you have this arrow going from the input being added there and then you normalize it send it along and do it again and out comes the output. So all right now just to be very clear on what is being optimized during back propagation in this complex flow right now clearly the the embeddings that you started out with both the standalone embeddings as well as the positional uh the position embeddings those things are going to get optimized right those are just weights they're going to get optimized clearly everything inside the transformer encoder block is going to get get nominized right and what are they well they are the aqa v matrices for Each attention head layer norm has parameters as well. The next like the little feed forward layer has weights as well. All these things are going to get optimized and then it goes through this relu which again has a bunch of weights. It's going to get optimized and then the final softmax has a bunch of weights. That's going to get optimized. All these things are going to get optimized by back prop. Okay. So in that sense you just step back for a second and look at the whole thing. It is just a mathematical model with a lot of parameters and we're just going to use gradient descent or stoastic gradient descent to optimize it. That's it. Yeah. >> For those eight matrices we train the model, are we calculating weights for like each cell of every possible matrix based on the number of inputs like every possible dimension up to the max number of inputs? Um actually the the weights themselves um don't depend on how long your input sentence is because remember what we're doing is for each sentence that comes in let's say the sentence has say three words there are three embeddings for that sentence each of those embeddings gets multiplied by say AK right so AK only needs to work needs to know how long is each embedding it doesn't need to know how many words do I have and that's a I'm glad you raised that question Ben because that's what makes a transformer's number of weights independent of the number of words in your sentence. It only depends on the vocabulary that you're going to work with because the vocabulary determines how many embeddings you need, how many embeddings you need. It the length only matters in terms of the positional embedding because if you have a thousand long sentence, you need a thousand long positional embedding matrix. But beyond that, it doesn't care. And that's why for example Google uh Gemini 1.5 Pro which is a million it can accommodate basically a million long million token context window right it can it's still very compute heavy but it does not change the number of parameters uh yeah >> conceptually which weights are optimized first but in sequential order or are they optimizing the weights at the very same time all >> simultaneously because if you think of back propagation ultimately you have a loss function right and you calculate the gradient of that loss function so if you have a say a billion parameters that gradient is basically a billion long vector right and we're going to take the gradient and we're going to do w new equals w old minus alpha times the gradient so all the w's are going to update instantaneously now the way it actually works in computation is you're going to do it the because of the back and back propagation it's going to start at the end and slowly flow backwards but when it's done everything will be updated. Yeah. >> We take uh two attention heads and we have the matrices of AK, A2 and AV in them. Uh why would the parameters of all three of them all the weights of the three matrices on this side and this side would be different because finally the things you're inputting from this side and the output is same. So the learning process should be ideally the same unlike like a CNN where we had put filters which were different. So what different thing we have to >> because the initialization is different. >> What do we mean? >> Like what I mean is if you have two heads right each head has three matrices. The starting values of those six matrix is different. >> Starting value of A aka B AQ and A is different for both the heads >> right? Much like for all the weights typically the values are randomly chosen. If they were all the same thing you're right. It won't you don't make a difference right? They will all change the same way. Yeah. U is the input of the transformer of the sentence or the the array of embedding of each word. >> Uh the in the transformer itself is expecting embeddings in and so what basically happens is that we get some sentence we run it through a tokenizer which connects it to a bunch of tokens which are just integers and then it goes through the embedding layer which maps the integers to these embeddings and then you feed it to the transformer. But when you do back propagation, it comes all the way back to the starting embedding layer and updates those weights. >> Okay. So they can be trainable. So the twist at the beginning must be input here, but they can train. >> They're trainable. Exactly. Exactly. >> Uh yeah. >> Are the attention heads solely parallel or can you have like a stack of attention heads? >> Typically they are parallelized. Um and because you can always stack the block itself to get more and more power. All right. So um so now to apply the transformer right there are common use cases are that you have a whole sentence that comes in and then you just want to classify it right the the canonical thing being hey movie sentiment classification boom positive or negative right classification another common one is labeling where every word gets labeled as a multiclass label and that's basically what we saw with our slot filling problem and then there is another thing called sequence generation where you give it a sequence you wanted to continue the sequence right generate more stuff i.e. large language models and all that good stuff. So, so this we know already know how to do because we actually literally built a collab with this with the transformer stack. Now the question is how can we do that right? How can you do basic classification with these things? So now if you again when you send a sentence in after all that stuff is done and when I say encoder here I'm assuming that you may have one one block you may have 106 blocks I don't care at the end of the day you send something in you get a bunch of contextual embeddings out right so at this point we need to take these contextual embeddings and somehow make it work for classification for just classifying something into yes or no positive or negative so it'll be nice if we can actually take all these embeddings and like essentially summarize them into a single embedding, a single vector because if you have a single vector then we can run it through maybe a relu and then we do a sigmoid and boom we can do a you know a binary classification problem super easy right so this begs the question okay how are we going to go from the all the many blue things to one green thing okay now of course um what we can do is we can simply average them we can take each of the embeddings just simply average them element by element, you'll get a nice green thing. Okay. Um any shortcomings from doing that? >> You would lose the ordering of the words. >> You do uh well in some sense the positional embedding, the positional encoding you have in the input does have this notion of position, right? So you're not necessarily losing the order necessarily, but you're sort of averaging all this information into something and averaging is going to lose some richness. Okay. >> I think it's going to be skewed to the one that has like the biggest number, right? So something is influencing your >> Yeah, the biggest ones are going to dominate. But hopefully we won't have too much of that because all the layer nom business at the beginning has hopefully made sure the numbers are all in a reasonably small and well behaved range. But the the point really is that you're going to lose richness in the information because you're just like mushing it down. So there's a much better and more elegant way to do this which is that what you do is for every sentence when you train it you add an artificial token called the class token. Okay, literally it's an artificial token and it's designated as you know CLS in the literature and then this token is getting trained with everything else. Okay. And so once you once you finish training that token has its own embedding too. And because it has been trained with everything else and this token is remember it's a contextual embedding which means that it's very much aware of all the other words in the sentence. So in some sense this context this CLS tokens contextual embedding sort of captures everything that's going on about that sentence right and so what we do is once we are done training we just grab this thing alone and then send that through a relu and a sigmoid and boom you're done. So this is a very clever trick to somehow you know instead of averaging everything at the end let's just have something just for the whole thing the sentence and just learn it anyway along with everything else. So in like a meta principle in deep learning is that whenever you think you're making an ad hoc decision about something like averaging a bunch of stuff you should always stop and say is there a better way to do it where it doesn't have to be ad hoc where the right way is learnable from the data directly using back propagation. Um there was a hand. Yeah. >> Is there a reason that you added the CLS at the start? Why not add it at the >> You can do it at the end. Is there any difference? >> Um the only thing to remember is that um it's a good question. So different centers are going to be of different length, right? So there might be short sentences, there might be long sentences. In particular, the lot the short sentences are going to get padded, right? I remember I talked about padding to make it to fit to one length. So what internally the transformer will do is ignore all the padded tokens because it doesn't do it's just padding doesn't really matter for anything. So if you have the serless at the very end we have to have much more administrative bookkeeping to take everything but the last one ignore it and only do the last one just much easier just to get in the beginning that's the reason. Yeah. >> What would be just a practical application of this would be something like sentiment analysis like a positive or negative. >> Yeah. So basically any kind of text comes in and you want to figure out some labeling problem like a classification problem. The easiest example I could think of was sentiment. But you can imagine for example an email comes into a like a call center operation and you want to take the email and automatically figure out which department should I send it to. Okay. So now now if the input data for a task is natural language text, right? We don't have to restrict ourselves to only the input training data we have. Right? Would it be great to learn from all the text that's out there? So, for example, to go back to that call center thing I just mentioned, you know, why clearly, let's say it's coming in English, the ability to take that English email and route it to one of 10 things. You know, you should have to learn English just for your call center application. You should learn English generally and use it for other things, right? So, why can't we just learn from all the text that's out there? And so, that brings us to something called self-supervised learning. And the idea of sens supervised learning is this. So if you recall the transfer learning example from lecture four right where we had restnet right and we took restn net we chopped off the final thing we make made it sort of headless and then we attached that output of the headless restn net to a little hidden layer and output and we did the handbags and shoes and you will recall that we were able to build a very good classifier for handbags and shoes with just like a 100 examples. Right? So the question is why was this so effective? Why was this so effective? And turns out the reason why any of this stuff actually works is because neural networks or they learn representations automatically when you train them. So what I mean by that is when you imagine a network, you feed in a bunch of stuff, it goes through all the layers, it comes out. Uh you can think of each layer as transforming the raw input in some different alternate representation of the input. Okay? And so and these are called representations. That's actually a technical term. Um, and so you can from this perspective when you train a a neural network, a deep network with lots of layers, what you're really learning is you're learning a way to you're learning how to represent the input in many different ways. Each of these arrows is a different way of representing things. Plus, you're learning a final regression model, either a linear regression model or a logistic regression model. Fundamentally, that's what's going on. Because the final layers tend to be sigmoid, soft max, or just linear, right? So the final layer if you just look at the this part alone whatever is coming in it's just going through essentially a linear regression model or a logistic regression model that's it. So fundamentally you're learning representations and a final little model. Okay. But the reason why all these things work so much better than logistic regression is because those representations have learned all kinds of useful things about the input data. They have sort of automatically feature engineered for you. So, so from this perspective you can imagine that each layer here is like an encoder. It encodes the input, right? The first layer encodes it. The first two layers encode something. The first three layers encode something and so on and so forth. So a deep network contains many encoders. And so the question is what do these representations actually embody right? What do they capture? Is it like specific knowledge about the particular problem that you train the thing train the network on or is it like general knowledge about the input data because if it is general knowledge about the input we can use it to solve other problems unrelated problems. So is it specific knowledge or general knowledge and it turns out they actually capture a lot of general knowledge about the input and that's why you can get reuse out of them you can reuse them for other unrelated things because they have captured general stuff. So if you look at this, I think I've shown you before, right? If you if you lo