Pytorch Seq2Seq with Attention for Machine Translation

Key Takeaways

what's going on guys hope you're doing awesome and welcome back for another video in the last video we implemented a sequence the sequence model on the task of translation and in this video I want to improve on that model by introducing this idea of attention so even before we get to actually implementing this let's try to first convey the intuition behind this important idea so this is the figure I used in my last video well you know we have our encoder and it produced this context vector and we had this decoder which takes his input in this context vector well one thing we can imagine is that if we have very long sentences then you know the this context vector is going to become a bottleneck in that you know if we're if we were very far along from there from the beginning of the decoder from the you know where the context vector is sent in then we might not remember what the context vector actually said so this figure right here is sort of conveying intuition of what we're gonna do instead of just having a context vector be sent as input to the decoder we're going to have a context vector which is sent to every cell of the of the RNN all right I don't want to spend too much time on this figure because I don't think it's very good but so what we yeah so this is I want to show this one right here first so we have the blue score on the y-axis and we have the sentence length as our x-axis and so using these you know the Sigma 6 models we implemented in the last video the problem is that it loses you know the the blue score if the sentence lengths are very long which resembles this idea that it's losing attention it's losing the its it's not able to memorize what the what the source sentence actually said but implementing this with attention we can see that even though we have very long sentence lengths we can still maintain a high blue score and so I want to go up here and try to you know explain how this works and you know I think most of this is going to be clear as well in in you know when we actually start coding it but it's good to have a first idea and I'm also going to link a couple resources that I think are great if you want you know more in-depth exploration so I'm going to explain from this figure from the paper and I think a lot of so you know we have this encoder right here at the bottom which is going to be a bi-directional RNN you know so it has hidden states going from the from the left and it has hidden states going to the right so and then we have our encoder right here at the top which is trying to predict the you know the target words Y T for a specific time step T so you know let's say that we're translating a sentence and we are translating the last word of that sentence then it probably makes sense that we want you know to to you know keep in our memory or you know try to remember what the source sentence said on the last few words so what we're gonna do when we send the context vector to this you know the Latin at this last time step then we're going to multiply these hidden states from the source sentence with this attention score values right here these scalar values and so these attention score values are going to be elementwise multiplied with the hidden state and they will signify you know how much are we going to pay attention to this hidden state from the source sentence so let's say that you know we're translating the last word then we will probably want to pay most attention to you know one of these last couple of words so let's say this this this last attention score will probably have a very high value so you know we element-wise multiply this attention scores with the hidden state from the source sentence and these attention scores are specific for the this time step T so we're doing this for every I'm step and the sum of all of those attention score multiplied with the hidden state is going to be summed up and that's going to be a context vector and that is going to be sent in to the a this step in our decoder all right hopefully that was clear but you know you might have one question still which is you know how do we compute these attention scores and so from the paper here what we do first is that we compute this energy state from a function a which takes as input si minus one which is the previous hidden state from our decoder you know when we're actually trying to translate and then the hidden state from the encoder so this a function is going to be a a small neural network probably like a like a one layered neural network for example and what this is trying to learn is which hidden state should we actually pay attention to so you know I said that if we're at the last sentence a last word in a sentence we would probably pay more attention to the to the you know the last words but that's just my intuition you know we're going to have the their network actually learned this from from you know a small neural network by itself and then we're going to take this energy state and we're gonna run it through a softmax and this is going to be the output from that softmax is going to be these attention score values so the great thing about you know running it through our self Mac's is that it's not going to be normalized so the sum of attention is going to you know add up to one so essentially if a if a attention score is you know 0.8 then that means that we're paying 80% attention to that hidden state and then lastly to produce a context vector as I said before we're going to element-wise multiplied our attention scores with our hidden state and so yeah I'm going to link you you know more in depth explanation in the description and I think a lot of this is going to be become more clear when we actually started coding it so let's go and try to code this now so what I have here is the code from the last video so I don't want to I don't know how many lines this is but it's like 225 lines that we wrote in the last video and I don't want to you know repeat all of this again since we're gonna just build on this code so if you want to know how to build a sequence sequence model watch my last video and in this video we're gonna expand on that video and and add this you know this attention part alright so we're gonna start in the in the encoder and what we gotta do is we gotta add this bi-directional so we're gonna do bi-directional equals true and now what's going to happen is that this hidden and this cell is going to have you know one extra dimensions it's going to have one going forward and one going backward and it wasn't super clear to me in the paper if you know what they did I think they just used one of them so the the one going forward for example they just took that one and then send that to the decoder which is not going to be bi-directional so you know to have the shapes add up you know what I'm what I did is that I added two fully connected layers and essentially you know we have our hidden size but since we have bi-directional we're going to have two times that and the decoder is just going to take you know hated size times one so I was thinking we can do a linear layer which just takes these two hidden states from forward and from backwards and send them through a linear layer and the this you know this linear layer can itself choose the network can choose what's important for actually you know if the forward or the backward and choose the most relevant part so instead of limiting the information to just forward or just backward we we sent it through a linear layer all right hopefully that was clear and then we're gonna do the same for the cell so we're gonna do a nonlinear and it's also going to be hidden size times two and hidden size and so you know all we got to do then is when we send it through the RN n right here is that we gotta compute this new hidden as self dot fully connected hidden of torture concatenate and we're gonna add hidden the first the first hidden which is for the forward one and I'm going to do zero to one just to keep the dimensions and so we're gonna do zero to one and then he then one to two all right this will be the hidden states for one going forward this will be the hidden stage for the one going backwards and then we're gonna do dimension equals two because we want to add right so this will be I think the shape of this is going to be 2 comma and comma hidden size like this and so you know when we concatenate them it's going to be 2 times the it in size so we just concatenate that dimension and send that through the fully connected layer and then we're going to do the same for the cell state and again you don't have to do it this way it was a little bit unclear to me just how the paper did it but this definitely works you can also just take two forward one or the backward one and I guess come check and see which one does the best but so we're gonna do this for the cell state as well you know so in this way we're not losing any information necessarily we just let the network choose by itself which one it things is best and then so we're gonna change this to the encoder States and so we're going to output this encoder state because you know we want the hidden States for each time step right and this hidden right here is just outputting they aren't like it's only outputting the rightmost hidden one it doesn't have an hidden state for each time step that that's what encoder state has but we need to send in you know all three to our decoder and so this one is let's see in the paper so the encoder states are these H J right here since they are you know specific for the time step as we see here we have an hidden state for each time step and that's exactly what these encoder states right here are going to be alright so what we then want to add in our decoder we're gonna have our embedding we're gonna have our RN n but the RN n now is going to take in hidden size times two plus the embedding size so why this makes sense is because we have the encoder States which has a forward one and a backward one and since this is you know both of them are included in the first hidden locate a specific hidden state from our encoder then it's hidden size times two and we're also going to send in the embedding size so we're just gonna so this will be from the contacts vector from the encoder from the encoder and this is just going to be as normal we're just gonna send the embedding as we did in the previous video so then what we don't want to do is we want to add a let's see here so we want to add self dot energy and we're gonna do this you know what I mentioned previously a small neural network so we're going to do hidden size times 3 comma 1 because here we're going to first add the hidden States from the from the encoder but we're also going to take the hidden hidden from you know our previous step in our decoder so you know in the paper they said si minus 1 and then the h JS so that's why it's times 3 right here and then we're gonna do our yeah so we already have the fully connected we're just gonna define our our softmax and it's going to be ended up softmax of the dimension 0 so this is gonna make sense exactly whites temperatures year later and then we're gonna do septic relu is n n dot rel loop and so what we got to do now is we gotta first you know compute the a goal is compute these attention scores right but what we got to do first is we got to compute the energy state so the compute energy states we want to send in the hidden states from our encoder and also you know they hidden from the decoder and well this is a bit confusing maybe but hidden here is from the decoder and then what we're gonna have is we're gonna have another which we're gonna call encoder States just it cannot confuse the hidden from the encoder and decoder this is from the encoder and this is from the decoder we want to add them together concatenate them and then send them through this energy layer right here and what we got to do first is we gotta do can first get the sequence length which is the encoder state shape 0 and then we're gonna do a reshape of this hidden state so we're gonna do hidden the repeat sequence length comma 1 comma 1 so essentially so we have this encoder state which is for all of our hidden hidden time steps but to add them together we need to have this you know have the same dimension along that axis so we need to you know repeat that you know the hidden state from our decoder sequence length times to be able to concatenate them together so when we have this we're going to compute energy by self dot relu of self dot energy torch concatenate and we're going to send in this H reshaped and the encoder States and then we're gonna add them a long dimension to essentially it's going to be then you know hidden size times three so you know I guess this one has a hidden side size times two because it's bi-directional and this one just has hidden size and working captain aiding them and we're sending them through this energy layer and then we have these energy right here then we're going to you know compute this attention values by doing the the softmax of the energy and I think so attention here will be shaped sequence length comma and comma one and that's why we also have the dimension equals zero which is you know we're gonna normalize through the sequence length essentially so that the some of the attention scores will be one for a specific batch because we have we have you know a batch size of n sending you know same thing and simultaneously so then what we want to do so our goal right now is to elementwise multiply the attention with the encoder states but to do that we gotta first do a little bit of a reshape so what we're gonna use is we're gonna use torch that'd be mm you know if we look at the encoder states then they're going to be n comma or let's see essentially the encoder states are going to be you know three-dimensional tensors so to be able to multiply in the way that we want which I'm going to explain how we do just a second but that's why we used towards that Bamm just for these three dimensional to do the matrix multiplication essentially so we're going to do attention equals attention dot permute one zero two and then so what we have after this is we have n comma 1 comma sequence length all right so we just swapped the order and we're going to do the same thing for the encoder states so we go to encoder state start permute 102 and what we have now is n comma sequence length how am i hidden sighs the times - all right so you can look on the torch that bmm documentation but essentially you know if we have them in this shape right here then that if we do now torch that bmm on attention encoder state this is going to do the matrix multiplication that we want and this is going to be our context vector and we're going to also do dot permute 102 right here so let's see if we multiply so if we multiply this with this we're going to obtain n comma 1 comma hidden size times 2 right so that's just what it says in the documentation if we do this multiplied with this using torch PMM we're gonna get this as our output size so we want 1 comma n comma hidden size times 2 and essentially what we're gonna add then is this dot permute 102 so that's what what it's going to do so then now that we have our context vector and remember this is for a specific time step in our decoder since we we're decoding one word at a time so we're gonna do the RN and input is going to be torched catenate of the context vector and also the embedding and we're going to do it along the dimension to the second or I guess the third dimension of the tensor so along the hidden size so that we obtain this hidden science size times three since the embedding is going to have hidden sighs context vector hidden size times two and then we want to send this through our RN n and everything here looks alright so everything else looks fine and so try to think if there's something I'm missed here so just a quick recap what we did we want to you know multiply we want to concatenate this hidden states from our decoder with our encoder States but to be able to do that we got a repeat the hidden States along each time step which was in from the encoder and when we do that we repeated sequence length times we then can concatenate these two and we can send them through this small neural network like you know one layer Network and to compute this energy state then we just run it through a soft Max and we get this attention scores then we got to do some you know shape reshaping so that we can use this torch PMM essentially so we can multiply this attention scores with all of the encoder states and we want to do that for all of our training examples so that that's why we do it in this way and then we have our in an input which is then essentially just a concatenation of the contacts vector and the embedding we send this through the RNN we get our output with our predictions as usual and then we get you know predictions and we also get a send out the hidden and the cell then let's see so in our in our sequence sequence we have our output right here we have a hidden we have our cell and one thing we gotta send in now is the encoder States to our decoder and so we have to send this out from our encoder as well encoder States and that was sent out right yet encoder States and the hidden and the cell so now that we have our encoder state and we're going to send that in to our decoder at every time step right at the first time step we're also going to send in the hidden and the cell but you know when we predict one word this is going to be overwritten by the by the new hidden in the cell from the decoder but the encoder states right here is always going to be sent in so it's always going to have this context in the way that we saw in you know in the beginning of the video and so I think that's that's it see if there's anything we want to change here so I actually think we don't have to do any other changes so you know we can actually little we can try to run this first and we're probably gonna get some errors but let's see yeah so I one thing I didn't mention is that we're doing this for number of layers equals one I have done an implementation where you can you know choose the number of layers but I found number of layers to equal one to be best anyways and I thought it would be you know simpler for the video because it would take quite a lot of time otherwise but I'm going to have it in the description if you want to check if you can expand it to several layers but we're going to have number of layers equals one and then we got to do also is we got to remove the drop out from the lsdm since it won't work if we only have one layer and then it's also giving us another error and then the drop out of a drop out and I think that's because we send in drop out here but the parameter is P so if we run this now it should probably work oh yeah so it won't work actually we need to tell us remove the drop out here and now it might work yeah so one more error is that here we had the sequence length comma n comma one if we want they want to be on this position which we do want then we need to do one - oh so I just wrote the wrong thing here and let's now try to rerun it again and hopefully it works okay so we can see that it's running and it it seems to work what we can do now is that I have already trained a model on this so we can do load model equals true and let's do this blue score right here I'm gonna take it up here and we can do blue score on the entire data and we can do just exit after this so this is gonna take some while I'm gonna run this blue score and we'll see what we get so just for some context in the last video we got just below 21 in blues core so you know when we when we have now added this attention mechanism let's see what we get so we can see that training with this added attention mechanisms so substantially improves the performance and then now we obtain a blue score of about twenty seven point five which is a you know a lot better than twenty one so yeah thank you for watching this video if you have any questions leave them in the comment below and I'll have a link to the github where you can read the code yeah hope to see in the next video [Music]