Self-Attention with Relative Position Representations – Paper explained

Key Takeaways

hello there it's finally here the long promised video explaining relative positional embeddings as they were introduced in prehistoric times by our ancestors in machine learning so if you're not yet sick of positional embeddings in Transformers and want more of them you are spending your coffee break with the right video because well we did a video about positional embeddings as introduced by the attention is all you need pay paper then we had a video about concatenating versus adding positional encodings and there we also discussed learned positional embeddings M Coffee Bean is now literally 50% caffeine and 50% positional information let's begin the attention is only un need paper introduces positional embeddings that encode the absolute position they encode the exact order of tokens such that the Transformer is informed about the Segal nature of data otherwise the Transformer that is processing everything in parallel would be invariant to order meaning that without positional encodings its output would not change after complete sequence reordering but what if we are dealing with other kinds of data where it is not about absolute position but about relative positions think about a graph for example it would be quite arbitrary to say that a certain node is the first one one what would make this node the first node and this the second that would be unmotivated for many problems and in some cases even misleading then let's forget about absolute order and move to relative positional encodings that are about the distances between elements in either a graph or a sequence which is in fact a degenerate graph a chain a subset of the authors of the attention is all un need paper also have a follow-up paper doing exactly this they introduce relative positional encodings where not the order but the relation or better said the relative position the distance between tokens is important so how does this work if you do not know anything about positional embeddings first go and watch our two videos about them and then come back don't worry they're short the idea of relative embeddings is moving away from the classical position embedding where each token has its own positional embedding with relative representations each word or token does not have only one positional embedding but as many positional embeddings as there are tokens in the sequence in order to describe the relationship between them this is because relative positional representations do not encode absolute Order anymore but a positional relationship in which each token stands to the other tokens so let's take an example sequence of five tokens in classical positional embeddings each token let's say X4 here has one encoding informing the Transformer about the position of X4 now to best visualize relative representations let's copy the sequence again like this in the relative variant each token has five positional embeddings one embedding for describing its relative position to its itself then four others for the rest of the sequence and to keep the same notation as the original paper the positional embedding describing the relationship of one token with itself is a vector w0 s0 is the distance between the token and itself then W1 and W2 as we move to the right and wus one W-2 and so on as we move to the left and the vector wi takes the same value independently of what tokens we are currently looking at because W wherever we are says that we are at zero hops away in the graph W1 says that we are one hop away to the left in a sequence or in a directed graph in general we go one hop following outgoing edges so you see this approach is not made only for sequences but for graphs too okay great for a sequence of length five positional embeddings should range from wus 4 to W4 so we have in total nine positional embeddings to either handcraft or learn for this sequence these W vectors can be written one under the other in a table like this notice that these vectors come from the relationship or distance between one token and the other so we can use a pairwise notation for this the authors also experiment with clipping at k meaning that after a certain distance positional embeddings get the same value so for k = 2 W3 and W4 all take the value of W2 okay but now for each token we are stuck with as many positional embeddings as the sequence has tokens what to do with all of these add them all up that wouldn't be a great idea addition is complicating things even with sinusoidal embeddings where we have just one positional encoding for each token because we could mix up the semantic and positional information with five positional embeddings per token the mixing up problem would be only five times larger H if only we had a mechanism that given a token computes new representations for that token in relationship to all other tokens well we do have this it's the self attention mechanism a clever idea would be to modify the self attention formulas to capture the relative positional representations too we remember that through self attention each token gets a new representation Z which is passed further in the Transformer module each token Vector is first transformed by a linear transformation then the new representation for each token is a weighted sum over all tokens where the weights are sort of an important score so fellow tokens that matter more are weighted more so here where the new representations are computed we can add the positional information now the token representation after the linear transformation is further shifted in the high dimensional space this means that each token XJ gets pushed into a Subspace saying that look I carry this semantic information but I'm also your second neighbor to your right because my position in some Dimension is similar to all other second neighbors okay great now the new representation Z is informed about the relative position but the self attention weight coefficients are not therefore the weight coefficients also receive their own positional push such that the important scorers make position informed decisions too so what do we have in the end now every token has a many relative embeddings as there are tokens in the sequence in two variants one relative embedding for the values and one for the keys to inform the attention weights and the exact values of the vectors are learned which kind of makes sense to let the model figure out for itself what the best balance is it would be quite difficult to handcraft all of these and what did this whole thing help with well this paper has only experimented with text which is a sequence obviously and they gain some performance in machine translation but word chains are only a flat graph and Beyond this the method can be applied to graph representations in general or anywhere where you have pairwise relationships between your elements because remember these relative representations depend only on how far tokens are from one another independently of whether we are in a sequence or in a more complex graph but miss Coffee Bean why didn't this paper try this out immediately well it's called marking your territory but there are other papers that implemented this successfully for graphs too anyway another upside of relative positional representations is that especially with clipping at K this learned positional information generalizes to any sequence length which is also a feature of the handcrafted sinusoidal embeddings but not NE necessarily of any learned positional embedding clipping at K seems a little odd because with small K it takes away the information of long-term dependencies now we can only store that something is close by or far away but we don't know how far away that is but this is similar with graph neuron networks where the best results are delivered with a small number of iterations many iterations allow for information from further away to diffuse used to your point of interest but multiple hops are not always beneficial as it can aggregate noisy signals from far away so clipping at K in relative positional representations acts a little like a relative position Event Horizon which does not hurt tasks which mostly rely on local dependencies well we hope that these relative embeddings and the associated paper are now more digestible to you thanks for staying until the end of this explanation and if you're still here do not forget to leave a like And subscribe you know to help with the YouTube algorithm okay bye [Music]