Long-Short Transformer

Key Takeaways

this video will provide a quick overview of the latest and efficient transformers the long shore transformer efficient transformers for language and vision the motivation behind this is we want to do this attention operation over longer input sequences most of these transformers for natural language processing can only take as input 512 tokens and we want to expand them to maybe take in an entire scientific paper as input or maybe an entire legal document or the entire pixel grid of images as input so we want to be able to attend over more than 512 tokens and this is problematic because the n squared computation of the current design of the attention layer so people have come up with these different uh designs strided attention like the sparse transformers or you have say some kind of either local window or some kind of alternating pattern of masking out the spatial grid of the query and value matrices we have low rank projections like how singular value decomposition can decompose say the query key value matrices into the most salient diagonal row to compress it and have that more uh information packed multiplication or we have the low rank projections like this paper where you have a parameterization of it to compress the project the query key value matrices into a vector space something like that and then we have recurrence like transformer xl or compressive transformer where you add the uh hidden state idea to attend over say the last 512 tokens and compress that into the hidden state at t and then the next uh 512 tokens hidden state t plus one attending back over the hidden state of t so these are some of these ideas of how we can attend over longer than 512 tokens to take in longer inputs for transformers so in attention we have these parameterizations that blow up the input sequence into query key and value matrices and then those matrix matrices can multiply together to do the processing of the attention computation so we might be able to take these uh query matrices at the key or the value matrices and decompose them into the most salient row and in this technique like singular value decomposition where you can compress it into this diagonal and then just maybe multiply the diagonals by each other and some idea like that and there's also this idea of strided sparse attention this is the sparse transformers paper from open ai where you have this stride so you don't intend over the entire this is the autoregressive task where you mask out the future inputs and then only a 10 behind you this would be a way where you either set this local window so you only look at say the last five sequentially looping it or you go up with how you're indexing the spatial grid of the value matrix projection and then other ideas like having a sparser pattern so it isn't contiguous like this and and so these other ways of designing this sparse attention this local window to apply the attention on rather than doing the full matrix multiplication so the idea behind this new long short term attention in this paper is to combine the outputs from a short term strided attention layer with a dynamic low rank projection matrix transformation so the strata detention only tends over this filtered kernel rather than the entire value key multiplication it has these local windows to attend over and then the dynamic projection is where you take the key matrix projection compress it with a weight matrix that says say takes a m by n matrix into n by k where k is smaller than m or n by n into n by k and that kind of idea so then you transpose it and then you multiply it by the original uh value or something but you compress it with this parametric uh compression operation so it's kind of like this down sampling uh like when you have a strided convolution and you desample the layers so say it goes from 32 by 32 down to 30 and 30 these kinds of ideas of compressing the representation with a weighted matrix multiplication so then here's the next big idea is introducing these dual layer normalization strategy in order to combine these two different outputs so in the paper they describe how these the outputs from the short-term retention and the low-rank projection attention have different scales so that it's hard to just concatenate them like this and not have this misalignment of the overall uh like mean and variance parameters say if it's normally distributed but generally like the scale of these parameters are going to be too different from the short term and long term retention so you have to apply these special layer normalizations into unifying the scale of these features for further computation into stacking this together and making a gigantic transformer out of it the authors evaluate this efficient transformer on the long range arena benchmark and these are some examples of different models that have been evaluated on this benchmark the reformer architecture linformer the long form which has a similar idea of combining sparse and low rank projection attention and this other idea big bird and so on these different models for attending over longer sequences so these are different tasks like long list ops byte level text classification where instead of tokenizing the text input you leave it at the byte level and therefore you have this really long input without the tokenization other ideas like image classification on a sequence of pixels and then these uh pathfinder tasks where you have this grid and you have to reason about i think something about how to navigate out of a maze or some idea like that so we see a result on using the transformer the longshore transformer performing better than these other models like reformer and on these uh long range arena tests as well as language modeling benchmarks of this end wiki data set and then showing the parameter count with the transformer ls and then the complexity that it achieves and also the results with image classification and this is a pretty interesting result achieving an 84.4 percent image and accuracy compared to these other designs and it's not really fair to compare this to say that like the true imageness state-of-the-art because that's using like a 2 billion parameter vision transformer whereas this is 39.9 million parameters in a very interesting different direction for the vision transformers and making them efficient rather than just scaling up the uh this style of vision transformer uh in the original where you chunk it into patches and that kind of idea so a really interesting result overall i hope from this video you're able to get a quick sense of this idea of having strided attention which they reason has short-term attention because this strided attention local window attention compared with this dynamic projection long-term retention and then combining this through the use of this layer normalization and overall just the state of this efficient transformer design continues to advance and it's a very exciting area of research being able to attend over longer inputs would allow more applications so thanks for watching and please stay tuned for the rest of the ai weekly update series