Do Transformers process sequences of FIXED or of VARIABLE length? | #AICoffeeBreakQuiz

Key Takeaways

hi there we had this question on our ai coffee break quiz recently and we must discuss its answer here because we never had such a contested question before on our quiz so if you do not know what i'm talking about here follow miss coffee bean to the community tab of this channel because we have quite the set of machine learning related quiz questions and the new question is coming each day but let's get to our question about the sequence length of the transformer it's not a simple question not gonna lie the answer depends a bit on how one reads it christian garcia noted very well that transformers do not process just sequences but something more general a sequence without order which is a set the sequential nature of the data becomes clear to the transformer only because we add positional embeddings to encode the order in which the data comes in if it is the case for more about this check out some of our previous videos but supposing we are processing a sequence with a given model for example sequence of words does the sequence always have to have the same length or can this length vary and it's a valid question because there are architectures like convolutional architectures with a fully connected layer in the end where the input must have always the same size if the image is not the right size it gets resized during pre-processing so we want to answer this question for the transformer in two ways first by looking into the code and then theoretically because well one just does not need to play around a lot with a transformer from hugging phase to notice that inputs which have different lengths are possible so yes the things we want to process with a transformer can have variable lengths the tokenizer just gives us more token ids as output and an attention mask full of ones but with more ones as the sequence grows bigger but because of the quadratically scaling attention in compute we also notice parameters in the transformers library limiting the sequence length which has the effect of truncating everything which exceeds that length to keep memory and computation in bounds looking at the bird tokenizer documentation we also see the pad token and its explanation by padding the sequence to a maximum length we can produce inputs of fixed length for the transformer even when the sequence is shorter than the maximum length our input length has always the same size because the sequence is padded with the pad token and the attention mask is padded with zeros and all this trouble is for efficiency reasons in the implementation when we want to train or do inference on more than just one sample at once because in that case tensors in a batch should have the same size such that the gpu can handle this efficiently therefore the answer to this question of whether transformers process sequences of a predefined length or a variable length is both yes and no yes sequence length can vary as in yes the transformer can process any length but also no because efficient implementations fix the length to a maximum length anyway but in principle and i think that it is the most important thing about transformers if everything with positional encodings and training works well for every sequence length there's no reason for why transformers should be bound to a specific set size or sequence length because the feed forward layers we see here are the same no matter the position of the token we are looking at so even if we add here one more token at the end this does not care as we take the same feed forward module and its weights and apply them on the newly added tokens and the attention is also not caring about the sequence length as it just computes a pairwise score given a vector and another vectors so it just looks at vector pairs at a time therefore miss coffee bean would rather say that transformers process sequences of variable length in principle okay bye [Music] [Music] you