Transformers | Basics of Transformers I/O

Key Takeaways

we also have the inputs and the outputs so all the inputs that go in either the encoder or the decoder are embedded embeddings are a way to represent these words in a n length vector and on top of these word embeddings we are adding positional encodings transformers do not have any recurrence so the model has no way of understanding which word comes first and the other one comes second or which word comes where in the sentence so by adding a positional encoding you are adding some information with each word that tells the modal where this word in the sentence comes in and lastly for the output we have a linear layer and a softmax layer at the end of the decoders so the output of the decoders can be transformed into something that we can understand and basically what they turn into is a vector that has the length of the amount of words that we have in our vocabulary and each of these cells tells us how likely it is that this word in this cell is going to be the next word in our sequence