Coding Position Encoding in Transformer Neural Networks

Key Takeaways

this is the code for positional encoding in Transformer neural networks Max sequence length is the maximum number of words in a sentence D model is the embedding Dimension length we create a vector of even numbers from 0 to D model for every one of these elements we will take 10 000 to the power of each element divided by the embedding Dimension length for every word in the sentence we will write out the integer position in a vector of number of words in the sentence cross one we then compute the even dimensions and the odd dimensions of the positional encoding using the parameters that we just created we'll stack them all together in order to get the final position and coding of the entire input sequence