Multi Head Attention Code for Transformer Neural Networks

Key Takeaways

how has multi-headed attention coded out in Transformers in the encoder every word Vector is encoded into three vectors each a query key and value vector in code this is equivalent to passing the input word embedding through a feed forward layer that has the qkv vectors stack on top of each other the green comments are the shapes of the tensors in each step we then reshape the qkv vector to split the embedding Dimension into eight heads and the rest of the embeddings so the multiple heads act like another batch Dimension we then for every word extract the query key and value vectors perform attention and get this output tensor that consists of words that better encapsulates the context of a word