Cross Attention vs Self Attention

Key Takeaways

what is cross-attention versus self-attention let's say we want to perform translation from English to a language called kannada that looks like this during attention each word is encoded as a query key and value Vector query vectors encode what am I looking for key Vector encodes what can I offer valuing codes what I actually offer during attention during self-attention each English word is converted into a query key and value vector and these are used to create vectors that better understand context during cross-attention every English word is converted into a key and value Vector as before but every Canada word is converted to a query Vector the final vectors will encode the countertop words taking into account the English words