Multi-Head Attention Tensor Shapes

A sequence of tokens goes into multi-head attention as a [T, d_model] block and comes back out exactly the same shape, and almost everyone who gets stuck on attention is really stuck on what happens to the dimensions in between. This video is a tour of those shapes. Each token is projected into a query, a key, and a value, and the matmul rule [a,b] @ [b,c] = [a,c] tells you exactly what comes out: X[T, d_model] @ W_Q[d_model, d_head] = Q[T, d_head]. The scores Q@Kᵀ are [T, T], and the d_head axis you measured similarity in is consumed by the dot product, so it never appears in the score matrix. That vanishing dimension is the single most confused point in attention. Then we add heads. d_model = n_heads × d_head (512 = 8 × 64) just partitions the width: one fat W_Q[512, 512] projects every head at once, a reshape splits the 512 into (n_heads, d_head), a transpose slides the head axis next to the batch axis, and from there matmul treats heads as nothing more than another batch dimension, so there is no head loop in the code. The payoff is the part people get wrong in both directions: that fat W_Q is one shared tensor, but it holds eight distinct learned blocks side by side, so the heads run through one matrix yet still learn different projections. Not eight separate matrices, and not one matrix making identical heads, but one tensor, eight different blocks. *Related Videos* ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Self-Attention - Explained: https://youtu.be/chSVRI_tQXk Multi-Head Attention (MHA), Multi-Query Attention (MQA), Grouped-Query Attention (GQA): https://youtu.be/efg-qgQ5t88 Multi-Head Latent Attention (MLA) - Explained: https://youtu.be/DWBKSbj8CqA Attention Mechanism Variations: https://youtu.be/LremmIyKr0w Sliding Window Attention: https://youtu.be/_8Vcui3DSbQ Softmax function - Explained: https://youtu.be/oJU6-qW6xZU Word2Vec — How Words Became Vectors: https://youtu.be/9Rd6sh-PmlY *Contents* ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 00:00 - Same shape in, same shape out 00:34 - Q, K