What exactly is multi head attention?

Key Takeaways

What exactly is multi-headed attention? Let's try to understand in a simple way. Quick recap first. In attention, every word becomes three vectors. Watch one word split into a query, a key, and a value. The query asks, the key answers, the value carries meaning. Now, take a five-word sentence. One head scores every word against every other. That gives this grid. Brighter cells, stronger links. One head, one grid, one way of relating words. But, here is the thing. Words relate in many ways at once. Grammar is one. Meaning is another. Watch the same grid light up two totally different ways. One head learns only one. So, here is the fix. Slice the big embedding into pieces. See the matrix split into colored bands. Each band becomes its own little head with its own query, key, and value weights. Now, all the heads run at once in parallel. Three mini grids side by side. Each lights up differently. One tracks grammar. One tracks meaning. One tracks position. Then, glue it back together. Each head's output stacks side by side into one long vector. And one final weight matrix mixes all those views into a single answer. And that is the whole formula. Take the heads, stack them together, multiply by the output matrix. That one line is multi-headed attention. So, that is the trick. One head sees one pattern. Many small heads see many, all in parallel. Stack them, mix them, and the model reads a sentence from several angles at once. If attention finally makes sense, hit subscribe. A lot more is coming.