Build your own GPT project (TypeScript) · Chapter 7 of 12

Multi-head attention

By the end of this chapter you can cut a token's numbers into several smaller heads, run attention in all of them at once with one call to chapter 6's function, join them back, and read what the newest token looks at in each head. That is the attention layer ChotaGPT really uses. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Attention
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 6's head gives every token one mix of the tokens before it: one softmax row, one set of weights adding up to 1. But a token in Datta's notes needs several things at once. To write the price after "Kal ka show (Fri 6 Nov), Kala Talkies: Ghar Wapsi Express, 9:30 pm. Balcony Rs ", ChotaGPT wants to know:

  • which venue this note is about (many tokens back);
  • which seat class came just before ("Balcony" or "Front");
  • that this is a show note at all ("Kal ka show" at the start).

One softmax row can put weight on all three, but then it blends them into one average and the separate facts are lost. Put most of the weight on one, and the others are missed.

Nikhil's answer is what every GPT does: run several smaller attentions side by side, each with its own weights, so each one can look for something different. ChotaGPT's final model has a width of 48 numbers per token, cut into 4 heads of 12. GPT-2 small has 768 numbers cut into 12 heads of 64.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    multihead.ts: split into heads and merge back

    In src/gpt/multihead.ts write splitHeads(x, nHead): [B, T, C] to [B, nHead, T, C / nHead]: reshape to [B, T, nHead, D] first, then permute([0, 2, 1, 3]) to swap the T and head dimensions. Then mergeHeads(x): [B, H, T, D] back to [B, T, H * D]. mergeHeads(splitHeads(x, h)) gives x back.

    The Check splits a [2, 3, 8] tensor numbered 0 to 47 into 4 heads, checks where position 2's numbers went, merges back, then (once all three tasks run) calls POST /attention.

  2. Task 2

    multihead.ts: all the heads at once

    In src/gpt/multihead.ts write multiHeadAttention(x, p, nHead): q, k and v are matmul(x, p.wq), matmul(x, p.wk) and matmul(x, p.wv), each split into heads; attention(q, k, v) (chapter 6's, given) runs every head at once; merge the heads' outputs, then matmul with p.wo. Return [out, cache] with cache { x, q, k, v, weights, merged }: q, k, v and weights split into heads, merged BEFORE wo. The given src/gpt/multihead-grad.ts reads that cache.

    The Check runs a [2, 4, 6] x with 2 heads, checks the shapes and the cache, and feeds your cache to the given backward: its gradients must match finite differences of your forward. Then (once all three tasks run) it calls POST /attention.

  3. Task 3 · interview question

    Interview: multihead.ts, what the last token looks at

    In src/gpt/multihead.ts write topAttended(weights, pieces, k = 3): weights is [H, T, T]. For each head h, take the LAST position's row (weights.row(h * T + T - 1)) and return its k largest weights, largest first (equal weights: the earlier position first), each as { position: j, piece: pieces[j], weight: Math.round(w * 1000) / 1000 }. One array per head.

    The Check tries two hand-made heads with ties, a weight to round, k = 1, then calls POST /attention (4 random heads).

    Nikhil will ask: four heads of 12 numbers cost the same as one head of 48. Why do GPTs use many small heads?

What you will learn

  • Heads: a token's numbers cut into groups, one group per head.
  • reshape and permute: moving numbers into a head dimension without mixing positions.
  • All heads in one call: why chapter 6's attention already works with a head dimension.
  • The output matrix wo, which mixes what the heads found back together.
  • The cache a forward keeps for a given backward.
  • Reading the weights: the few tokens the last token looks at most, head by head.

More in Attention

Build "Multi-head attention" yourself

Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.