Build your own GPT project · Chapter 7 of 12
Multi-head attention
By the end of this chapter you can cut a token's numbers into several smaller heads, run attention in all of them at once with one call to chapter 6's function, join them back, and read what the newest token looks at in each head. That is the attention layer ChotaGPT really uses. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Attention
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.
Why you need this
Chapter 6's head gives every token one mix of the tokens before it: one softmax row, one set of weights adding up to 1. But a token in Bhavna's notes needs several things at once. To write the next dish after "Thane kitchen: Rajma, ", ChotaGPT wants to know:
- which kitchen this note is about (a few tokens back);
- what the last dish was (just before);
- that this is a menu line at all ("Kal ka dabba" at the start).
One softmax row can put its weight on all three, but then it blends them into one average and the separate facts are lost. Put most of the weight on one, and the others are missed.
Kunal's answer is what every GPT does: run several smaller attentions side by side, each with its own weights, so each one can look for something different. ChotaGPT's final model has a width of 64 numbers per token, cut into 4 heads of 16. GPT-2 small has 768 numbers cut into 12 heads of 64.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
multihead.py: split into heads and merge back
In gpt/multihead.py write split_heads(x, n_head): (B, T, C) to (B, n_head, T, C // n_head): reshape to (B, T, n_head, D) first, then swap the T and head axes. Then merge_heads(x): (B, H, T, D) back to (B, T, H * D). merge_heads(split_heads(x, h)) gives x back.
The Check splits a (2, 3, 8) array into 4 heads, checks where position 2's numbers went, and merges back.
- Task 2
multihead.py: all the heads at once
In gpt/multihead.py write multi_head_attention(x, p, n_head): q, k and v are x times p["wq"], p["wk"], p["wv"], each split into heads; attention(q, k, v) (chapter 6's, given) runs every head at once; merge the heads' outputs, then times p["wo"]. Return (out, cache) with cache {"x", "q", "k", "v", "weights", "merged"}: q, k, v and weights split into heads, merged BEFORE wo. The given gpt/multihead_grad.py reads that cache.
The Check runs (2, 4, 6) inputs with 2 heads, checks the shapes and the cache, and feeds your cache to the given backward: its gradients must match finite differences of your forward.
- Task 3 · interview question
Interview: multihead.py, what the last token looks at
In gpt/multihead.py write top_attended(weights, pieces, k=3): weights is (H, T, T). For each head, take the LAST position's row and return its k largest weights, largest first (equal weights: the earlier position first), each as {"position": j, "piece": pieces[j], "weight": round(float(w), 3)}. One list per head.
The Check tries two hand-made heads with ties, k = 1, then calls POST /attention (4 random heads).
Kunal will ask: four heads of 16 numbers cost the same as one head of 64. Why do GPTs use many small heads?
What you will learn
- Heads: a token's numbers cut into groups, one group per head.
- reshape and transpose: moving numbers into a heads axis without mixing positions.
- All heads in one call: why chapter 6's attention already works with a heads axis.
- The output matrix wo, which mixes what the heads found back together.
- The cache a forward keeps for a given backward.
- Reading the weights: the top few tokens the last token looks at, head by head.
More in Attention
Build "Multi-head attention" yourself
DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.