Build your own GPT project · Chapter 6 of 12
Self-attention, step by step
By the end of this chapter you have written the idea at the heart of every GPT: self-attention, where each token looks back at the tokens before it and takes a mix of what they carry, with a mask so that it can never look ahead. You run one head of it on a DabbaDrop note and read its weights. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Attention
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.
Why you need this
Chapter 5's MLP reads three characters and scores 1.54, far better than the bigram's 2.25. To read more, the obvious move is to join more rows: 64 tokens of 64 numbers each would be 4,096 numbers going into the first layer. That plan has three problems:
- the first layer's matrix grows with the context: 4,096 rows of weights where the MLP had 48;
- every position has its own weights, so "Thane" at position 5 and "Thane" at position 40 are learned separately;
- the context is fixed: a text of 10 tokens and a text of 64 need different networks.
Bhavna's notes need more than three characters. "Kal ka dabba (Tue 3 Nov), Thane kitchen: Rajma..." only makes sense if the tokens after "kitchen:" can see which kitchen it was, more than three characters back. ChotaGPT reads a block of 64 tokens.
Kunal's answer is attention: the same small set of weights, used at every position, letting each token choose which earlier tokens matter to it.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
attention.py: the causal mask
In gpt/attention.py write causal_mask(T): a (T, T) array of booleans, True where position i may look at position j, which is when j <= i (itself and every earlier position), False above the diagonal.
The Check tries T = 4 and T = 1.
- Task 2
attention.py: scaled dot-product attention
In gpt/attention.py write attention(q, k, v), each (..., T, d): the scores are q times k with its last two axes swapped (np.swapaxes(k, -1, -2)), divided by sqrt(d); where causal_mask(T) is False the score becomes -np.inf (np.where); weights = softmax(scores) over the last axis; return (weights @ v, weights).
The Check compares with its own attention on (5, 4) arrays and (2, 3, 6, 8) arrays, and looks for a lookahead, rows that do not add up to 1, and missing scaling.
- Task 3 · interview question
Interview: attention.py, one head of self-attention
In gpt/attention.py write self_attention(x, Wq, Wk, Wv): one head. q, k and v are x times Wq, Wk and Wv; return what attention(q, k, v) returns.
The Check runs a (6, 8) input through (8, 4) matrices, then calls POST /attention, where the first token may only look at itself.
Kunal will ask: why divide the scores by sqrt(d)? What happens to softmax without it when d is 64?
What you will learn
- The causal mask: a table of which positions may look at which, and why a GPT must never see the future.
- Queries, keys and values: what each token asks, offers as a label, and passes on.
- Dot products as matches, and softmax turning matches into weights.
- Masking with minus infinity, before softmax and not after.
- Why the scores are divided by sqrt(d).
- Self-attention: q, k and v all made from the same tokens.
More in Attention
Build "Self-attention, step by step" yourself
DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.