Build your own GPT project · Chapter 8 of 12
A transformer block
By the end of this chapter you have built the unit a GPT is made of: a layer norm that puts every token's numbers on the same scale, a small network that works on each token alone, and a block that joins them with chapter 7's attention, each part adding its result onto the token's numbers instead of replacing them. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Attention
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.
Why you need this
Chapter 7's multi-head attention lets every token gather information from the tokens before it. That is one move. A GPT makes many: GPT-2 small stacks 12 blocks, and ChotaGPT's long run stacks 3. Stacking raw attention layers does not work well, for three reasons you can measure:
- The numbers drift. Fresh embeddings are tiny. On a note, the last token's 32 numbers have a standard deviation of 0.03 (you will see it in
POST /block). Each layer would grow or shrink them, and after a few layers they are far too big or close to zero. - Attention only mixes. It takes weighted averages of the tokens' values. Nothing yet does real work on what a token has gathered.
- Deep stacks are hard to train. The gradient has to travel back through every layer, and it fades on the way.
A transformer block fixes all three with three ideas: layer norm, a feed-forward network, and residuals. Kunal says it plainly: "This is the whole GPT, minus the embeddings at the start and the scores at the end. Chapter 9 adds those."
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
block.py: layer norm
In gpt/block.py write layer_norm(x, gamma, beta, eps=EPS): over the last axis of x, subtract each row's mean and multiply by inv_std = 1 / sqrt(var + eps); that is xhat. out = xhat * gamma + beta. Return (out, {"xhat": xhat, "inv_std": inv_std, "gamma": gamma}), the mean and inv_std with keepdims=True. The given gpt/block_grad.py reads that cache.
The Check normalises a (2, 3, 6) array (every row: mean 0, std 1 with gain 1 and shift 0), checks the cache's shapes, and feeds it to the given backward.
- Task 2
block.py: the feed-forward network
In gpt/block.py write feed_forward(x, p): linear_forward with p["fc_w"], p["fc_b"] (C to 4C), gelu_forward, then linear_forward with p["proj_w"], p["proj_b"] (4C back to C). Return (out, {"fc": ..., "act": ..., "proj": ...}), the three caches those functions returned.
The Check compares your output with its own and feeds your cache to the given feed_forward_backward.
- Task 3 · interview question
Interview: block.py, the block with residuals
In gpt/block.py write block_forward(x, p, n_head): x1 = x + multi_head_attention of layer_norm(x) with ln1_g, ln1_b; out = x1 + feed_forward of layer_norm(x1) with ln2_g, ln2_b. Return (out, {"ln1": ..., "attn": ..., "ln2": ..., "ffn": ...}): the four caches.
The Check compares with its own block, looks for missing residuals and swapped layer norms, feeds your caches to the given block_backward, then calls POST /block.
Kunal will ask: why add each part's output back onto its input, instead of just passing it on?
What you will learn
- Layer norm: one mean and one spread per token, and why there is an eps.
- Gain and shift (gamma, beta): how training can undo the normalising where it helps.
- The feed-forward network: chapter 5's Linear, GELU, Linear, four times wider in the middle.
- The residual stream: the row of numbers each token carries through the whole model.
- Pre-norm: normalise a copy, run the part, add the result back.
- Why residuals let a gradient reach the first block.
More in Attention
Build "A transformer block" yourself
DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.