Build your own GPT project (TypeScript) · Chapter 8 of 12
A transformer block
By the end of this chapter you have built the unit a GPT is made of: a layer norm that puts every token's numbers on the same scale, a small network that works on each token alone, and a block that joins them with chapter 7's attention, each part adding its result onto the token's numbers instead of replacing them. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Attention
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Chapter 7's multi-head attention lets every token gather information from the tokens before it. That is one move. A GPT makes many: GPT-2 small stacks 12 blocks, and ChotaGPT's long run stacks 3. Stacking plain attention layers one after another does not work well, for three reasons you can measure:
- The numbers drift. Fresh embeddings are tiny: on Datta's note "Kal ka show (Fri 6 Nov), Kala Talkies", the last token's 32 numbers have a standard deviation of 0.025 (you will see it in
POST /block). Each layer can grow or shrink them, and after a few layers they are far too big or close to zero. - Attention only mixes. It takes weighted averages of the tokens' values. Nothing yet does real work on what a token has gathered.
- Deep stacks are hard to train. The gradient has to travel back through every layer, and it fades on the way.
A transformer block fixes all three with three ideas: layer norm, a feed-forward network, and residuals. Nikhil puts it plainly to Sayali: "This is the whole GPT, minus the embeddings at the start and the scores at the end. Chapter 9 adds those."
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
block.ts: layer norm
In src/gpt/block.ts write layerNorm(x, gamma, beta, eps = EPS): for each row of x (x.rows rows of x.last numbers), take the row's mean and variance (the mean of (v - mean) ** 2); invStd for the row is 1 / Math.sqrt(variance + eps), and xhat is (v - mean) * invStd. out is xhat * gamma + beta, number j using gamma.data[j] and beta.data[j]. Return [out, { xhat, invStd, gamma }]: out and xhat new Tensors of x's shape, invStd a Float64Array with one number a row. The given src/gpt/block-grad.ts reads that cache.
The Check normalizes a [2, 3, 6] tensor (every row: mean 0, std 1 with gain 1 and shift 0), checks the cache, feeds it to the given layerNormBackward, then calls POST /block once Task 3 is written.
- Task 2
block.ts: the feed-forward network
In src/gpt/block.ts write feedForward(x, p): linearForward with p.fc_w and p.fc_b (C to 4C numbers), geluForward, then linearForward with p.proj_w and p.proj_b (4C back to C). Return [out, { fc, act, proj }]: the three caches those functions returned, under those names.
The Check compares your output with its own on a [2, 3, 4] tensor and feeds your caches to the given feedForwardBackward.
- Task 3 · interview question
Interview: block.ts, the block with residuals
In src/gpt/block.ts write blockForward(x, p, nHead): x1 = x + multiHeadAttention of layerNorm(x) (with p.ln1_g, p.ln1_b); out = x1 + feedForward of layerNorm(x1) (with p.ln2_g, p.ln2_b). Return [out, { ln1, attn, ln2, ffn }]: the four caches.
The Check compares with its own block, looks for missing residuals and a layer norm used twice, feeds your caches to the given blockBackward, then calls POST /block on "Kal ka show (Fri 6 Nov), Kala Talkies" (13 tokens, change 0.385).
Nikhil will ask: why add each part's output back onto its input, instead of just passing the output on?
What you will learn
- Layer norm: one mean and one spread per token, and why there is an eps.
- Gain and shift (gamma, beta): how training can undo the normalizing where it helps.
- The feed-forward network: chapter 5's Linear, GELU, Linear, four times wider in the middle.
- The residual stream: the row of numbers each token carries through the whole model.
- Pre-norm: normalize a copy, run the part, add the result back.
- Why residuals let the gradient reach the first block, measured.
More in Attention
Build "A transformer block" yourself
Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.