Build your own GPT project (TypeScript) · Chapter 8 of 12

A transformer block

By the end of this chapter you have built the unit a GPT is made of: a layer norm that puts every token's numbers on the same scale, a small network that works on each token alone, and a block that joins them with chapter 7's attention, each part adding its result onto the token's numbers instead of replacing them. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Attention
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 7's multi-head attention lets every token gather information from the tokens before it. That is one move. A GPT makes many: GPT-2 small stacks 12 blocks, and ChotaGPT's long run stacks 3. Stacking plain attention layers one after another does not work well, for three reasons you can measure:

  • The numbers drift. Fresh embeddings are tiny: on Datta's note "Kal ka show (Fri 6 Nov), Kala Talkies", the last token's 32 numbers have a standard deviation of 0.025 (you will see it in POST /block). Each layer can grow or shrink them, and after a few layers they are far too big or close to zero.
  • Attention only mixes. It takes weighted averages of the tokens' values. Nothing yet does real work on what a token has gathered.
  • Deep stacks are hard to train. The gradient has to travel back through every layer, and it fades on the way.

A transformer block fixes all three with three ideas: layer norm, a feed-forward network, and residuals. Nikhil puts it plainly to Sayali: "This is the whole GPT, minus the embeddings at the start and the scores at the end. Chapter 9 adds those."

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    block.ts: layer norm

    In src/gpt/block.ts write layerNorm(x, gamma, beta, eps = EPS): for each row of x (x.rows rows of x.last numbers), take the row's mean and variance (the mean of (v - mean) ** 2); invStd for the row is 1 / Math.sqrt(variance + eps), and xhat is (v - mean) * invStd. out is xhat * gamma + beta, number j using gamma.data[j] and beta.data[j]. Return [out, { xhat, invStd, gamma }]: out and xhat new Tensors of x's shape, invStd a Float64Array with one number a row. The given src/gpt/block-grad.ts reads that cache.

    The Check normalizes a [2, 3, 6] tensor (every row: mean 0, std 1 with gain 1 and shift 0), checks the cache, feeds it to the given layerNormBackward, then calls POST /block once Task 3 is written.

  2. Task 2

    block.ts: the feed-forward network

    In src/gpt/block.ts write feedForward(x, p): linearForward with p.fc_w and p.fc_b (C to 4C numbers), geluForward, then linearForward with p.proj_w and p.proj_b (4C back to C). Return [out, { fc, act, proj }]: the three caches those functions returned, under those names.

    The Check compares your output with its own on a [2, 3, 4] tensor and feeds your caches to the given feedForwardBackward.

  3. Task 3 · interview question

    Interview: block.ts, the block with residuals

    In src/gpt/block.ts write blockForward(x, p, nHead): x1 = x + multiHeadAttention of layerNorm(x) (with p.ln1_g, p.ln1_b); out = x1 + feedForward of layerNorm(x1) (with p.ln2_g, p.ln2_b). Return [out, { ln1, attn, ln2, ffn }]: the four caches.

    The Check compares with its own block, looks for missing residuals and a layer norm used twice, feeds your caches to the given blockBackward, then calls POST /block on "Kal ka show (Fri 6 Nov), Kala Talkies" (13 tokens, change 0.385).

    Nikhil will ask: why add each part's output back onto its input, instead of just passing the output on?

What you will learn

  • Layer norm: one mean and one spread per token, and why there is an eps.
  • Gain and shift (gamma, beta): how training can undo the normalizing where it helps.
  • The feed-forward network: chapter 5's Linear, GELU, Linear, four times wider in the middle.
  • The residual stream: the row of numbers each token carries through the whole model.
  • Pre-norm: normalize a copy, run the part, add the result back.
  • Why residuals let the gradient reach the first block, measured.

More in Attention

Build "A transformer block" yourself

Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.