Build your own GPT project (TypeScript) · Chapter 4 of 12

Training by hand: gradients

By the end of this chapter you know what a gradient is and can measure one by nudging, you have written the gradient of the loss by hand and proved it right, and you have trained your first model: a table of numbers that starts knowing nothing and learns, step by step, to guess the next character almost as well as chapter 3's counting did. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Learning from mistakes
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 3's bigram table was counted: no learning, just a tally. That works for one table and stops working the moment a model has layers. Nobody can count what ChotaGPT's 99,696 numbers should be. They have to be learned: start from numbers that know nothing, measure the loss, and change every number a little in the direction that makes the loss smaller. Repeat a thousand times.

To do that, for every number in the model you need to know which way is "smaller loss", and how strongly. That is the gradient, and this chapter computes it two ways: by hand with a short formula, and by nudging, to prove the formula right.

Sayali asks: "If counting already gives 2.2969, why learn the same table?" Nikhil: because the learning loop you write today is the one that trains ChotaGPT in chapter 10. On a table you can check it against the counted answer.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    grad.ts: the gradient of the loss

    In src/gpt/grad.ts write crossEntropyBackward(logits, targets): the gradient of crossEntropy (the mean loss over N = targets.length rows) with respect to every logit: softmax of each row, minus 1 at the row's target, all divided by N. A new Tensor shaped like logits; do not change logits.

    The Check compares your gradient with finite differences of crossEntropy on [4, 5] and [2, 3, 6] logits.

  2. Task 2

    grad.ts: finite differences

    In src/gpt/grad.ts write numericalGrad(f, w, eps = 1e-5): f() computes a loss from the numbers in w as they are right now. For every index i of w.data: set it to old + eps and call f, set it to old - eps and call f, put the old value back; the gradient there is (plus - minus) / (2 * eps). Return a new Tensor shaped like w; w ends exactly as it started.

    The Check runs it on the sum of the squares of w plus w[0][1] * w[1][0] at [[1, -2], [0.5, 3]], whose gradient is [[2, -3.5], [-1, 6]], and on the sum of sin(v).

  3. Task 3 · interview question

    Interview: grad.ts, one step of gradient descent

    In src/gpt/grad.ts write bigramStep(W, x, y, lr): W is a [V, V] table of logits, one row per token. The batch's logits are a [x.length, V] tensor whose row r is a copy of row x[r] of W; the loss is crossEntropy of them and y; dlogits comes from crossEntropyBackward. dW is zeros shaped like W with each row r of dlogits ADDED to row x[r] (+=: a token may appear many times in x). Change W itself (every number minus lr times its gradient) and return the loss from before the step.

    The Check runs one step on a 4 x 4 table where two tokens appear twice, compares W with finite differences, then calls POST /bigram/train for 60 steps.

    Nikhil will ask: numericalGrad needs no calculus and is always right. Why not train with it?

What you will learn

  • A gradient: how much the loss changes when one number grows a little, measured by nudging.
  • Gradient descent: the loop of loss, gradient, small step, and what the learning rate does.
  • The gradient of cross-entropy: probabilities minus 1 at the right token, from the chain rule in two steps.
  • Finite differences: checking a hand-written gradient, and why central differences.
  • The bigram as a table of logits, trained from zeros, and why its gradient must be added, not written.
  • Why nobody trains with finite differences, in measured seconds.

More in Learning from mistakes

Build "Training by hand: gradients" yourself

Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.