Build your own GPT project · Chapter 4 of 12
Training by hand: gradients
By the end of this chapter you know what a gradient is and can measure one by nudging, you have written the gradient of the loss by hand and proved it right, and you have trained your first model: a table of numbers that starts knowing nothing and learns, step by step, to guess the next character as well as chapter 3's counting did. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Learning from mistakes
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.
Why you need this
Chapter 3's bigram table was counted: no learning, just a tally. That works for one table and stops working the moment the model has layers. Nobody can count what a GPT's 169,792 numbers should be. They have to be learned: start from numbers that know nothing, measure the loss, and change every number a little in the direction that makes the loss smaller. Repeat a thousand times.
To do that, for every number in the model you need to know which way is "smaller loss", and how strongly. That is the gradient, and this chapter computes it two ways: by hand with a short formula, and by nudging, to prove the formula right.
Farah asks: "If counting already gives 2.2461, why learn the same table?" Kunal: because the learning loop you write today is the one that trains ChotaGPT in chapter 10. On a table you can check it against the counted answer.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
grad.py: the gradient of the loss
In gpt/grad.py write cross_entropy_backward(logits, targets): the gradient of cross_entropy (the mean loss) with respect to every logit: softmax of each row, minus 1 at the row's target, all divided by the number of guesses N, in the shape of logits. Work on the flattened (N, V) rows and reshape back; do not change logits.
The Check compares your gradient with finite differences on (4, 5) and (2, 3, 6) logits.
- Task 2
grad.py: finite differences
In gpt/grad.py write numerical_grad(f, w, eps=1e-5): f() computes a loss from the current numbers in the array w. For every element of w (np.ndindex(w.shape)): set it to old + eps and call f, set it to old - eps and call f, put the old value back; the gradient there is (plus - minus) / (2 * eps). Return an array shaped like w; w ends exactly as it started.
The Check runs it on sum(w**2) + w[0,1] * w[1,0] at [[1, -2], [0.5, 3]], whose gradient is [[2, -3.5], [-1, 6]], and on sum(sin(v)).
- Task 3 · interview question
Interview: grad.py, one step of gradient descent
In gpt/grad.py write bigram_step(W, x, y, lr): W is a table of logits, one row per token. The batch's logits are W[x]; the loss is cross_entropy of them and y; dlogits comes from cross_entropy_backward. dW is zeros like W with each dlogits row ADDED to row x[i] (np.add.at: a token may appear many times in x). Change W in place (W -= lr * dW) and return the loss from before the step.
The Check runs one step on a 4 x 4 table where one token appears twice, then calls POST /bigram/train for 60 steps.
Kunal will ask: numerical_grad needs no calculus and is always right. Why not train with it?
What you will learn
- A gradient: how much the loss changes when one number grows a little, measured by nudging.
- Gradient descent: the loop of loss, gradient, small step, and what the learning rate does.
- The gradient of cross-entropy: probabilities minus 1 at the right token, from the chain rule in two steps.
- Finite differences: checking a hand-written gradient, and why the check uses central differences.
- The bigram as a table of logits, trained from zeros, and why np.add.at again.
- Why nobody trains with finite differences, in measured seconds.
More in Learning from mistakes
Build "Training by hand: gradients" yourself
DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.