Build your own GPT project · Chapter 5 of 12
Embeddings and a tiny MLP
By the end of this chapter you have written the three layers every neural network in this project is built from (an embedding table, a linear layer and a GELU), each with its forward and its backward, and you have watched them train a small network that reads three characters and beats the bigram model by a wide margin. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Learning from mistakes
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.
Why you need this
Chapter 4 trained the bigram model by gradient descent: 2.32 after 100 steps, close to the counted table's validation loss of 2.2461. That is about as good as a model can get when it sees one character. It cannot do much better, because after a it always gives the same guess, whether the a is in "Dal", "Kal" or "aur".
Chapter 1 showed the fix: more context, fewer choices. But the bigram table cannot simply grow. A table for three characters would need one row for every run of three: 67 x 67 x 67 = 300,763 rows of 67 numbers, and most of those runs never appear in the notes, so their rows would never learn anything.
Kunal wants a model that reads three characters with only a few thousand numbers, and that can use what it learned about one character for another. That model is a small neural network, and this chapter builds its pieces.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
layers.py: the embedding table
In gpt/layers.py write embedding_forward(E, ids): the rows E[ids], so out has the shape of ids plus E.shape[1]; return (out, cache) with the cache your backward needs. Then embedding_backward(dout, cache): dE, zeros shaped like E, with each dout row ADDED to the row of its id (np.add.at).
The Check looks up (2, 3) ids in a (6, 3) table where id 1 appears twice and compares dE with finite differences.
- Task 2
layers.py: a linear layer
In gpt/layers.py write linear_forward(x, W, b): x @ W, plus b when b is not None; return (out, cache). Then linear_backward(dout, cache): (dx, dW, db) where dx = dout @ W.T (shaped like x), dW = x2.T @ d2 with x and dout flattened to 2-D (reshape(-1, last axis)), and db = d2 summed over rows, or None when the forward had no bias.
The Check runs (2, 3, 4) inputs through a (4, 5) layer, with and without a bias, and compares each gradient with finite differences.
- Task 3 · interview question
Interview: layers.py, GELU
In gpt/layers.py write gelu_forward(x): with t = tanh(GELU_C * (x + 0.044715 * x**3)), gelu(x) = 0.5 * x * (1 + t); return (out, cache). Then gelu_backward(dout, cache): dout times the derivative of gelu at x, by the chain rule (the derivative of tanh(u) is 1 - tanh(u)**2, and u's derivative is GELU_C * (1 + 3 * 0.044715 * x**2)).
The Check tries five values, compares the backward with finite differences, then calls POST /mlp/train for 300 steps: the MLP must beat the bigram.
Kunal will ask: why put a GELU between the two linear layers? What would two linear layers in a row be without it?
What you will learn
- Embeddings: a short list of numbers for every token, looked up by id, and changed by training.
- Why an id used twice collects two gradients, and how np.add.at adds them.
- A linear layer: x @ W + b, the weighted sums that hold most of a network's numbers.
- Its backward: which gradient goes to the input, which to the weights, and why the weights' gradient is a sum over the batch.
- GELU: the smooth bend between two linear layers, its derivative by the chain rule, and why a network needs a bend at all.
- A tiny MLP language model (MLP: multi-layer perceptron, layers in a row) trained on the notes.
More in Learning from mistakes
Build "Embeddings and a tiny MLP" yourself
DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.