Build your own GPT project (TypeScript) · Chapter 5 of 12
Embeddings and a tiny MLP
By the end of this chapter you have written the three layers every neural network in this project is built from (an embedding table, a linear layer and a GELU), each with its forward and its backward, and you have watched them train a small network that reads three characters and beats the bigram model by a wide margin. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Learning from mistakes
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Chapter 4 trained the bigram model by gradient descent: 2.4025 on the validation text after 100 steps, heading for the counted table's 2.2969. That is about as good as a model can get when it sees one character. After a it always gives the same guess, whether the a is in "Kala", "Rs 240 hai" or "Datta".
Chapter 1 showed the fix: more context, fewer choices. But the bigram table cannot simply grow. A table for three characters would need one row for every run of three: 66 x 66 x 66 = 287,496 rows of 66 numbers, nearly 19 million numbers, and most of those runs never appear in the notes, so their rows would never learn anything.
Sayali asks the plain question: "if a bigger table is hopeless, what do real models do?" Nikhil's answer: they stop keeping a row for every context. They give every character a few numbers of its own, and learn a small function that reads them. That function is a neural network, and this chapter builds its pieces.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
layers.ts: the embedding table
In src/gpt/layers.ts write embeddingForward(E, ids): ids is B rows of T token ids; out has the shape [B, T, C] (C = E.last), and row r of out (the ids read in order) is row ids[...] of E. Return [out, cache] with the cache type given ({ ids, shape }). Then embeddingBackward(dout, cache): dE, zeros shaped like E, with each row of dout ADDED (+=) to the row of dE of its id.
The Check looks up 2 rows of 3 ids in a [6, 3] table where id 1 appears twice, compares dE with finite differences, then (once all three layers run) calls POST /mlp/train for 1 step.
- Task 2
layers.ts: a linear layer
In src/gpt/layers.ts write linearForward(x, W, b): matmul(x, W), plus b when b is not null; return [out, cache] with { x, W, hasBias }. Then linearBackward(dout, cache): [dx, dW, db]. dx is matmul(dout, W transposed), x's shape. For dW, flatten x and dout to 2-D (reshape([t.rows, t.last])), then x2 transposed times d2: W's shape. db is d2's rows added up (shape [C]), or null when the forward had no bias.
The Check runs a [2, 3, 4] x through a [4, 5] layer, with and without a bias, compares each gradient with finite differences, then (once all three layers run) calls POST /mlp/train for 1 step.
- Task 3 · interview question
Interview: layers.ts, GELU
In src/gpt/layers.ts write geluForward(x): number by number, with t = Math.tanh(GELU_C * (x + 0.044715 * x ** 3)), gelu(x) = 0.5 * x * (1 + t); return [out, { x, t }]. Then geluBackward(dout, cache): dout times the slope of gelu at x, number by number, by the product rule and the chain rule (the slope of tanh(u) is 1 - tanh(u) ** 2, and u's slope is GELU_C * (1 + 3 * 0.044715 * x ** 2)).
The Check tries five values, compares the backward with finite differences, then calls POST /mlp/train for 200 steps: the MLP must beat the bigram.
Nikhil will ask: why put a GELU between the two linear layers? What would two linear layers in a row be without it?
What you will learn
- Embeddings: a short list of numbers for every token, looked up by id, changed by training.
- Why an id used twice collects two gradients, and why that means +=.
- Matrix products with matmul, and a linear layer: weighted sums plus a bias.
- The linear layer's backward: which gradient goes to the input, which to the weights, and why the weights' gradient is a sum over the batch.
- GELU: the smooth bend between two linear layers, its slope by the chain rule, and why a network needs a bend at all.
- A tiny MLP language model (MLP: multi-layer perceptron, layers in a row) trained on the notes.
More in Learning from mistakes
Build "Embeddings and a tiny MLP" yourself
Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.