Build your own GPT project (TypeScript) · Chapter 12 of 12
Tests and the take-home
By the end of this chapter ChotaGPT has tests that catch real bugs: tests for the tokenizers, gradient checks for the backward passes you were given, and tests for the API, random generation included. Then you take the whole project home: run it on your laptop, train it longer, and see how each piece maps to a real GPT. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Shipping it
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
ChotaGPT is 99,696 numbers, trained by code you did not all write line by line: src/gpt/block-grad.ts and src/gpt/multihead-grad.ts were given, and every one of the 1,000 training steps trusted them. Sayali wants the team to keep working on ChotaGPT after this project: a bigger model, more notes. The first person who changes a line of a backward pass needs to know, in seconds, whether they broke it.
So Nikhil asks for tests. But a test that passes on broken code is worse than no test: it makes you feel safe. How do you know your tests are good? You break the code on purpose and see whether they notice. That is how this chapter's Check works, and it is a habit worth keeping.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
tokenizer.test.ts: test the tokenizers
In tests/tokenizer.test.ts, replace the starter's it() with tests (each an it() with expect): bpe().decode(bpe().encode(line)) is the line for the first 50 lines of the validation text (splits()[1]); new CharTokenizer("chai biscuit") decodes encode("chai Z") as "chai ?"; a hand-made new BpeTokenizer(["a", "b", "c"], [[0, 1], [1, 2]]) encodes "abc" as [3, 2] and "bc" as [4]. Run npm test in the terminal.
The Check runs your file on our code, where every test must pass, and on two broken copies of src/gpt/tokenizer.ts, where at least one test must fail: an encode that drops unknown characters, and an encodeChunk that applies the newest merge first.
- Task 2
gradients.test.ts: test the given backward passes
In tests/gradients.test.ts, test layerNormBackward (x [2, 3, 5], gamma and beta [5]) and multiHeadAttentionBackward (x [2, 4, 6], 2 heads, p with four random [6, 6] matrices wq, wk, wv, wo) against finite differences: with a random dout of the output's shape, the loss () => forward(...)[0].mul(dout).sum() has the backward's gradients for that dout. Expect gradCheck(loss, w, gradient) toBeLessThan(1e-6) for every gradient (dx, dgamma, dbeta; dx and each matrix's), with w the very tensor the loss reads and the numbers from Tensor.randn with one seeded Rng.
The Check runs your file on our code and on two broken copies: a layer norm backward that forgets one term, and an attention backward that forgets the 1 / sqrt(d).
- Task 3 · interview question
Interview: api.test.ts, test the API
In tests/api.test.ts, test with client() (supertest): POST /generate {"prompt": "Kal ka show", "max_new_tokens": 20, "seed": 3, "stream": false} answers 200, twice the same "text", which starts with the prompt; POST /generate with "max_new_tokens": 5, "stop_at_newline": false and stream left out answers a text/event-stream content type with 5 "event: token" and an "event: done"; "temperature": 0 answers 422; POST /tokenize gives the text back as "decoded" and its "pieces" joined.
The Check runs your file on our code and on two broken copies of the generate routes: a /generate whose text leaves the prompt out, and one that lets a temperature of 0 through (a 500).
Nikhil will ask: generation is random. How do you write a test for it that passes every time and still catches real bugs?
What you will learn
- Vitest: describe, it and expect, and npm test.
- Tests that fail for one reason you can name, on cases small enough to work out by hand.
- Gradient checks as tests: one number whose slope proves a whole backward pass.
- supertest: the whole API in a test, with no server and no port.
- Testing randomness: what a seed pins down, and what stays true for every seed.
- Mutation testing: how the Check judges tests, by breaking the code.
More in Shipping it
Build "Tests and the take-home" yourself
Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.