Build your own GPT project · Chapter 12 of 12

Tests and the take-home

By the end of this chapter ChotaGPT has tests that catch real bugs: tests for the tokenizers, gradient checks for the backward passes you were given, and tests for the API, random generation included. Then you take the whole project home: run it on your laptop, train it longer, and see how each piece maps to a real GPT. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Shipping it
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.

Why you need this

ChotaGPT is 169,792 numbers, trained by code you mostly did not write line by line: gpt/block_grad.py and gpt/multihead_grad.py were given, and every one of the 1,200 training steps trusted them. Farah wants the team to keep working on ChotaGPT after this project: a bigger model, more notes. The first person who changes a line of a backward pass needs to know, in seconds, whether they broke it.

So Kunal asks for tests. But a test that passes on broken code is worse than no test: it makes you feel safe. How do you know your tests are good? You break the code on purpose and see whether they notice. That is how this chapter's Check works, and it is a habit worth keeping.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    test_tokenizer.py: test the tokenizers

    In tests/test_tokenizer.py, write tests (functions named test_...): bpe.decode(bpe.encode(line)) == line for lines of the validation text (app.state.splits()[1]); CharTokenizer("dal chawal") decodes encode("dal Q") as "dal ?"; a hand-made BpeTokenizer(["a", "b", "c"], [(0, 1), (1, 2)]) encodes "abc" as [3, 2] and "bc" as [4]. Run pytest in the terminal.

    The Check runs your file on our code, where every test must pass, and on two broken copies, where at least one test must fail: an encode that drops unknown characters, and an encode_chunk that applies the newest merge first.

  2. Task 2

    test_gradients.py: test the given backward passes

    In tests/test_gradients.py, test layer_norm_backward (x (2, 3, 5)) and multi_head_attention_backward (x (2, 4, 6), 2 heads, four random (6, 6) matrices) against finite differences: with a random dout of the output's shape, the loss float((forward(...)[0] * dout).sum()) has the backward's gradients for that dout. Each gradient (dx, dgamma, dbeta; dx and each matrix's) with grad_check(loss, the array, gradient) < 1e-6, the arrays from a seeded numpy Generator.

    The Check runs your file on our code and on two broken copies: a layer norm backward that forgets one term, and an attention backward that forgets the 1/sqrt(d).

  3. Task 3 · interview question

    Interview: test_api.py, test the API

    In tests/test_api.py, test with the client fixture: POST /generate {"prompt": "Kal ka dabba", "max_new_tokens": 20, "seed": 3, "stream": false} answers 200, twice the same "text", which starts with the prompt; POST /generate with stream left out answers text/event-stream with an "event: token" line per token and an "event: done"; "temperature": 0 answers 422; POST /tokenize gives the text back as "decoded" and its "pieces" joined.

    The Check runs your file on our code and on two broken copies: a /generate whose text leaves the prompt out, and one that lets a temperature of 0 through (a 500).

    Kunal will ask: generation is random. How do you write a test for it that passes every time and still catches real bugs?

What you will learn

  • pytest fixtures: values a test asks for by name.
  • Tests that fail for one reason you can name, on cases small enough to work out by hand.
  • Gradient checks as tests: one number whose slope proves a whole backward pass.
  • TestClient: the whole API in a test, with no server and no port.
  • Testing randomness: what a seed pins down, and what stays true for every seed.
  • Mutation testing: how the Check judges tests, by breaking the code.

More in Shipping it

Build "Tests and the take-home" yourself

DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.