Build your own GPT project · Chapter 11 of 12
Sampling and perplexity
By the end of this chapter ChotaGPT writes DabbaDrop notes: you turn the model's probabilities into one chosen token, shape that choice with temperature, top-k and top-p, write the loop that adds token after token, and measure the model with a number Farah can understand: perplexity. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The whole GPT
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.
Why you need this
Chapter 10 ended with the long run's weights: a validation loss of 0.494. That is a good number, but nobody at DabbaDrop has read a single line ChotaGPT wrote, because it has not written one yet. The model only ever answers one question: for each of the 256 tokens, how likely is it to come next? After Rider update: it says:
[(' Thane', 0.33), (' Dadar', 0.229), (' B', 0.106), (' Andheri', 0.101), (' M', 0.091)]
(the five likeliest of 256; the other 251 share what is left)
Two things are missing. Something must pick one of those tokens, add it, and ask again, forty times, until the note ends. And Bhavna, who will judge the notes, and Farah, who asked for this project, need a number that means something to them. "0.494" does not. "It hesitates between about 1.6 tokens" does.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
sample.py: temperature, top-k and top-p
In gpt/sample.py write apply_temperature(logits, temperature): logits / temperature, a ValueError when temperature is 0 or less. top_k_filter(logits, k): keep the k highest scores (and any equal to the k-th highest), the rest -np.inf; k None or k >= V gives an unchanged copy. top_p_filter(logits, p): p None or p >= 1 gives an unchanged copy; else softmax, order the tokens from most to least likely (np.argsort(-probs, kind="stable")), keep the fewest from the top whose probabilities add up to at least p (at least one), the rest -np.inf. None of them changes logits.
The Check tries each filter on small rows, with ties, None, unsorted scores, and checks the input is untouched.
- Task 2
sample.py: generate, one token at a time
In gpt/sample.py write generate(params, config, ids, max_new_tokens, temperature=1.0, top_k=None, top_p=None, seed=0) as a generator: rng = np.random.default_rng(seed) once; each round the context is the last config.block_size ids, forward on np.array([context]), the LAST position's logits, then temperature, top_k, top_p in that order, softmax, int(rng.choice(V, p=probs)); append it to a copy of the ids and yield it.
The Check compares 25 tokens with its own sampler for a seed, a 100-token prompt (only the last 64 count), and temperature 2.0 with top_p 0.5 (the order matters), then calls POST /generate.
Kunal will ask: nothing changes in the model between two runs. Why do two runs with different seeds write different notes?
- Task 3 · interview question
Interview: sample.py, perplexity
In gpt/sample.py write perplexity(params, config, ids): for s = 0, block_size, 2 * block_size ... while s < len(ids) - 1, x = ids[s : s + block_size] and y = ids[s + 1 : s + block_size + 1], x cut to len(y); add cross_entropy(logits, y) times len(y); divide by the number of targets for the mean loss per token, and return e to that power as a float.
The Check compares with its own on 150 validation tokens and on 2 tokens, then calls POST /perplexity on a rider's update and on an English office sentence.
Kunal will ask: the long run has a perplexity of about 1.6 on the validation notes. What does that number mean, and why is it so low?
What you will learn
- Sampling: picking the next token at random, in proportion to its probability, and why not always the likeliest one.
- Temperature: one number that makes the picks safer or bolder.
- Top-k and top-p: cutting off the long tail of unlikely tokens.
- The writing loop: a Python generator that yields each token as soon as it is picked.
- Perplexity: the loss turned back into "how many tokens was the model choosing between".
More in The whole GPT
Build "Sampling and perplexity" yourself
DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.