Build your own GPT project (TypeScript) · Chapter 11 of 12
Sampling and perplexity
By the end of this chapter ChotaGPT writes Parda's notes: you turn the model's probabilities into one chosen token, shape that choice with temperature, top-k and top-p, write the loop that adds token after token, and measure the model with a number Sayali can understand: perplexity. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The whole GPT
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Chapter 10 ended with the long run's weights: a validation loss of 0.4966. That is a good number, but nobody at Parda has read a single line ChotaGPT wrote, because it has not written one yet. The model only ever answers one question: for each of the 256 tokens, how likely is it to come next? After Box office update: it says (the five likeliest; the other 251 share what is left):
[" S", 0.249], [" b", 0.146], [" K", 0.125], [" The", 0.122], [" R", 0.113]
Two things are missing. Something must pick one of those tokens, add it, and ask again until the note ends. And Sayali needs a number that means something to her. "0.4966" does not. "It hesitates between about 1.7 tokens" does.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
sample.ts: temperature, top-k and top-p
In src/gpt/sample.ts write applyTemperature(logits, temperature): a new Tensor, every score divided by temperature; throw an Error when temperature is not more than 0. topKFilter(logits, k): keep the k highest scores (and any score equal to the k-th highest), the rest -Infinity; k null or k >= V gives an unchanged copy (logits.clone()). topPFilter(logits, p): p null or p >= 1 gives an unchanged copy; else softmax, order the token ids from most to least likely (equal ones by the smaller id), keep the fewest from the top whose probabilities add up to at least p (at least one), every other score -Infinity. None of them changes logits.
The Check tries each one on small rows (a tie at the k-th place, null, unsorted scores, equal scores), checks the input is untouched, then sends POST /generate with a temperature of 0 and a top_p of 1.5 (both 422).
- Task 2
sample.ts: generate, one token at a time
In src/gpt/sample.ts write generate(params, config, ids, maxNewTokens, options) as a generator (function*): options default to temperature 1, topK null, topP null, seed 0. Make one new Rng(seed) before the loop. Each round: the context is the last config.blockSize ids, forward(params, [context], config), the LAST position's row of scores as a Tensor of shape [V], then applyTemperature, topKFilter, topPFilter in that order, softmax, rng.choice of the probabilities. Add the id to a copy of ids and yield it.
The Check compares 25 tokens with its own sampler for a seed, temperature 2 with topP 0.5 for six seeds (the order matters), no options, and a 100-token prompt (only the last 64 count), then reads the token events of POST /generate.
- Task 3 · interview question
Interview: sample.ts, perplexity
In src/gpt/sample.ts write perplexity(params, config, ids): for start = 0, blockSize, 2 * blockSize ... while start < ids.length - 1, the targets y are ids.slice(start + 1, start + blockSize + 1) and x is the same number of ids from start. Add crossEntropy(logits, y) times y.length for each piece; divide by the number of targets for the mean loss per token, and return Math.exp of it.
The Check compares with its own on 150 validation tokens (pieces of 64, 64 and 21 targets) and on 2 tokens, then sends POST /perplexity a box office update and an English office sentence.
Nikhil will ask: the long run's perplexity on the validation notes is about 1.7. What does that number mean, and why is it so low?
What you will learn
- Sampling: picking the next token at random, in proportion to its probability, and why not always the likeliest one.
- Temperature: one number that makes the picks safer or bolder.
- Top-k and top-p: cutting off the long tail of unlikely tokens.
- The writing loop: a generator function that yields each token as soon as it is picked.
- Perplexity: the loss turned back into "how many tokens was the model choosing between".
More in The whole GPT
Build "Sampling and perplexity" yourself
Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.