Build your own GPT project (TypeScript) · Chapter 3 of 12
A bigram model and its loss
By the end of this chapter you can read and use ChotaGPT's tensor helper, you can turn a row of scores into probabilities without anything overflowing, you can measure any model with one number (its loss), and you have built and scored the first real model of the project: a table that guesses the next character from the one before it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Learning from mistakes
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Chapter 1's counting model looked up the whole context in the notes. It answered Pizza with a 404, and almost every long context is new. A model must give a probability to every possible next token, always, even after text it has never seen.
And there is a second need. Nikhil will build several models in this project, each better than the last. "Better" must be a number, measured the same way every time, on the validation text. Sayali asks: "How will we know ChotaGPT is improving, and not just looking busy?" That number is the loss, and this chapter defines it.
The first model to measure is the smallest one that is still a real language model: the bigram model, which guesses the next character from the one before it and nothing else. This chapter uses characters (66 tokens), not chapter 2's BPE tokens, so that a row of the table is a row you can read.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
functional.ts: scores to probabilities
In src/gpt/functional.ts write softmax(x): a new Tensor of x's shape where each row of x (its last dimension: x.rows rows of x.last numbers, x.row(r)) becomes probabilities: Math.exp of each score divided by the row's sum. Subtract the row's own largest score first, so Math.exp never overflows. Do not change x.
The Check tries [1, 2, 3], [[1, 1], [0, 2]], [[1000, 1000, 999]], [[0, 2], [1000, 999]] and a [2, 3, 4] tensor.
- Task 2
functional.ts: the loss
In src/gpt/functional.ts write crossEntropy(logits, targets): logits has logits.rows rows of scores, targets one id per row (the token that really came next). The loss is the mean, over every row, of minus the log of the probability its row gives its target, as a number. Compute the log directly (log-softmax: the target's shifted score minus the log of the sum of the exps of the shifted scores), never Math.log of softmax's answer.
The Check tries 10 equal scores (ln 10), a small example, scores of 800 and 900, and [2, 5, 7] logits with 10 targets.
- Task 3 · interview question
Interview: bigram.ts, a model by counting
In src/gpt/bigram.ts write countBigram(ids, vocabSize, smoothing = 1): a Tensor of shape [vocabSize, vocabSize] where every cell starts at smoothing, then row a, column b gets 1 more for every time id b comes right after id a; then each row is divided by its own sum. Then bigramLoss(probs, ids): the mean over every neighbouring pair (a, b) of ids of minus the log of probs.at(a, b), as a number.
The Check counts small sequences with and without smoothing, scores a 2 x 2 table, then calls GET /bigram/loss and GET /bigram/next?char=Q (422).
Nikhil will ask: a model that guesses every character equally has a loss of ln 66, about 4.19. Why is the loss a log at all, and what would a loss of 0 mean?
What you will learn
- The tensor helper: a block of numbers with a shape, read row by row.
- Seeded random numbers: Rng, and why not Math.random.
- Logits and softmax: a score for every token, turned into probabilities that add up to 1.
- Why softmax subtracts the largest score, and what Math.exp(1000) does without it.
- The loss (cross-entropy): minus the log of the probability the model gave to what really came next, and why log-softmax.
- The bigram table, counted with smoothing, and ln 66 = 4.1897: the loss of guessing.
More in Learning from mistakes
Build "A bigram model and its loss" yourself
Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.