Build your own GPT project · Chapter 3 of 12

A bigram model and its loss

By the end of this chapter you can turn a row of scores into probabilities without numpy overflowing, you can measure any model with one number (its loss), and you have built and scored the first real model of ChotaGPT's project: a table that guesses the next character from the one before it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Learning from mistakes
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.

Why you need this

Chapter 1's counting model looked up the whole context in the notes. It answered Pizza with a 404, and almost every long context is new. A model must give a probability to every possible next token, always, even after text it has never seen.

And there is a second need. Kunal will build five models in this project, each better than the last. "Better" must be a number, measured the same way every time, on the validation text. Farah asks: "How will we know ChotaGPT is improving, and not just looking busy?" That number is the loss, and this chapter defines it.

The first model to measure is the smallest one that is still a real language model: the bigram model, which guesses the next character from the one before it and nothing else. This chapter uses characters (67 tokens), not chapter 2's BPE tokens, so that a row of the table is a row you can read.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    functional.py: scores to probabilities

    In gpt/functional.py write softmax(x): for every row of x (its last axis, whatever the shape), e to the power of each score divided by the row's sum. Subtract each row's largest score first (keepdims=True), so np.exp never overflows.

    The Check tries [1, 2, 3], a (2, 2) array, [1000, 1000, 999] and a (2, 3, 4) array.

  2. Task 2

    functional.py: the loss

    In gpt/functional.py write cross_entropy(logits, targets): logits is (..., V), targets has the same shape without V. The loss is the mean, over every guess, of minus the log of the probability its row gives its target, as a Python float. Compute log-softmax directly (the shifted scores minus the log of the sum of their exps), never log(softmax(...)).

    The Check tries 10 equal scores (ln 10), a small example, scores of 800 and 900, and (2, 5, 7) logits with (2, 5) targets.

  3. Task 3 · interview question

    Interview: bigram.py, a model by counting

    In gpt/bigram.py write count_bigram(ids, vocab_size, smoothing=1.0): a (vocab_size, vocab_size) float table where every cell starts at smoothing, then row a, column b gets 1 more for every time id b comes right after id a (np.add.at); then each row is divided by its sum. Then bigram_loss(probs, ids): the mean over every neighbouring pair (a, b) of ids of minus the log of probs[a, b], as a float.

    The Check counts small sequences with and without smoothing, scores a 2 x 2 table, then calls GET /bigram/loss and GET /bigram/next?char=Q (422).

    Kunal will ask: a model that guesses every character equally has a loss of ln 67, about 4.20. Why is the loss a log at all, and what would a loss of 0 mean?

What you will learn

  • Logits and softmax: a score for every token, turned into probabilities that add up to 1.
  • Why softmax subtracts the largest score, and what np.exp(1000) does without it.
  • The loss (cross-entropy): minus the log of the probability the model gave to what really came next.
  • Why log-softmax, and never the log of softmax's answer.
  • The bigram table: counting what follows what with np.add.at, and why every cell starts at 1.
  • ln 67 = 4.2047: the loss of guessing, and what the bigram scores against it.

More in Learning from mistakes

Build "A bigram model and its loss" yourself

DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.