Build your own GPT project · Chapter 10 of 12

Training ChotaGPT

By the end of this chapter ChotaGPT learns. You write the three pieces of its training: random batches of windows, Adam (the rule that moves every weight), and the loop that runs them and measures the loss as it falls. You train a tiny ChotaGPT in seconds, and you read the record of Kunal's long run, the one whose weights the API now uses. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The whole GPT
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.

Why you need this

Chapter 9's model scores 5.5548 on the validation text: as good as guessing (ln 256 = 5.5452). Chapter 4 trained the bigram table with gradient steps, and the idea is the same here: measure the loss, find the gradient of every weight, step every weight downhill. What changes is the size. ChotaGPT has 169,792 weights in 16 kinds of arrays, and the training text is 60,390 tokens. Three things are needed that chapter 4 did not have:

  • batches: a step cannot read all 60,390 tokens; it reads a few random windows;
  • a better step rule: one learning rate for every weight is too fast for some and too slow for others;
  • a record: the training loss and the validation loss, step by step, so you can see learning, and see when it stops helping.

The backward pass of the whole model is given (gpt/model_grad.py): it walks chapter 9's cache from the scores back to the embeddings. You write what drives it.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    train.py: a batch of windows

    In gpt/train.py write get_batch(ids, batch_size, block_size, rng): ONE call rng.integers(0, len(ids) - block_size, size=batch_size) gives the starts; x stacks ids[s : s + block_size] for each start s, y the same windows moved one token to the right. Return (x, y), both (batch_size, block_size).

    The Check compares your windows with its own for a seeded rng and makes sure the last token can be reached.

  2. Task 2

    train.py: Adam

    In gpt/train.py write adam_step(params, grads, state, lr, beta1=0.9, beta2=0.99, eps=1e-8), changing params in place: state["t"] += 1 first (start at 0); per name, m = beta1 * m + (1 - beta1) * g and v = beta2 * v + (1 - beta2) * g * g (zeros the first time), kept in state["m"][name] and state["v"][name]; m_hat = m / (1 - beta1t), v_hat = v / (1 - beta2t); param -= lr * m_hat / (sqrt(v_hat) + eps).

    The Check runs three steps on two arrays and compares every number with its own Adam.

  3. Task 3 · interview question

    Interview: train.py, the training loop

    In gpt/train.py write train(config, train_ids, val_ids, steps, batch_size=16, lr=3e-3, seed=0, eval_every=50, on_eval=None): params = init_params(config, seed); rng = np.random.default_rng(seed); state = {}. For step 1 to steps: get_batch, loss_and_grads (given), clip_grads (given), adam_step with lr_at(step, steps, lr) (given). At every eval_every-th step and the last, add {"step", "train_loss": round(loss, 4), "val_loss": round(evaluate(params, val_ids, config), 4), "seconds": round(time since the start, 2)} to history and call on_eval(row) when it is not None. Return (params, history).

    The Check trains a small model for 7 steps and compares the weights and every row with its own loop, then calls POST /train.

    Kunal will ask: the long run's training loss kept falling after step 1,100, but its validation loss went up. What was happening, and what would you do?

What you will learn

  • A batch: random windows of the token ids, as one array.
  • The learning rate schedule and gradient clipping (both given).
  • Adam: a step of about the same size for every weight, in a steady direction.
  • The training loop, and why the validation loss is the number to watch.
  • The tiny run and the long run: the same code with bigger numbers.
  • Overfitting: when the training loss falls and the validation loss rises.

More in The whole GPT

Build "Training ChotaGPT" yourself

DabbaDrop wants its team to know what a GPT really does inside. You build ChotaGPT, piece by piece, and teach it to write the kitchen's Hinglish notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build your own GPT project are free. This one comes with the all labs plan.