Build your own GPT project (TypeScript) · Chapter 10 of 12

Training ChotaGPT

By the end of this chapter ChotaGPT learns. You write the three pieces of its training: random batches of windows, Adam (the rule that moves every weight), and the loop that runs them and measures the loss as it falls, live, through an API that sends each measurement the moment it exists. You train a tiny ChotaGPT in seconds, and you read the record of Nikhil's long run, the one whose weights the API now uses. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The whole GPT
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 9's model scores 5.5501 on the validation text: as good as guessing (ln 256 = 5.5452). Chapter 4 trained the bigram table with gradient steps, and the idea is the same here: measure the loss, find the gradient of every weight, step every weight downhill. What changes is the size. ChotaGPT has 99,696 weights in 16 kinds of tensors, and the training text is 73,319 tokens. Three things are needed that chapter 4 did not have:

  • batches: a step cannot read all 73,319 tokens; it reads a few random windows;
  • a better step rule: one learning rate for every weight is too fast for some and too slow for others;
  • a record, sent live: the training loss and the validation loss, step by step, so you can watch learning happen, and see when it stops helping.

The backward pass of the whole model is given (src/gpt/model-grad.ts): it walks chapter 9's cache from the scores back to the embeddings. You write what drives it.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    train.ts: a batch of windows

    In src/gpt/train.ts write getBatch(ids, batchSize, blockSize, rng): for each of batchSize windows, ONE call rng.int(ids.length - blockSize) gives the start s; x gets ids.slice(s, s + blockSize), y the same window moved one token to the right, ids.slice(s + 1, s + blockSize + 1). Return [x, y], each batchSize rows of blockSize ids.

    The Check compares your windows with its own for new Rng(7), makes sure the last id can be reached, then calls GET /train/curve.

  2. Task 2

    train.ts: Adam

    In src/gpt/train.ts write adamStep(params, grads, state, lr, beta1 = 0.9, beta2 = 0.99, eps = 1e-8), changing params' numbers in place: state.t += 1 first; for each name of grads, state.m[name] and state.v[name] (new Tensors of zeros the first time, kept after) and, for every number i with gradient g: m = beta1 * m + (1 - beta1) * g; v = beta2 * v + (1 - beta2) * g * g; mHat = m / (1 - beta1 t); vHat = v / (1 - beta2 t); param -= lr * mHat / (Math.sqrt(vHat) + eps).

    The Check runs three steps on two tensors and compares every number with its own Adam, then calls GET /model/loss (the long run's weights: 0.4966).

  3. Task 3 · interview question

    Interview: train.ts, the training loop

    In src/gpt/train.ts write the async train(config, trainIds, valIds, steps, options): with options' batchSize = 16, lr = 3e-3, seed = 0, evalEvery = 50, onEval: params = initParams(config, seed), rng = new Rng(seed), state = newAdamState(). For step 1 to steps: getBatch, lossAndGrads (given), clipGrads, adamStep with lrAt(step, steps, lr). At every evalEvery-th step and the last, push { step, train_loss: round4(loss), val_loss: round4(evaluate(params, valIds, config)), seconds: since the start, to 2 places } into history and await onEval?.(row). Return [params, history].

    The Check trains a small model for 7 steps and compares the weights and every row with its own loop, checks that train waits for onEval's promise, then calls POST /train (40 steps, a row every 20).

    Nikhil will ask: the long run's validation loss was still falling at step 1,000. What does a longer run show, and when would you stop training?

What you will learn

  • A batch: random windows of the token ids, drawn from a seeded Rng.
  • The learning rate schedule and gradient clipping (both given).
  • Adam: a step of about the same size for every weight, in a steady direction.
  • The training loop, and why the validation loss is the number to watch.
  • Streaming on one thread: why train is async and waits for each row to be sent.
  • The tiny run and the long run: the same code with bigger numbers, and what a longer run shows.

More in The whole GPT

Build "Training ChotaGPT" yourself

Parda wants its team to know what a GPT really does inside. You build ChotaGPT from scratch and teach it to write Parda's Hinglish show notes. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build your own GPT project (TypeScript) are free. This one comes with the all labs plan.