Fine-tuning basics (TypeScript) · Chapter 14 of 20

Overfitting and early stopping

By the end of this chapter you can measure a model on held-out examples while it trains, read a training curve and a validation curve side by side, say when a model has stopped learning the job and started learning its training examples by heart, and stop training at the right moment with the right weights in hand. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Inside
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Mangala has a new request. Chulha is adding a seventh queue next month, and she will only have time to tag about sixty examples for it. "Can chota learn from sixty?" she asks. "And if it trains longer, does it get better?"

Irfan's answer is "measure it". So far you have chosen the epochs (one epoch is one pass over the training examples) from a table someone else made. With a thousand examples, two or three epochs were right. With sixty, nobody knows, and the training loss will not tell you: it is measured on the very examples the model is learning, so it keeps going down whatever happens. You need a second number, measured on examples the model never trains on, after every epoch, and a rule for when to stop.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    val-loss.ts: the loss on held-out examples

    Write and export valLoss(model, set, adapters = {}): the loss training uses, on set, without training: each example through chatIds (the train marks keep the loss on the reply), makeBatch, and lossOf(forward(model, batch.ids, adapters).logits, batch). Return the average over every trained position of the whole set (if you use several batches, weight each batch's loss by its count). Do not change the model.

    Under if (import.meta.main), print chota-base's loss on examples('val').slice(64, 128) with 4 decimals:

    chota-base, validation examples 65 to 128: loss L

  2. Task 2

    curve.ts: a loss curve, epoch by epoch

    Write and export trainCurve(model, set, val, { epochs, lr, seed? }) returning one { epoch, train, val } per epoch: train the model fully with the given train (seed default 0) and, in its onEpoch, record the epoch (from 1), the average of that epoch's step losses, and valLoss(model, val) right then.

    Under if (import.meta.main), a fresh baseModel() on the first 60 training examples, measured on the first 64 validation examples, 15 epochs, lr 0.003. Print one line per epoch (3 decimals):

    epoch E: train T, val V

  3. Task 3

    stop-at.ts: where early stopping stops

    Write and export stopAt(val, patience) returning { best, stop }: val[i] is the validation loss after epoch i + 1. best is the epoch (from 1) with the lowest loss so far (an equal loss is not better); stop is the first epoch that is patience epochs past the best with no new low, or the last epoch if that never happens.

    Under if (import.meta.main), for the curve [2.4, 1.1, 0.8, 0.82, 0.79, 0.81, 0.83, 0.9], print one line for each patience 1, 2 and 3:

    patience P: best epoch B, stop after epoch S

  4. Task 4 · interview question

    Interview: best-or-last.ts, which weights do you ship?

    Mangala asks: "Training stopped. Is the model you have now the one to ship?" Fully fine-tune a fresh baseModel() with the given train on the first 60 training examples, lr 0.003, up to 30 epochs. In onEpoch, measure valLoss on the first 64 validation examples; on a new low, keep the epoch, the loss and a copy of the weights (saveModel); once 2 epochs pass with no new low, return true. Restore the copy with loadModel. Print:

    stopped after epoch S; best epoch B (val V)

    last weights: val loss L, accuracy A%, LEAK C/N

    best weights: val loss L, accuracy A%, LEAK C/N

    V and L with 3 decimals; the last two lines measure each model on the whole validation set (valLoss, and the given evaluate). Never the test set.

What you will learn

  • Validation loss: the training loss, measured on held-out examples, with no step taken.
  • Overfitting: what it looks like on two curves, and why it happens.
  • Early stopping: stop when the validation loss has not improved for a few epochs (patience).
  • Keep the best, not the last: a copy of the weights at the best epoch, restored at the end.
  • Loss is not accuracy: what a rising validation loss means, and what it does not.

More in Inside

Build "Overfitting and early stopping" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.