Fine-tuning basics · Chapter 16 of 20

Catastrophic forgetting

By the end of this chapter you can measure what fine-tuning costs a model in what it could already do, explain why a model forgets, read why LoRA forgot more than full fine-tuning on chota, and fix it with replay: mixing the old job back into training. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Inside
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.

Why you need this

chota-base had a job before you met it. Chulha trained it on two years of customers' messages to guess the next word as an agent types a reply: the inbox's type-ahead. That is still running, on chota-base, beside the tagger.

Rashmi has an idea to save money: "We fine-tuned chota to tag. Can the same model also do the type-ahead? Then we run one model, not two." Irfan does not guess. He measures the type-ahead on the fine-tuned models, and it is bad. Type "booking kiye 4 din" into the inbox with the fully fine-tuned model behind it, and it suggests:

ho gaye abhi AAYEGA

With the LoRA model, adapters on:

aaj delivery DELIVERY

chota-base suggested ho gaye abhi tak. Training on a new job made the model worse at its old one. That is catastrophic forgetting, and every fine-tuned model has some: a big model fine-tuned on a narrow task can lose some of its general skill the same way, so it is worth measuring every time.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    typeahead.py: chota's first job, as a number

    Write typeahead_row(text): Row(ids, train) (chota's Row) with ids [USER, *encode(text), END] and train False for <user>, True for every id after it; and typeahead_loss(model, texts, adapters=None): the loss of loss_of(forward(model, batch.ids, adapters)[0], batch) for batch = make_batch of the texts' rows.

    Under if __name__ == "__main__":, print chota-base's loss on lines 256 to 511 of typeahead-val.txt (lines("typeahead-val.txt")[256:512]), 4 decimals:

    chota-base, type-ahead loss on 256 messages: L

  2. Task 2

    forgetting.py: before and after

    On the first 400 training examples, train one fresh chota-base fully (train, 3 epochs, lr 0.001) and another with LoRA (train_lora, 3 epochs, lr 0.004). Print the type-ahead loss on the 256 messages of task 1 (4 decimals) and, for the trained models, line(evaluate(...)) on the whole validation set:

    chota-base: type-ahead T

    full: type-ahead T, accuracy A%, format F%, LEAK C/L

    LoRA, adapters on: type-ahead T, accuracy A%, format F%, LEAK C/L

    LoRA, adapters off: type-ahead T

  3. Task 3

    replay.py: mix the old job back in

    Write train_with_replay(model, set_, *, epochs, lr, replay, batch=16, seed=0, clip=1.0) returning {"losses": [...]}: full fine-tuning exactly like the given train, except that after taking each batch's examples you add replay rows typeahead_row(corpus[rng.int(len(corpus))]) with the same rng, corpus = lines("corpus.txt"), and take the step on the mixed batch.

    Under if __name__ == "__main__":, train a fresh chota-base with replay 4 on the first 400 training examples, 4 epochs, lr 0.001, and print:

    replay 4, 4 epochs: type-ahead T, accuracy A%, format F%, LEAK C/L

    (type-ahead on the 256 messages of task 1, 4 decimals; the scores on the whole validation set)

  4. Task 4 · interview question

    Interview: why_lora_forgot.py, what is it guessing?

    Irfan asks: "LoRA trained 4,096 numbers and full fine-tuning all 45,152, yet LoRA's type-ahead is far worse. What is it guessing instead?" On the 256 messages of task 1, in one batch of typeahead rows, count the positions the type-ahead scores (weight 1) whose top guess (the highest score, the first on a tie) is one of the six codes ([encode(t)[0] for t in TAGS]). Do it for chota-base, for chota trained fully (3 epochs, lr 0.001) and with LoRA (3 epochs, lr 0.004, adapters on), both on the first 400 training examples:

    chota-base: top guess is a code at K of N positions

    full: top guess is a code at K of N positions

    LoRA: top guess is a code at K of N positions

What you will learn

  • Catastrophic forgetting: a model trained on a new job loses part of an old one, because nothing in the new training asks it to keep it.
  • How to measure it: the old job's loss on held-out text, before and after.
  • Full fine-tuning against LoRA: both forget; here LoRA forgets more, and its base stays untouched.
  • Why: what each way of training can change, and the rare code tokens.
  • Replay: mix a few rows of the old job into every batch.

More in Inside

Build "Catastrophic forgetting" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.