Fine-tuning basics (TypeScript) · Chapter 19 of 20

Capstone: a tagger Mangala trusts

By the end of this chapter you have trained Chulha's tagger from the given splits with everything this lab built (LoRA, oversampling, replay, early stopping that keeps the best epoch), saved the adapters with the story of how they were chosen, and proved on the held-out test set, once, that they clear Mangala's bar. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Proof
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Irfan's ready model has been tagging Chulha's messages in a trial. It is accurate and cheap, and it missed 4 of the 18 test leaks. Chapter 18 showed the best prompt catching every one of them. Chapter 17 showed the other cost: merged, the tagger wrecks the inbox's type-ahead.

Mangala writes down what she will accept before she sees any result:

Mangala's barWhy
accuracy above 90.9%better than the best prompt, or why fine-tune at all
every leak caught (18/18)a missed leak is a kitchen full of gas
format at least 98%the router compares exact codes
type-ahead loss at most 1.30chota-base is 1.1614: its first job must survive

Rashmi adds one line: "And I want to know how you got it, so we can do it again next quarter." Every piece you need was built in an earlier chapter. This one puts them in order and proves the result.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    oversample.ts: more leaks to learn from

    Export oversample(set, times): a new list with each example of set, in order, repeated times[its tag] times (1 when the tag is not named, 0 leaves it out), its copies right after it.

    Under if (import.meta.main), print for examples('train') and oversample(train, { LEAK: 2 }) one line per tag in TAGS order, then the total:

    LEAK 94 -> 188

    total 1000 -> N

  2. Task 2

    tagger.ts: train the tagger, keep the best, save it

    Export trainTagger(o) with o { train, val, epochs, patience, seed, times, replay, lr }, using the given pieces in lora.ts in this order: rows = oversample(o.train, o.times); run = startLora(o.seed); for each epoch from 1, train = trainEpoch(run, rows, o.replay, o.lr), val = valLoss(run, o.val), record { epoch, train, val }; a val lower than the best so far makes this the best epoch and snapshots copyAdapters(run.adapters); otherwise stop once o.patience epochs have passed since the best. Return { best, bestEpoch, epochs }.

    Under if (import.meta.main), run Irfan's recipe on train and val (epochs 6, patience 2, seed 0, times { LEAK: 2 }, replay 8, lr 0.004), print each epoch and the best, and save C.saveAdapters(best, { base: 'chota-base', rank: 8, alpha: 16, bestEpoch, epochs }) to tagger-adapters.json. Run it once in the terminal (about half a minute): the check reads the file, and also runs trainTagger on small sets of its own.

  3. Task 3

    report.ts: Mangala's bar

    Load tagger-adapters.json (C.loadAdapters) and measure it on chota-base with the given evaluate and typeaheadLoss (lora.ts) on examples('test'), once. Print:

    accuracy A%, format F%, LEAK C/L, type-ahead T (one decimal, and four for T)

    Mangala's bar: met (or: Mangala's bar: not met (accuracy, LEAK, format, type-ahead), only those that failed, in that order)

    The bar: accuracy above 90.9% (the best prompt), every leak caught, format at least 98%, type-ahead at most 1.30. The check measures your saved adapters against it too.

What you will learn

  • A recipe from parts: which piece fixes which number, measured one piece at a time.
  • Early stopping that keeps the best: a snapshot, not the last weights.
  • Adapters with their story: the file says how it was chosen.
  • One honest test: the test set touched once, on adapters already chosen.

More in Proof

Build "Capstone: a tagger Mangala trusts" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.