Fine-tuning basics (TypeScript) · Chapter 13 of 20

Training with LoRA

By the end of this chapter you can fine-tune chota with LoRA, using the training loop you already have, and compare it fairly with full fine-tuning: accuracy, leaks caught, numbers trained, seconds and the size of what you ship. You can also choose a rank and the weights to put adapters on, from measurements rather than habit. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Inside
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 12 built LoRA's two matrices. Now Irfan wants them trained, and he wants to know what LoRA costs. His own job on the lab's API was LoRA rank 8, and it tags 95.2% of the test set right. Full fine-tuning, chapter 11's loop on the whole training set, gets 97.3%. "Is LoRA just a bit worse?" asks Rashmi. "Then why does every service use it?"

Mangala has a narrower question. Full fine-tuning caught all 18 test leaks; Irfan's LoRA model caught 14. Four kitchens.

Your training loop from chapter 11 trains every number. To answer them you need a loop that trains only the adapters, and a fair comparison: same data, same validation set, same scoring, only the training different.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    train-lora.ts: the training loop with LoRA

    Write and export trainLora(model, set, options) returning { adapters, losses }. options: { epochs, lr, rank?, alpha?, kinds?, batch?, seed?, clip? } with the defaults rank 8, alpha 2 × rank, kinds all four attention weights (ATTENTION), batch 16, seed 0, clip 1 (0 means none).

    It is the given train with three changes: new adapters before the loop (newAdapters(model, attentionNames(model.config, kinds), rank, alpha, seed)); each step's gradients from lossAndGrads(model, batch, adapters, 'adapters'), named NAME.A and NAME.B; and Adam steps on adapterParams(adapters) only. Shuffle and batch exactly as train does. chota's own weights must not change. losses: every step's loss, in order.

    Under if (import.meta.main), train LoRA on the first 200 training examples, 2 epochs, lr 0.004, and print two lines:

    N steps: loss F at the first, L at the last (4 decimals)

    trained N numbers in K adapters

  2. Task 2

    compare.ts: full fine-tuning or LoRA?

    Train chota-base two ways on the first 400 training examples, each from a fresh baseModel(): full with train(model, set, { epochs: 3, lr: 0.001 }), LoRA with trainLora(model, set, { epochs: 3, lr: 0.004 }). Score each on the whole validation set with the given evaluate (pass the adapters for LoRA) and time each training. Print, full first:

    full: trained N numbers, accuracy A%, format F%, LEAK C/L, file B bytes, S s

    lora: trained N numbers, accuracy A%, format F%, LEAK C/L, file B bytes, S s

    N: countParams of what trained (model.params, or the adapters). The middle is line(score). B: saveModel(model).length for full, saveAdapters(adapters).length for LoRA. S: seconds, one decimal.

  3. Task 3

    ranks.ts: ranks and targets

    Train LoRA four times, each on a fresh baseModel(), on the first 400 training examples, 3 epochs, lr 0.004 (alpha 2 × rank): rank 2, rank 4 and rank 8 on all four attention weights, then rank 8 on wq and wv only. Score each on the whole validation set and print one line per run, in that order:

    rank R on KINDS: trained N numbers, accuracy A%, format F%, LEAK C/L

    (KINDS joined with commas: wq,wk,wv,wo or wq,wv)

  4. Task 4 · interview question

    Interview: frozen.ts, what did each way change?

    Rashmi asks: "Next month Mangala wants a second tagger, for complaints. Do we keep two copies of chota?" On the first 200 training examples, train one fresh chota-base fully (train, 2 epochs, lr 0.001) and another with LoRA (trainLora, 2 epochs, lr 0.004). Compare each model's numbers, weight by weight, with a third fresh chota-base, and print:

    full fine-tuning changed X of 45152 numbers

    LoRA changed Y of 45152 numbers

    files: adapters A bytes, the model M bytes (saveAdapters(adapters).length, saveModel(model).length)

    cylinder book karna hai: with the adapters "R1", without "R2"

    R1 and R2: the LoRA model's reply, generate(model, promptIds(message), 4, adapters), then with no adapters, each written with JSON.stringify.

What you will learn

  • The LoRA loop: the same loop with three changes: adapters, their gradients, and steps on them only.
  • Frozen means frozen: chota-base does not change at all, and how to prove it.
  • A fair comparison: full fine-tuning against LoRA on accuracy, LEAK, numbers trained, seconds and file size.
  • Rank and targets: how many numbers the adapters have, and where they sit, both change the result.
  • What LoRA really saves, on chota and on a big model.

More in Inside

Build "Training with LoRA" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.