Fine-tuning basics · Chapter 12 of 20

LoRA: two small matrices

By the end of this chapter you can say what LoRA is and why most fine-tuning services use it, compute a layer with a LoRA adapter on it by hand, work out the gradients that train the adapter and test them by nudging, and count how many numbers LoRA trains for any rank and any model. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Inside
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.

Why you need this

In chapter 11 you fine-tuned all of chota: every one of its 45,152 numbers moved. Trained that way on the whole training set, chota tags 97.3% of the test set right and catches 18 of 18 leaks. It worked. Irfan is still not happy with it as a habit, and Rashmi has a question.

Rashmi's question first: "If Mangala wants a second tagger next month, for complaints, do we keep a second copy of the whole model?" With full fine-tuning, yes. Every job you run makes a new model the size of the old one. chota is 241,993 bytes, so that is fine here. A real model has billions of numbers, so each copy is many gigabytes.

Irfan's worry is memory. To train a number, the training loop keeps three more numbers beside it: its gradient, and Adam's two running averages. On a model of seven billion numbers that is four copies of the model in memory at once. That is why most fine-tuning services, and the lab's own API, do not train every number. Look at the oldest events of the ready model's job, Irfan's ftjob-c0ffee01tagsv1ready000:

Validating training file: file-ready0train and validation file: file-ready0valid
Files validated, moving job to queued state
Fine-tuning job started: LoRA rank 8 on chota-base, 189 steps
Step 1/189: training loss=9.8047

It trained LoRA adapters, rank 8, on chota's attention weights. This chapter opens that up.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    lora_forward.py: x W + scale (x A) B

    Write lora_forward(x, W, A, B, scale): a new array y = x W + scale × (x A) B, for x (n × d_in), W (d_in × d_out), A (d_in × r), B (r × d_out). Do not change the inputs.

    Under if __name__ == "__main__":, put an adapter on chota-base's h0.wq with new_adapters(model, ["h0.wq"], 8, 16, 0), run forward(model, [prompt_ids("gas cha vaas yetoy")], adapters), and take cache.layers[0] (its "n1" is the x h0.wq reads; chota's own y is its "qh", put back side by side as the starter shows). Print three lines:

    B = 0: largest change from x W: D (D: the largest |difference| between lora_forward(n1, W, A, B, scale) and n1 @ W)

    B random: largest change from x W: D (after filling B with Rng(5).normal() × 0.1, row by row, and running forward again; 4 decimals)

    largest difference from chota's own q: D

  2. Task 2

    lora_backward.py: the gradients of A and B

    Write lora_backward(x, A, B, scale, dy) returning (dA, dB): dB = scale × (x A)ᵀ dy (r × d_out) and dA = scale × xᵀ (dy Bᵀ) (d_in × r), where dy is the loss's gradient for y. New arrays; do not change the inputs.

    Under if __name__ == "__main__":, make x, W, A, B and dy with the starter's given Rng(7) block, take loss = (y * dy).sum() for y = lora_forward(x, W, A, B, 2), and compare your formulas with nudging (each number up and down by 1e-6, (loss up − loss down) / 2e-6). Print with 6 decimals:

    dA[0, 0]: formula F, nudging N

    dB[1, 1]: formula F, nudging N

  3. Task 3

    lora_count.py: how many numbers does LoRA train?

    Write lora_numbers(d_in, d_out, rank, weights): how many numbers LoRA trains with one adapter (A d_in × rank, B rank × d_out) on each of weights matrices of d_in × d_out.

    Under if __name__ == "__main__":, print one line for each rank 1, 2, 4, 16 and 32 for chota (8 weights of 32 × 32), then one for a 4096-wide model with 32 layers, rank 8 on wq and wv (64 weights of 4096 × 4096):

    chota, rank R on all attention: N numbers, P% of 45,152

    a 4096-wide model, 32 layers, rank 8 on wq and wv: N numbers, P% of the T in those weights

    Counts with commas (f"{n:,}"); shares with one decimal for chota, two for the big model. T: how many numbers those 64 weights hold.

  4. Task 4 · interview question

    Interview: why_b_zero.py, why does B start at zero?

    Irfan asks: "new_adapters fills A with random numbers and B with zeros. Why not both random? Show me." Read the first 50 lines of typeahead-val.txt, each on its own as [USER, *encode(line)], with chota-base and no adapters; then with new_adapters(model, attention_names(model.config), 8, 16, 0); then with those adapters' every B filled with Rng(1).normal() × 0.1 (adapters in the dict's order, each B row by row). Compare each with chota-base and print:

    B = 0: largest change C, top guess changed at K of N positions

    B random: largest change C, top guess changed at K of N positions

    C: the largest |difference| of any score from chota-base's (2 decimals); the top guess is the id with the highest score (np.argmax: the first on a tie); N: every position of the 50 lines.

What you will learn

  • Where LoRA goes: the attention weights, eight 32 × 32 matrices in chota.
  • The idea: keep a weight W frozen and learn its change as two thin matrices, A and B.
  • The forward: x W + scale (x A) B, and why B starts at zero.
  • The gradients: what a gradient of a weight is, the two formulas for A and B, and how to test any gradient by nudging.
  • Rank and alpha: the two settings every LoRA library asks for, and how many numbers they train.

More in Inside

Build "LoRA: two small matrices" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.