Fine-tuning basics (TypeScript) · Chapter 12 of 20
LoRA: two small matrices
By the end of this chapter you can say what LoRA is and why most fine-tuning services use it, compute a layer with a LoRA adapter on it by hand, work out the gradients that train the adapter and test them by nudging, and count how many numbers LoRA trains for any rank and any model. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Inside
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.
Why you need this
In chapter 11 you fine-tuned all of chota: every one of its 45,152 numbers moved. Trained that way on the whole training set, chota tags 97.3% of the test set right and catches 18 of 18 leaks. It worked. Irfan is still not happy with it as a habit, and Rashmi has a question.
Rashmi's question first: "If Mangala wants a second tagger next month, for complaints, do we keep a second copy of the whole model?" With full fine-tuning, yes. Every job you run makes a new model the size of the old one. chota is 241,993 bytes, so that is fine here. A real model has billions of numbers, so each copy is many gigabytes.
Irfan's worry is memory. To train a number, the training loop keeps three more numbers beside it: its gradient, and Adam's two running averages. On a model of seven billion numbers that is four copies of the model in memory at once. That is why most fine-tuning services, and the lab's own API, do not train every number. Look at the oldest events of the ready model's job, Irfan's ftjob-c0ffee01tagsv1ready000:
Validating training file: file-ready0train and validation file: file-ready0valid
Files validated, moving job to queued state
Fine-tuning job started: LoRA rank 8 on chota-base, 189 steps
Step 1/189: training loss=9.8047
It trained LoRA adapters, rank 8, on chota's attention weights. This chapter opens that up.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
lora-forward.ts: x W + scale (x A) B
Write and export loraForward(x, W, A, B, scale): a new Matrix y = x W + scale × (x A) B, for x (n × dIn), W (dIn × dOut), A (dIn × r), B (r × dOut). Do not change the inputs.
Under if (import.meta.main): put an adapter on chota-base's h0.wq with newAdapters(model, ['h0.wq'], 8, 16, 0), run forward(model, [promptIds('gas cha vaas yetoy')], adapters), and take cache.layers[0] (n1 is the x h0.wq reads, q is chota's own y). Print three lines:
B = 0: largest change from x W: D (D: the largest |difference| between loraForward(n1, W, A, B, scale) and matmul(n1, W))
B random: largest change from x W: D (after filling B with new Rng(5).normal() × 0.1 and running forward again; 4 decimals)
largest difference from chota's own q: D
- Task 2
lora-backward.ts: the gradients of A and B
Write and export loraBackward(x, A, B, scale, dy) returning { dA, dB }: dB = scale × (x A)ᵀ dy (r × dOut) and dA = scale × xᵀ (dy Bᵀ) (dIn × r), where dy is the loss's gradient for y. New matrices; do not change the inputs.
Under if (import.meta.main): make x, W, A, B and dy with the starter's given Rng(7) block, take loss = the sum of y[i] × dy[i] for y = loraForward(x, W, A, B, 2), and compare your formulas with nudging (each number up and down by 1e-6, (loss up − loss down) / 2e-6). Print with 6 decimals:
dA[0]: formula F, nudging N
dB[5]: formula F, nudging N
- Task 3
lora-count.ts: how many numbers does LoRA train?
Write and export loraNumbers(dIn, dOut, rank, weights): how many numbers LoRA trains with one adapter (A dIn × rank, B rank × dOut) on each of
weightsmatrices of dIn × dOut.Under if (import.meta.main), print one line for each rank 1, 2, 4, 16 and 32 for chota (8 weights of 32 × 32), then one for a 4096-wide model with 32 layers, rank 8 on wq and wv (64 weights of 4096 × 4096):
chota, rank R on all attention: N numbers, P% of 45,152
a 4096-wide model, 32 layers, rank 8 on wq and wv: N numbers, P% of the T in those weights
Counts with commas (toLocaleString('en-US')); shares with one decimal for chota, two for the big model. T: how many numbers those 64 weights hold.
- Task 4 · interview question
Interview: why-b-zero.ts, why does B start at zero?
Irfan asks: "newAdapters fills A with random numbers and B with zeros. Why not both random? Show me." Read the first 50 lines of typeahead-val.txt, each on its own as [USER, ...encode(line)], with chota-base and no adapters; then with newAdapters(model, attentionNames(model.config), 8, 16, 0); then with those adapters' every B filled with new Rng(1).normal() × 0.1 (adapters in Object.values order). Compare each with chota-base and print:
B = 0: largest change C, top guess changed at K of N positions
B random: largest change C, top guess changed at K of N positions
C: the largest |difference| of any score from chota-base's (2 decimals); the top guess is the id with the highest score (the first on a tie); N: every position of the 50 lines.
What you will learn
- Where LoRA goes: the attention weights, eight 32 × 32 matrices in chota.
- The idea: keep a weight W frozen and learn its change as two thin matrices, A and B.
- The forward: x W + scale (x A) B, and why B starts at zero.
- The gradients: what a gradient of a weight is, the two formulas for A and B, and how to test any gradient by nudging.
- Rank and alpha: the two settings every LoRA library asks for, and how many numbers they train.
More in Inside
Build "LoRA: two small matrices" yourself
Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.