Fine-tuning basics (TypeScript) · Chapter 15 of 20
Learning rate and epochs
By the end of this chapter you can say what the learning rate, a step, an epoch and a batch are, count the steps of any run, sweep the learning rate and read what the losses say, explain what clipping does and what it cannot do, and choose a learning rate the way a careful engineer does: on the validation set. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Inside
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Every training run in this lab so far came with its settings given: LoRA at learning rate 0.004, full fine-tuning at 0.001, two or three epochs. Irfan chose them. Now he wants you to be able to choose them, and he has a reason.
Last week a teammate trained a LoRA model at 0.02 "because faster is better". It tagged fewer than half of the test messages right. The same job at 0.004 tags 86.6%. Nothing else was different. Mangala does not care which number was typed into the job; she cares that one of them misses leaks. Rashmi cares that every failed run is ₹ and minutes thrown away.
The lab's API takes these settings too: n_epochs, batch_size and learning_rate_multiplier (it multiplies the API's base rate, 0.004). So does every provider's fine-tuning job. A setting you do not understand is a setting you leave on auto and hope.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
steps.ts: epochs and steps
Write and export stepsPerEpoch(n, batch) (how many batches one pass over n examples takes; a last, smaller batch is a step too), totalSteps(n, batch, epochs) and autoBatch(n) (the lab API's batch size when a job sets none: 8 under 200 examples, 16 from 200 on).
Under if (import.meta.main), for the runs (120 examples, 3 epochs), (200, 2), (650, 4) and (5000, 2), with the batch from autoBatch, print one line each:
N examples, E epochs: batch B, S steps an epoch, T steps, each example seen E times
- Task 2
sweep.ts: a learning-rate sweep
Write and export sweep(lrs, train, val, { epochs, clip }) returning one { lr, train, val } per learning rate, in order: each a fresh baseModel() trained with the given trainLora(model, train, { epochs, lr, clip }) (rank 8). train is the mean loss of the last epoch's steps (onEpoch gives them); val is the loss on val after training, over every trained id of the set, with no step taken.
Under if (import.meta.main), sweep 0.0003, 0.001, 0.004, 0.02 and 0.1 on the first 200 training examples and the first 64 validation examples, 2 epochs, clip 0, and print one line per rate (3 decimals):
lr L: train T, val V
- Task 3
clip.ts: clipping
On the first 200 training examples, train LoRA four times, each on a fresh baseModel(), 2 epochs: lr 0.004 with clip 0 then clip 1, and lr 0.1 with clip 0 then clip 1. After each, on the first 64 validation examples, work out the validation loss and the accuracy (the given evaluate, with the adapters). Print, in that order:
lr L, clip C: val V, accuracy A% (val with 3 decimals, accuracy with 1)
- Task 4 · interview question
Interview: choose-lr.ts, which rate do you ship?
Irfan asks: "Your sweep is done. Which rate do we use, and why that one?" Write and export chooseLr(rows) for rows { lr, train, val, test? }: return the lr of the row with the lowest val (the first on a tie), never choosing by train or by test.
Under if (import.meta.main), choose from the starter's table and print one line:
chosen: lr L
What you will learn
- Steps, epochs and batches: what each means and how to count them.
- The learning rate: how far each step moves the numbers; what too low and too high look like in the losses.
- A sweep: one run per rate, the same in every other way, read side by side.
- Clipping: a limit on the gradient's length; what it rescues and what it does not.
- Choosing on validation: never on the training loss, never on the test set.
More in Inside
Build "Learning rate and epochs" yourself
Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Fine-tuning basics (TypeScript) are free. This one comes with the all labs plan.