Fine-tuning basics · Chapter 15 of 20
Learning rate and epochs
By the end of this chapter you can say what the learning rate, a step, an epoch and a batch are, count the steps of any run, sweep the learning rate and read what the losses say, explain what clipping does and what it cannot do, and choose a learning rate the way a careful engineer does: on the validation set. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Inside
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.
Why you need this
Every training run in this lab so far came with its settings given: LoRA at learning rate 0.004, full fine-tuning at 0.001, two or three epochs. Irfan chose them. Now he wants you to be able to choose them, and he has a reason.
Last week a teammate trained a LoRA model at 0.02 "because faster is better". It tagged fewer than half of the test messages right. The same job at 0.004 tags 86.6%. Nothing else was different. Mangala does not care which number was typed into the job; she cares that one of them misses leaks. Rashmi cares that every failed run is ₹ and minutes thrown away.
The lab's API takes these settings too: n_epochs, batch_size and learning_rate_multiplier (it multiplies the API's base rate, 0.004). So does every provider's fine-tuning job. A setting you do not understand is a setting you leave on auto and hope.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
steps.py: epochs and steps
Write steps_per_epoch(n, batch) (how many batches one pass over n examples takes; a last, smaller batch is a step too), total_steps(n, batch, epochs) and auto_batch(n) (the lab API's batch size when a job sets none: 8 under 200 examples, 16 from 200 on).
Under if __name__ == "__main__":, for the runs (120 examples, 3 epochs), (200, 2), (650, 4) and (5000, 2), with the batch from auto_batch, print one line each:
N examples, E epochs: batch B, S steps an epoch, T steps, each example seen E times
- Task 2
sweep.py: a learning-rate sweep
Write sweep(lrs, train_set, val, *, epochs, clip) returning one dict {"lr", "train", "val"} per learning rate, in order: each a fresh base_model() trained with the given train_lora(model, train_set, epochs=epochs, lr=lr, clip=clip) (rank 8). train is the mean loss of the last epoch's steps (on_epoch gives them); val is the loss on val after training, over every trained id of the set, with no step taken (write it as val_loss(model, set_, adapters=None) in this file: task 3 imports it).
Under if __name__ == "__main__":, sweep 0.0003, 0.001, 0.004, 0.02 and 0.1 on the first 200 training examples and the first 64 validation examples, 2 epochs, clip 0, and print one line per rate (3 decimals):
lr L: train T, val V
- Task 3
clip.py: clipping
On the first 200 training examples, train LoRA four times, each on a fresh base_model(), 2 epochs: lr 0.004 with clip 0 then clip 1, and lr 0.1 with clip 0 then clip 1. After each, on the first 64 validation examples, work out the validation loss (val_loss from sweep.py) and the accuracy (the given evaluate, with the adapters). Print, in that order:
lr L, clip C: val V, accuracy A% (val with 3 decimals, accuracy with 1)
- Task 4 · interview question
Interview: choose_lr.py, which rate do you ship?
Irfan asks: "Your sweep is done. Which rate do we use, and why that one?" Write choose_lr(rows) for rows of dicts {"lr", "train", "val", maybe "test"}: return the lr of the row with the lowest val (the first on a tie), never choosing by train or by test.
Under if __name__ == "__main__":, choose from the starter's table and print one line:
chosen: lr L
What you will learn
- Steps, epochs and batches: what each means and how to count them.
- The learning rate: how far each step moves the numbers; what too low and too high look like in the losses.
- A sweep: one run per rate, the same in every other way, read side by side.
- Clipping: a limit on the gradient's length; what it rescues and what it does not.
- Choosing on validation: never on the training loss, never on the test set.
More in Inside
Build "Learning rate and epochs" yourself
Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.