Fine-tuning basics · Chapter 9 of 20

Checkpoints, failures and cancelling

By the end of this chapter you can read a job's checkpoints and use any of them as a model, find out exactly why a job failed, stop a job you no longer want without losing what it made, and choose which checkpoint to ship the right way: by validation, never by test. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: A fine-tuning job
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.

Why you need this

Chapter 8's jobs all succeeded, and the model you used was always the last one a job made. Real jobs are messier. Mangala's next export has one broken line in it, and the job fails after you have gone for tea. Irfan starts a twenty-epoch job by mistake and wants it stopped, but not thrown away. And the last model a job makes is not always its best: training can go too far.

follow from chapter 8 prints whatever happens, but it cannot tell you which of a job's models to keep, and it does nothing about a job that went wrong.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    checkpoints.py: a job's checkpoints

    Write checkpoints(job_id): the job's checkpoints from client.fine_tuning.jobs.checkpoints.list(job_id, limit=100), oldest first (the API lists them newest first), each as a dict {"step", "epoch", "model", "valid_loss", "token_accuracy"} from step_number, metrics.epoch, fine_tuned_model_checkpoint, metrics.full_valid_loss and metrics.full_valid_mean_token_accuracy.

    Under if __name__ == "__main__":, print one line for each checkpoint of "ftjob-c0ffee01tagsv1ready000":

    step S (epoch E): valid loss L, token accuracy A%, MODEL (L with 4 decimals, A with 1)

  2. Task 2

    failed_job.py: why did the job fail?

    Write train_on(path): upload the file (upload, from jobs.py, given), start a job of "chota-base" on it with nothing else set, wait for it to finish (wait_for) and return a dict {"status", "error"}: the finished job's status and job.error.message, or None when it has no error.

    Under if __name__ == "__main__":, write the first 20 train examples to /tmp/broken.jsonl with line 5's reply given the role "bot", call train_on on it and print:

    STATUS: ERROR

  3. Task 3

    cancel.py: cancel a job, keep what it made

    Write cancel_after_checkpoint(job_id): check the running job's checkpoints every 500 ms until it has one, cancel it with client.fine_tuning.jobs.cancel(job_id), retrieve it, and return a dict {"status", "checkpoints"} (its status and how many checkpoints it has now).

    Under if __name__ == "__main__":, start a job of "chota-base" on train.jsonl and val.jsonl with suffix "long" and n_epochs 10, call cancel_after_checkpoint on it, print STATUS with N checkpoint(s), then ask each checkpoint's model "gas ki smell aa rahi hai" and print MODEL: REPLY for each, oldest first.

  4. Task 4 · interview question

    Interview: pick.py, which checkpoint ships?

    Irfan: "Eight epochs at three times the learning rate, on only 200 examples. Which checkpoint do we ship?" Write the first 200 train examples to /tmp/first-200.jsonl, start a job of "chota-base" on it with val.jsonl as the validation file, suffix "pick", n_epochs 8 and learning_rate_multiplier 3, and wait for it. Print each checkpoint oldest first as epoch E: valid loss L (4 decimals), then chosen: epoch E, MODEL for the one with the lowest valid_loss, then score only that model on examples("test") with score_model:

    test: accuracy A%, format F%, LEAK C/L

What you will learn

  • Checkpoints: the model saved at the end of each epoch, with its validation numbers, and usable like any model.
  • Failed jobs: where a job says why it failed, and the line it stopped at.
  • Cancelling: stopping a job, and what you keep.
  • Choosing a checkpoint: by validation loss, and why the test set must play no part.

More in A fine-tuning job

Build "Checkpoints, failures and cancelling" yourself

Chulha, a Nagpur startup, tags WhatsApp messages for 40 LPG gas agencies, and a missed gas leak is not an option. You fine-tune chota, a tiny transformer, until it beats the best prompt while forgetting little. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Fine-tuning basics are free. This one comes with the all labs plan.