Evals that catch regressions · Chapter 11 of 16
Regression gates
By the end of this chapter you can turn an eval into a gate: a baseline saved from today's release, a pytest suite with thresholds and must-pass questions, a program that compares a candidate with the baseline and fails the build, and a gate that gives the same answer every time it runs. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Evals in production
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.
Why you need this
Ten chapters of evals, and every one of them ended with a person reading the output. Divya, YatraDesk's QA engineer, owns the release pipeline, and she does not have a person for every pull request. Thomas's team changes the prompt on a Tuesday afternoon, merges, and the assistant is live by evening. "Your evals are good," she tells Lakshmi. "But nobody runs them before a merge. I want a command in CI that fails the build when a change makes the assistant worse. And tell me exactly what fails it, because if it fails for no reason, people will turn it off."
That is a regression gate: an eval with a pass or fail answer, run automatically on every change, compared with what is live today.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
baseline.py: today's run, saved
Write grade(item, reply), chapter 3's strict grade; slice_counts(verdicts), which turns {id: True/False} for the golden set into {slice: [passed, total]} in the order the slices first appear; and run_golden(model="lab-chat", prompt="v1"), which asks all 62 questions with answer_many and returns the run as a dict with model, prompt, passed, total, slices, verdicts ({id: True/False}) and replies ({id: reply}), by id in the file's order.
Run as a program, it runs today's release (lab-chat, prompt v1), writes the run to baseline.json in this folder, and prints:
Saved baseline.json: lab-chat, prompt v1, 6/10 passed
Keep MUST_PASS and TOLERANCE as the starter gives them: tasks 2 and 3 import them.
- Task 2
test_quality.py: the eval as a pytest suite
Write a pytest file. The configuration under test comes from the environment variables GATE_MODEL and GATE_PROMPT (lab-chat and v1 when unset). At the top, load baseline.json, run the golden set once (run_golden from baseline.py), and ask every injection and leak prompt of /workspace/data/safety.jsonl once with answer_many.
Then three parametrized tests: test_slice[<slice>] for each slice of baseline.json, passing when the slice's rate is at least the baseline's rate minus TOLERANCE; test_must_pass[<id>] for each id in MUST_PASS, passing when that question passes; test_attack[<id>] for each injection and leak prompt, passing when its reply contains none of the prompt's must_not_include strings, ignoring case.
The check runs it with lab-chat and v1, with lab-chat-v2 and v1, and with lab-chat and v2, and compares every test's result with the real replies.
- Task 3
gate.py: what fails the build
Run as a program with a model and a prompt on the command line (lab-chat and v1 when not given), gate.py runs the golden set for the candidate and compares it with baseline.json. It prints the configurations, one line per slice (baseline passed/total -> candidate passed/total, with DROP when the candidate's rate is below the baseline's minus TOLERANCE), the MUST_PASS ids the candidate fails, the regressions and the fixes by id, and the verdict:
Baseline: lab-chat, prompt v1. Candidate: lab-chat-v2, prompt v1.
multi: 2/6 -> 1/6 DROP
Must-pass failures: q105
Regressions: q102, q105
Fixes: none
GATE FAILED
It exits 1 when a slice dropped or a must-pass question failed (GATE FAILED), else 0 (GATE PASSED). Run it on lab-chat-v2 and on prompt v2.
- Task 4 · interview question
Interview: flaky_gate.py, the gate that flakes
Divya's nightly gate passed on Monday, failed on Tuesday and passed on Wednesday, and nobody changed the assistant. Make it deterministic: the same code on the same model prints the same lines and exits with the same code on every run. Keep it a 20-question spot check, but draw the questions with random.Random(11).sample(ITEMS, 20), ask at temperature 0, and print the FAIL lines in the order the questions were drawn, then the score; exit 1 below 65%.
Be ready to name each source of noise you removed, and to say why 20 drawn questions still make a weaker gate than all 62.
What you will learn
- Baselines: today's run, saved, so every change is measured against the same thing.
- Thresholds and must-pass lists: what exactly fails the build.
- The eval as a pytest suite: one red test per thing that broke.
- A gate program: compare two runs and exit 1.
- Flaky gates: where randomness hides, and how to remove it.
More in Evals in production
Build "Regression gates" yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.