Evals that catch regressions · Chapter 5 of 16
Metrics and reports
By the end of this chapter you can turn an eval run into numbers you can defend: pass rates per slice, a 95% interval for each by bootstrap, a paired comparison of two versions with its own interval, a report anyone can read, and an honest answer to "is this difference real?". Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Measuring
- 5 tasks
- Python
- In the all labs plan
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.
Why you need this
Lakshmi ran the golden set on both models and posted the two scores in YatraDesk's team chat: lab-chat 68%, lab-chat-v2 60%. Thomas replied within a minute: "8 points on 62 questions. Is that real, or is it noise? Last week you told me a small change in the score was nothing."
It is the right question. A pass rate on 62 questions is an estimate: 62 other questions of the same kinds would give a slightly different number. Before a difference decides a release, you need to know how much the number could move by chance, and whether the difference is bigger than that. You also need to show where it comes from, because "60%" does not say which questions fail.
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
per_slice.py: the run as data, per slice
Write grade(item, reply), chapter 3's strict grade; golden_run(model="lab-chat", prompt="v1"), which asks all 62 questions with answer_many and returns one dict per question in the file's order, {"id", "slice", "topic", "passed"}; and pass_rates(run, key), which returns {value: (passed, total)} for each value of that key, in the order the values first appear.
Run as a program, it evaluates the model and prompt named on the command line (lab-chat-v2 and v1 when none are given) and prints the overall rate, then each slice, then each topic:
lab-chat-v2, prompt v1: 6/10 (60%)
slice single: 4/4 (100%)
topic visa: 3/5 (60%)
- Task 2
bootstrap.py: a 95% interval by resampling
Write bootstrap(passed, resamples=1000, seed=7): with rng = random.Random(seed), each resample is rng.choices(passed, k=len(passed)) and its rate the share of True; sort the rates and return the ones at index round(resamples * 0.025) and round(resamples * 0.975) - 1. Write normal(passed): p plus or minus 1.96 times the square root of p(1 - p)/n, kept between 0 and 1.
Run as a program, it runs the golden set with the model on the command line (lab-chat when none is given) and prints one line for all 62 questions and one for the 12 of smoke.jsonl:
62 questions: 50% (bootstrap 37% to 63%, normal 38% to 62%)
12 questions: 50% (bootstrap 25% to 75%, normal 22% to 78%)
- Task 3
paired.py: the difference, question by question
Write paired(a, b, resamples=1000, seed=7) for two verdict dicts with the same ids: diff is b's pass rate minus a's; with ids = list(a) and rng = random.Random(seed), each resample is rng.choices(ids, k=len(ids)) and its difference the mean of b[id] - a[id] over the sample; low and high are at the same indexes as in task 2. Return (diff, low, high).
Run as a program, it runs the golden set on lab-chat and lab-chat-v2 (prompt v1) and prints both scores, the difference and its interval in points with a sign, the regressions and the fixes:
lab-chat: 6/10 (60%)
lab-chat-v2: 5/10 (50%)
Difference: -10 points, 95% interval -30 to +10
Regressions: q102, q107
Fixes: q105
- Task 4
report.py: one page for release 2.0
Run the golden set for today (lab-chat, prompt v1) and release 2.0 (lab-chat-v2, prompt v2), and write report.md: a title, one sentence saying what was measured, and a markdown table with the configurations in columns and the slices in rows (in the order they first appear, then a row called all). Each cell is passed/total, the rate, and the bootstrap interval of that slice in brackets:
| Slice | today (lab-chat, v1) | release 2.0 (lab-chat-v2, v2) |
|---|---|---|
| single | 20/20 100% [100%, 100%] | 19/20 95% [85%, 100%] |
Then the lines Regressions: and Fixes: with the ids, in the file's order, or none. Print: Wrote report.md
- Task 5 · interview question
Interview: thomas_claim.py, two more questions
Prompt v2 passes two more questions than v1 on lab-chat, so Thomas says it is better. Change his script to decide from the paired interval (paired from paired.py, v1 as a, v2 as b) and print exactly:
prompt v1: 40/62
prompt v2: 41/62
Difference: +2 points, 95% interval -5 to +8
Changed: 9 questions (5 fixes, 4 regressions)
Verdict: no clear difference
The verdict is better when the whole interval is above 0, worse when it is all below 0, and no clear difference otherwise.
Be ready to explain what "two more questions" hides, and what Thomas should read before he decides.
What you will learn
- A run as data: the verdicts, kept per question, sliced per group.
- Confidence intervals by bootstrap, written as a loop.
- The normal approximation, and how many questions an eval needs.
- Paired comparisons: why resampling questions beats two intervals.
- A report: models in columns, slices in rows, intervals, regressions.
More in Measuring
Build "Metrics and reports" yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.