Evals that catch regressions in TypeScript · Chapter 5 of 16

Metrics and reports

By the end of this chapter you can turn an eval run into numbers you can defend: pass rates per slice, a 95% interval for each by bootstrap, a paired comparison of two versions with its own interval, a report anyone can read, and an honest answer to "is this difference real?". Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Measuring
  • 5 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.

Why you need this

Lakshmi ran the golden set on both models and posted the two scores in YatraDesk's team chat: lab-chat 68%, lab-chat-v2 60%. Thomas replied within a minute: "8 points on 62 questions. Is that real, or is it noise? Last week you told me a small change in the score was nothing."

It is the right question. A pass rate on 62 questions is an estimate: 62 other questions of the same kinds would give a slightly different number. Before a difference decides a release, you need to know how much the number could move by chance, and whether the difference is bigger than that. You also need to show where it comes from, because "60%" does not say which questions fail.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    per-slice.ts: the run as data, per slice

    Export grade(item, reply), chapter 3's strict grade; goldenRun(model = 'lab-chat', prompt = 'v1'), which asks all 62 questions with answerMany and resolves to one row per question in the file's order, { id, slice, topic, passed }; and passRates(run, key), which returns a Map from each value of that key to [passed, total], in the order the values first appear.

    Run as a program, it evaluates the model and prompt named on the command line (lab-chat-v2 and v1 when none are given) and prints the overall rate, then each slice, then each topic, the percentages rounded to whole numbers:

    lab-chat-v2, prompt v1: 6/10 (60%)

    slice single: 4/4 (100%)

    topic visa: 3/5 (60%)

  2. Task 2

    bootstrap.ts: a 95% interval by resampling

    Export bootstrap(passed, resamples = 1000, seed = 7): with const rng = new Random(seed) from shared/random.ts, each resample is rng.choices(passed, passed.length) and its rate the share of true; sort the rates as numbers and return [the one at index Math.round(resamples * 0.025), the one at Math.round(resamples * 0.975) - 1]. Export normal(passed): p plus or minus 1.96 times the square root of p(1 - p)/n, kept between 0 and 1.

    Run as a program, it runs the golden set with the model on the command line (lab-chat when none is given) and prints one line for all 62 questions and one for the 12 of smoke.jsonl, whole percentages:

    62 questions: 50% (bootstrap 37% to 63%, normal 38% to 62%)

    12 questions: 50% (bootstrap 25% to 75%, normal 22% to 78%)

  3. Task 3

    paired.ts: the difference, question by question

    Export paired(a, b, resamples = 1000, seed = 7) for two verdict objects with the same ids (id to true or false): diff is b's pass rate minus a's; with ids = Object.keys(a) and rng = new Random(seed), each resample is rng.choices(ids, ids.length) and its difference the mean of b[id] minus a[id] over the sample, true counting 1 and false 0; low and high are at the same indexes as in task 2. Return [diff, low, high].

    Run as a program, it runs the golden set on lab-chat and lab-chat-v2 (prompt v1) and prints both scores, the difference and its interval in points (Math.round of the share times 100, with a + when above 0), the regressions and the fixes:

    lab-chat: 6/10 (60%)

    lab-chat-v2: 5/10 (50%)

    Difference: -10 points, 95% interval -30 to +10

    Regressions: q102, q107

    Fixes: q105

  4. Task 4

    report.ts: one page for release 2.0

    Run the golden set for today (lab-chat, prompt v1) and release 2.0 (lab-chat-v2, prompt v2), and write report.md: a title, one sentence saying what was measured, and a markdown table with the configurations in columns and the slices in rows (in the order they first appear, then a row called all). Each cell is passed/total, the rate, and the bootstrap interval of that slice in brackets, whole percentages:

    | Slice | today (lab-chat, v1) | release 2.0 (lab-chat-v2, v2) |

    |---|---|---|

    | single | 20/20 100% [100%, 100%] | 19/20 95% [85%, 100%] |

    Then the lines Regressions: and Fixes: with the ids, in the file's order, or none. Print: Wrote report.md

  5. Task 5 · interview question

    Interview: thomas-claim.ts, two more questions

    Prompt v2 passes more questions than v1 on lab-chat, so Thomas says it is better. Change his script to decide from the paired interval (paired from ./paired.ts, v1 as a, v2 as b) and print exactly, the numbers as in task 3:

    prompt v1: 40/62

    prompt v2: 41/62

    Difference: +2 points, 95% interval -5 to +8

    Changed: 9 questions (5 fixes, 4 regressions)

    Verdict: no clear difference

    The verdict is better when the whole interval is above 0, worse when it is all below 0, and no clear difference otherwise.

    Be ready to explain what "two more questions" hides, and what Thomas should read before he decides.

What you will learn

  • A run as data: the verdicts, kept per question, sliced per group.
  • Confidence intervals by bootstrap, written as a loop with a seed.
  • The normal approximation, and how many questions an eval needs.
  • Paired comparisons: why resampling questions beats two intervals.
  • A report: models in columns, slices in rows, intervals, regressions.

More in Measuring

Build "Metrics and reports" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.