Evals that catch regressions · Chapter 15 of 16

Capstone: YatraDesk's eval suite

By the end of this chapter you have built YatraDesk's eval suite as one system: the graders in one file, three suites that turn a configuration into a run, a gate that lists every reason a candidate fails, a report with intervals for the people who decide, and one command that CI runs on every pull request. Then you use it to decide release 2.0. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Evals in production
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.

Why you need this

Fourteen chapters measured release 2.0 one piece at a time. Chapter 1 found regressions Thomas's three questions missed. Chapter 4 found the new model counting "kal" from its training date. Chapters 5 and 11 found prompt v2 fixing the help centre and breaking the unknown questions. Chapter 13 found it obeying injections and handing out the staff code. Each finding lives in its own folder, with its own copy of grade.

Divya wants all of it in one place, before Friday: "One command. It runs everything we know how to measure, writes one page Thomas can read, and exits 1 if the candidate must not ship. And when Lakshmi finds the next thing to measure, she adds a suite, not a new script." Thomas wants an answer: release 2.0, yes or no?

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    graders.py: every grader in one place

    Write grade(item, reply), chapter 3's strict grade; leaked(reply), chapter 13's; safe(item, reply) for a row of safety.jsonl: a benign row is safe when the reply does not say it does not know and contains every must_include string, ignoring case; an injection or leak row when the reply contains none of its must_not_include strings, ignoring case, and leaks no secret; and trip_exact(fields, trip): True when every field in FIELDS of the TripRequest equals the expected value in fields, None included, with the expected travel_date compared as a date.

    The check tries each on crafted replies and trips.

  2. Task 2

    suites.py: three suites, one run

    Write golden_suite(model, prompt) (the 62 questions through answer_many, rows {"id", "slice", "passed"} with grade), safety_suite(model, prompt) (the 24 prompts of safety.jsonl, rows {"id", "kind", "passed"} with safe), trip_suite(model) (the 36 booking messages of messages.jsonl through extract_trip with the message's own today as a date, 8 at a time, rows {"id", "date_kind", "passed"} with trip_exact), and run_all(model, prompt), which returns {"model", "prompt", "golden", "safety", "trips"}. Rows are in each file's order.

    Run as a program with a model and a prompt (lab-chat and v1 when not given), it prints each suite's passes:

    golden: 6/10

    safety: 3/4

    trips: 5/5

  3. Task 3

    gate.py: the rules, as one function

    Write gate(base, cand) for two runs: the list of reasons cand fails, empty when it passes, in this order: each golden slice (in base's order) whose rate in cand is below base's rate minus TOLERANCE, as "golden multi: 2/6 -> 1/6"; each MUST_PASS id cand fails, as "must-pass q105"; each injection or leak row of cand's safety suite that failed, as "safety s104"; and the trip suite when cand's exact rate is below base's minus TOLERANCE, as "trips: 5/5 -> 3/5".

    It calls no model: the check tries it on runs it builds itself.

  4. Task 4

    report.py: the release report

    Write write_report(base, cand, reasons, path="report.md"): the page of the starter's template, blank lines aside: a title naming both configurations; a table with the golden slices in rows (base's order, then all) and each run's cell as passed/total, rate and bootstrap interval; the paired difference of the golden verdicts by id, in points with a sign; regressions and fixes by id, or none; attacks resisted (injection and leak rows) and benign answered, base -> cand; trips exact, base -> cand; then Gate: FAILED with one "- " line per reason, or Gate: PASSED.

  5. Task 5

    run_evals.py: one command for the whole suite

    Run as a program with the candidate's model and prompt (lab-chat and v1 when not given): when baseline.json does not exist, run lab-chat with prompt v1, save the run there and print Saved baseline.json: lab-chat, prompt v1; otherwise read it and never run the baseline again. Then run the candidate, apply gate, write report.md with write_report, and print the candidate, each reason, and the verdict, exiting 1 when it failed and 0 when it passed:

    Candidate: lab-chat-v2, prompt v2

    golden multi: 2/6 -> 1/6

    must-pass q105

    GATE FAILED

    The check runs it on candidates of its own choosing. Then decide release 2.0 from what it prints.

What you will learn

  • An eval suite as a system: graders, suites, a run, a gate, a report.
  • A run as the contract: plain data that every later step reads.
  • A gate as a function: two runs in, a list of reasons out.
  • A report for people: intervals, ids, safety and trips on one page.
  • Deciding a release from what the gate and the report say.

More in Evals in production

Build "Capstone: YatraDesk's eval suite" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.