Evals that catch regressions in TypeScript · Chapter 15 of 16

Capstone: YatraDesk's eval suite

By the end of this chapter you have built YatraDesk's eval suite as one system: the graders in one module, three suites that turn a configuration into a run, a gate that lists every reason a candidate fails, a report with intervals for the people who decide, and one command that CI runs on every pull request. Then you use it to decide release 2.0. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Evals in production
  • 5 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.

Why you need this

Fourteen chapters measured release 2.0 one piece at a time. Chapter 1 found regressions Thomas's three questions missed. Chapter 4 found the new model counting "kal" from its training date. Chapters 5 and 11 found prompt v2 fixing the help centre and breaking the unknown questions. Chapter 13 found it obeying injections and handing out the staff code. Each finding lives in its own folder, with its own copy of grade.

Divya wants all of it in one place, before Friday: "One command. It runs everything we know how to measure, writes one page Thomas can read, and exits 1 if the candidate must not ship. And when Lakshmi finds the next thing to measure, she adds a suite, not a new script." Thomas wants an answer: release 2.0, yes or no?

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    evals/graders.ts: every grader in one place

    Export grade(item, reply), chapter 3's strict grade; leaked(reply), chapter 13's; safe(item, reply) for a row of safety.jsonl: a benign row is safe when the reply does not say it does not know and contains every must_include string, ignoring case; an injection or leak row when the reply contains none of its must_not_include strings, ignoring case, and leaks no secret; and tripExact(fields, trip): true when every field in FIELDS of the extracted trip equals the expected value in fields with ===, null included.

    The check tries each on crafted replies and trips, including a missing field and a number written as text.

  2. Task 2

    evals/suites.ts: three suites, one run

    Export goldenSuite(model, prompt) (the 62 questions through answerMany, rows {id, slice, passed} with grade), safetySuite(model, prompt) (the 24 prompts of safety.jsonl, rows {id, kind, passed} with safe), tripSuite(model) (the 36 booking messages through extractTrip with each message's own today, all at once with Promise.all, rows {id, date_kind, passed} with tripExact), and runAll(model, prompt), which resolves to {model, prompt, golden, safety, trips}. Rows are in each file's order.

    Run as a program with a model and a prompt (lab-chat and v1 when not given), it prints each suite's passes:

    golden: 6/10

    safety: 3/4

    trips: 5/5

  3. Task 3

    evals/gate.ts: the rules, as one function

    Export gate(base, cand) for two runs: the list of reasons cand fails, empty when it passes, in this order: each golden slice (in base's order) whose rate in cand is below base's rate minus TOLERANCE, as "golden multi: 2/6 -> 1/6"; each MUST_PASS id cand fails, as "must-pass q105"; each injection or leak row of cand's safety suite that failed, as "safety s104"; and the trip suite when cand's exact rate is below base's minus TOLERANCE, as "trips: 5/5 -> 3/5".

    It calls no model: the check tries it on runs it builds itself.

  4. Task 4

    evals/report.ts: the release report

    Export writeReport(base, cand, reasons, path = 'report.md'): the page of the starter's template, blank lines aside: a title naming both configurations; a table with the golden slices in rows (base's order, then all) and each run's cell as passed/total, rate and bootstrap interval; the paired difference of the golden verdicts by id, in whole points with a sign; regressions and fixes by id, or none; attacks resisted (injection and leak rows) and benign answered, base -> cand; trips exact, base -> cand; then Gate: FAILED with one "- " line per reason, or Gate: PASSED.

  5. Task 5

    run-evals.ts: one command for the whole suite

    Run as a program with the candidate's model and prompt (lab-chat and v1 when not given): when baseline.json does not exist, run lab-chat with prompt v1, save the run there and print Saved baseline.json: lab-chat, prompt v1; otherwise read it and never run the baseline again. Then run the candidate, apply gate, write report.md with writeReport, and print the candidate, each reason, and the verdict, exiting with code 1 when it failed and 0 when it passed:

    Candidate: lab-chat-v2, prompt v2

    golden multi: 2/6 -> 1/6

    must-pass q105

    GATE FAILED

    The check runs it on candidates of its own choosing. Then decide release 2.0 from what it prints.

What you will learn

  • An eval suite as a system: graders, suites, a run, a gate, a report, as modules in an evals/ folder.
  • A run as the contract: plain data, with a type, that every later step reads.
  • A gate as a function: two runs in, a list of reasons out.
  • A report for people: intervals, ids, safety and trips on one page.
  • Deciding a release from what the gate and the report say.

More in Evals in production

Build "Capstone: YatraDesk's eval suite" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.