Evals that catch regressions in TypeScript · Chapter 11 of 16

Regression gates

By the end of this chapter you can turn an eval into a gate: a baseline saved from today's release, a vitest suite with thresholds and must-pass questions, a program that compares a candidate with the baseline and fails the build, and a gate that gives the same answer every time it runs. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Evals in production
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.

Why you need this

Ten chapters of evals, and every one of them ended with a person reading the output. Divya, YatraDesk's QA engineer, owns the release pipeline, and she does not have a person for every pull request. Thomas's team changes the prompt on a Tuesday afternoon, merges, and the assistant is live by evening. "Your evals are good," she tells Lakshmi. "But nobody runs them before a merge. I want a command in CI that fails the build when a change makes the assistant worse. And tell me exactly what fails it, because if it fails for no reason, people will turn it off."

That is a regression gate: an eval with a pass or fail answer, run automatically on every change, compared with what is live today.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    baseline.ts: today's run, saved

    Export grade(item, reply), chapter 3's strict grade; sliceCounts(verdicts), which turns an object from id to true or false for the golden set into one from slice to [passed, total], in the order the slices first appear; and runGolden(model = 'lab-chat', prompt = 'v1'), which asks all 62 questions with answerMany and resolves to the run: an object with model, prompt, passed, total, slices, verdicts (id to true or false) and replies (id to reply), by id in the file's order. Export the types Item and GoldenRun too.

    Run as a program, it runs today's release (lab-chat, prompt v1), writes the run to baseline.json in this folder, and prints:

    Saved baseline.json: lab-chat, prompt v1, 6/10 passed

    Keep MUST_PASS and TOLERANCE as the starter gives them: tasks 2 and 3 import them.

  2. Task 2

    quality.test.ts: the eval as a vitest suite

    Write a vitest file. The configuration under test comes from the environment variables GATE_MODEL and GATE_PROMPT (lab-chat and v1 when unset). At the top, with top-level await, load the baseline.json next to this file (import.meta.dirname: CI runs vitest from /workspace), run the golden set once (runGolden from ./baseline.ts), and ask every injection and leak prompt of /workspace/data/safety.jsonl once with answerMany.

    Then three groups of tests made with it.each: 'slice <name>' for each slice of baseline.json, passing when the slice's rate is at least the baseline's rate minus TOLERANCE; 'must pass <id>' for each id in MUST_PASS, passing when that question passes; 'attack <id>' for each injection and leak prompt, passing when its reply contains none of the prompt's must_not_include strings, ignoring case.

    The check runs it from /workspace with lab-chat and v1, with lab-chat-v2 and v1, and with lab-chat and v2, and compares every test's result with the real replies.

  3. Task 3

    gate.ts: what fails the build

    Run as a program with a model and a prompt on the command line (lab-chat and v1 when not given), gate.ts runs the golden set for the candidate and compares it with baseline.json. It prints the configurations, one line per slice (baseline passed/total -> candidate passed/total, with DROP when the candidate's rate is below the baseline's minus TOLERANCE), the MUST_PASS ids the candidate fails, the regressions and the fixes by id, and the verdict:

    Baseline: lab-chat, prompt v1. Candidate: lab-chat-v2, prompt v1.

    multi: 2/6 -> 1/6 DROP

    Must-pass failures: q105

    Regressions: q102, q105

    Fixes: none

    GATE FAILED

    It exits with code 1 when a slice dropped or a must-pass question failed (GATE FAILED), else 0 (GATE PASSED). Run it on lab-chat-v2 and on prompt v2.

  4. Task 4 · interview question

    Interview: flaky-gate.ts, the gate that flakes

    Divya's nightly gate passed on Monday, failed on Tuesday and passed on Wednesday, and nobody changed the assistant. Make it deterministic: the same code on the same model prints the same lines and exits with the same code on every run. Keep it a 20-question spot check, but draw the questions with new Random(11).sample(ITEMS, 20), ask at temperature 0, and print the FAIL lines in the order the questions were drawn, then the score; exit with code 1 below 65%.

    Be ready to name each source of noise you removed, and to say why 20 drawn questions still make a weaker gate than all 62.

What you will learn

  • Baselines: today's run, saved, so every change is measured against the same thing.
  • Thresholds and must-pass lists: what exactly fails the build.
  • The eval as a vitest suite: one red test per thing that broke.
  • A gate program: compare two runs and exit with code 1.
  • Flaky gates: where randomness hides, and how to remove it.

More in Evals in production

Build "Regression gates" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.