Evals that catch regressions in TypeScript · Chapter 2 of 16

Golden datasets

By the end of this chapter you can treat a golden set as data with a schema: validate every row with zod before an eval reads it, split it into dev and test so a new row never moves an old one, add new questions as a new version with a manifest, and keep near-duplicates from leaking across the split. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Measuring
  • 4 tasks
  • TypeScript
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Chapter 1's eval read smoke.jsonl and trusted every row. That was fine for 12 questions somebody checked by hand. YatraDesk's golden set is 62 questions now, and it grows: every week Arif, the support lead, copies new questions from WhatsApp, the ones customers really ask and the bot got wrong. Every new row is a chance to measure something the eval missed. It is also a chance to break the eval without anyone noticing.

Lakshmi has two worries. A typo in a row ("expect": "answr", or a must_include of "Dubai" that a lower-cased reply can never contain) changes a score, and the score is all anyone reads. And when the set grows, last month's 68% and this month's 66% must still mean something: if the questions changed, which number moved because the assistant changed?

The lesson

A golden set is data

A golden set is a file, but it is not a pile of text: every row has the same fields, and an eval depends on each of them. expect decides which grader runs; must_include is what the grader looks for; slice decides which group a verdict counts in. So describe a row once, as a schema, and check every row against it before any eval reads it.

In chapter 1 you wrote an Item type. A type helps while you write code, but it is erased before the code runs: JSON.parse hands back whatever is in the file, and TypeScript believes you. A row with "expect": "answr" type-checks perfectly. To check the data itself, at run time, you need a schema, and in TypeScript that is usually zod.

A zod schema is a value you build from small pieces. z.enum allows a fixed set of values, .refine adds any rule you can write as a function, and z.strictObject is an object that also rejects keys it does not name:

import { z } from 'zod';

const Row = z.strictObject({
  id: z.string().refine((id) => id.startsWith('q'), 'an id starts with q'),
  expect: z.enum(['answer', 'abstain']),
});

const result = Row.safeParse({ id: 'Q12', expect: 'answr', slise: 'single' });
if (!result.success) {
  for (const issue of result.error.issues) console.log(issue.path, issue.message);
}
[ 'id' ] an id starts with q
[ 'expect' ] Invalid option: expected one of "answer"|"abstain"
[] Unrecognized key: "slise"

Three mistakes, three issues, each with a path that says where. safeParse never throws: it returns { success: true, data } or { success: false, error }, which suits code that wants to say why a row was refused. parse returns the data or throws a ZodError with the same issues, which suits code that should stop at the first bad row.

Two more pieces matter here:

  • Strict objects. A plain z.object quietly drops keys it does not know. A typo like must_includes then vanishes, and the real must_include is reported missing, or worse, a field you meant to add never reaches the eval. z.strictObject turns an unknown key into an issue (the one with the empty path above: it belongs to the whole row).
  • A rule over two fields. A .refine on the whole object sees every field at once, so it can say "an answer item needs at least one must_include string". Give it a path and the issue names the field you choose.

The schema is also the type: type GoldenItem = z.infer<typeof GoldenItem> gives you the row's type without writing it twice, so the two can never disagree.

Which rules matter? The ones that would make an eval silently wrong. Chapter 1's grader lower-cases the reply, so a must_include with a capital letter can never match, and that question fails forever. An answer item with no must_include at all passes every reply. Neither raises an error in the eval; both change its score.

One rule is not about one row at all: every id is unique. zod checks one value at a time and never sees the other rows, so this rule lives in the code that loads the file: keep a Set of the ids seen so far, and throw when one comes back. A duplicate id is not harmless: in an object from id to verdict, the second row's verdict overwrites the first's, and the score counts one question fewer than the file holds.

Slices

Before reading any score, count what the set holds. A slice is a group of questions of one kind, and YatraDesk's golden set has four: single (one rule answers it), multi (it takes two, or a rule the bot tends to miss), faq (only the help centre answers it), and unknown (nothing YatraDesk wrote answers it). The smoke set from chapter 1 is lopsided:

const slices = new Map<string, number>();
for (const row of smoke) slices.set(row.slice, (slices.get(row.slice) ?? 0) + 1);
console.log(slices);
Map(3) { 'single' => 6, 'multi' => 5, 'unknown' => 1 }

No faq question, and one unknown. An eval on it says nothing about help-centre questions, and one unknown question is a coin toss. The same count per topic (refunds, visa, baggage...) shows which parts of the policy the set covers at all. A Map keeps its keys in the order they first appeared, and sort is stable, so sorting its entries by count keeps equal counts in the file's order.

Dev and test

Lakshmi will try prompt changes, read the failures and try again. Every time she looks at a question's failure and adjusts the prompt for it, that question stops being a fair test: the prompt was shaped to pass it. So the set is split in two:

  • dev: the questions you look at, tune on and debug with;
  • test: the questions you only score, at the end, to report.

A score on test says how the assistant does on questions nobody tuned it for, which is what customers send.

The split has to be the same every time. Pick rows with Math.random() and next week's run puts different questions in test, so two test scores compare different questions. A seeded shuffle is the same every run, but the set grows: shuffle 62 rows with new Random(7) and send the first 30% to test, then add Arif's three rows and shuffle again with the same seed, and 24 of the 62 old rows switch sides. The fix is to decide each row's side from the row alone, with a hash of its id:

import { createHash } from 'node:crypto';

const hex = createHash('sha256').update('q044').digest('hex');
console.log(hex.slice(0, 16));
console.log(parseInt(hex.slice(0, 8), 16) % 100);
06248554a24bb55d
24

The number depends only on the id, so q044 lands on the same side on every run, on every machine, and whatever rows are added around it. Take the number mod 100 and put a row in test when it is under some share, and about that share of rows land in test.

Why only the first 8 hex digits? The whole digest is 64 hex digits, a 256-bit number. A JavaScript number holds about 16 significant digits, so parseInt(hex, 16) gives 2.778403799822999e+75, and its last digits are rounded away: % 100 of it gives 16, a number that has nothing to do with the hash. Eight hex digits are a 32-bit number, read exactly, and still spread evenly. (BigInt('0x' + hex) % 100n reads all of it; the first 8 digits are simpler and just as good for a split.)

"About that share" is the catch. A hash spreads rows evenly over many rows, not over a slice of nine. Stratified means each slice is split in the same proportion, and a hash split is only roughly stratified, so print each slice's counts and look. If a slice has one or two rows in test, its test score is a coin toss: the fix is more questions in that slice, which is what Arif's batches are for.

Versions and a manifest

When Arif's new questions go in, the golden set is a new version, and scores on version 1 and version 2 are not directly comparable: version 2 has questions version 1 never asked. So every score is reported with the version it came from, and each version gets a manifest: a small JSON file saying what it is.

const bytes = readFileSync('/workspace/data/smoke.jsonl');
console.log(createHash('sha256').update(bytes).digest('hex').slice(0, 16));
17e4c12c4dc64c55

A sha256 of the file's bytes is its fingerprint: change one character and it changes completely. With the hash in the manifest, anyone can check that the file they are scoring is the one the report was made from. Read the bytes back after writing the file: hashing the rows you meant to write is not the same as hashing the file on disk.

New rows also need checking against the set, not just the schema: an id that is already used overwrites a verdict, and a question that is already there in other capitals or punctuation counts the same thing twice. safeParse is the right tool for a batch: it never throws, so you can give each refused row its reason, the field of its first issue, and carry on with the next.

Leaks

The last trap is the quiet one. q012 asks "How long does a Dubai visa take and how much does it cost?"; q026 asks "Dubai visa kitne din mein milta hai?". Different ids, different words, the same rule (4.3) and the same answer. If one is in dev and the other in test, tuning the prompt on q012 also teaches it q026, and the test score rises for a reason that has nothing to do with unseen questions. That is contamination: the test set leaked into what you tuned on. The same happens to a model trained on a public benchmark, which is why a score on a famous dataset can say less than it seems.

The fix is to split groups, not rows: give near-duplicates one key and hash the key, so the whole group lands on one side.

Your turn

Validate the golden set and count its slices in task 1, split it in task 2, and add Arif's batch as version 2 in task 3. The interview question is a split that leaks: find the near-duplicates on both sides, keep them together, and be ready to say why a leaked test set flatters a prompt tuned on dev.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    golden.ts: a schema for a row

    One bad row quietly changes every score. Export a zod schema GoldenItem for a row of /workspace/data/questions.jsonl, a strict object (an unknown key is an error): slice is single, multi, faq or unknown; language is en or hinglish; expect is answer or abstain; every must_include and must_not_include string is lower-case; an answer item has at least one must_include string.

    Export loadGolden(path), which returns one parsed GoldenItem per non-empty line and throws when a row is invalid or an id appears twice; with no argument it reads questions.jsonl.

    Run as a program, it prints the number of valid items, then the count of each slice and of each topic, most common first, equal counts in the file's order:

    40 valid items

    By slice: single 20, multi 12, faq 8

    By topic: visa 9, payment 7

  2. Task 2

    split.ts: dev and test by a hash

    Lakshmi tunes on dev and reports on test. Export inTest(qid): hash the id with createHash('sha256') from node:crypto, read the first 8 characters of the hex digest as a number in base 16, and the row is in test when that number mod 100 is under 30.

    Run as a program, it reads the .jsonl file named on the command line (questions.jsonl when none is given), writes every row, unchanged and in the file's order, to dev.jsonl or test.jsonl in this folder, and prints each slice's counts in the order the slices first appear, then the totals:

    single: 15 dev, 5 test

    faq: 6 dev, 3 test

    Total: 21 dev, 8 test

    The check imports inTest, runs the file twice, and again with a row appended: no old row may move.

  3. Task 3

    add-items.ts: Arif's batch, versioned

    Check each row of Arif's BATCH in order and print ACCEPT and its id, or REJECT, its id, a colon and the first rule it breaks: invalid (with the first issue's field in brackets) when GoldenItem.safeParse rejects it, duplicate id when the id is already in the set, duplicate question when the question is already there once both are lower-cased, with every character that is not a letter, a digit or a space replaced by a space, and runs of spaces collapsed. Accepted rows join the set.

    Write questions.v2.jsonl (every old row unchanged, then the accepted ones) and manifest.json with version 2, file, count, sha256 (of the written file's bytes) and added (the accepted ids). End with:

    Wrote questions.v2.jsonl: 64 items, version 2

  4. Task 4 · interview question

    Interview: leaky-split.ts, near-duplicates on both sides

    Lakshmi's test score looks too good: the split puts near-duplicates on both sides, like q012 (the Dubai visa in English) and q026 (the same in Hinglish). Fix groupKey so they always land together: rows with the same sources, in any order, share a key; rows with no sources share a key when their questions are the same once lower-cased, with every character that is not a letter, a digit or a space replaced by a space, and spaces collapsed. Keep inTest exported and hashing the key with sha256, and keep the printing.

    It should end with: Leaked groups: none

    Be ready to explain why a leaked test set flatters a prompt tuned on dev.

What you will learn

  • A schema for a row: a zod schema that rejects bad rows, and the one rule zod cannot check.
  • Slices: counting the set by kind of question before trusting a score.
  • Dev and test splits by a hash, stable when the set grows.
  • Versions and manifests: knowing exactly which file a score came from.
  • Contamination: why near-duplicates on both sides of a split flatter you.

More in Measuring

Build "Golden datasets" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.