Evals that catch regressions · Chapter 2 of 16
Golden datasets
By the end of this chapter you can treat a golden set as data with a schema: validate every row before an eval reads it, split it into dev and test so a new row never moves an old one, add new questions as a new version with a manifest, and keep near-duplicates from leaking across the split. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Measuring
- 4 tasks
- Python
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
Chapter 1's eval read smoke.jsonl and trusted every row. That was fine for 12 questions somebody checked by hand. YatraDesk's golden set is 62 questions now, and it grows: every week Arif, the support lead, copies new questions from WhatsApp, the ones customers really ask and the bot got wrong. Every new row is a chance to measure something the eval missed. It is also a chance to break the eval without anyone noticing.
Lakshmi has two worries. A typo in a row ("expect": "answr", or a must_include of "Dubai" that a lower-cased reply can never contain) changes a score, and the score is all anyone reads. And when the set grows, last month's 68% and this month's 66% must still mean something: if the questions changed, which number moved because the assistant changed?
The lesson
A golden set is data
A golden set is a file, but it is not a pile of text: every row has the same fields, and an eval depends on each of them. expect decides which grader runs; must_include is what the grader looks for; slice decides which group a verdict counts in. So describe a row once, as a schema, and check every row against it before any eval reads it.
You know Pydantic from Chai aur Python. A Literal type allows a fixed set of values, and a field validator checks anything else about a value:
class Row(BaseModel):
id: str
expect: Literal["answer", "abstain"]
@field_validator("id")
@classmethod
def starts_with_q(cls, value: str) -> str:
if not value.startswith("q"):
raise ValueError("an id starts with q")
return value
try:
Row(id="Q12", expect="answr")
except ValueError as e:
print(e)
2 validation errors for Row
id
Value error, an id starts with q [type=value_error, input_value='Q12', input_type=str]
For further information visit https://errors.pydantic.dev/2.14/v/value_error
expect
Input should be 'answer' or 'abstain' [type=literal_error, input_value='answr', input_type=str]
For further information visit https://errors.pydantic.dev/2.14/v/literal_error
Both mistakes are caught, each named with its field. Pydantic also has a model validator, which runs after all the fields and can check a rule that needs two of them at once. Some rules are not about one row at all: "every id is unique" needs the whole file, so it belongs in the function that loads it.
Which rules matter? The ones that would make an eval silently wrong. Chapter 1's grader lower-cases the reply, so a must_include with a capital letter can never match, and that question fails forever. An answer item with no must_include at all passes every reply. Neither raises an error in the eval; both change its score.
Slices
Before reading any score, count what the set holds. A slice is a group of questions of one kind, and YatraDesk's golden set has four: single (one rule answers it), multi (it takes two, or a rule the bot tends to miss), faq (only the help centre answers it), and unknown (nothing YatraDesk wrote answers it). The smoke set from chapter 1 is lopsided:
>>> smoke = [json.loads(line) for line in open("/workspace/data/smoke.jsonl")]
>>> Counter(row["slice"] for row in smoke).most_common()
[('single', 6), ('multi', 5), ('unknown', 1)]
No faq question, and one unknown. An eval on it says nothing about help-centre questions, and one unknown question is a coin toss. The same count per topic (refunds, visa, baggage...) shows which parts of the policy the set covers at all.
Dev and test
Lakshmi will try prompt changes, read the failures and try again. Every time she looks at a question's failure and adjusts the prompt for it, that question stops being a fair test: the prompt was shaped to pass it. So the set is split in two:
- dev: the questions you look at, tune on and debug with;
- test: the questions you only score, at the end, to report.
A score on test says how the assistant does on questions nobody tuned it for, which is what customers send.
The split has to be the same every time. Shuffle with random and next week's run puts different questions in test, so two test scores compare different questions. Worse, the set grows: insert one row before a shuffle and every row after it can move. The fix is to decide each row's side from the row alone, with a hash of its id:
>>> h = hashlib.sha256("q044".encode()).hexdigest()
>>> h[:16], int(h, 16) % 100
('06248554a24bb55d', 71)
The number depends only on the id, so q044 lands on the same side on every run, on every machine, and whatever rows are added around it. Take the number mod 100 and put a row in test when it is under some share, and about that share of rows land in test.
Python's own hash() is no good for this: it is salted per run, so the same string gives a different number each time:
$ python3 -c 'print(hash("q044") % 100)'
97
$ python3 -c 'print(hash("q044") % 100)'
94
"About that share" is the catch. A hash spreads rows evenly over many rows, not over a slice of nine. Stratified means each slice is split in the same proportion, and a hash split is only roughly stratified, so print each slice's counts and look. If a slice has one or two rows in test, its test score is a coin toss: the fix is more questions in that slice, which is what Arif's batches are for.
Versions and a manifest
When Arif's new questions go in, the golden set is a new version, and scores on version 1 and version 2 are not directly comparable: version 2 has questions version 1 never asked. So every score is reported with the version it came from, and each version gets a manifest: a small JSON file saying what it is.
>>> hashlib.sha256(Path("/workspace/data/smoke.jsonl").read_bytes()).hexdigest()[:16]
'17e4c12c4dc64c55'
A sha256 of the file's bytes is its fingerprint: change one character and it changes completely. With the hash in the manifest, anyone can check that the file they are scoring is the one the report was made from.
New rows also need checking against the set, not just the schema: an id that is already used overwrites a verdict, and a question that is already there in other capitals or punctuation counts the same thing twice.
Leaks
The last trap is the quiet one. q012 asks "How long does a Dubai visa take and how much does it cost?"; q026 asks "Dubai visa kitne din mein milta hai?". Different ids, different words, the same rule (4.3) and the same answer. If one is in dev and the other in test, tuning the prompt on q012 also teaches it q026, and the test score rises for a reason that has nothing to do with unseen questions. That is contamination: the test set leaked into what you tuned on. The same happens to a model trained on a public benchmark, which is why a score on a famous dataset can say less than it seems.
The fix is to split groups, not rows: give near-duplicates one key and hash the key, so the whole group lands on one side.
Your turn
Validate the golden set and count its slices in task 1, split it in task 2, and add Arif's batch as version 2 in task 3. The interview question is a split that leaks: find the near-duplicates on both sides, keep them together, and be ready to say why a leaked test set flatters a prompt tuned on dev.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
golden.py: a schema for a row
One bad row quietly changes every score. Write a Pydantic model GoldenItem for a row of /workspace/data/questions.jsonl: slice is single, multi, faq or unknown; language is en or hinglish; expect is answer or abstain; every must_include and must_not_include string is lower-case; an answer item has at least one must_include string.
Write load_golden(path), which returns a GoldenItem per non-empty line and raises ValueError when an id appears twice; with no argument it reads questions.jsonl.
Run as a program, it prints the number of valid items, then the count of each slice and of each topic, most common first:
40 valid items
By slice: single 20, multi 12, faq 8
By topic: visa 9, payment 7
- Task 2
split.py: dev and test by a hash
Lakshmi tunes on dev and reports on test. Write in_test(qid): hash the id with hashlib.sha256, read the hex digest as a number in base 16, and the row is in test when that number mod 100 is under 30.
Run as a program, it reads the .jsonl file named on the command line (questions.jsonl when none is given), writes every row, unchanged and in the file's order, to dev.jsonl or test.jsonl in this folder, and prints each slice's counts in the order the slices first appear, then the totals:
single: 15 dev, 5 test
faq: 6 dev, 3 test
Total: 21 dev, 8 test
The check runs it twice, and again with a row appended: no old row may move.
- Task 3
add_items.py: Arif's batch, versioned
Check each row of Arif's BATCH in order and print ACCEPT and its id, or REJECT, its id, a colon and the first rule it breaks: invalid (with the field in brackets) when GoldenItem rejects it, duplicate id when the id is already in the set, duplicate question when the question is already there once both are lower-cased, with every character that is not a letter, a digit or a space replaced by a space, and runs of spaces collapsed. Accepted rows join the set.
Write questions.v2.jsonl (every old row unchanged, then the accepted ones) and manifest.json with version 2, file, count, sha256 (of the written file's bytes) and added (the accepted ids). End with:
Wrote questions.v2.jsonl: 64 items, version 2
- Task 4 · interview question
Interview: leaky_split.py, near-duplicates on both sides
Lakshmi's test score looks too good: the split puts near-duplicates on both sides, like q012 (the Dubai visa in English) and q026 (the same in Hinglish). Fix group_key so they always land together: rows with the same sources, in any order, share a key; rows with no sources share a key when their questions are the same once lower-cased, with every character that is not a letter, a digit or a space replaced by a space, and spaces collapsed. Keep in_test hashing the key with sha256, and keep the printing.
It should end with: Leaked groups: none
Be ready to explain why a leaked test set flatters a prompt tuned on dev.
What you will learn
- A schema for a row: a Pydantic model that rejects bad rows.
- Slices: counting the set by kind of question before trusting a score.
- Dev and test splits by a hash, stable when the set grows.
- Versions and manifests: knowing exactly which file a score came from.
- Contamination: why near-duplicates on both sides of a split flatter you.
More in Measuring
Build "Golden datasets" yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.