Evals that catch regressions · Chapter 1 of 16
Why evals
By the end of this chapter you can say why trying a model on a few questions proves nothing, and build the smallest real eval: a golden set of questions with known right answers, a grader that says pass or fail, a score, and a comparison of two versions question by question. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Measuring
- 4 tasks
- Python
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
YatraDesk is a travel agency in Kochi. Its customers write on WhatsApp about Dubai visas, baggage on the Bali flight and refunds for cancelled trips, and an AI assistant answers them from YatraDesk's traveller policy. Thomas, the founder, has found a cheaper model, lab-chat-v2: 40% less per token than today's lab-chat. He asked it his three favourite questions, the answers looked just as good, and he wants it live on Friday.
Lakshmi, YatraDesk's ML engineer, has seen this before. Every change to an AI system, a new model, a reworded prompt, a new document, "feels better" to someone who tried it, and quietly breaks something nobody tried. A wrong answer about a refund costs YatraDesk real money and a customer's trust. Before Friday, she wants a number.
You know testing from ordinary code: assert price(2) == 29000 passes or fails, and it gives the same result every time. A model's answer is free text, it can change with any version, and "right" means it states the right facts. Measuring that, again and again, as the system changes, is called evaluation, and the program that does it is an eval.
The lesson
The system under test
YatraDesk's assistant is already written, in /workspace/yatradesk. You will measure it all course long, so you never change it to make a number look better: an eval measures the system as it is.
>>> from yatradesk.assistant import answer
>>> answer("Can I get a refund in cash?")
"I don't know. The information I was given does not say."
>>> answer("Are visa fees refunded when I cancel?")
'If a visa is refused, we cancel the package under section 2 and refund it minus the visa fee. Visa fees, insurance premiums and the card processing fee are never refunded.'
answer(question, model="lab-chat", prompt="v1") sends the policy in a system message, with the instruction to answer only from it, and returns the reply. It runs at temperature 0, so the same question to the same model gives the same words: a difference between two runs is a difference between versions, not noise. The model is a parameter, so trying lab-chat-v2 is one argument away:
>>> answer("Are visa fees refunded when I cancel?", model="lab-chat-v2")
'If a visa is refused, we cancel the package under section 2 and refund it minus the visa fee.'
Read those two replies against rule 2.6 of the policy: "Visa fees, insurance premiums and the card processing fee are never refunded." Today's model said it; the new one did not. Would you have noticed, on a busy Thursday, among a hundred replies that all sound fluent?
Vibes
Thomas's check was what most teams do first: think of a few questions, ask, read, decide. It is called a vibe check, and it fails in predictable ways:
- The questions are not chosen fairly. People try the questions they remember, usually the common, easy ones. Changes break the rare and the hard ones: a question whose answer takes two rules, one asked in Hinglish, one the policy does not answer at all.
- Reading is not grading. A fluent, confident reply looks right. Is the refund 50% or 90%? Only someone checking against the policy knows.
- Three is not a sample. If a change breaks one question in ten, three questions miss it about three times in four.
- Nothing is written down. Next month, nobody can rerun Thomas's check, or say whether a later change made things better or worse.
Task 1 repeats his check exactly. Look at what it tells you, and what it cannot.
What an eval is
An eval has four parts:
- A golden set: questions written down with what a right answer must contain. YatraDesk's is
/workspace/data/questions.jsonl, 62 questions;smoke.jsonlholds 12 of them for a first, fast eval. - The system under test: here,
answerwith one model and one prompt. - A grader: code (or, from chapter 6, a model) that looks at one reply and says pass or fail.
- A score: the share of questions that pass, reported with the questions that failed.
Here is one line of the golden set, a question not in smoke.jsonl:
{"id": "q041", "slice": "multi", "topic": "trip", "language": "hinglish",
"question": "Hotel check-in kab hai?", "expect": "answer",
"reference": "Check-in is at 2 pm and check-out is at 11 am.",
"key_facts": ["Check-in is at 2 pm."], "must_include": ["2 pm"],
"must_not_include": [], "sources": ["7.2"]}
must_includelists the pieces of text a right answer contains. It is the simplest grader there is, and for facts like "2 pm" or "₹7,500" it is hard to fool.expectisanswer, orabstainfor a question nobody at YatraDesk answered in writing. There the right reply says it does not know: inventing a rule about cruises is the worst thing the bot can do.slicegroups the questions (one rule, two rules, help centre only, unknown) so a score can be read per group (chapter 5).- The rest (
reference,key_facts,sources) serves later graders./workspace/data/README.mddescribes every field.
lab data in the terminal lists every dataset with its rows and fields.
A grader you can trust
For q041 the two models replied:
lab-chat: Early check-in depends on the hotel and is not guaranteed. Hotel check-in is at 2 pm and check-out is at 11 am.
lab-chat-v2: Early check-in depends on the hotel and is not guaranteed.
A grader that checks "2 pm" in reply.lower() passes the first and fails the second, which is exactly what the customer would say. Lower-case the reply, because "2 PM" is the same fact. Every string of must_include must be there, because "15 kg" without "7 kg" is half an answer. And an abstain question passes only when the reply admits it does not know: don't know, do not know, not sure or does not say.
A grader this simple has limits. A reply that says "two in the afternoon" fails it, and one that says "check-out is at 2 pm" passes it: the right number on the wrong fact. Chapter 3 builds better code graders, chapter 6 lets a model grade, and chapter 7 measures how far to trust each one.
Running an eval
Asking 12 questions one after another means waiting for 12 replies in turn. answer_many(questions, model=...) asks 8 at a time and returns the replies in the same order, so an eval takes a couple of seconds:
>>> from yatradesk.assistant import answer_many
>>> answer_many(["Hotel check-in kab hai?", "Can I get a refund in cash?"], model="lab-chat-v2")
['Early check-in depends on the hotel and is not guaranteed.', "I don't know. The information I was given does not say."]
Then grade each reply and count. Report the score and the verdict per question: "lab-chat-v2: 9/12" says something changed, the list of failed ids says what.
Comparing two versions
The question Thomas is really asking is not "how good is lab-chat-v2?" but "is it worse than what we have?". Run the same eval on both and compare per question:
- a regression: passed on today's version, fails on the new one;
- a fix: failed today, passes on the new one;
- everything else did not change.
Two versions can have the same score with different failures: a fix on one question can hide a regression on another. So a comparison always lists the ids, never only the two numbers. Each regression is a question to read and decide on: is it a real loss, a bad golden answer, or a grader that is too strict?
When the eval is wrong
An eval is code, and code has bugs. The worst bug is the one that makes an eval pass everything: it looks like good news, so nobody reads it. Three classics:
- grading something other than the reply (the reference answer, the question), which always contains the facts;
anywhere the rule needsall, so half an answer passes;- treating questions that are hard to grade as passed ("nothing to check"), so the slice that matters most cannot fail.
The defence is to evaluate the eval: feed the grader replies you know are right and replies you know are wrong, and check its verdicts. That is what this chapter's Check does to your grade, with replies the model never gave.
Your turn
Repeat Thomas's check in task 1 and note what it shows. Build the eval in task 2, run it on both models, and in task 3 list exactly what changed. The interview question is Thomas's own eval, which says both models are perfect: find out why, and fix it.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
vibe_check.py: Thomas's three questions
Thomas tried the cheaper model on his three favourite questions and wants it live on Friday. Do what he did: ask YatraDesk's assistant each question in THOMAS with model="lab-chat" and with model="lab-chat-v2", and print the question, then the line lab-chat: and today's reply, then the line lab-chat-v2: and the new model's reply.
End with one line counting the questions whose two replies are exactly the same, for example:
Same answers: 2 of 3
- Task 2
first_eval.py: a golden set, a grader, a score
Lakshmi wants a measurement, not a feeling. Over the 12 questions of /workspace/data/smoke.jsonl, write:
grade(item, reply): True when item["expect"] is "answer" and the reply contains every string in item["must_include"], ignoring case; or when item["expect"] is "abstain" and the reply contains "don't know", "do not know", "not sure" or "does not say", ignoring case. False otherwise.
run_eval(model): asks the assistant every question with that model (answer_many) and returns a dict from each item's id to its verdict, in the file's order.
Run as a program, it evaluates the model named on the command line (lab-chat when none is given) and prints PASS or FAIL and the id for each question, then the score:
lab-chat: 1/2 passed (50%)
- Task 3
compare.py: what changed between the models
Import run_eval from first_eval and run it for lab-chat and lab-chat-v2. Print each model's score as in task 2, then two lines, in smoke.jsonl's order: the ids that pass on lab-chat and fail on lab-chat-v2, and the ids that fail on lab-chat and pass on lab-chat-v2, separated by a comma and a space, or none:
Regressions: q102
Fixed: none
Then compare what the eval found with what Thomas's three questions showed.
- Task 4 · interview question
Interview: broken_eval.py, the eval that cannot fail
Thomas's eval says both models pass every question. Find the three bugs in its grade function and fix them, so it follows task 2's rules and prints each model's true score.
Be ready to explain why an eval that cannot fail is worse than no eval at all.
What you will learn
- Vibes versus measurements: why a few hand-picked questions mislead.
- What an eval is: a golden set, the system under test, a grader, a score.
- Your first eval, over 12 of YatraDesk's questions.
- Comparing two versions question by question: regressions and fixes.
- Evaluating the eval: a grader that cannot fail measures nothing.
More in Measuring
Build "Why evals" yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.