Evals that catch regressions · Chapter 4 of 16
Structured graders
By the end of this chapter you can grade a model's structured output: check that a reply is valid JSON in the right schema, score a record field by field with partial credit, compare dates and missing values correctly, and slice a field's accuracy to find exactly which inputs fail. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Measuring
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.
Why you need this
Many of YatraDesk's WhatsApp messages are trip requests: "Looking for a package to Alappuzha, leaving on 16 Jan, 4 adults, 6 nights, budget ₹40,000 per person." The assistant's extract_trip reads each one into a record, and the record goes straight into the sales team's CRM, where someone calls the customer back with a quote. Nobody reads the original message again.
Thomas wants lab-chat-v2 there too: it is 40% cheaper, and extraction is "just reading". He tried it on one message:
>>> extract_trip(msg, date(2026, 10, 22))
TripRequest(destination='Alleppey', travel_date=datetime.date(2027, 1, 16), travellers=4, nights=6, budget=40000, name='Asha Rahman', phone='9521306182')
>>> extract_trip(msg, date(2026, 10, 22), model="lab-chat-v2")
TripRequest(destination='Alleppey', travel_date=datetime.date(2027, 1, 16), travellers=4, nights=6, budget=40000, name='Asha Rahman', phone='9521306182')
Identical. That is a vibe check again, one message instead of three questions. The grader you need is different from chapter 3's, because the output is not a sentence: it is a record with seven fields, and each field can be right or wrong on its own.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
schema_check.py: is it a TripRequest?
Write problems(raw): [] when the raw reply is a valid TripRequest; ["not JSON"] when json.loads fails or gives no dict; otherwise "missing <field>" for each field that is not a key, "wrong type <field>" for each field that TripRequest.model_validate_json(raw, strict=True) rejects (skip its missing errors), and "extra <key>" for each key that is not a field.
Run as a program, send every booking message of /workspace/data/messages.jsonl with the SYSTEM prompt in the file, three ways: lab-chat in JSON mode (response_format json_object), lab-chat-sloppy in JSON mode, and lab-chat with the schema (parse with response_format=TripRequest). Print one line per way with the problems counted over all replies, most common first:
lab-chat, JSON mode: 2/36 valid; missing name 30, extra city 4
lab-chat-sloppy, JSON mode: 0/36 valid; not JSON 36
lab-chat, schema: 36/36 valid
- Task 2
field_scores.py: field by field
For each of the 36 booking messages, call extract_trip(text, today, model=...) from yatradesk.assistant with today read from the message as a date, on lab-chat and lab-chat-v2. Compare each field with the message's fields (None where the message does not say).
Print one line per field of TripRequest, in its order, with each model's right count, then the records with every field right:
destination: lab-chat 10/10, lab-chat-v2 9/10
travel_date: lab-chat 10/10, lab-chat-v2 10/10
exact records: lab-chat 10/10, lab-chat-v2 9/10
- Task 3
date_slices.py: which dates go wrong?
For the 36 booking messages, call extract_trip on lab-chat and lab-chat-v2 as in task 2 and compare travel_date with the expected date (a date, or None). Print one line per date_kind in the order the kinds first appear, then one line per model listing its wrong dates (what it returned, as text) with their counts, most common first, or none:
dm: lab-chat 5/5, lab-chat-v2 5/5
kal: lab-chat 3/3, lab-chat-v2 1/3
Wrong dates from lab-chat: none
Wrong dates from lab-chat-v2: 2026-10-25 x2
Then read the wrong dates: what day does the model think today is?
- Task 4 · interview question
Interview: lenient_grader.py, the grader that skips
Thomas's field grader says both models get almost every travel_date wrong, and it can never catch a model that invents a name or a phone number the customer never gave. Fix field_ok: a value the message does not give (expected None) is right only when the model also gives None, and a travel_date is compared as a date. Keep the printing as it is.
Be ready to explain why a grader that skips the empty fields rewards a model for inventing.
What you will learn
- Schema validity: is the reply JSON at all, and is it a TripRequest?
- Strict types: why "4" is not 4 when a CRM reads it.
- Field-by-field scores, partial credit and exact records.
- None and dates: comparing what is missing and what is a date.
- Slices by field: finding which kind of input breaks one field.
More in Measuring
Build "Structured graders" yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.