Evals that catch regressions in TypeScript · Chapter 4 of 16

Structured graders

By the end of this chapter you can grade a model's structured output: check that a reply is valid JSON in the right schema, score a record field by field with partial credit, compare dates and missing values correctly, and slice a field's accuracy to find exactly which inputs fail. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Measuring
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.

Why you need this

Many of YatraDesk's WhatsApp messages are trip requests: "Looking for a package to Alappuzha, leaving on 16 Jan, 4 adults, 6 nights, budget ₹40,000 per person." The assistant's extractTrip reads each one into a record, and the record goes straight into the sales team's CRM, where someone calls the customer back with a quote. Nobody reads the original message again.

Thomas wants lab-chat-v2 there too: it is 40% cheaper, and extraction is "just reading". He tried it on one message:

console.log(await extractTrip(message, '2026-10-22'));
console.log(await extractTrip(message, '2026-10-22', { model: 'lab-chat-v2' }));
{
  destination: 'Alleppey',
  travel_date: '2027-01-16',
  travellers: 4,
  nights: 6,
  budget: 40000,
  name: 'Asha Rahman',
  phone: '9521306182'
}

Both models printed exactly that. That is a vibe check again, one message instead of three questions. The grader you need is different from chapter 3's, because the output is not a sentence: it is a record with seven fields, and each field can be right or wrong on its own.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    schema-check.ts: is it a TripRequest?

    Export problems(raw): [] when the raw reply is a valid TripRequest; ['not JSON'] when JSON.parse throws or gives no plain object (an array or null counts as no object); otherwise 'missing <field>' for each field that is not a key, 'wrong type <field>' once for each other field that TripRequest.safeParse has an issue with, and 'extra <key>' for each key that is not a field.

    Run as a program, send every booking message of /workspace/data/messages.jsonl with the SYSTEM prompt in the file, three ways: lab-chat in JSON mode (response_format { type: 'json_object' }), lab-chat-sloppy in JSON mode, and lab-chat with the schema (parse with zodResponseFormat(TripRequest, 'trip_request')). Print one line per way with the problems counted over all replies, most common first:

    lab-chat, JSON mode: 2/36 valid; missing name 30, extra city 4

    lab-chat-sloppy, JSON mode: 0/36 valid; not JSON 36

    lab-chat, schema: 36/36 valid

  2. Task 2

    field-scores.ts: field by field

    For each of the 36 booking messages, call extractTrip(text, today, { model }) from the yatradesk assistant with today from the message's today field, on lab-chat and lab-chat-v2. Compare each field with the message's fields (null where the message does not say).

    Print one line per field of TripRequest, in its order, with each model's right count, then the records with every field right:

    destination: lab-chat 10/10, lab-chat-v2 9/10

    travel_date: lab-chat 10/10, lab-chat-v2 10/10

    exact records: lab-chat 10/10, lab-chat-v2 9/10

  3. Task 3

    date-slices.ts: which dates go wrong?

    For the 36 booking messages, call extractTrip on lab-chat and lab-chat-v2 as in task 2 and compare travel_date with the expected date (a YYYY-MM-DD string, or null). Print one line per date_kind in the order the kinds first appear, then one line per model listing its wrong dates (what it returned, as text) with their counts, most common first, or none:

    dm: lab-chat 5/5, lab-chat-v2 5/5

    kal: lab-chat 3/3, lab-chat-v2 1/3

    Wrong dates from lab-chat: none

    Wrong dates from lab-chat-v2: 2026-10-25 x2

    Then read the wrong dates: what day does the model think today is?

  4. Task 4 · interview question

    Interview: lenient-grader.ts, the grader that skips

    Thomas's field grader says both models get almost every travel_date wrong, and it can never catch a model that invents a name or a phone number the customer never gave. Fix fieldOk: a value the message does not give (expected null) is right only when the model also gives null, and a travel_date is right when it is the same day. Keep fieldOk exported and the printing as it is.

    Be ready to explain why a grader that skips the empty fields rewards a model for inventing.

What you will learn

  • Schema validity: is the reply JSON at all, and is it a TripRequest?
  • Strict types: why "4" is not 4 when a CRM reads it.
  • Field-by-field scores, partial credit and exact records.
  • Null and dates: comparing what is missing and what is a day.
  • Slices by field: finding which kind of input breaks one field.

More in Measuring

Build "Structured graders" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.