Evals that catch regressions in TypeScript · Chapter 6 of 16

LLM as judge

By the end of this chapter you can grade free-text answers with a model in TypeScript: write a judge prompt over key facts, get a typed verdict back with zod, run the judge over a whole golden set, read where it disagrees with a code grader, and make sure a judge that is not judging can never pass in silence. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Measuring
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.

Why you need this

Arif brings new questions from WhatsApp every week, and more and more of them want answers with several parts. "Is insurance included, and what does it cover?" needs three facts: insurance up to 70 years, medical emergencies up to USD 50,000, lost baggage up to USD 500. A customer can be told those in a dozen wordings, and the assistant can get two right and forget the third.

The must_include lists of chapters 1 to 5 are hard to fool on numbers like "2 pm" and "₹7,500", but they say pass or fail and nothing else. They cannot say "two of three facts", and writing a list of strings for every way of saying a fact does not scale. Lakshmi proposes the tool most teams reach for next: let a model read the answer and grade it, against facts written down in the golden set. That model is called a judge, and this chapter builds one.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    judge.ts: a model as a grader

    Export Verdict, a zod schema with score (an integer), passed (a boolean), reason (a string) and missing (an array of strings), with its type, and judge(item, reply, { model = 'lab-chat' }), which resolves to the model's verdict on one reply to a question of /workspace/data/questions.jsonl.

    The system message tells the model to grade the answer against the key facts, to score it from 1 to 5 by how many key facts it states, and to reply in JSON with score, passed, reason and missing. The user message is exactly the line Question: and the question, the line Answer: and the reply, the line Key facts:, then one line per key fact starting with a dash and a space. Ask with client.chat.completions.parse, response_format: zodResponseFormat(Verdict, 'verdict') and temperature: 0, and throw an Error when the reply has no parsed verdict.

    Run as a program, it judges lab-chat-v2's replies to the questions in IDS, in the file's order, and prints one line each:

    q101: 3/5 FAIL (Covers 1 of 2 key facts. Missing: "7 kg of cabin baggage".)

  2. Task 2

    judge-eval.ts: the judge over the golden set

    Import judge from ./judge.ts. Ask the assistant the 52 questions of questions.jsonl whose expect is answer, with lab-chat and then with lab-chat-v2 (answerMany), judge every reply, and print one line per model with the judge's passes and its mean score, to two decimals:

    lab-chat: 8/10 passed, mean score 4.20

    lab-chat-v2: 6/10 passed, mean score 3.70

    Then compare with what the strict code grader said about the same models in chapter 3.

  3. Task 3

    disagree.ts: two graders, one set of replies

    Export strict(item, reply), chapter 3's strict grade for an answer item: every must_include string in the reply and none of the must_not_include strings, ignoring case. Run as a program, ask the 52 answerable questions with the model named on the command line (lab-chat when none is given), grade each reply with strict and with judge, and for each question where they disagree, in the file's order, print the id with both verdicts and the judge's score, then the reply indented by two spaces. End with the count:

    q101: code PASS, judge FAIL (1/5)

    Children under 5 travel free on a parent's lap.

    Disagree: 1 of 10

    Then read each one: is the judge wrong, the code grader, or the golden row?

  4. Task 4 · interview question

    Interview: silent-judge.ts, the judge that never judged

    Thomas's judge passes every answer of lab-chat-v2. Find out why, and fix it so it can never pass in silence: a zod Verdict whose score is an integer from 1 to 5 and whose reason is not empty; readVerdict(text), which returns the verdict in a reply's text and throws an Error for anything that is not a valid verdict (text that is not JSON, or JSON that Verdict.safeParse rejects); and a judge that asks lab-chat to grade, with zodResponseFormat(Verdict, 'verdict'), and returns readVerdict of the reply's text. Keep the printing, with the score added:

    PASS q019 (5/5)

    Judge passed 1 of 6

    Be ready to explain why an eval that passes in silence is worse than one that crashes.

What you will learn

  • When code graders are not enough, and what a judge adds.
  • A judge prompt: instructions, a scale, and labelled parts.
  • Key facts and references: what the judge grades against.
  • Typed verdicts with zod and zodResponseFormat, at temperature 0.
  • The cost of a judge, and what judges get wrong.

More in Measuring

Build "LLM as judge" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.