Evals that catch regressions · Chapter 6 of 16
LLM as judge
By the end of this chapter you can grade free-text answers with a model: write a judge prompt over key facts, get a structured verdict back, run the judge over a whole golden set, read where it disagrees with a code grader, and make sure a judge that is not judging can never pass in silence. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Measuring
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.
Why you need this
Arif brings new questions from WhatsApp every week, and more and more of them want answers with several parts. "Is insurance included, and what does it cover?" needs three facts: insurance up to 70 years, medical emergencies up to USD 50,000, lost baggage up to USD 500. A customer can be told those in a dozen wordings, and the assistant can get two right and forget the third.
The must_include lists of chapters 1 to 5 are hard to fool on numbers like "2 pm" and "₹7,500", but they say pass or fail and nothing else. They cannot say "two of three facts", and writing a list of strings for every way of saying a fact does not scale. Lakshmi proposes the tool most teams reach for next: let a model read the answer and grade it, against facts written down in the golden set. That model is called a judge, and this chapter builds one.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
judge.py: a model as a grader
Write Verdict, a Pydantic model with score (int), passed (bool), reason (str) and missing (a list of str), and judge(item, reply, model="lab-chat"), which returns the model's Verdict on one reply to a question of /workspace/data/questions.jsonl.
The system message tells the model to grade the answer against the key facts, to score it from 1 to 5 by how many key facts it states, and to reply in JSON with score, passed, reason and missing. The user message is exactly the line Question: and the question, the line Answer: and the reply, the line Key facts:, then one line per key fact starting with a dash and a space. Ask with client.chat.completions.parse, response_format=Verdict and temperature=0.
Run as a program, it judges lab-chat-v2's replies to the questions in IDS and prints one line each:
q101: 3/5 FAIL (Covers 1 of 2 key facts. Missing: "7 kg of cabin baggage".)
- Task 2
judge_eval.py: the judge over the golden set
Import judge from judge.py. Ask the assistant the 52 questions of questions.jsonl whose expect is answer, with lab-chat and then with lab-chat-v2 (answer_many), judge every reply, and print one line per model with the judge's passes and its mean score, to two decimals:
lab-chat: 8/10 passed, mean score 4.20
lab-chat-v2: 6/10 passed, mean score 3.70
Then compare with what the strict code grader said about the same models in chapter 3.
- Task 3
disagree.py: two graders, one set of replies
Write strict(item, reply), chapter 3's strict grade for an answer item. Run as a program, ask the 52 answerable questions with the model named on the command line (lab-chat when none is given), grade each reply with strict and with judge, and for each question where they disagree, in the file's order, print the id with both verdicts and the judge's score, then the reply indented by two spaces. End with the count:
q101: code PASS, judge FAIL (1/5)
Children under 5 travel free on a parent's lap.
Disagree: 1 of 10
Then read each one: is the judge wrong, the code grader, or the golden row?
- Task 4 · interview question
Interview: silent_judge.py, the judge that never judged
Thomas's judge passes every answer of lab-chat-v2. Find out why, and fix it so it can never pass in silence: a Verdict whose score is an integer from 1 to 5 and whose reason is not empty (Pydantic's Field); read_verdict(text), which returns a Verdict from a reply's text and raises a ValueError for anything that is not a valid verdict; and a judge that asks lab-chat to grade, with Verdict as the response format, and returns read_verdict of the reply's text. Keep the printing, with the score added:
PASS q019 (5/5)
Judge passed 1 of 6
Be ready to explain why an eval that passes in silence is worse than one that crashes.
What you will learn
- When code graders are not enough, and what a judge adds.
- A judge prompt: instructions, a scale, and labelled parts.
- Key facts and references: what the judge grades against.
- Structured verdicts with Pydantic, at temperature 0.
- The cost of a judge, and what judges get wrong.
More in Measuring
Build "LLM as judge" yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.