Evals that catch regressions in TypeScript · Chapter 8 of 16
Pairwise
By the end of this chapter you can compare two answers with a judge instead of grading each one: ask which is better, against a reference, catch position bias by asking in both orders, count wins, losses and ties, and read a win rate without being fooled by it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Measuring
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.
Why you need this
Thomas has stopped asking whether each answer is right. "I read both models side by side for an hour," he says, "and the cheaper one's answers are just as good. Shorter, even. Which model gives better answers?" It is a fair question, and a different one: not "does this answer pass?" but "which of these two is better?".
Many teams ask it this way, because people find it easier to compare two answers than to score one on a scale. "Is this a 3 or a 4?" is hard; "is the left one better than the right one?" is easy. A judge can be asked the same way. It is called pairwise judging, and it has one trap that every team falls into once.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
pairwise.ts: which answer is better?
Export Choice, a zod schema with winner (z.enum of A, B and tie) and reason (a string), PAIRS, the rows of /workspace/data/pairs.jsonl, and prefer(question, reference, a, b, { model = 'lab-chat' }), which resolves to 'A', 'B' or 'tie'. The system message asks the model to compare the two answers with the reference answer, which answer is better for the customer, and to reply in JSON with the winner (A, B, or tie if they are equally good) and a reason. The user message is exactly the lines Question:, Reference:, Answer A: and Answer B:, each followed by its text. Ask with client.chat.completions.parse, response_format: zodResponseFormat(Choice, 'choice') and temperature: 0, and throw an Error when the reply has no parsed choice.
Run as a program, it judges every pair (a as A, b as B) with the model named on the command line (lab-chat when none is given) and prints how many verdicts equal arif_prefers, and how many were ties:
lab-chat: agrees with Arif on 7/10 (ties 1)
- Task 2
swap.ts: both orders
For lab-chat and then lab-chat-biased, judge every pair twice with prefer: a as A and b as B, then b as A and a as B. A pair is consistent when the second verdict is the first one swapped (A then B, B then A, tie then tie). Print one line per judge with the consistent pairs, how many of all the verdicts picked the answer shown first, and how many pairs are consistent and agree with Arif (the first order's verdict equals arif_prefers):
lab-chat: consistent 9/10, first shown picked 11/20, agrees with Arif 7/10
Then explain what each judge's numbers say about it.
- Task 3
win-rate.ts: release 2.0's model, answer against answer
Ask the 52 questions of questions.jsonl whose expect is answer with lab-chat and with lab-chat-v2 (answerMany, prompt v1). Compare each pair of replies with prefer (lab-chat judging, the item's reference) in both orders: a model wins a question when it wins in both orders; anything else is a tie. Print the number of questions with exactly the same two replies, the wins and ties, and lab-chat-v2's win rate with ties counted as half, to one decimal:
Identical replies: 6 of 10
lab-chat wins 3, lab-chat-v2 wins 1, ties 6
lab-chat-v2 win rate: 40.0% (ties count half)
Then compare it with chapter 5's paired comparison of the same two models.
- Task 4 · interview question
Interview: one-order.ts, the win rate that proves anything
Thomas's script asks lab-chat-biased to compare the two models with lab-chat's reply always first, and concludes that lab-chat-v2 is worse. Keep lab-chat-biased as the judge and print three lines: the wins with lab-chat first, the wins with lab-chat-v2 first, and the result when a win needs both orders to agree:
lab-chat first: lab-chat wins 9 of 10
lab-chat-v2 first: lab-chat-v2 wins 9 of 10
Both orders: lab-chat wins 1, lab-chat-v2 wins 0, ties 9
Be ready to explain what this judge's verdicts measure, and how you would catch it before anyone read a win rate.
What you will learn
- Pairwise judging: A versus B, against a reference.
- Position bias, and swapping the order to catch it.
- Ties: when two answers are equally good, or identical.
- Win rates: wins, losses and ties over a golden set.
- When pairwise beats a score, and when it does not.
More in Measuring
Build "Pairwise" yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.