Evals that catch regressions in TypeScript · Chapter 7 of 16

Trusting the judge

By the end of this chapter you can measure a judge before you trust it: its agreement with human labels, Cohen's kappa, a 2x2 table of where it goes wrong, its agreement per kind of answer, and its length bias. And you can answer the question every team asks about a judge: is this number good? Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Measuring
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.

Why you need this

Chapter 6's judge gives a score to every answer, with a reason that reads well. Thomas likes it: "Now Arif can stop reading replies and do something useful." Lakshmi is not so sure. The judge is a grader, and nobody has graded the grader. If it fails right answers, the eval cries wolf; if it passes wrong ones, it hides the regressions it was built to catch.

So Arif did what only a person can do: he graded 72 answers by hand, pass or fail, and saved them in labels.jsonl. Some are the policy's own words, some say the same thing in other words, some are wrong, half right, or have two numbers swapped, and some are padded with a long friendly closing. His grades are the ground truth. This chapter measures the judge against them.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    agreement.ts: the judge against Arif

    Copy chapter 6's Verdict and judge(item, reply, { model = 'lab-chat' }) and export them (a label of /workspace/data/labels.jsonl works as the item), with LABELS, the rows of labels.jsonl. Export kappa(a, b) for two equal-length arrays of true/false verdicts: observed is the share of positions where they are equal; chance is pa * pb + (1 - pa) * (1 - pb), with pa and pb the shares of true; kappa is (observed - chance) / (1 - chance), or 1 when chance is exactly 1.

    Run as a program, it judges every label's answer with the model named on the command line (lab-chat when none is given), compares each verdict's passed with Arif's grade (arif is pass or fail), and prints the agreement to one decimal and kappa to two:

    lab-chat: agreement 7/10 (70.0%), kappa 0.21

  2. Task 2

    confusion.ts: which way the judge is wrong

    Judge every label with lab-chat (import judge and LABELS from ./agreement.ts) and print the four cells of the 2x2 table, Arif's grade first, then the agreement for each kind of answer in the order the kinds first appear:

    Arif pass, judge pass: 5

    Arif pass, judge fail: 2

    Arif fail, judge pass: 1

    Arif fail, judge fail: 2

    verbatim: 4/4 agree

    Then read it: which kinds of answer does the judge get wrong, and which way?

  3. Task 3

    length-bias.ts: padding for points

    For every label whose kind contains padded, make the short version by removing a space and FILLER from its answer. Judge both versions with lab-chat, then with lab-chat-biased, and print one line per judge with the mean score of each version to two decimals, and how many answers pass when padded but fail when short:

    lab-chat: mean short 3.00, mean padded 3.00, passes gained 0 of 4

    Then run agreement.ts with lab-chat-biased and compare.

  4. Task 4 · interview question

    Interview: is-it-good.ts, more often than not

    Thomas says a judge that agrees with Arif more often than not can replace him. Make his script print, for lab-chat's judge on the 72 labels, the judge's agreement and kappa, the same for a judge that passes every answer without reading it, and the chance agreement of the real judge (pa * pj + (1 - pa) * (1 - pj)), percentages to one decimal and kappa to two:

    Judge: 7/10 agree (70.0%), kappa 0.21

    Always pass: 8/10 agree (80.0%), kappa 0.00

    Chance agreement: 62.0%

    Be ready to explain what the agreement means, why a judge that reads nothing can score higher, what kappa adds, and what you would do with this judge.

What you will learn

  • Agreement with human labels, and why it misleads on its own.
  • Chance agreement and Cohen's kappa, written as code.
  • A 2x2 table and slices by kind: which way the judge is wrong.
  • Length bias, and a preview of position bias.
  • What to do with a judge you have measured.

More in Measuring

Build "Trusting the judge" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.