Evals that catch regressions · Chapter 16 of 16

Interview round

By the end of this chapter you have answered five timed questions, one idea each, from across the whole lab: kappa from a table, an eval tested on replies of known verdict, a split that leaks, a judge's position bias, and a gate that flakes. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Evals in production
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.

Why you need this

YatraDesk is hiring a second evals engineer, and Lakshmi wrote the interview from this course. "I am not checking that they remember a formula," she tells Divya. "I am checking that they never trust a number before they know what it measures, and that they test their own eval like any other code." Divya wants the round to be part of the lab, so every new engineer takes it once.

Each question uses one idea from an earlier chapter, with no hand-holding: the file has the goal, the check has the edge cases. The ideas are below, as a refresher and as what an interviewer listens for. The answers are not.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1 · interview question

    Interview: kappa.py, kappa by hand

    Time yourself: 10 minutes, no libraries. Write kappa_table(both_pass, a_only, b_only, both_fail) for two graders' 2x2 table, returning (agreement, kappa): the observed agreement as a share, and Cohen's kappa with chance agreement from each grader's own pass rate; kappa is 1.0 when chance agreement is exactly 1.

    The check tries balanced tables, lopsided ones, total disagreement and tables where one verdict is all there is.

  2. Task 2 · interview question

    Interview: meta_eval.py, the eval that cannot fail

    Time yourself: 10 minutes. Write meta_check(grader): for every item of the golden set, grader(item, right) must return True, where right is the item's reference for an answer item and RIGHT_UNKNOWN for an abstain item, and grader(item, WRONG) must return False. Return the ids, in the file's order and each once, where it returned anything else.

    The check gives it a correct grader and several broken ones, including one that cannot fail.

  3. Task 3 · interview question

    Interview: overlap.py, the leaky split

    Time yourself: 10 minutes. Write normalise(question): lower-case, every character that is not a letter a to z, a digit or a space becomes a space, then the words joined by single spaces. Write overlap(dev, test) for two lists of rows with id and question: every (dev id, test id) pair whose normalised questions are equal, in test's order, and for one test row in dev's order.

    The check tries capitals, punctuation, extra spaces and an empty test set.

  4. Task 4 · interview question

    Interview: position_bias.py, first shown, first chosen

    Time yourself: 10 minutes. Write position_bias(model): ask prefer (given) about every pair of pairs.jsonl in both orders, a then b and b then a, and return (first, verdicts, consistent): the verdicts that picked the answer shown first, the number of verdicts, and the pairs that got the same winner in both orders.

    Run as a program, it prints a line for lab-chat, then one for lab-chat-biased:

    lab-chat: picks the first shown 5/10 (50%), consistent on 5/5 pairs

  5. Task 5 · interview question

    Interview: same_every_time.py, does the gate flake?

    Time yourself: 10 minutes. Write same_every_time(file, runs=3, args=None): run the Python program file with those arguments runs times, each in its own process, and return True when every run printed exactly the same output and exited with the same code, else False.

    The check gives it a steady program, one that prints a set of strings, an unseeded and a seeded random number, and one that prints the same line but exits with a random code.

What you will learn

  • How to work a timed question: the plan out loud, the smallest thing that runs, then the edges.
  • Five ideas again, at interview depth: kappa (chapter 7), evaluating the eval (1 and 3), leaky splits (2), position bias (8), and flaky gates (11).

More in Evals in production

Build "Interview round" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.