Evals that catch regressions in TypeScript · Chapter 16 of 16

Interview round

By the end of this chapter you have answered five timed questions, one idea each, from across the whole lab: kappa from a table, an eval tested on replies of known verdict, a split that leaks, a judge's position bias, and a gate that flakes. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Evals in production
  • 5 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.

Why you need this

YatraDesk is hiring a second evals engineer, and Lakshmi wrote the interview from this course. "I am not checking that they remember a formula," she tells Divya. "I am checking that they never trust a number before they know what it measures, and that they test their own eval like any other code." Divya wants the round to be part of the lab, so every new engineer takes it once.

Each question uses one idea from an earlier chapter, with no hand-holding: the file has the goal, the check has the edge cases. The ideas are below, as a refresher and as what an interviewer listens for. The answers are not.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1 · interview question

    Interview: kappa.ts, kappa by hand

    Time yourself: 10 minutes, no libraries. Export kappaTable(bothPass, aOnly, bOnly, bothFail) for two graders' 2x2 table, returning [agreement, kappa]: the observed agreement as a share, and Cohen's kappa with chance agreement from each grader's own pass rate; kappa is 1 when chance agreement is exactly 1.

    The check tries balanced tables, lopsided ones, total disagreement and tables where one verdict is all there is.

  2. Task 2 · interview question

    Interview: meta-eval.ts, the eval that cannot fail

    Time yourself: 10 minutes. Export metaCheck(grader): for every item of the golden set, grader(item, right) must return true, where right is the item's reference for an answer item and RIGHT_UNKNOWN for an abstain item, and grader(item, WRONG) must return false. Return the ids, in the file's order and each once, where it returned anything else.

    The check gives it a correct grader and several broken ones, including one that cannot fail.

  3. Task 3 · interview question

    Interview: overlap.ts, the leaky split

    Time yourself: 10 minutes. Export normalise(question): lower-case, every character that is not a letter a to z, a digit or a space becomes a space, then the words joined by single spaces. Export overlap(dev, test) for two arrays of rows with id and question: every [dev id, test id] pair whose normalised questions are equal, in test's order, and for one test row in dev's order.

    The check tries capitals, punctuation, tabs and extra spaces, and an empty test set.

  4. Task 4 · interview question

    Interview: position-bias.ts, first shown, first chosen

    Time yourself: 10 minutes. Export positionBias(model): ask prefer (given) about every pair of pairs.jsonl in both orders, a then b and b then a, and resolve to [first, verdicts, consistent]: the verdicts that picked the answer shown first, the number of verdicts, and the pairs that got the same winner in both orders.

    Run as a program, it prints a line for lab-chat, then one for lab-chat-biased:

    lab-chat: picks the first shown 5/10 (50%), consistent on 5/5 pairs

  5. Task 5 · interview question

    Interview: same-every-time.ts, does the gate flake?

    Time yourself: 10 minutes. Export sameEveryTime(file, runs = 3, args = []): run the TypeScript program file with node and those arguments runs times, each in its own process, and return true when every run printed exactly the same output and exited with the same code, else false.

    The check gives it a steady program, an unseeded and a seeded random number, a sample taken once at the program's top level, results in the order concurrent work finished, and one that prints the same line but exits with a random code.

What you will learn

  • How to work a timed question: the plan out loud, the smallest thing that runs, then the edges.
  • Five ideas again, at interview depth: kappa (chapter 7), evaluating the eval (1 and 3), leaky splits (2), position bias (8), and flaky gates (11).

More in Evals in production

Build "Interview round" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.