Evals that catch regressions · Chapter 14 of 16
Online evals
By the end of this chapter you can evaluate an assistant after it is live: log every request it serves, choose the few replies a person should label, notice when customers start asking different things, and build a dashboard that measures production instead of the golden set. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Evals in production
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.
Why you need this
Everything so far was an offline eval: a fixed golden set, run before a release. It answers "is this version better than the last one, on the questions we wrote down?". It cannot answer "how is the assistant doing with what customers are asking this week?", because nobody wrote this week's questions down.
YatraDesk's assistant has been live on WhatsApp and the website for two weeks of October, still on lab-chat with prompt v1, since release 2.0 did not pass its gate. Arif reads replies all day and can label 20 of them properly, against the policy. "Which 20?" he asks. "If I pick, I will pick the ones I remember, and you taught me in chapter 1 what that does." And Thomas's dashboard says "stable" every morning, while Arif's team answers more complaints than ever.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
log_traffic.py: every request, logged
Write serve(row): send one row of /workspace/data/traffic.jsonl with the model and prompt in its served_by (the assistant's system prompt, then the question, temperature 0), timing the call, and return the record {"id", "ts", "week", "model", "prompt", "question", "reply", "latency_ms", "prompt_tokens", "completion_tokens"}, with the tokens from the response's usage.
Run as a program, it serves every row, 8 at a time, writes one JSON record per line to traffic_log.jsonl in traffic.jsonl's order, and prints:
Logged 240 replies to traffic_log.jsonl
- Task 2
label_queue.py: Arif's 20
Write build_queue(log, week, size=20, seed=14): log rows with a "reason" added, one row per question (the first of that week that asked it), at most size. First every question questions.jsonl does not have ("not in the golden set"), then every other question whose reply abstains ("abstained"), then the rest ("sampled"): grouped by the slice traffic.jsonl gives the row's id, groups in the order their slices first appear, each group shuffled in that order with one random.Random(seed), then the first of each group, the second of each group, and so on.
Run as a program, it builds week 2's queue from traffic_log.jsonl, writes label_queue.jsonl ({"id", "question", "reply", "reason"} per line) and prints:
not in the golden set: 2
abstained: 3
sampled: 15
Wrote 20 rows to label_queue.jsonl
- Task 3
drift.py: did the traffic change?
Write shares(rows, key, values): {value: share of rows with that value}, 0.0 for a value no row has.
Run as a program, it prints, for each slice and then each topic of /workspace/data/traffic.jsonl in the order they first appear, the share in week 1 and in week 2, with ALERT when they differ by 10 points or more, then the share of each week's questions that questions.jsonl does not have, with the same rule:
slice single: 50% -> 45%
topic visa: 10% -> 25% ALERT
not in the golden set: 0% -> 5%
- Task 4 · interview question
Interview: dashboard.py, stable on the dashboard
Thomas's dashboard runs the golden set every night and has said stable for two weeks while complaints doubled. Rewrite it to measure production from traffic_log.jsonl, without asking the model: for each week, the replies, how many the golden set can grade (their question is in questions.jsonl), how many of those pass the strict grade, and how many replies abstained:
week 1: 50 replies, 50 graded (100%), 40 passed (80%), 2 abstained (4%)
week 2: 50 replies, 40 graded (80%), 31 passed (78%), 6 abstained (12%)
Quality: CHANGED
passed is a share of graded; graded and abstained are shares of replies. CHANGED when the graded share fell by 5 points or more, the pass rate fell by 5 points or more, or the abstention rate rose by 5 points or more; stable otherwise.
Be ready to explain why the old dashboard could not see the change, and what a pass rate on graded replies alone hides.
What you will learn
- Logging traffic: what each request must leave behind for an eval.
- Sampling for labels: spending scarce human labels where code is blind.
- Drift: noticing that what customers ask has changed.
- Production metrics: coverage, abstentions and pass rates from the log.
More in Evals in production
Build "Online evals" yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.