Evals that catch regressions in TypeScript · Chapter 1 of 16

Why evals

By the end of this chapter you can say why trying a model on a few questions proves nothing, and build the smallest real eval in TypeScript: a golden set of questions with known right answers, a grader that says pass or fail, a score, and a comparison of two versions question by question. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Measuring
  • 4 tasks
  • TypeScript
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

YatraDesk is a travel agency in Kochi. Its customers write on WhatsApp about Dubai visas, baggage on the Bali flight and refunds for cancelled trips, and an AI assistant answers them from YatraDesk's traveller policy. Thomas, the founder, has found a cheaper model, lab-chat-v2: 40% less per token than today's lab-chat. He asked it his three favourite questions, the answers looked just as good, and he wants it live on Friday.

Lakshmi, YatraDesk's ML engineer, has seen this before. Every change to an AI system, a new model, a reworded prompt, a new document, "feels better" to someone who tried it, and quietly breaks something nobody tried. A wrong answer about a refund costs YatraDesk real money and a customer's trust. Before Friday, she wants a number.

You know testing from ordinary code: expect(price(2)).toBe(29000) passes or fails, and it gives the same result every time. A model's answer is free text, it can change with any version, and "right" means it states the right facts. Measuring that, again and again, as the system changes, is called evaluation, and the program that does it is an eval.

The lesson

The system under test

YatraDesk's assistant is already written, in /workspace/yatradesk. You will measure it all course long, so you never change it to make a number look better: an eval measures the system as it is. A chapter's file imports it from two folders up, with the .ts at the end, as Node needs:

import { answer } from '../../yatradesk/assistant.ts';

console.log(await answer('Can I get a refund in cash?'));
console.log(await answer('Are visa fees refunded when I cancel?'));
I don't know. The information I was given does not say.
If a visa is refused, we cancel the package under section 2 and refund it minus the visa fee. Visa fees, insurance premiums and the card processing fee are never refunded.

answer(question, { model, prompt }) sends the policy in a system message, with the instruction to answer only from it, and resolves to the reply's text. The options are an object, and both have defaults: lab-chat and prompt v1, today's assistant. It runs at temperature 0, so the same question to the same model gives the same words: a difference between two runs is a difference between versions, not noise. Trying the cheaper model is one option away:

await answer('Are visa fees refunded when I cancel?', { model: 'lab-chat-v2' });
If a visa is refused, we cancel the package under section 2 and refund it minus the visa fee.

Read those two replies against rule 2.6 of the policy: "Visa fees, insurance premiums and the card processing fee are never refunded." Today's model said it; the new one did not. Would you have noticed, on a busy Thursday, among a hundred replies that all sound fluent?

answer returns a promise, because the request goes over the network, so you await it. Every file in this lab is a module, so await works at the top of the file with no function around it, and node file.ts runs it: Node strips the types and runs what is left.

Vibes

Thomas's check was what most teams do first: think of a few questions, ask, read, decide. It is called a vibe check, and it fails in predictable ways:

  • The questions are not chosen fairly. People try the questions they remember, usually the common, easy ones. Changes break the rare and the hard ones: a question whose answer takes two rules, one asked in Hinglish, one the policy does not answer at all.
  • Reading is not grading. A fluent, confident reply looks right. Is the refund 50% or 90%? Only someone checking against the policy knows.
  • Three is not a sample. If a change breaks one question in ten, three questions miss it about three times in four.
  • Nothing is written down. Next month, nobody can rerun Thomas's check, or say whether a later change made things better or worse.

Task 1 repeats his check exactly. Look at what it tells you, and what it cannot.

What an eval is

An eval has four parts:

  1. A golden set: questions written down with what a right answer must contain. YatraDesk's is /workspace/data/questions.jsonl, 62 questions; smoke.jsonl holds 12 of them for a first, fast eval.
  2. The system under test: here, answer with one model and one prompt.
  3. A grader: code (or, from chapter 6, a model) that looks at one reply and says pass or fail.
  4. A score: the share of questions that pass, reported with the questions that failed.

Here is one line of the golden set, a question not in smoke.jsonl:

{"id": "q041", "slice": "multi", "topic": "trip", "language": "hinglish",
 "question": "Hotel check-in kab hai?", "expect": "answer",
 "reference": "Check-in is at 2 pm and check-out is at 11 am.",
 "key_facts": ["Check-in is at 2 pm."], "must_include": ["2 pm"],
 "must_not_include": [], "sources": ["7.2"]}
  • must_include lists the pieces of text a right answer contains. It is the simplest grader there is, and for facts like "2 pm" or "₹7,500" it is hard to fool.
  • expect is answer, or abstain for a question nobody at YatraDesk answered in writing. There the right reply says it does not know: inventing a rule about cruises is the worst thing the bot can do.
  • slice groups the questions (one rule, two rules, help centre only, unknown) so a score can be read per group (chapter 5).
  • The rest (reference, key_facts, sources) serves later graders. /workspace/data/README.md describes every field.

The file is JSON Lines: one JSON object per line, not one big array. A new question is one more line, and a reader can stream it. To load it, read the text, split it on new lines, drop the empty last line, and JSON.parse each one. JSON.parse returns any, so give the rows a type that names the fields you read: id, question, expect as 'answer' | 'abstain', must_include as string[]. TypeScript then catches a misspelt field before any question is asked.

lab data in the terminal lists every dataset with its rows and fields.

A grader you can trust

For q041 the two models replied:

lab-chat:    Early check-in depends on the hotel and is not guaranteed. Hotel check-in is at 2 pm and check-out is at 11 am.
lab-chat-v2: Early check-in depends on the hotel and is not guaranteed.

A grader that checks whether the lower-cased reply includes "2 pm" passes the first and fails the second, which is exactly what the customer would say. Lower-case the reply, because "2 PM" is the same fact. Every string of must_include must be there, because "15 kg" without "7 kg" is half an answer: that is every, where some would be satisfied by one. And an abstain question passes only when the reply admits it does not know: don't know, do not know, not sure or does not say.

A grader this simple has limits. A reply that says "two in the afternoon" fails it, and one that says "check-out is at 2 pm" passes it: the right number on the wrong fact. Chapter 3 builds better code graders, chapter 6 lets a model grade, and chapter 7 measures how far to trust each one.

Running an eval

Asking 12 questions one after another means waiting for 12 replies in turn. answerMany(questions, { model }) asks 8 at a time and resolves to the replies in the same order, so an eval takes a couple of seconds:

import { answerMany } from '../../yatradesk/assistant.ts';

console.log(await answerMany(['Hotel check-in kab hai?', 'Can I get a refund in cash?'], { model: 'lab-chat-v2' }));
[
  'Early check-in depends on the hotel and is not guaranteed.',
  "I don't know. The information I was given does not say."
]

Then grade each reply and count. Report the score and the verdict per question: "lab-chat-v2: 9/12" says something changed, the list of failed ids says what. An object from id to verdict keeps both, and a JavaScript object keeps string keys like q041 in the order you added them, so the file's order survives.

One thing to forget once and never again: without await, answerMany gives you a promise, not the replies. typecheck says so before you run anything: Property 'map' does not exist on type 'Promise<string[]>', and then, helpfully, Did you forget to use 'await'?.

A file that is also a module

Task 3 imports runEval from task 2's file. In TypeScript, importing a file runs its top level. So if first-eval.ts evaluates lab-chat and prints at the top level, the import alone runs that whole eval before task 3's own code starts. Put the program part inside if (import.meta.main) { ... }: import.meta.main is true only for the file you ran with node, and false when another file imports it. It is the twin of Python's if __name__ == "__main__":. The model to evaluate comes from the command line: node first-eval.ts lab-chat-v2 puts it in process.argv[2] (argv[0] is node, argv[1] the file), and ?? gives the default when it is missing.

Comparing two versions

The question Thomas is really asking is not "how good is lab-chat-v2?" but "is it worse than what we have?". Run the same eval on both and compare per question:

  • a regression: passed on today's version, fails on the new one;
  • a fix: failed today, passes on the new one;
  • everything else did not change.

Two versions can have the same score with different failures: a fix on one question can hide a regression on another. So a comparison always lists the ids, never only the two numbers. Each regression is a question to read and decide on: is it a real loss, a bad golden answer, or a grader that is too strict?

When the eval is wrong

An eval is code, and code has bugs. The worst bug is the one that makes an eval pass everything: it looks like good news, so nobody reads it. Three classics:

  • grading something other than the reply (the reference answer, the question), which always contains the facts;
  • some where the rule needs every, so half an answer passes;
  • treating questions that are hard to grade as passed ("nothing to check"), so the slice that matters most cannot fail.

The defence is to evaluate the eval: feed the grader replies you know are right and replies you know are wrong, and check its verdicts. That is what this chapter's Check does to your grade, with replies the model never gave.

Your turn

Repeat Thomas's check in task 1 and note what it shows. Build the eval in task 2, run it on both models, and in task 3 list exactly what changed. The interview question is Thomas's own eval, which says both models are perfect: find out why, and fix it.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    vibe-check.ts: Thomas's three questions

    Thomas tried the cheaper model on his three favourite questions and wants it live on Friday. Do what he did: ask YatraDesk's assistant each question in THOMAS with the model lab-chat and with lab-chat-v2, and print the question, then the line lab-chat: and today's reply, then the line lab-chat-v2: and the new model's reply.

    End with one line counting the questions whose two replies are exactly the same, for example:

    Same answers: 2 of 3

  2. Task 2

    first-eval.ts: a golden set, a grader, a score

    Lakshmi wants a measurement, not a feeling. Over the 12 questions of /workspace/data/smoke.jsonl, export two functions:

    grade(item, reply): true when item.expect is 'answer' and the reply contains every string in item.must_include, ignoring case; or when item.expect is 'abstain' and the reply contains "don't know", "do not know", "not sure" or "does not say", ignoring case. false otherwise.

    runEval(model): asks the assistant every question with that model (answerMany) and resolves to an object from each item's id to its verdict, in the file's order.

    Run as a program, it evaluates the model named on the command line (lab-chat when none is given) and prints PASS or FAIL and the id for each question, then the score, the percentage rounded to a whole number:

    lab-chat: 1/2 passed (50%)

  3. Task 3

    compare.ts: what changed between the models

    Import runEval from ./first-eval.ts and run it for lab-chat and lab-chat-v2. Print each model's score as in task 2, then two lines, in smoke.jsonl's order: the ids that pass on lab-chat and fail on lab-chat-v2, and the ids that fail on lab-chat and pass on lab-chat-v2, separated by a comma and a space, or none:

    Regressions: q102

    Fixed: none

    Then compare what the eval found with what Thomas's three questions showed.

  4. Task 4 · interview question

    Interview: broken-eval.ts, the eval that cannot fail

    Thomas's eval says both models pass every question. Find the three bugs in its grade function and fix them, so it follows task 2's rules and prints each model's true score.

    Be ready to explain why an eval that cannot fail is worse than no eval at all.

What you will learn

  • Vibes versus measurements: why a few hand-picked questions mislead.
  • What an eval is: a golden set, the system under test, a grader, a score.
  • Your first eval, over 12 of YatraDesk's questions, in TypeScript.
  • Comparing two versions question by question: regressions and fixes.
  • Evaluating the eval: a grader that cannot fail measures nothing.

More in Measuring

Build "Why evals" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.