Evals that catch regressions in TypeScript · Chapter 12 of 16
The cost of evals
By the end of this chapter you can say what an eval run costs in tokens and rupees, cut that cost without cutting what the eval sees: a cache that never asks an unchanged question twice, a cheaper judge used only where it was measured, a stratified sample when you must ask fewer questions, and a budget per run. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Evals in production
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.
Why you need this
Divya's gate from chapter 11 runs on every pull request, and the team loves it: last week it stopped two prompt changes before they reached a customer. Then Thomas opened the API bill. "Every pull request asks 62 questions on two models, and then a judge reads every answer. Forty pull requests a week. The evals cost more than the assistant!"
He is half right. Evals are not free, and a gate that runs on every change multiplies whatever one run costs. But the answer is not to run fewer evals. It is to know exactly where the money goes, and to stop paying for work that was already done.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
cache.ts: never ask twice
Export cacheKey(model, prompt, question): the sha256 hex digest (createHash from node:crypto) of the model, the prompt version's full system prompt (systemPrompt from ../../yatradesk/assistant.ts) and the question, joined by newlines. Export cachedAnswers(questions, model = 'lab-chat', prompt = 'v1'), which resolves to { replies, asked }: it reads cache.json in this folder (key to reply, empty when missing), asks answerMany only the questions whose key is not there, adds their replies, writes cache.json again with every earlier entry kept, and returns the replies in the questions' order and how many it asked.
Run as a program, it runs the golden set with the model and prompt on the command line (lab-chat and v1 when not given):
lab-chat-v2, prompt v1: 10 questions, 10 asked, 0 from the cache
The same command again asks nothing.
- Task 2
cost.ts: rupees per run
Export prices(), which resolves to model id -> price from client.models.list() (each lab model has price_inr_per_million with input, cached_input and output), and cost(responses, price), which returns { tokensIn, cached, tokensOut, rupees }: the sums of usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens and usage.completion_tokens, and the rupees with uncached prompt tokens at input, cached ones at cached_input and completion tokens at output, per million.
Run as a program, it asks all 62 questions with ask() on lab-chat and on lab-chat-v2, judges each model's replies to the answerable questions with judgeCall(), and prints:
lab-chat, prompt v1: 10 calls, 12,390 in (10,368 cached), 210 out, ₹1.04
lab-chat-v2, prompt v1: 10 calls, 12,390 in (10,368 cached), 160 out, ₹0.61
judge (lab-chat): 16 calls, 1,580 in (0 cached), 540 out, ₹0.75
One run: ₹2.40, the judge's share 31%
- Task 3
sample.ts: fewer questions, wider intervals
Export stratified(items, share, seed = 7): one rng = new Random(seed); for each slice, in the order slices first appear, rng.sample of that slice's items with k = Math.ceil(share * the slice's size); return the chosen items in the order of items.
Run as a program, it asks the golden set with lab-chat-v2 once and prints the whole set, then for shares 0.5 and 0.25 the sample's size per slice and its verdicts, each with chapter 5's bootstrap interval and its width in points:
all 62: 30/62 48% [36%, 61%], width 25 points
share 0.5: single 3, multi 2
sample 5: 3/5 60% [20%, 100%], width 80 points
- Task 4 · interview question
Interview: stale-cache.ts, the cache that never misses
Thomas's cached nightly eval says lab-chat-v2 has no regressions at all. Fix its cache so each run gets the replies of its own model and prompt, while the third run, lab-chat again, still comes entirely from the cache. It prints the three scores, then lab-chat-v2's regressions and fixes against lab-chat:
lab-chat: 6/10
lab-chat-v2: 5/10
lab-chat again: 6/10
Regressions: q102
Fixes: none
Be ready to say what a cache key must contain, and what else could make a cached reply stale.
What you will learn
- Tokens and rupees: pricing a run from the usage the API reports.
- Caching: why an unchanged question never needs asking twice, and what the key must hold.
- Cheap judges first: code graders where they work, a small judge only where it was calibrated.
- Sampling: fewer questions, stratified, and what it does to the interval.
- Budgets: a number per run that the eval must stay under.
More in Evals in production
Build The cost of evals yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.