Evals that catch regressions in TypeScript · Chapter 3 of 16

Code graders

By the end of this chapter you can pick the right code grader for a check, from exact match to regular expressions and numbers within a tolerance, normalise text before comparing it, catch answers that say too much, and test a grader with replies whose right verdict you already know. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Measuring
  • 5 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.

Why you need this

Chapter 1's grader gave lab-chat every point on the smoke set, and Lakshmi's team was pleased for a day. Then Arif forwarded a reply from WhatsApp. A customer asked how much refund they get for cancelling 30 days before departure, and the assistant answered:

await answer('How much refund do I get if I cancel 30 days before departure?');
Cancel 45 days or more before departure and we refund 90% of what you paid. Cancel 15 to 44 days before departure and we refund 50% of what you paid.

Both sentences are in the policy. The second is the answer; the first is a different rule, and a customer who reads "90%" first will expect it. A grader that only checks that "50%" is present passes this reply. The cheaper model is no better, just wrong differently:

await answer('How much refund do I get if I cancel 30 days before departure?', { model: 'lab-chat-v2' });
Cancel 45 days or more before departure and we refund 90% of what you paid.

A grader is a decision about what "right" means. Chapter 1 made one such decision. This chapter builds the others a real eval needs.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    graders.ts: five graders and a normaliser

    Export normalise(text): lower-case it; remove a currency mark before a number (₹, rs, rs. or inr, with any spaces after it); remove every comma between two digits; replace every full stop that is not between two digits with a space; replace every character that is not a letter, a digit, a space, % or . with a space; collapse whitespace and trim.

    Then export five graders, each returning true or false: exact(reply, expected), equal after trimming both; normalised(reply, expected), equal once both are normalised; containsAll(reply, parts) and containsNone(reply, parts), every part or no part (normalised) in the normalised reply; matches(reply, pattern), the regular expression written in the string pattern found anywhere, ignoring case.

    For example normalised('₹7,500', 'Rs 7500') is true.

  2. Task 2

    route-eval.ts: an exact-match eval

    Route every message of /workspace/data/messages.jsonl with route from the yatradesk assistant and compare with its intent. Print the accuracy (a whole percentage), then one line per intent in the order of INTENTS with its right answers out of its messages, then the confusions: the true intent, the word as, the routed intent and the count, most common first, ties in the order they first happen, or none:

    Accuracy: 18/20 (90%)

    booking: 8/8

    visa: 3/4

    Confused: visa as booking 1, refund as payment 1

  3. Task 3

    strict-eval.ts: must_not_include

    Ask the assistant all 62 questions of /workspace/data/questions.jsonl (answerMany) with the model named on the command line, lab-chat when none is given, and grade each reply loose (chapter 1's grade) and strict (loose, and none of the must_not_include strings in the reply, ignoring case).

    Print one line per slice in the order they first appear, with the loose and the strict count, then the totals, then the ids that pass loose and fail strict, in the file's order, or none:

    single: 20/20 -> 20/20

    multi: 6/10 -> 5/10

    All: 26/30 -> 25/30

    Caught: q102

  4. Task 4

    amounts.ts: numbers within a tolerance

    Export amounts(text): every number in the text, in order, commas removed, multiplied by 1,000, 1,00,000 or 1,00,00,000 when k, lakh, lakhs or crore follows it (with or without a space, any case). Export within(got, want, tolerance = 1): true when they differ by at most the tolerance.

    Run as a program: for every booking of /workspace/data/bookings.json with status confirmed or deposit_paid, work out the policy's refund for cancelling on TODAY (from the yatradesk agent): 90% of paid at 45 days or more before travel_date, 50% at 15 to 44, nothing under 15, not rounded. Compare it with refundQuote({ booking_ref })'s refund and print:

    Within ₹1: 10 of 10

    Exactly equal: 9 of 10

    Not equal: YDAB12CD (policy 922.5, quote 922)

    listing each booking whose quote is not exactly the policy's refund, in the file's order, or none.

  5. Task 5 · interview question

    Interview: fragile-regex.ts, 2 pm or 12 pm?

    Thomas's regex grader for "check-in is at 2 pm" keeps its pattern in a string, and it gets half of its eight known verdicts wrong. Fix PATTERN and says2pm so 2 pm written as 2 pm, 2 PM, 2pm or 2 p.m. passes, and 12 pm, 22 pm, 2 am or no time at all fails. Keep says2pm exported, and CASES and the printing; it should end with:

    Wrong verdicts: 0 of 8

    Be ready to explain why a grader you only tried on replies that should pass tells you nothing about the replies that should fail.

What you will learn

  • Exact match for labels, and normalised match for text.
  • Contains all, contains none: required facts and forbidden ones.
  • Regular expressions for patterns, and where they break in JavaScript.
  • Numbers within a tolerance, read from text in any format.
  • When code graders are enough, and when they are not.

More in Measuring

Build "Code graders" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.