Building with LLMs (TypeScript) · Chapter 16 of 24

Testing prompts

By the end of this chapter you can write vitest tests for code that calls a model: golden examples with it.each, expectations that survive a model's changing wording, and an accuracy test with a threshold, the first step towards evals. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Reliable output
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.

Why you need this

SafarLine's support helpers work. Then someone edits a prompt to make the replies friendlier, Harpreet asks to try the cheaper lab-chat-mini, and Lata wants a ninth label. Each change is one line. Does the classifier still label lost bags? Does the extractor still get "kal" right? Nobody knows, unless something checks, every time.

You know vitest, and it works here unchanged: it, expect, it.each, beforeAll. What is new is the thing under test: a function whose answer comes from a model. Its wording can change between two calls, it costs money to call, and "correct" is sometimes a share of messages rather than one value. typecheck cannot help either: a prompt that lost a word still has exactly the same types.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    intents.test.ts: golden examples for classify

    Lata wants a ninth label, and nobody may break the other eight. In intents.test.ts, write vitest tests for support.ts's classify: one it.each over at least 8 golden rows, each an object with a message and its intent, covering all eight intents (write your own messages, or copy clear ones from messages.jsonl).

    The check runs your tests with vitest: they must all pass on the real support.ts, and at least one must fail on each of four broken copies (one has no lost_item label, two rename a label, one lost its labels altogether).

    Run them yourself with npx vitest run intents.test.ts. Do not change support.ts.

  2. Task 2

    extract.test.ts: golden records for extract

    In extract.test.ts, write vitest tests for support.ts's extract(text, today): one it.each, each row a message and the whole record you expect as an object of its six fields (fromCity, toCity, travelDate, seats, name, phone), null for what the message does not say. Use one fixed today you choose, written as YYYY-MM-DD, never the computer's clock.

    Write at least 3 rows, among them a message that says kal or parso, one with a date but no year (like 14 November), and one that leaves fields out.

    The check runs your tests on the real support.ts (all pass) and on three broken copies: one forgets today's date, one reads the computer's clock, one cannot say null.

  3. Task 3 · interview question

    Interview: answers.test.ts, the test that fails on Fridays

    answers.test.ts passed on Nikhil's laptop and failed in the next run. Rewrite it so its tests pass every time on the real support.ts and catch broken copies of answer. Test three questions: How much luggage can I take? (policy 6.1), Can I bring my dog on the bus? (policy 6.3), and Is there Wi-Fi on the bus? (the policy does not say).

    The check runs your tests 5 times on the real support.ts (every run must pass), then on three broken copies: one sends no policy, one no longer says only, one sends only the first part of the policy.

    Be ready to explain why the first test failed, and why temperature 0 is not the whole answer.

  4. Task 4

    accuracy.test.ts: an eval as a test

    In accuracy.test.ts, test classify on all 240 messages in /workspace/data/messages.jsonl against their true intents: one test that the accuracy is at least 90%, and one that every intent on its own is right at least 50% of the time (right for that intent divided by its messages). Classify the 240 messages once, in a beforeAll, and let both tests use the results.

    The check runs your tests on the real support.ts (all pass) and on three broken copies: no lost_item label, praise called feedback, and a prompt that lost its labels.

What you will learn

  • Golden examples: a small set of inputs with answers everyone agrees on, run with it.each.
  • Expecting what matters: facts, labels and fields, never the exact wording of free text.
  • Breaking the prompt on purpose to see whether a test catches it.
  • An eval as a test: accuracy over a dataset, held above a threshold.

More in Reliable output

Build "Testing prompts" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant, in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.