Building with LLMs · Chapter 16 of 24
Testing prompts
By the end of this chapter you can write pytest tests for code that calls a model: golden examples with parametrize, assertions that survive a model's changing wording, and an accuracy test with a threshold, the first step towards evals. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Reliable output
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.
Why you need this
SafarLine's support helpers work. Then someone edits a prompt to make the replies friendlier, Harpreet asks to try the cheaper lab-chat-mini, and Lata wants a ninth label. Each change is one line. Does the classifier still label lost bags? Does the extractor still get "kal" right? Nobody knows, unless something checks, every time.
You know pytest from Chai aur Python, and it works here unchanged. What is new is the thing under test: a function whose answer comes from a model. Its wording can change between two calls, it costs money to call, and "correct" is sometimes a share of messages rather than one value.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
test_intents.py: golden examples for classify
Lata wants a ninth label, and nobody may break the other eight. In test_intents.py, write pytest tests for support.py's classify: one test function with @pytest.mark.parametrize and at least 8 golden rows, each a message and its intent, covering all eight intents (write your own messages, or copy clear ones from messages.jsonl).
The check runs your tests with pytest: they must all pass on the real support.py, and at least one must fail on each of four broken copies (one has no lost_item label, two rename a label, one lost its labels altogether).
Run them yourself with pytest test_intents.py. Do not change support.py.
- Task 2
test_extract.py: golden records for extract
In test_extract.py, write pytest tests for support.py's extract(text, today): one test function with @pytest.mark.parametrize, each row a message and the whole record you expect as a dictionary of its six fields (from_city, to_city, travel_date, seats, name, phone), None for what the message does not say. Use one fixed today you choose, never date.today().
Write at least 3 rows, among them a message that says kal or parso, one with a date but no year (like 14 November), and one that leaves fields out.
The check runs your tests on the real support.py (all pass) and on three broken copies: one forgets today's date, one reads the computer's clock, one cannot say None.
- Task 3 · interview question
Interview: test_answers.py, the test that fails on Fridays
test_answers.py passed on Nikhil's laptop and failed in the next run. Rewrite it so its tests pass every time on the real support.py and catch broken copies of answer. Test three questions: How much luggage can I take? (policy 6.1), Can I bring my dog on the bus? (policy 6.3), and Is there Wi-Fi on the bus? (the policy does not say).
The check runs your tests 5 times on the real support.py (every run must pass), then on three broken copies: one sends no policy, one no longer says only, one sends half the policy.
Be ready to explain why the first test failed, and why temperature 0 is not the whole answer.
- Task 4
test_accuracy.py: an eval as a test
In test_accuracy.py, test classify on all 240 messages in /workspace/data/messages.jsonl against their true intents: one test that the accuracy is at least 90%, and one that every intent on its own is right at least 50% of the time (right for that intent divided by its messages). Classify the 240 messages once, in a fixture, and let both tests use the results.
The check runs your tests on the real support.py (all pass) and on three broken copies: no lost_item label, praise called feedback, and a prompt that lost its labels.
What you will learn
- Golden examples: a small set of inputs with answers everyone agrees on, run with parametrize.
- Asserting what matters: facts, labels and fields, never the exact wording of free text.
- Breaking the prompt on purpose to see whether a test catches it.
- An eval as a test: accuracy over a dataset, held above a threshold.
More in Reliable output
Build "Testing prompts" yourself
SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.