Pydantic AI · Chapter 18 of 23
Evals
By the end of this chapter you can turn real farmers' messages into a dataset of cases, score an agent on it with built-in and your own evaluators, read the report case by case, and compare two versions of an agent with numbers instead of a feeling. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Graphs
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Pydantic AI are free. This one comes with the all labs plan.
Why you need this
Chapter 17's tests check KhetBazaar's code: the right tool, the right mandi, the right retry. They cannot say how good the assistant is. Is the new prompt better than the old one? Does lab-chat-mini classify Marathi as well as lab-chat? Did adding a tool help, or only feel helpful?
Pradnya rewrote the classifier's prompt last week because the old one "sounded cold". Ganesh read three replies and liked them. Nobody checked the other 26 messages. Kedar's rule: a change to a prompt, a model or a tool ships when the numbers on the same cases say it is better. Collecting those numbers is an eval, and pydantic-evals, from the Pydantic team, is built for it.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
dataset.py: a dataset of farmers' messages
/workspace/data/messages.jsonl holds 29 messages Ganesh labelled, one JSON object per line with an id, the text and the label. Make them a pydantic-evals Dataset named intents, with one Case per line in the file's order (its name the id, its inputs the text, its expected output the label) and the evaluator EqualsExpected for the whole dataset. Write it as Dataset[str, str, Any].
Save it to intents.yaml in this folder, load it back from that file, and print two lines: the number of cases loaded and the word cases (29 cases), then the first case's name, inputs and expected output joined by " | ".
- Task 2
evaluators.py: an evaluator of your own
Make an agent for openai-chat:lab-chat with the instructions You are KhetBazaar's farmer assistant. and a mandi_price(crop, mandi) tool from khetbazaar, and a dataset of the four cases in the starter, each with its crop and mandi in its metadata. Write an evaluator, QuotesTodaysPrice, that passes when the output holds today's modal price for the case's crop and mandi, from khetbazaar.
Evaluate the agent with an async task (progress=False) and print one line per case: its name, a colon and a space, then True or False from the evaluator.
- Task 3
report.py: reading a report
Make the intents dataset from /workspace/data/messages.jsonl again (as in task 1, with EqualsExpected), and an agent for openai-chat:lab-chat whose output type is a Literal of the nine intents (price, sell, weather, pest, order, claim, payment, complaint, praise), with the instructions Classify the farmer's WhatsApp message by what they want. Evaluate it with an async task (progress=False).
From the report, print one line per case it got wrong, sorted by name, as W013: expected pest, got order, then a last line Accuracy: and the average of the assertions as a percentage with one decimal (Accuracy: 93.1%).
- Task 4 · interview question
Interview: compare.py, a prompt that feels better
Measure the two prompts in the starter on the intents dataset (with EqualsExpected): for each, an agent for openai-chat:lab-chat with the prompt as its instructions and plain text output. Evaluate prompt A, then prompt B (async tasks, progress=False), and print two lines: A: and its average of the assertions, then B: and its, each as a percentage with one decimal.
The interviewer will ask: why not trust that B "feels better", and what would you check before shipping it?
What you will learn
- Case and Dataset: inputs, the expected output, metadata; saved as YAML.
- evaluate_sync with an async task.
- Evaluators: EqualsExpected, and your own.
- The report: cases, assertions, averages, failures.
- Comparing two runs with a baseline.
More in Graphs
Build "Evals" yourself
KhetBazaar, a Nashik agritech startup, answers farmers on WhatsApp from a bike. You build its farmer assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Pydantic AI are free. This one comes with the all labs plan.