Pydantic AI · Chapter 17 of 23
Testing agents
By the end of this chapter you can test an agent without a real model: block real requests, swap the model with agent.override, run the agent on TestModel and on a FunctionModel you script yourself, read every message of a run with capture_run_messages, and say what still needs the real model. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Graphs
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Pydantic AI are free. This one comes with the all labs plan.
Why you need this
Kedar changed one line of the farmer assistant last week: the instructions now name the farmer's village. The next morning Ganesh found that every farmer in Dindori got their onion price from Lasalgaon, not from Pimpalgaon, their own mandi. Nothing crashed. The answers just looked right and were wrong.
A test would have caught it. But a test that asks the real model is slow (a second or more per request), costs money on every run, needs a network, and gets slightly different words each time. You want hundreds of tests that run in a second, offline, on every change, and give the same answer every time. Pydantic AI comes with fake models for exactly that, and a switch that makes sure no test reaches a real one by accident.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
test_reply.py: the agent offline with TestModel
assistant.py (do not change it) is KhetBazaar's farmer assistant: an agent whose instructions name the farmer and their language, three tools, and reply(farmer_id, message), the answer the farmer reads. Write pytest tests for it with TestModel, swapped in with agent.override, and block real model requests in the file with models.ALLOW_MODEL_REQUESTS = False. They must pass on assistant.py as it is.
The check runs them again on three copies with one bug planted in each: the instructions say English for every farmer; credit_balance is not registered as a tool; reply keeps the spaces around the model's output. Each copy must make at least one test fail, and your tests must send no request to the lab model.
- Task 2
test_scripted.py: tool calls with FunctionModel
Write pytest tests for assistant.py's tools with a FunctionModel you write: a function of the messages so far that first asks for one tool (a ToolCallPart) and then answers in text. Check the tool's result that went back to the model (the ToolReturnPart) against khetbazaar. They must pass on assistant.py, send no request to the lab model, and fail on each of three copies: mandi_price always uses Lasalgaon; book_collection ignores graded; credit_balance always reads F1001's credit.
- Task 3
test_tools.py: when a tool fails
Write pytest tests for the tools' errors: FunctionModels that ask for bad arguments, with the run's messages read through capture_run_messages. A tool's error goes back to the model as a RetryPromptPart. They must pass on assistant.py, send no request to the lab model, and fail on each of three copies: mandi_price lets khetbazaar's ValueError out for a crop KhetBazaar does not buy; book_collection tells the model only Please try again.; reply crashes when the model never stops asking for a bad crop, instead of returning SORRY.
- Task 4 · interview question
Interview: test_real.py, what the real model chooses
Write one test that runs assistant.py's agent on the real lab model (no override) for farmer F1003 and the message aaj kanda bhav kay ahe?, and checks the model's first tool call: mandi_price, with the crop onion.
The check runs it on assistant.py (it must pass), on a correct copy whose tool result has one more key so the model's words differ (it must pass too), and on a copy whose mandi_price calls its parameter item (it must fail). The interviewer will ask: what do you test with a fake model, what with a real one, and how often do you run each?
What you will learn
- ALLOW_MODEL_REQUESTS: tests that cannot reach a real model.
- agent.override: a different model for the code under test.
- TestModel: a model that calls your tools and answers on its own.
- FunctionModel: a model you write, step by step.
- capture_run_messages: every message of a run, even a failed one.
More in Graphs
Build "Testing agents" yourself
KhetBazaar, a Nashik agritech startup, answers farmers on WhatsApp from a bike. You build its farmer assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Pydantic AI are free. This one comes with the all labs plan.