OpenAI Agents SDK · Chapter 20 of 24
Testing agents
By the end of this chapter you can test an agent's tools without a model, write a scripted model that plays the model's part in a run, check the instructions, tools and inputs the model was given, test a guardrail on its own, keep the tests from ever calling a real model, and say what still needs a real one. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of OpenAI Agents SDK are free. This one comes with the all labs plan.
Why you need this
Karthik changed one line of the helpdesk last week: the instructions now mention the student's batch. Did anything else break? Kavita wants to know before every change goes live, not after a parent complains. Running the assistant by hand on twenty messages takes an afternoon, costs tokens, and gives slightly different words every time, so "it looks the same" is hard to judge.
Most of what can break in an agent is your code, not the model: a tool that reads the wrong student, instructions with the wrong language, a guardrail that never fires, a reply with stray spaces. None of that needs a model to test. Tests are small functions that run your code and assert what it must do; pytest runs them all in a second, every time anything changes.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
test_tools.py: test the tools, without a model
assistant.py (do not change it) is Shikhar's helpdesk: an agent with three tools, a guard agent and its off-limits guardrail, and reply(student_id, message), the answer a student reads. Write pytest tests for its three tools, fee_status, refund_quote and class_schedule, with no model: run each one the way the SDK does, with its on_invoke_tool, a ToolContext (from agents.tool_context) holding a Student, and the arguments as JSON text, and compare what it returns, read as JSON, with shikhar's own function. They must pass on assistant.py as it is.
The Check runs them again on three copies with one bug planted in each: fee_status reads another student's fees; refund_quote returns a dictionary, not JSON text; class_schedule ignores the day it is given. Each copy must make at least one test fail.
- Task 2
test_scripted.py: a scripted model
Write pytest tests for assistant.py's helpdesk with a model you write: a subclass of Model (from agents.models.interface) whose async get_response returns a ModelResponse of the output items you scripted, one reply per request, and keeps what it was given (the instructions, the input, the tools); its stream_response can raise NotImplementedError. Script replies with openai.types.responses items: a ResponseOutputMessage of ResponseOutputText for text, a ResponseFunctionToolCall for a tool call. Give the helpdesk your model through its model attribute, and the guard one too (reply runs both). One test scripts a call to fee_status, then text, and checks the tool result the model saw in its next input.
They must pass on assistant.py, send no request to the lab model, and fail on each of three copies: the instructions say English for every student; refund_quote is not offered to the model; reply keeps the spaces around the model's output.
- Task 3
test_guardrail.py: the guardrail, tested alone
Write pytest tests for assistant.py's off_limits guardrail. Give the guard agent a scripted model that answers with its verdict as JSON text (like {"off_limits": true, "reason": "asks for a test paper"}), run the guardrail alone with its run method (the helpdesk, the message and a RunContextWrapper of a Student), and read from the result's output whether the tripwire fired. Then test reply: an off-limits message gets REFUSAL.
They must pass on assistant.py, send no request to the lab model, and fail on each of three copies: the tripwire never fires; the tripwire fires on every message; reply lets the tripwire's exception out. reply runs the helpdesk as well as the guard: give both a scripted model.
- Task 4 · interview question
Interview: test_real.py, what still needs a real model
Write one test that runs assistant.py's helpdesk on the real lab model (do not change its model) for student S2061 and the message JEE-28X ka kal ka timetable kya hai?, and checks the model's first tool call, read from the result's items: class_schedule, with the arguments batch_id JEE-28X and day tomorrow.
The Check runs it on assistant.py (it must pass), on a correct copy whose tool result has one more key so the model's words differ (it must pass too), and on a copy whose class_schedule calls its day parameter when (it must fail). The interviewer will ask: what do you test with a scripted model, what with a real one, and how often do you run each?
What you will learn
- pytest in a minute: test functions, assert, running them.
- Tools as plain functions: running a function tool the way the SDK does, with on_invoke_tool.
- A scripted model: a Model subclass whose get_response returns the output items you wrote, and keeps what it was sent.
- Guardrails alone: a guardrail's run, with a scripted guard.
- The real model: the one kind of test that needs it, and what it should assert.
More in Production
Build "Testing agents" yourself
Shikhar Classes, a Kota coaching institute, answers parents and students on WhatsApp all day. You build its helpdesk assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of OpenAI Agents SDK are free. This one comes with the all labs plan.