Agents from scratch · Chapter 16 of 18

Testing agents

By the end of this chapter you can test an agent at two levels: its loop with a fake client and scripted replies, in milliseconds and with no model; and whole runs against the model, by the tools they called and by what really changed in GharFix's data, never by the words of the reply. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Agents from scratch are free. This one comes with the all labs plan.

Why you need this

Every chapter so far changed the loop: validation, budgets, approval, permissions, traces, streaming, async. Vikram has a rule for code that books plumbers and moves money: nothing changes without a test that would have caught the mistake. Meenakshi has her own version: "Tell me it still books, reschedules and cancels, before you ship it, not after a customer tells me."

Testing an agent is not like testing a function. The model's reply varies from run to run, a run takes seconds, and the interesting failures are quiet: a loop that answers only the first call, an agent that says it booked when it did not. You need tests that are fast where they can be, real where they must be, and never fooled by a friendly sentence.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    fake_client.py: a fake client

    agent.py's run_agent(question, client=None) takes the client it talks to. In fake_client.py, write text(content): a reply shaped like the API's (a ChatCompletion) that answers with that text, finish reason stop; and tool_calls(*calls): a reply that asks for tools, each call a pair of a tool's name and its arguments as a dictionary, each with an id of its own and its arguments as JSON text, finish reason tool_calls.

    Then a class FakeClient(replies): fake.chat.completions.create(**kwargs) returns the next reply in the list, and fake.requests is a list of every request's keyword arguments, as they were when sent. A request with no reply left raises an error.

    The check builds replies with your functions, and runs the real agent.py with your FakeClient.

  2. Task 2

    test_loop.py: test the loop without the model

    In test_loop.py, write at least 4 pytest tests of agent.py's run_agent with your FakeClient: what its docstring promises. For example: a text reply is the answer in one request; every call of a reply gets its own tool message, with its id, after the model's own message, holding the tool's result as JSON; an unknown tool becomes an error result; the loop stops at max_steps.

    The check runs your tests with the lab model switched off. They must all pass on the real agent.py, and at least one must fail on each of five broken copies: one forgets the model's own message, one answers only the first call, one crashes on an unknown tool, one allows more requests than max_steps, one sends results that are not JSON.

  3. Task 3

    test_trajectories.py: golden trajectories, and what really happened

    In test_trajectories.py, write pytest tests of run_agent with the real lab model (no client given), with pytest.mark.parametrize for at least one: a booking that names a slot (its trajectory is find_technicians then book_slot, and a booking with the customer's phone exists for the right date, slot and job, with that slot now booked), a reschedule (reschedule_booking, and the booking's date and slot are the new ones) and a cancellation (cancel_booking, and the booking's status is cancelled). Take them from Meenakshi's jobs in /workspace/data/jobs.jsonl.

    Every test starts from fresh data, and a customer may already have older bookings with the same phone: look for the one the run made. The check runs your tests on the real agent.py (all pass) and on three broken copies: one never really books, one stops after one request, one runs get_booking for reschedules and cancellations.

  4. Task 4 · interview question

    Interview: test_flaky.py, the test that fails sometimes

    Vikram's two tests in test_flaky.py passed, then failed, then passed. Rewrite them so that they pass every time on the real agent.py, alone or together, in any order, and still catch an agent that only says it booked and one that stops before booking. Keep test_booking, test_second_booking and their questions. Do not change agent.py.

    The check runs your tests twice, with the second test first, and the second alone, then on the two broken copies. Be ready to explain why each one failed.

What you will learn

  • A fake client: scripted replies, so the loop runs with no model.
  • Loop tests: what the loop sends and returns, checked exactly.
  • Golden trajectories: the tools a job must call, from Meenakshi's jobs.
  • Outcomes: asserting on GharFix's data, not on the reply's words.

More in Production

Build "Testing agents" yourself

GharFix, a Bengaluru home-services company, needs an assistant that books, reschedules and refunds safely. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Agents from scratch are free. This one comes with the all labs plan.