Agents from scratch (TypeScript) · Chapter 16 of 18
Testing agents
By the end of this chapter you can test an agent at two levels with vitest: its loop with a fake client and scripted replies, in milliseconds and with no model; and whole runs against the model, by the tools they called and by what really changed in GharFix's data, never by the words of the reply. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Agents from scratch (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Every chapter so far changed the loop: validation, budgets, approval, permissions, traces, streaming, concurrency. Vikram has a rule for code that books plumbers and moves money: nothing changes without a test that would have caught the mistake. Meenakshi has her own version: "Tell me it still books, reschedules and cancels, before you ship it, not after a customer tells me."
Testing an agent is not like testing a function. The model's reply varies from run to run, a run takes seconds, and the interesting failures are quiet: a loop that answers only the first call, an agent that says it booked when it did not. You need tests that are fast where they can be, real where they must be, and never fooled by a friendly sentence.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
fake-client.ts: a fake client
agent.ts's runAgent(question, { client }) takes the client it talks to. In fake-client.ts, export text(content): a reply shaped like the API's (a ChatCompletion) that answers with that text, finish reason stop; and toolCalls(...calls): a reply that asks for tools, each call a [name, arguments object] pair, each with an id of its own, type function and its arguments as JSON text, content null, finish reason tool_calls.
Then export a class FakeClient: new FakeClient(replies); fake.chat.completions.create(body) resolves to the next reply in the list, and fake.requests is an array of every request's body, as it was when sent. A request with no reply left throws an error.
The check builds replies with your functions, and runs the real agent.ts with your FakeClient.
- Task 2
loop.test.ts: test the loop without the model
In loop.test.ts, write at least 4 vitest tests of agent.ts's runAgent with your FakeClient: what its comment promises. For example: a text reply is the answer in one request; every call of a reply gets its own tool message, with its id, after the model's own message, holding the tool's result as JSON; an unknown tool becomes an error result; the loop stops at maxSteps.
The check runs your tests with the lab model switched off. They must all pass on the real agent.ts, and at least one must fail on each of five broken copies: one forgets the model's own message, one answers only the first call, one crashes on an unknown tool, one allows more requests than maxSteps, one sends results that are not JSON.
- Task 3
trajectories.test.ts: golden trajectories, and what really happened
In trajectories.test.ts, write vitest tests of runAgent with the real lab model (no client given), with it.each over rows of /workspace/data/jobs.jsonl for at least one: a booking that names a slot (its trajectory is find_technicians then book_slot, and a booking with the customer's phone exists for the right date, slot and job, with that slot now booked), a reschedule (reschedule_booking, and the booking's date and slot are the new ones) and a cancellation (cancel_booking, and the booking's status is cancelled).
Every test starts from fresh data, and a customer may already have older bookings with the same phone: look for the one the run made. The check runs your tests on the real agent.ts (all pass) and on three broken copies: one never really books, one stops after one request, one runs get_booking for reschedules and cancellations.
- Task 4 · interview question
Interview: flaky.test.ts, the test that fails sometimes
Vikram's two tests in flaky.test.ts passed, then failed, then passed. Rewrite them so that they pass every time on the real agent.ts, alone or together, in any order, and still catch an agent that only says it booked and one that stops before booking. Keep both tests, their titles and their questions. Do not change agent.ts.
The check runs your tests twice, then with the second test first, then the second alone, then on the two broken copies. Be ready to explain why each one failed.
What you will learn
- A fake client: scripted replies, so the loop runs with no model.
- Loop tests: what the loop sends and returns, checked exactly.
- Golden trajectories: the tools a job must call, from Meenakshi's jobs, with it.each.
- Outcomes: asserting on GharFix's data, not on the reply's words.
More in Production
Build "Testing agents" yourself
GharFix, a Bengaluru home-services company, needs an assistant that books, reschedules and refunds safely. You build it in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Agents from scratch (TypeScript) are free. This one comes with the all labs plan.