Evals that catch regressions · Chapter 10 of 16

Agent evals

By the end of this chapter you can evaluate an agent by what it did, not only by what it said: grade the tools it called and their arguments, spot loops and extra calls, match a trajectory against the calls a correct run makes, and show why an eval of the final answer alone lets a dangerous agent through. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Evals in production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.

Why you need this

YatraDesk's booking agent is going on WhatsApp next month. It is a model with six tools: it searches packages, looks up bookings, reads the visa and baggage rules, and quotes refunds. Today every tool only reads. The next version will change bookings: cancel a trip, move a date.

Divya owns the release gate, and she has one question: "What does it call, before it touches a booking?" An answer can be right after a run that called the wrong tool, passed a wrong booking reference, or asked the same thing five times. On a tool that only reads, that costs time and money. On a tool that cancels a trip, the first wrong call is the one that matters, whatever the agent says afterwards.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    logged.py: grade the logged runs

    Write matches(step, call) (same tool, and every argument of the call with the same value in the step) and grade_run(steps, expected), which returns the first label that applies, in this order: no tool call (no steps); loop (the same tool with the same arguments twice or more); wrong tool (some expected tool never called); wrong argument (for some expected call, the first step using its tool does not match it); extra calls (more steps than the expected calls plus ALLOWED_EXTRA, which is 1); none.

    Run as a program, grade the 14 runs of /workspace/data/trajectories.jsonl and print each label next to the logged fault, then how many agree:

    t101: none (logged: none)

    t102: loop (logged: loop)

    Matches the logged fault: 2 of 2

  2. Task 2

    live.py: the agent on three models

    Write live_runs(model), run_agent from yatradesk.agent on every logged run's question in the file's order, 8 at a time with a ThreadPoolExecutor, and as_steps(trajectory), the trajectory's steps as dicts with tool and args. Run as a program, for lab-chat, lab-chat-confused and lab-chat-loopy, grade every live run with grade_run and print how many call the first expected tool first, the total tool calls, the runs stopped at max_steps, and the count of each label that occurs, in the order none, wrong tool, wrong argument, loop, extra calls, no tool call:

    lab-chat: right first tool 9/10, steps 12, stopped at max_steps 0; none 8, extra calls 2

  3. Task 3

    trajectory_match.py: exact, in order, any order

    Write exact(steps, expected) (as many steps as calls, each matching the call at its position), in_order(steps, expected) (the calls appear among the steps in order, anything in between) and any_order(steps, expected) (every call matches some step, each step used at most once), all with matches from logged.py. Run as a program, count each kind of match for the logged runs, then for live runs with lab-chat and lab-chat-loopy:

    logged: exact 6/10, in order 8/10, any order 9/10

    lab-chat: exact 7/10, in order 9/10, any order 9/10

    lab-chat-loopy: exact 0/10, in order 9/10, any order 9/10

    Then decide which matcher suits YatraDesk's booking agent.

  4. Task 4 · interview question

    Interview: final_answer_only.py, the right answer the wrong way

    Divya's first agent eval only checks the final answer (answer_ok), and says lab-chat-confused is right on almost every run. Keep answer_ok and add the trajectory: a run is fully right only when answer_ok is True and grade_run says none. Print both counts, then every run whose answer is right but whose trajectory is not, with its label and number of tool calls:

    Answer only: 9/10 right

    Answer and trajectory: 7/10 right

    t004: right answer, wrong argument, 2 calls

    Be ready to explain what an answer-only eval misses on an agent that books and cancels trips.

What you will learn

  • Trajectories: the record of an agent's run, step by step.
  • Grading a run: the right tool, the right arguments, loops, extra calls.
  • Live runs on three models, graded the same way.
  • Matching trajectories exactly, in order, or in any order.
  • Why the final answer is not enough.

More in Evals in production

Build "Agent evals" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.