OpenAI Agents SDK · Chapter 21 of 24

Evals

By the end of this chapter you can turn Shikhar's real messages into a dataset of cases, run an agent on every case, grade each output with an exact match, with the tools the run called, and with a judge agent, and compare two versions of an agent with numbers instead of a feeling. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of OpenAI Agents SDK are free. This one comes with the all labs plan.

Why you need this

Chapter 20's tests check the helpdesk's code: the right tool, the right student, the approval that waits. They run on a scripted model, so they cannot say how good the real assistant is. Does the new prompt classify better than the old one? Would lab-chat-mini do as well, for a fraction of the price? Did the refund answers improve, or only sound nicer?

Bhanu rewrote the classifier's prompt last week because the old one "sounded cold". He read three replies and liked them. Nobody looked at the other 29 messages. Kavita's rule from now on: a change to a prompt, a model or a tool ships when the numbers on the same cases say it is better. Collecting those numbers is an eval.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    intent_eval.py: a dataset and an intent score

    /workspace/data/messages.jsonl holds 32 messages Bhanu labelled, one JSON object per line with an id, the text and the label. Make them a dataset: one case per line, in the file's order, with its id, its message and its expected label.

    Make a classifier agent for lab-chat with the instructions Classify the WhatsApp message by what the sender wants. and the output type Triage, a Pydantic model with one field, intent, a Literal of the 11 intents (admission, fees, refund, schedule, doubt, results, hostel, complaint, praise, off_limits, counsellor). Run it on every case and grade each with an exact match: the intent equals the label. Print one line per case it got wrong, as M014: expected doubt, got schedule, then a last line Score: right/32 and the share with one decimal in brackets, like Score: 31/32 (96.9%).

  2. Task 2

    trajectory.py: a trajectory grader

    Make a helpdesk agent for lab-chat with the instructions You are Shikhar Classes' helpdesk assistant. Use the tools. and five tools that return json.dumps of shikhar's functions of the same names: fee_status(student_id), refund_quote(student_id), refund_status(refund_id), test_results(student_id) and book_doubt_session(student_id, subject).

    Run the seven cases in the starter, each a name, a message and the tools a right run calls. Read the tools each run called from its new items. A case passes when they equal the expected list, in order (no tools expected: the run called none). Print one line per case, in order: its name, a colon, PASS or FAIL, then the tools called joined by a comma and a space, or (no tools); then a last line Trajectory: passed/7.

  3. Task 3

    judge.py: an LLM judge

    Make a helpdesk agent for lab-chat with the instructions You are Shikhar Classes' helpdesk assistant. Use the tools. and a refund_quote(student_id) tool. For S2061, S2053, S2072 and S2052, in that order, ask it How much refund will S2061 get if they leave today? (with that student's id).

    Make a judge agent for lab-chat with the instructions Grade the answer against the reference answer. Score it from 1 to 5 by how many key facts of the reference it states. and the output type Verdict with score (int), passed (bool) and reason (str). Send it three lines: Question: the question, Answer: the helpdesk's answer, Reference: The refund is ₹N. where N is refund_quote's total from shikhar. Print one line per student, as S2061: 1 FAIL (the score, then PASS or FAIL), then Judge: passed/4 passed.

  4. Task 4 · interview question

    Interview: compare.py, a prompt that feels better

    Measure the two prompts in the starter on the 32 messages: for each, an agent for lab-chat with the prompt as its instructions and plain text output. A case is right when the reply, stripped of spaces, equals its label. Run every case with prompt A, then every case with prompt B, and print three lines: A: right/32, B: right/32, then B misses: and the ids of the cases A got right and B got wrong, in the file's order, joined by a comma and a space (B misses: none when there are none).

    The interviewer will ask: why not trust that B "feels better", and what would you check before shipping it?

What you will learn

  • What an eval is: a dataset, the task, a grader, a score.
  • Exact graders: the typed output against a label.
  • Trajectory graders: the tools a run called, from its items.
  • An LLM judge: a second agent that grades free text against a reference.
  • Comparing versions: the same cases, two agents, the cases that changed.

More in Production

Build "Evals" yourself

Shikhar Classes, a Kota coaching institute, answers parents and students on WhatsApp all day. You build its helpdesk assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of OpenAI Agents SDK are free. This one comes with the all labs plan.