AI ordering agent project · Chapter 9 of 12

Tracing

By the end of this chapter every step your agent takes is written to Postgres the moment it happens: each model call with what it asked for, its tokens and its time, and each tool call with its arguments, its result and its error. A member reads their own trace, Bhavna reads anyone's, and the totals say how many calls, tokens and milliseconds a conversation cost. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ready for members
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI ordering agent project are free. This one comes with the all labs plan.

Why you need this

In the pilot week, when an order went wrong, the question was always the same: did the member say 1 pm, or did someone type 1 pm? Bhavna had WhatsApp on one screen and the API's log on the other, and for four of the 19 wrong orders nobody could tell.

With the agent, the complaints will sound like "the agent booked the wrong day" or "it said the slot was full". The server log from the ordering backend shows that POST /orders answered 201 in 9 ms; it does not show which date the model wrote, what find_slots returned, or which slot id the model then picked. Chapter 6 stores the messages, but not how long each step took, how many tokens it used, or which steps failed. Bhavna needs the run itself, step by step, and Rafiq needs its token count: a trace.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    tracing.py: trace a model step

    In agent/tracing.py, write Tracer.model_step(reply, sent, ms): one row through self.add(...) with kind "model", name agent_settings.model, input {"messages": sent}, output {"tool_calls": [{"name": c.name, "arguments": arguments_of(c)}, ...]} when the reply has calls or {"text": reply.text} when it does not, prompt_tokens and completion_tokens from reply.usage, and ms.

    The Check traces a reply that asks for find_slots (4 messages sent, 810 and 52 tokens, 731 ms) and an answer, then reads agent_steps: two rows with the run's id and every column as above.

  2. Task 2

    tracing.py: trace a tool step

    In agent/tracing.py, write Tracer.tool_step(call, result, ms): one row with kind "tool", name call.name, input arguments_of(call), output result, error result["error"] when result is a dict with an "error" key (else None), and ms.

    The Check traces find_dish with a list as its result, place_order with a slot_full error result and a call whose arguments are not JSON, and wants the arguments as JSON (or the text), the whole result as output, the error's text in error, and the ms.

  3. Task 3 · interview question

    Interview: tracing.py, read the trace

    In agent/tracing.py, write trace_view(db, conversation_id, user): when the conversation does not exist, or is not the user's own and the user is not an admin, raise not_found("There is no conversation <id>"). Return {"conversation_id", "steps": [{"run_id", "kind", "name", "input", "output", "error", "prompt_tokens", "completion_tokens", "ms"}, ...] in the order they were written (by id), "totals": {"model_calls", "tool_calls", "tool_errors", "prompt_tokens", "completion_tokens", "ms"}}, the token sums counting a missing count as 0.

    The Check runs Tara's booking through the app, then reads her trace: model, find_slots, model, find_dish, model, place_order, model, with the step keys above and totals equal to what the lab model really received (4 model calls, 3 tool calls, the tokens, the sum of the ms). Advait gets a 404, conversation 999 a 404, Bhavna (admin) the whole trace; GET /agent/conversations/latest/trace answers 200.

    Kunal will ask: Tara says the agent booked the wrong day. What do you look at in her trace, and in what order?

What you will learn

  • A trace: one row per step of a run, model or tool, in the order they happened.
  • Hooks: how the loop reports each step without knowing where the report goes.
  • What to keep for a step: inputs, outputs, errors, tokens and time, and what not to copy again.
  • Committing as you go: a run that fails half way still leaves its steps.
  • Reading it back: only the member's own trace, or an admin's view; totals from the same rows.

More in Ready for members

Build "Tracing" yourself

DabbaDrop members want to order by chat. You build the agent that books their tiffins through DabbaDrop's own API, and asks before it pays. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI ordering agent project are free. This one comes with the all labs plan.