Pydantic AI · Chapter 19 of 23

Tracing

By the end of this chapter you can say what a trace and a span are, switch on Pydantic AI's OpenTelemetry instrumentation and export a run's spans to the console, keep spans in memory and read them in code, find which step of a run was slow, and decide what a trace may hold and what it must never hold. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Pydantic AI are free. This one comes with the all labs plan.

Why you need this

The farmer assistant answers real farmers now, and the first complaints have reached Ganesh: "Sometimes it takes ten seconds to say whether it will rain." Sneha wants to know what each conversation costs. Pradnya wants to know, when a farmer gets a wrong price, which tool said it.

print will not do. A run is several requests and tool calls, some of them at the same time, on a server where nobody reads the terminal. You need a record of every step: what it was, when it started, how long it took, and what it cost, kept and sent somewhere you can search. That record is a trace, and Pydantic AI can write it for you.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    instrument.py: a run, traced to the console

    Ganesh wants to see what one run really does. Set up OpenTelemetry: a TracerProvider whose span processor exports each span to the console as it ends, made the global provider, and switch instrumentation on for every agent. Then make an agent for openai-chat:lab-chat with the instructions You are KhetBazaar's farmer assistant. and run it on the farmer's question exactly:

    Do you buy pomegranate that has cracked skin?

    Print the run's output. The console exporter prints the spans as JSON: the check reads them, so print nothing else.

  2. Task 2

    spans.py: a run's spans as a tree

    Keep a run's spans in memory instead: an in-memory exporter on a provider of the agent's own (not the global one). Make an agent for openai-chat:lab-chat named khetbazaar, with the instructions You are KhetBazaar's farmer assistant. Use the tools. and two tools that call khetbazaar.mandi_price(crop, mandi) and khetbazaar.weather_forecast(taluka), each with a docstring. Run it on: Will it rain in Sinnar this week, and what is the tomato rate in Pimpalgaon?

    Then print one line per span, oldest first (by start time): the span's name, indented two spaces when it has a parent; a chat span's line ends with a colon, a space and its tokens as N in, N out (from its gen_ai.usage.input_tokens and gen_ai.usage.output_tokens attributes).

  3. Task 3

    slow_tool.py: find the slow tool

    Write slowest_tool(spans): given the finished spans of one run (ReadableSpan objects, as an in-memory exporter returns them), return a tuple of the name of the tool whose execution took longest and how long it took in milliseconds, rounded to a whole number: ("weather_forecast", 1212).

    The check passes spans of runs where different tools are slow, and where requests to the model took longer than any tool. Take the tool's name from the span's gen_ai.tool.name attribute.

  4. Task 4 · interview question

    Interview: private.py, what never goes in a trace

    Export spans to the console as in task 1, but with Pydantic AI's instrumentation settings told to leave out the content of messages. Run an agent for openai-chat:lab-chat with the instructions You are KhetBazaar's farmer assistant. on the farmer's message exactly:

    My payment is late. Account 50100234567812, IFSC HDFC0001234, phone 9822012345.

    Print the run's output. The check reads the spans: they must keep the model and the token counts, and hold neither the account number, the phone number nor the reply.

    The interviewer will ask: what should a trace hold, and what should it never hold?

What you will learn

  • Traces and spans: what OpenTelemetry records, and why it is a tree.
  • Instrumenting agents: a tracer provider, an exporter, and one switch.
  • Reading spans in code: names, parents, times and attributes.
  • Finding the slow step, comparing like with like.
  • What a trace holds, and leaving the farmer's words out of it.

More in Production

Build "Tracing" yourself

KhetBazaar, a Nashik agritech startup, answers farmers on WhatsApp from a bike. You build its farmer assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Pydantic AI are free. This one comes with the all labs plan.