Building with LLMs · Chapter 12 of 24

Structured output

By the end of this chapter you can describe the reply you want as a Pydantic model, have the API hold the model to it with a strict schema, validate what comes back, and repair a bad reply by telling the model what was wrong. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Reliable output
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.

Why you need this

Chapter 11 ended on a warning: JSON mode promises valid JSON, not your keys or your types. Nikhil has seen what that costs. A reply with "seats": "2" passes every test with the number 2 and breaks on the day someone adds it to another number; a reply with an extra key or a renamed one fails three functions away from the request that caused it.

You already know the tool that fixes this in Python: Pydantic. A model class says what each field is, and Pydantic refuses data that does not fit. Structured output takes the same class to the API, so the model is held to it while it writes.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    parse_booking.py: a booking as a typed object

    Write a Pydantic model Booking with the fields origin, destination and name (text) and seats (a whole number), each of which may be None. Ask lab-chat for it with the parse method and response_format=Booking: a system message saying to extract the booking, and Dhruv's message (in the starter) as the user message.

    Print one line from the parsed Booking: the name, a colon, a space, the seats, the words seat(s), a comma, then origin -> destination, for example:

    Asha Rao: 2 seat(s), Pune -> Goa

  2. Task 2

    triage.py: enums for labels

    Write a Pydantic model Triage: intent, exactly one of booking_request, cancellation, refund_status, delay_complaint, fare_query, lost_item, date_change, praise; urgency, exactly one of high and normal; pnr, text or None; language, text.

    For each of the first 10 messages in /workspace/data/messages.jsonl, ask lab-chat with parse and response_format=Triage (a system message saying to triage the message, the text as the user message). Print the id, intent, urgency, the PNR (or - when it is None) and the language, separated by spaces, for example:

    M200 lost_item normal SL7K2Q9M en

  3. Task 3

    repair.py: repair by asking again

    Write a Pydantic model Booking (name and phone: text or None; seats: a whole number or None) and a function get_booking(client, text) that asks lab-chat-sloppy (no response format; a system message to extract the booking and reply in JSON with the keys name, phone and seats; the text as the user message), validates the reply with Booking.model_validate_json, and on ValidationError adds the bad reply as an assistant message and Pydantic's error as a user message and asks again with the whole list. It returns the Booking and the attempt number, and gives up with an error after 3 attempts. Do not clean the reply in code.

    When the file is run, call it for Dhruv's message and print the attempt, its number, a colon, a space and booking.model_dump(), for example:

    attempt 2: {'name': 'Asha Rao', 'phone': None, 'seats': 2}

  4. Task 4 · interview question

    Interview: fix_schema.py, fix the strict schema

    fix_schema.py sends a hand-written strict schema and validates each reply with the Booking model, printing booking.model_dump() for two messages. It has three mistakes: two the API refuses and one Pydantic refuses. Fix them until it prints two lines; a message with no seat count must still work, with seats None.

    Keep the messages, strict mode and the three keys.

What you will learn

  • JSON schemas and the json_schema response format.
  • Strict mode and its two rules.
  • Pydantic with .parse: a class in, an object out; Literal for labels.
  • Validation and repair: what the schema cannot promise, and asking again.

More in Reliable output

Build "Structured output" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.