Building with LLMs · Chapter 14 of 24

Classification and routing

By the end of this chapter you can make a model put each message into one of your own labels, stop it from answering with anything else, send each message to the right team, and measure how often the label is right. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Reliable output
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.

Why you need this

SafarLine's WhatsApp queue is one long list: a booking, then a lost bag, then an angry delay, then a refund question. Lata reads every message twice, once to understand it and once to decide who should handle it. The refund team waits while she forwards messages to them by hand.

What she does first is sort. That is a job a model does well, and it is different from answering: the result is not a sentence for a customer but one word from a short list, which your code then acts on. A word your code acts on has to be exactly right, every time: "Refund" is not refund_status, and a sentence is not a label at all.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    label.py: one label per message

    Lata sorts every message before anyone answers it. In label.py, write classify(text) that asks lab-chat, at temperature 0, to classify the customer's message as exactly one of these intents, all listed in the system message: booking_request, cancellation, refund_status, delay_complaint, fare_query, lost_item, date_change, praise. The customer's text is the user message.

    Return the label the model replied with (without spaces or newlines around it), or "unknown" when the reply is not one of the eight.

    The check calls classify on a few messages, one of them from Lata's queue whose reply is not a label at all.

  2. Task 2

    intent_schema.py: labels the model cannot get wrong

    In intent_schema.py, write a Pydantic model Intent with one field, intent, whose type allows only the eight intents of task 1. Write classify(text) that asks lab-chat to classify the message with Intent as the structured output schema, and returns the intent as text, such as lost_item.

    The check reads the schema your request sent: intent must be limited to exactly those eight values.

  3. Task 3

    route.py: send each message to its team

    In route.py, write team_for(intent), which returns the team for an intent: sales for booking_request, fare_query and date_change; refunds for cancellation and refund_status; operations for delay_complaint and lost_item; feedback for praise; and human for anything else (unknown, a typo, an empty string).

    Then write route(text), which classifies the message with your classifier from task 2 and returns its team.

  4. Task 4 · interview question

    Interview: accuracy.py, how good is the classifier?

    In accuracy.py, classify all 240 messages in /workspace/data/messages.jsonl with your classifier from task 2 and compare each label with the message's true intent.

    Print first the accuracy: accuracy, a colon, a space, right/total and the share in brackets with one decimal, for example accuracy: 200/240 (83.3%) (made-up numbers).

    Then one line per kind of mistake: the true intent, an arrow, the label given, a colon and the count, for example fare_query -> booking_request: 3, the most common first (equal counts by the true intent, then the label).

    Be ready to explain the biggest mistake: read the messages behind it. Is the model wrong, or is the message?

What you will learn

  • Classification: one label from a fixed list, given in the prompt.
  • Checking the label your code receives, and an enum that makes a wrong one impossible.
  • Routing: from a label to a team or a function, with a safe default.
  • Accuracy, and reading the mistakes behind the number.

More in Reliable output

Build "Classification and routing" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.