Building with LLMs (TypeScript) · Chapter 14 of 24

Classification and routing

By the end of this chapter you can make a model put each message into one of your own labels, stop it from answering with anything else, send each message to the right team with a switch the compiler checks, and measure how often the label is right. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Reliable output
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.

Why you need this

SafarLine's WhatsApp queue is one long list: a booking, then a lost bag, then an angry delay, then a refund question. Lata reads every message twice, once to understand it and once to decide who should handle it. The refund team waits while she forwards messages to them by hand.

What she does first is sort. That is a job a model does well, and it is different from answering: the result is not a sentence for a customer but one word from a short list, which your code then acts on. A word your code acts on has to be exactly right, every time: "Refund" is not refund_status, and a sentence is not a label at all.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    label.ts: one label per message

    Lata sorts every message before anyone answers it. In label.ts, export the eight intents as one array, as const, and the union type that comes from it:

    booking_request, cancellation, refund_status, delay_complaint, fare_query, lost_item, date_change, praise.

    Then export classify(text: string): Promise<Intent | 'unknown'>, which asks lab-chat, at temperature 0, to classify the customer's message as exactly one of the eight, all listed in the system message; the customer's text is the user message.

    Return the label the model replied with (without spaces or newlines around it), or 'unknown' when the reply is not one of the eight.

    The check calls classify on a few messages, one of them from Lata's queue whose reply is not a label at all.

  2. Task 2

    intent-schema.ts: labels the model cannot get wrong

    In intent-schema.ts, export a zod schema Intent with one field, intent, that allows only the eight intents of task 1 (z.enum), and its type: export type Intent = z.infer<typeof Intent>.

    Export classify(text: string): Promise<Intent['intent']>, which asks lab-chat to classify the message with Intent as the structured output schema, and returns the intent, such as lost_item.

    The check reads the schema your request sent: intent must be limited to exactly those eight values.

  3. Task 3

    route.ts: send each message to its team

    In route.ts, export the type Team = 'sales' | 'refunds' | 'operations' | 'feedback' | 'human', and teamFor(intent: Intent['intent'] | 'unknown'): Team (Intent from ./intent-schema.ts), which returns sales for booking_request, fare_query and date_change; refunds for cancellation and refund_status; operations for delay_complaint and lost_item; feedback for praise; and human for unknown.

    Write it as a switch with a case for each label and a default that assigns the intent to a never (so typecheck fails the day a ninth intent is added) and still returns human for anything else that arrives at run time (a typo, an empty string).

    Then export route(text: string): Promise<Team>, which classifies the message with your classifier from task 2 and returns its team.

    The check also adds a ninth intent to a copy of your schema and runs typecheck on route.ts: it must complain.

  4. Task 4 · interview question

    Interview: accuracy.ts, how good is the classifier?

    In accuracy.ts, classify all 240 messages in /workspace/data/messages.jsonl with your classifier from task 2 and compare each label with the message's true intent.

    Print first the accuracy: accuracy, a colon, a space, right/total and the share in brackets with one decimal, for example accuracy: 200/240 (83.3%) (made-up numbers).

    Then one line per kind of mistake: the true intent, an arrow, the label given, a colon and the count, for example fare_query -> booking_request: 3, the most common first (equal counts by the true intent, then the label, in alphabetical order).

    Be ready to explain the biggest mistake: read the messages behind it. Is the model wrong, or is the message?

What you will learn

  • Classification: one label from a fixed list, given in the prompt.
  • Checking the label your code receives, with a type guard, and a z.enum that makes a wrong one impossible.
  • Routing: from a label to a team, with an exhaustive switch and a safe default.
  • Accuracy, and reading the mistakes behind the number.

More in Reliable output

Build "Classification and routing" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant, in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.