Build an AI gateway · Chapter 5 of 21

Routing by task, cost and latency

By the end of this chapter you can measure which models do which of Jawab's jobs right, how fast and for how much, turn those measurements into a route table, and send each request to the model that fits its job. You will cut Jawab's bill by a third without one wrong answer. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: One API over three providers
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.

Why you need this

Chapter 4 gave Lakshmi her numbers, and they are not good: everything goes to nimbus-large, the most expensive model of the five, and the whole day costs ₹54.28 against a budget of ₹27.50. Most of Jawab's requests are small jobs: put a ticket in a category, look up a fact, translate a short reply. A model that costs a sixteenth as much might do them just as well. Or it might not, and then customers get wrong answers, which Anjali will not accept to save a few rupees.

Joseph has a third concern: speed. A customer typing in Kaapi Kottai's chat widget is waiting for the FAQ answer; nobody is waiting for the summary that goes to the next agent's screen. The fastest model for one is not the cheapest for the other.

This is routing: choosing, for each request, the model that does its job right, as cheaply and as fast as the job needs. Every serious gateway does it (OpenRouter, Portkey and LiteLLM all route by name, by alias and by rules). Darwaza can now call any of five models by name (chapter 2) and price every answer (chapter 4). What it lacks is the knowledge of which model is good for what, and the rule that uses it.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    quality.py: which model is good enough for which job?

    Write async def measure(target, start, end) -> list[dict]: replay start..end with every request sent to target (a gateway that returns await handle({**req, "model": target}), handle from darwaza.py), and return one dict per feature (triage, reply, faq, summary, translate) from that feature's results with ok True: {"target", "feature", "requests", "wrong" (right False), "p50" (percentile of their ms, 50), "inr_per_answer" (their cost_inr added up over requests; 0 if none)}.

    Under if __name__ == "__main__":, measure nimbus/nimbus-large, nimbus/nimbus-mini, kestrel/kestrel-pro, kestrel/kestrel-swift and tara/tara-flash on 11:00 to 13:00, and print for each, features in that order (₹ with 4 decimals):

    TARGET: wrong triage A, reply B, faq C, summary D, translate E

    p50 triage N ms, reply N ms, faq N ms, summary N ms, translate N ms

    ₹ per answer triage X, reply X, faq X, summary X, translate X

    Trap: count only answered results: a failed request is not a wrong answer, and has no time or price that says anything about the model.

  2. Task 2

    routes.py: the route table

    From quality.py's measurements, write ROUTES (for each feature, the targets to use, best first), ALIASES (fast: the fastest model, cheap: the cheapest on the price sheet, smart: a large model) and def route(req) -> list[str]: the app's named model (req["model"], resolved through ALIASES) first, then the feature's route without it; a feature with no route uses reply's.

    The rules for each list: only models with no wrong answers for the feature; only small ones (nimbus-mini, kestrel-swift, tara-flash) when any small one is right; triage and faq fastest first (p50, a customer is waiting); every other feature cheapest first per answer, as measured.

    Trap: the price sheet is not the bill: compare the measured rupees per answer, which include the providers' caches. And route returns a new list: it never changes ROUTES.

  3. Task 3

    gateway.py: Darwaza, routed

    Write async def handle(req) -> GatewayResponse: send req with call (darwaza.py) to route(req)[0] (routes.py) and answer with respond(attempt) (darwaza.py), which prices an answer and gives a failure the app's status. One call per request.

    Under if __name__ == "__main__":, replay 10:00 to 12:00 through handle with the label "Darwaza, routed" and print the report.

    Trap: the app's named model or alias still comes first: use route, not the table directly.

  4. Task 4 · interview question

    Interview: pick.py, routes from measurements

    Write def pick_routes(rows) -> dict[str, list[str]]: for each feature in rows, the targets with requests > 0 and wrong 0; only the small ones (nimbus/nimbus-mini, kestrel/kestrel-swift, tara/tara-flash) if any of those is small; triage and faq ordered by p50, every other feature by inr_per_answer, lowest first.

    Under if __name__ == "__main__":, measure all five models (measure from quality.py) on 15:00 to 17:00 and print one line per feature, in the order triage, reply, faq, summary, translate:

    FEATURE: TARGET, TARGET, ...

    Trap: a model with no answers for a feature was not measured on it: leave it out, even though its wrong count is 0.

What you will learn

  • Measuring quality: each model on each job, on real traffic, counting wrong answers.
  • Speed and cost per job: p50 and rupees per answer, measured, not read from a price sheet.
  • A route table: for each feature, the models to use, best first.
  • Aliases and named models: fast, cheap, smart, and an app that asks for a model by name.
  • What routing saves, measured on Jawab's day.

More in One API over three providers

Build "Routing by task, cost and latency" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.