Build an AI gateway · Chapter 1 of 21

Meet Jawab, and Darwaza's first route

By the end of this chapter you can say what an AI gateway is and why companies put one in front of their models, call a provider by hand and read what it sends back, build Darwaza's first route, replay real traffic through it, and judge it with the numbers a team agreed on before anything broke. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: One API over three providers
  • 4 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Jawab, in Kochi, answers customers for six brands from Kerala: Kaapi Kottai's filter coffee, Nila Handlooms' kasavu sarees, Payyoli Snacks' banana chips, Thenga Naturals' coconut oil, Malabar Spice Box's pepper and Kayal Pickles' fish pickle. Its app does five jobs with AI: triage (what is this ticket about, how urgent), reply (a draft answer in the brand's voice), faq (a product question answered from the brand's facts), summary (a conversation, for the person taking it over) and translate (into Malayalam, Hindi or Tamil).

Every one of those features calls an AI provider itself. Last Tuesday, Nimbus, the provider they all use, went down for forty minutes after lunch. Every brand's customers got nothing, and each request hung for six seconds before it failed. Anjali, Jawab's founder, wants that never to happen again. Lakshmi, who pays the bills, wants to know what each brand's AI costs, and wants it to cost less. Joseph, who is on call, wants to know what failed and why, without reading five codebases.

Farhan, the platform engineer, has the answer most companies arrive at: put one door in front of every model. Every AI request goes through it; behind it, the door decides which provider and which model answers, what happens when one fails, what may be cached, who has used how much, and what gets logged. That door is an AI gateway (LiteLLM, Portkey, OpenRouter and Cloudflare's AI Gateway are well-known ones). Farhan has named Jawab's Darwaza, and you will build it with him, in Python.

The lesson

Three providers, three APIs

Jawab has accounts with three providers. They do the same work and agree on almost nothing else:

ProviderSpeaksModelsPrice (₹ per million tokens, in / out)
NimbusOpenAI's Chat Completions formatnimbus-large, nimbus-mini250 / 1,000 and 15 / 60
KestrelAnthropic's Messages formatkestrel-pro, kestrel-swift150 / 750 and 20 / 100
TaraGoogle's Gemini formattara-flash8 / 32

lab providers in the terminal lists them, with their limits and their cache prices. The lab has no internet, so these three are the lab's own stand-ins, at $NIMBUS_BASE_URL, $KESTREL_BASE_URL and $TARA_BASE_URL. Each speaks the real format it is named after, counts tokens its own way, bills at its own prices and fails in its own way, on a schedule the lab keeps (Nimbus's outage is at 14:00, every time). What they write is predictable, not a neural network: the same request always gets the same answer, which is how this lab can tell whether Darwaza gave a customer the right one. A gateway does not care what a model thinks; it cares where the request goes, how long it takes, what it costs and what happens when it fails. Those parts are real.

One request, by hand

This is what each feature of Jawab's app does today. The first request of Tuesday was a summary for Payyoli Snacks. Send it to Nimbus with httpx, the HTTP client this lab uses:

nimbus = PROVIDERS["nimbus"]
res = httpx.post(
    f"{nimbus.base_url}/v1/chat/completions",
    headers={"authorization": f"Bearer {nimbus.api_key}"},
    json={"model": "nimbus-large", "messages": req["messages"], "max_tokens": req["max_tokens"]},
)
body = res.json()
Kavya wrote about delivery for order PS-54624. They need it before Saturday, and the warehouse has not confirmed yet.
tokens: 162 in, 30 out
cost: ₹0.0705
took: 428 ms

Three things came back besides the answer. Usage: Nimbus read 162 tokens and wrote 30. Price: providers bill per million tokens, input and output at different prices, so this one call cost 162 x ₹250 + 30 x ₹1,000, over a million: seven paise. Time: 428 ms, most of it Nimbus thinking. Task 1 does this call and prints all four lines.

json= turns the dictionary into the request's JSON body and sets its content type; res.json() turns Nimbus's reply back into a dictionary, so the answer is body["choices"][0]["message"]["content"] and the tokens are in body["usage"]. A request in this lab is a dictionary too: day()[0].request["messages"].

What a gateway is

Today the same twenty lines live in every feature of Jawab's app, each with a copy of Nimbus's key. Changing provider means changing all of them. Retrying means five different retry loops. Nobody can say what Kaapi Kottai spent this week.

A gateway turns that inside out. The app speaks one shape to Darwaza, whatever model answers. Both shapes are plain dictionaries, described in /workspace/shared/jawab.py:

class GatewayRequest(TypedDict):
    team: str            # "kaapi", "nila", ...
    feature: str         # "triage", "reply", ...
    messages: list[Message]
    max_tokens: NotRequired[int]
    temperature: NotRequired[float]

# GatewayResponse, one of:
{"ok": True, "text": ..., "model": "nimbus/nimbus-large", "usage": {"input": 162, "output": 30}, "cost_inr": 0.0705}
{"ok": False, "status": 503, "error": "The server is temporarily unavailable. Please try again later."}

team is the brand the request is for, feature the job. Darwaza answers with the text, the model that wrote it ("nimbus/nimbus-large": provider, slash, model), what it used and what it cost. jawab.py also has PROVIDERS (the addresses and keys: PROVIDERS["nimbus"].base_url) and PRICES (Lakshmi's price sheet: PRICES["nimbus/nimbus-large"].input).

Notice the second shape. A provider that fails is not an exception to Darwaza: it is an answer with a status. The app gets {"ok": False, "status": 503, "error": ...} and decides what to show the customer; nothing is raised across the door. When the connection itself fails and there is no status at all, Darwaza uses 502 (bad gateway: the door could not reach what is behind it). httpx raises an httpx.HTTPError then (httpx.ConnectError when nothing answers): catch it and return the 502.

Darwaza v0

Farhan's first Darwaza has one route: everything to Nimbus's nimbus-large. That sounds like nothing, but it already moves the key, the provider's format and the cost into one place. Task 2 writes it: handle(req) sends the request to Nimbus and turns Nimbus's JSON into a GatewayResponse.

One thing is new: handle is async. Darwaza is one door for every request of the app, and most of a request's time is spent waiting for a provider. With async def handle and await client.post(...), while one request waits, Python works on the next: hundreds can wait at once on one thread. So Darwaza uses httpx.AsyncClient instead of httpx.post, made once at the top of the file and shared by every request (it keeps its connections to Nimbus open between requests):

client = httpx.AsyncClient(timeout=60)

async def handle(req: GatewayRequest) -> GatewayResponse:
    res = await client.post(...)

Why timeout=60? httpx stops waiting after 5 seconds unless told otherwise, which is a decision about Darwaza that nobody made yet. Jawab's features wait up to a minute today, so v0 does too; chapter 6 chooses better numbers, on purpose.

Jawab's day, replayed

One request tells you little. Jawab's real traffic does. day() in the shared code is Tuesday 3 November 2026, 09:00 to 21:00: 626 requests, in order, each with its time, its team, its feature and its messages (lab traffic):

626 requests, 09:02:03 to 20:58:07
by feature: summary 59, reply 187, triage 191, faq 121, translate 68
by team:    payyoli 173, kayal 83, kaapi 138, malabar 72, nila 75, thenga 85

replay(handle, "10:00", "11:00") is a load generator: it plays a slice of that day through your gateway. Twelve hours would take twelve hours, so it squeezes the day: a minute of Jawab's day passes in 25 ms, and the whole day in about 18 seconds. Only the gaps between requests shrink. Each request still takes its real time at the provider, and each is sent at its own moment whether or not earlier ones have answered, the way real customers behave. That is why handle must be async: a replay keeps dozens of requests waiting at once. Meanwhile the providers live through the same day: at 14:00 by the replay's clock, Nimbus goes down.

replay is async too, so a script that replays runs inside async def main(), started once with asyncio.run(main()). It returns a report (print_report prints it): what Jawab's customers got. Every result in report.results is one customer request, with its .status and how long it waited (.ms). Each expected answer is known in advance, so the report also counts answers that came back wrong.

The hour after lunch

Task 3 replays a quiet hour and the hour and a half around the outage:

Darwaza v0, 10:00 to 11:00: 37 requests
  answered  37 (100%)   failed 0   wrong 0
  latency   p50 383 ms   p95 2,196 ms   p99 2,208 ms
  spent     ₹4.96   provider calls 37 (0 errors)
Darwaza v0, 13:30 to 15:00: 102 requests
  answered  58 (56.9%)   failed 44   wrong 0
  latency   p50 469 ms   p95 6,011 ms   p99 6,013 ms
  spent     ₹5.85   provider calls 102 (44 errors)
503s after lunch: 44

In the morning every request was answered, but even then 2 of 37 took over 2 seconds: Nimbus's large model has a slow tail. After lunch, 44 of 102 customers got a 503, each after six seconds of waiting. Kestrel and Tara were up the whole time, idle, because v0 only knows one door behind the door. Most of this lab is about what Darwaza does instead.

The numbers that matter

"Is it good?" needs numbers agreed before anything broke. Joseph's are an SLO, a service level objective:

  • Answered: at least 99.5% of requests get an answer.
  • p95 latency: at most 1,000 ms. Sort every request's time; p95 is the one 95% of customers waited no longer than. Averages hide the slow tail; a percentile shows it. By nearest rank, p95 of n times is the one at position ceil(0.95 x n), counting from 1 (so index math.ceil(0.95 * n) - 1 in a Python list).

Lakshmi adds a third: rupees per 1,000 answers, and a daily budget for each brand (lab teams): ₹27.50 a day across all six.

One more word: refused. Later chapters give Darwaza rules of Jawab's own: a brand over the requests a minute it pays for, or over its budget for the day, is told no, on purpose. A refusal is not a failure, so the scorecard leaves it out of both numbers. The interview task writes the scorecard: v0 after lunch is 56.9% answered with p95 6,011 ms, and misses the SLO by a long way.

Over the whole day, measured: v0 answers 86.1% (44 outage 503s, 39 rate limits during Payyoli's evening sale, 4 server errors), with p95 6,003 ms, and spends ₹54.57. The Darwaza you finish in chapter 20 answers every request that is not refused, with p95 about 500 ms, for about a third of the rupees.

Your turn

Call Nimbus by hand in task 1. Write Darwaza's first route in task 2 and try it with lab replay gateway.py 10:00 10:30. Replay the morning and the outage in task 3. In the interview question, score v0 against the SLO.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    first_call.py: one request, by hand

    Before Darwaza exists, every feature of Jawab's app calls a provider itself. Do what they do, once: send the first request of Jawab's day (day()[0].request, from jawab) to Nimbus's nimbus-large with httpx.post, at f"{PROVIDERS['nimbus'].base_url}/v1/chat/completions", with the Bearer key and json={"model": "nimbus-large", "messages": ..., "max_tokens": ...}. Print, in this order:

    the answer (body["choices"][0]["message"]["content"])

    tokens: P in, C out (usage's prompt_tokens and completion_tokens)

    cost: ₹X (P times the input price plus C times the output price, over 1,000,000, from PRICES["nimbus/nimbus-large"]; 4 decimals)

    took: N ms (from before the request to after the JSON, a whole number)

  2. Task 2

    gateway.py: Darwaza v0, the first route

    Write async def handle(req: GatewayRequest) -> GatewayResponse, Darwaza's first version, with one client for every request at the top of the file: client = httpx.AsyncClient(timeout=60). Every request goes to Nimbus's nimbus-large with req["messages"], "max_tokens": req.get("max_tokens") and "temperature": req.get("temperature").

    A 200 becomes {"ok": True, "text", "model": "nimbus/nimbus-large", "usage": {"input", "output"}, "cost_inr"} (input is prompt_tokens, output completion_tokens, cost_inr in rupees at PRICES["nimbus/nimbus-large"]).

    Any other status becomes {"ok": False, "status", "error"}, with the provider's own body["error"]["message"]. handle never raises: when the request itself fails (httpx.HTTPError), return {"ok": False, "status": 502, "error": str(e)}.

  3. Task 3

    two_windows.py: a quiet morning, and after lunch

    Replay two slices of Jawab's day through handle (from gateway) with await replay(handle, start, end, label="Darwaza v0") inside async def main(), run with asyncio.run(main()), and print each with print_report: first "10:00" to "11:00", then "13:30" to "15:00". Then print one line from the second report's results:

    503s after lunch: N (how many results have status 503)

  4. Task 4 · interview question

    Interview: scorecard.py, is v0 good enough?

    Write def scorecard(results: list[Result], inr: float) -> dict returning {"answered", "p95", "inr_per_1000", "meets_slo"}. Leave out results with refused True, then: answered is the % with ok True (one decimal); p95 is their ms by nearest rank (sorted, the value at position ceil(0.95 x count), counting from 1; not with the given percentile()); inr_per_1000 is inr / answers x 1,000 (two decimals); meets_slo is answered >= 99.5 and p95 <= 1000.

    Under if __name__ == "__main__":, replay "13:30" to "15:00" through handle and print, from scorecard(report.results, report.inr):

    answered: X%

    p95: N ms

    ₹ per 1,000 answers: Y

    SLO: met (or SLO: missed)

What you will learn

  • Jawab's three providers, each with its own API, prices and troubles.
  • One request by hand: what a provider wants, and what it sends back.
  • What a gateway is: one shape in, many providers behind it.
  • Darwaza v0: the first route, async, with errors returned as answers.
  • Jawab's day, replayed: the load generator, and what Nimbus's outage did.
  • The numbers that matter: answered, p95 latency, rupees, and an SLO.

More in One API over three providers

Build "Meet Jawab, and Darwaza's first route" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.