Build an AI gateway · Chapter 20 of 21

Capstone: Darwaza runs Jawab's day

By the end of this chapter Darwaza is yours, whole, in four files you organised: it serves Jawab's entire Tuesday within the SLO and every brand's budget, sends no personal data anywhere, finishes every stream, and holds up on a day you have never seen. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Run it in production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.

Why you need this

Every chapter built one layer and measured it alone, with the rest given. Jawab is about to put Darwaza in front of every customer of six brands, and Anjali, Lakshmi and Joseph each have a line in the sand:

  • Anjali: at least 99.5% of customers answered (those Jawab's own rules refuse are counted apart), and never again forty minutes of nothing.
  • Joseph: p95 at most 1,000 ms, every stream ends properly, and no answer that is wrong.
  • Lakshmi: at most ₹27.50 for the day, and every brand within its own budget.
  • And all three: no personal data reaches a provider.

Layers interact. A cache in front of the guard leaks phone numbers into cache keys. A budget checked after the call cannot stop the call. A retry with no deadline turns a fast fallback into a slow one. Only the whole gateway, measured on the whole day, says whether the layers work together.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    providers.py: one shape over three APIs

    Your own Darwaza, in four files, from your work in chapters 2 to 19 (no given darwaza.py). This file talks to providers: the adapters (to_nimbus, to_kestrel, to_tara, from_nimbus, from_kestrel, from_tara), ProviderError, Attempt, normalize_error, retry_after_of, provider_of, build, async call (never raises), cost_inr, estimate_tokens, worst_case_inr, ROUTES, ALIASES, route, TIMEOUT_MS, DEADLINE_MS, sse_events and stream_from, with the names and shapes of chapters 2 to 6 and 16.

    The check calls each provider and compares usage and rupees with the providers' log and bill, forces a 529, two kinds of 429, a 404, a timeout and a dropped stream, and asks for routes: replies only on large models, no translation on Kestrel.

  2. Task 2

    resilience.py: when providers fail

    Write backoff (full jitter: a random wait between 0 and min(400, 50 x 2 ** attempt), with rand as its second parameter), retry_wait (retry-after first, else backoff), RetryBudget (starts full; on_request adds ratio; try_retry), Breaker (closed, open, half-open with one probe; a good probe closes it and forgets the old failures; injectable now) and async hedge (the second call only after after_ms; the first good answer wins; the other cancelled; returns (attempt, hedged)), as in chapters 7 to 10.

    The check runs each on check-made inputs and a fake clock.

  3. Task 3

    savings.py: spend less, share fairly

    Write cache_key, cacheable, ExactCache (TTL, size cap, never a cut-off answer), async embed, cosine, SemanticCache (threshold 0.85, faq only, scoped by team, feature and system prompt; get returns (answer or None, vector, score)), TokenBucket (per_minute, capacity, injectable now; take returns (ok, retry_after_ms)), TeamLimits and Budgets (reserve, settle, level ok, tight from 80%, over), as in chapters 11 to 15.

    The check uses each on check-made requests, a fake clock and real embeddings.

  4. Task 4

    darwaza.py: Darwaza runs Jawab's day

    Write create_darwaza(**options) (every layer on unless turned off; no options is the Darwaza Jawab runs, the semantic cache on) and gateway = create_darwaza(), with async handle and an async generator stream, the guard (chapter 18) and traces (chapter 17) in this file. Under if __name__ == "__main__":, replay the whole day through create_darwaza() with the scenario sys.argv[1] (or "normal") and the label f"Darwaza, {scenario} day", and print_report it.

    The check replays Jawab's whole day, then a stormy day you have not seen, each through a fresh create_darwaza(). At least 99.5% answered on Tuesday and 99% on the stormy day (of those not refused; an SLO is kept over a month, and a stormy day may use some of it), and on both: p95 at most 1,000 ms, 0 wrong, at most ₹27.50, every team within its daily budget, 0 personal data sent, every stream complete.

What you will learn

  • Organising a gateway by concern: what talks to providers, what survives their failures, what saves money, and what composes them.
  • The order of the layers, and why each sits where it does.
  • What each layer is worth, measured by turning it off.
  • Testing on a day you have not seen, so you build for failures, not for Tuesday.

More in Run it in production

Build "Capstone: Darwaza runs Jawab's day" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.