Build an AI gateway (TypeScript) · Chapter 20 of 21
Capstone: Darwaza runs Jawab's day
By the end of this chapter Darwaza is yours, whole, in four files you organised: it serves Jawab's entire Tuesday within the SLO and every brand's budget, sends no personal data anywhere, finishes every stream, and holds up on a day you have never seen. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Run it in production
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Every chapter built one layer and measured it alone, with the rest given. Jawab is about to put Darwaza in front of every customer of six brands, and Anjali, Lakshmi and Joseph each have a line in the sand:
- Anjali: at least 99.5% of customers answered (those Jawab's own rules refuse are counted apart), and never again forty minutes of nothing.
- Joseph: p95 at most 1,000 ms, every stream ends properly, and no answer that is wrong.
- Lakshmi: at most ₹27.50 for the day, and every brand within its own budget.
- And all three: no personal data reaches a provider.
Layers interact. A cache in front of the guard leaks phone numbers into cache keys. A budget checked after the call cannot stop the call. A retry with no deadline turns a fast fallback into a slow one. Only the whole gateway, measured on the whole day, says whether the layers work together.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
providers.ts: one shape over three APIs
Your own Darwaza, in four files, from your work in chapters 2 to 19 (no given darwaza.ts). This file talks to providers: export the adapters (toNimbus, toKestrel, toTara, fromNimbus, fromKestrel, fromTara), normalizeError, retryAfterOf, providerOf, build, call (never throws), costInr, estimateTokens, worstCaseInr, ROUTES, ALIASES, route, TIMEOUT_MS, sseEvents and streamFrom, with the shapes of chapters 2 to 6 and 16.
The check calls each provider and compares usage and rupees with the providers' log and bill, forces a 529, two kinds of 429, a 404, a timeout and a dropped stream, and asks for routes: replies only on large models, no translation on Kestrel.
- Task 2
resilience.ts: when providers fail
Export attemptSignal, sleep, backoff (full jitter: a random wait between 0 and min(cap, base x 2^attempt), with rand as its fourth parameter), waitFor (retry-after first, else backoff), RetryBudget (starts full; onRequest adds ratio; tryRetry), Breaker (closed, open, half-open with one probe; injectable now) and hedge (the second call only after afterMs; the first good answer wins; the other aborted; hedged: boolean), as in chapters 6 to 10.
The check runs each on check-made inputs and a fake clock.
- Task 3
savings.ts: spend less, share fairly
Export cacheKey, cacheable, ExactCache (TTL, size cap, never a cut-off answer), embed, cosine, SemanticCache (threshold 0.85, faq only, scoped by team, feature and system prompt), TokenBucket (perMinute, capacity, injectable now; take returns retryAfterMs), TeamLimits and Budgets (reserve, settle, level ok, tight from 80%, over), as in chapters 11 to 15.
The check uses each on check-made requests, a fake clock and real embeddings.
- Task 4
darwaza.ts: Darwaza runs Jawab's day
Export createDarwaza(options = {}) (every layer on unless turned off; no options is the Darwaza Jawab runs) and gateway = createDarwaza(), with handle and stream, the guard (chapter 18) and traces (chapter 17) in this file. Under if (import.meta.main), replay the whole day through createDarwaza() with the scenario process.argv[2] ?? 'normal' and the label
Darwaza, ${scenario} day, and printReport it.The check replays Jawab's whole day, then a stormy day you have not seen, each through a fresh createDarwaza(). At least 99.5% answered on Tuesday and 99% on the stormy day (of those not refused; an SLO is kept over a month, and a stormy day may use some of it), and on both: p95 at most 1,000 ms, 0 wrong, at most ₹27.50, every team within its daily budget, 0 personal data sent, every stream complete.
What you will learn
- Organising a gateway by concern: what talks to providers, what survives their failures, what saves money, and what composes them.
- The order of the layers, and why each sits where it does.
- What each layer is worth, measured by turning it off.
- Testing on a day you have not seen, so you build for failures, not for Tuesday.
More in Run it in production
Build "Capstone: Darwaza runs Jawab's day" yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.