Build an AI gateway (TypeScript) · Chapter 1 of 21
Meet Jawab, and Darwaza's first route
By the end of this chapter you can say what an AI gateway is and why companies put one in front of their models, call a provider by hand and read what it sends back, build Darwaza's first route, replay real traffic through it, and judge it with the numbers a team agreed on before anything broke. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: One API over three providers
- 4 tasks
- TypeScript
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
Jawab, in Kochi, answers customers for six brands from Kerala: Kaapi Kottai's filter coffee, Nila Handlooms' kasavu sarees, Payyoli Snacks' banana chips, Thenga Naturals' coconut oil, Malabar Spice Box's pepper and Kayal Pickles' fish pickle. Its app does five jobs with AI: triage (what is this ticket about, how urgent), reply (a draft answer in the brand's voice), faq (a product question answered from the brand's facts), summary (a conversation, for the person taking it over) and translate (into Malayalam, Hindi or Tamil).
Every one of those features calls an AI provider itself. Last Tuesday, Nimbus, the provider they all use, went down for forty minutes after lunch. Every brand's customers got nothing, and each request hung for six seconds before it failed. Anjali, Jawab's founder, wants that never to happen again. Lakshmi, who pays the bills, wants to know what each brand's AI costs, and wants it to cost less. Joseph, who is on call, wants to know what failed and why, without reading five codebases.
Farhan, the platform engineer, has the answer most companies arrive at: put one door in front of every model. Every AI request goes through it; behind it, the door decides which provider and which model answers, what happens when one fails, what may be cached, who has used how much, and what gets logged. That door is an AI gateway (LiteLLM, Portkey, OpenRouter and Cloudflare's AI Gateway are well-known ones). Farhan has named Jawab's Darwaza, and you will build it with him.
The lesson
Three providers, three APIs
Jawab has accounts with three providers. They do the same work and agree on almost nothing else:
| Provider | Speaks | Models | Price (₹ per million tokens, in / out) |
|---|---|---|---|
| Nimbus | OpenAI's Chat Completions format | nimbus-large, nimbus-mini | 250 / 1,000 and 15 / 60 |
| Kestrel | Anthropic's Messages format | kestrel-pro, kestrel-swift | 150 / 750 and 20 / 100 |
| Tara | Google's Gemini format | tara-flash | 8 / 32 |
lab providers in the terminal lists them, with their limits and their cache prices. The lab has no internet, so these three are the lab's own stand-ins, at $NIMBUS_BASE_URL, $KESTREL_BASE_URL and $TARA_BASE_URL. Each speaks the real format it is named after, counts tokens its own way, bills at its own prices and fails in its own way, on a schedule the lab keeps (Nimbus's outage is at 14:00, every time). What they write is predictable, not a neural network: the same request always gets the same answer, which is how this lab can tell whether Darwaza gave a customer the right one. A gateway does not care what a model thinks; it cares where the request goes, how long it takes, what it costs and what happens when it fails. Those parts are real.
One request, by hand
This is what each feature of Jawab's app does today. The first request of Tuesday was a summary for Payyoli Snacks. Send it to Nimbus:
const res = await fetch(`${baseUrl}/v1/chat/completions`, {
method: 'POST',
headers: { 'content-type': 'application/json', authorization: `Bearer ${apiKey}` },
body: JSON.stringify({ model: 'nimbus-large', messages: req.messages, max_tokens: req.maxTokens }),
});
Kavya wrote about delivery for order PS-54624. They need it before Saturday, and the warehouse has not confirmed yet.
tokens: 162 in, 30 out
cost: ₹0.0705
took: 400 ms
Three things came back besides the answer. Usage: Nimbus read 162 tokens and wrote 30. Price: providers bill per million tokens, input and output at different prices, so this one call cost 162 x ₹250 + 30 x ₹1,000, over a million: seven paise. Time: 400 ms, most of it Nimbus thinking. Task 1 does this call and prints all four lines.
res.json() is typed unknown in TypeScript: the compiler cannot know what a server sent. Write a small type for the parts you read and cast to it:
type NimbusReply = {
choices: { message: { content: string } }[];
usage: { prompt_tokens: number; completion_tokens: number };
};
What a gateway is
Today the same twenty lines live in every feature of Jawab's app, each with a copy of Nimbus's key. Changing provider means changing all of them. Retrying means five different retry loops. Nobody can say what Kaapi Kottai spent this week.
A gateway turns that inside out. The app speaks one shape to Darwaza, whatever model answers:
type GatewayRequest = { team: string; feature: string; messages: Message[]; maxTokens?: number; temperature?: number };
type GatewayResponse =
| { ok: true; text: string; model: string; usage: { input: number; output: number }; costInr?: number }
| { ok: false; status: number; error: string };
team is the brand the request is for, feature the job. Darwaza answers with the text, the model that wrote it ('nimbus/nimbus-large': provider, slash, model), what it used and what it cost. Both shapes are in /workspace/shared/jawab.ts, with PROVIDERS (the addresses and keys) and PRICES (Lakshmi's price sheet).
Notice the second half of GatewayResponse. A provider that fails is not an exception to Darwaza: it is an answer with a status. The app gets { ok: false, status: 503, error: '...' } and decides what to show the customer; nothing is thrown across the door. When the network itself fails and there is no status at all, Darwaza uses 502 (bad gateway: the door could not reach what is behind it).
Darwaza v0
Farhan's first Darwaza has one route: everything to Nimbus's nimbus-large. That sounds like nothing, but it already moves the key, the provider's format and the cost into one place. Task 2 writes it: handle(req) sends the request to Nimbus and turns Nimbus's JSON into a GatewayResponse.
Jawab's day, replayed
One request tells you little. Jawab's real traffic does. day() in the shared code is Tuesday 3 November 2026, 09:00 to 21:00: 626 requests, in order, each with its time, its team, its feature and its messages (lab traffic):
626 requests, 09:02:03 to 20:58:07
by feature: summary 59, reply 187, triage 191, faq 121, translate 68
by team: payyoli 173, kayal 83, kaapi 138, malabar 72, nila 75, thenga 85
replay(handle, { from, to }) is a load generator: it plays a slice of that day through your gateway. Twelve hours would take twelve hours, so it squeezes the day: a minute of Jawab's day passes in 25 ms, and the whole day in about 18 seconds. Only the gaps between requests shrink. Each request still takes its real time at the provider, and each is sent at its own moment whether or not earlier ones have answered, the way real customers behave. Meanwhile the providers live through the same day: at 14:00 by the replay's clock, Nimbus goes down.
The replay returns a report (printReport prints it): what Jawab's customers got. Every result in report.results is one customer request, with its status and how long it waited. Each expected answer is known in advance, so the report also counts answers that came back wrong.
The hour after lunch
Task 3 replays a quiet hour and the hour and a half around the outage:
Darwaza v0, 10:00 to 11:00: 37 requests
answered 37 (100%) failed 0 wrong 0
latency p50 384 ms p95 2,204 ms p99 2,214 ms
spent ₹4.96 provider calls 37 (0 errors)
Darwaza v0, 13:30 to 15:00: 102 requests
answered 58 (56.9%) failed 44 wrong 0
latency p50 471 ms p95 6,011 ms p99 6,012 ms
spent ₹5.85 provider calls 102 (44 errors)
503s after lunch: 44
In the morning every request was answered, but even then 2 of 37 took over 2 seconds: Nimbus's large model has a slow tail. After lunch, 44 of 102 customers got a 503, each after six seconds of waiting. Kestrel and Tara were up the whole time, idle, because v0 only knows one door behind the door. Most of this lab is about what Darwaza does instead.
The numbers that matter
"Is it good?" needs numbers agreed before anything broke. Joseph's are an SLO, a service level objective:
- Answered: at least 99.5% of requests get an answer.
- p95 latency: at most 1,000 ms. Sort every request's time; p95 is the one 95% of customers waited no longer than. Averages hide the slow tail; a percentile shows it. By nearest rank, p95 of n times is the one at position ceil(0.95 x n), counting from 1.
Lakshmi adds a third: rupees per 1,000 answers, and a daily budget for each brand (lab teams): ₹27.50 a day across all six.
One more word: refused. Later chapters give Darwaza rules of Jawab's own: a brand over the requests a minute it pays for, or over its budget for the day, is told no, on purpose. A refusal is not a failure, so the scorecard leaves it out of both numbers. The interview task writes the scorecard: v0 after lunch is 56.9% answered with p95 6,012 ms, and misses the SLO by a long way.
Over the whole day, measured: v0 answers 85.9% (44 outage 503s, 39 rate limits during Payyoli's evening sale, 5 server errors), with p95 6,002 ms, and spends ₹52.54. The Darwaza you finish in chapter 20 answers every request that is not refused, with p95 about 500 ms, for about a third of the rupees.
Your turn
Call Nimbus by hand in task 1. Write Darwaza's first route in task 2 and try it with lab replay gateway.ts 10:00 10:30. Replay the morning and the outage in task 3. In the interview question, score v0 against the SLO.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
first-call.ts: one request, by hand
Before Darwaza exists, every feature of Jawab's app calls a provider itself. Do what they do, once: send the first request of Jawab's day (day()[0].request, from ../../shared/jawab.ts) to Nimbus's nimbus-large with fetch, at
${PROVIDERS.nimbus.baseUrl}/v1/chat/completions, with the Bearer key and the body { model: 'nimbus-large', messages, max_tokens }. Print, in this order:the answer (choices[0].message.content)
tokens: P in, C out (usage.prompt_tokens and usage.completion_tokens)
cost: ₹X (P times the input price plus C times the output price, over 1,000,000, from PRICES['nimbus/nimbus-large']; 4 decimals)
took: N ms (from before the fetch to after the JSON)
- Task 2
gateway.ts: Darwaza v0, the first route
Write and export handle(req: GatewayRequest): Promise<GatewayResponse>, Darwaza's first version: every request goes to Nimbus's nimbus-large with req.messages, max_tokens: req.maxTokens and temperature: req.temperature.
A 200 becomes { ok: true, text, model: 'nimbus/nimbus-large', usage: { input, output }, costInr } (input is prompt_tokens, output completion_tokens, costInr in rupees at PRICES['nimbus/nimbus-large']).
Any other status becomes { ok: false, status, error }, with the provider's own error.message. handle never throws: when fetch itself fails, return { ok: false, status: 502, error: the error's message }.
- Task 3
two-windows.ts: a quiet morning, and after lunch
Replay two slices of Jawab's day through handle (from ./gateway.ts) with replay(handle, { from, to, label: 'Darwaza v0' }) and print each with printReport: first 10:00 to 11:00, then 13:30 to 15:00. Then print one line from the second report's results:
503s after lunch: N (how many results have status 503)
- Task 4 · interview question
Interview: scorecard.ts, is v0 good enough?
Write and export scorecard(results: Result[], inr: number) returning { answered, p95, inrPer1000, meetsSlo }. Leave out results with refused: true, then: answered is the % with ok true (one decimal); p95 is their ms by nearest rank (sorted, the value at position ceil(0.95 x count), counting from 1; not with the given percentile()); inrPer1000 is inr / answers x 1,000 (two decimals); meetsSlo is answered >= 99.5 and p95 <= 1000.
Under if (import.meta.main), replay 13:30 to 15:00 through handle and print, from scorecard(report.results, report.inr):
answered: X%
p95: N ms
₹ per 1,000 answers: Y
SLO: met (or SLO: missed)
What you will learn
- Jawab's three providers, each with its own API, prices and troubles.
- One request by hand: what a provider wants, and what it sends back.
- What a gateway is: one shape in, many providers behind it.
- Darwaza v0: the first route, and errors returned as answers.
- Jawab's day, replayed: the load generator, and what Nimbus's outage did.
- The numbers that matter: answered, p95 latency, rupees, and an SLO.
More in One API over three providers
Build "Meet Jawab, and Darwaza's first route" yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.