Build an AI gateway (TypeScript) · Chapter 2 of 21

Three providers, three APIs

By the end of this chapter you can call Nimbus, Kestrel and Tara through one function, call('provider/model', req), explain what each provider's API wants and why their answers look different, and turn all three answers, token counts included, into one shape that nothing after Darwaza has to think about. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: One API over three providers
  • 4 tasks
  • TypeScript
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

In chapter 1, Nimbus went down at 14:00 and 44 of Jawab's customers got a 503 after six seconds. Kestrel and Tara were up the whole time. Anjali asks the obvious question: why didn't Darwaza just send those requests to Kestrel?

Because Darwaza v0 can only speak to Nimbus. Its one route builds Nimbus's request and reads Nimbus's answer. Kestrel and Tara do the same work, but they want a different URL, a different key header, the system prompt in a different place, different role names, and they answer in a different JSON with token counts that mean slightly different things. Send Kestrel a request built for Nimbus and it says 400.

Every gateway starts here. LiteLLM, Portkey and OpenRouter all have one adapter per provider: a small piece of code that turns the gateway's one request into that provider's request, and that provider's answer back into the gateway's one answer. Farhan wants Darwaza to have three, so that every later chapter (routing, fallbacks, caching) can treat a model as just a name: 'kestrel/kestrel-swift'.

The lesson

Three APIs for one job

Here is the same summary request (the first of Jawab's day) as each provider wants it. Same work, three shapes:

NimbusKestrelTara
Real API it copiesOpenAI's Chat CompletionsAnthropic's MessagesGoogle's Gemini generateContent
URL/v1/chat/completions/v1/messages/v1beta/models/tara-flash:generateContent
Keyauthorization: Bearer ...x-api-key: ... and a kestrel-version headerx-tara-api-key: ...
System prompta message with role systema top-level system fieldsystemInstruction: { parts: [{ text }] }
Turnsmessages: [{ role, content }]messages, only user and assistantcontents: [{ role, parts: [{ text }] }], roles user and model
Most tokens outmax_tokens, optionalmax_tokens, requiredgenerationConfig.maxOutputTokens

Where the lab's names differ from the real ones, it is only the names: Anthropic's version header is anthropic-version, and Gemini's key header is x-goog-api-key. The shapes are the real ones, so an adapter you write here works against the real APIs with those two names changed.

Nimbus is the easy one: Darwaza's messages are already in OpenAI's shape, which is why so many gateways use it as their own format. The other two need translating.

What a provider says when the shape is wrong

Send Kestrel the Nimbus-shaped request, with the system prompt as a message, and this comes back (real output):

400 {"type":"error","error":{"type":"invalid_request_error","message":"messages: Unexpected role \"system\". The Messages API accepts a top-level `system` parameter, not \"system\" as an input message role."}}

Leave out max_tokens, or the version header, and Kestrel says which:

400 {"type":"error","error":{"type":"invalid_request_error","message":"max_tokens: Field required"}}
400 {"type":"error","error":{"type":"invalid_request_error","message":"kestrel-version: header is required"}}

Kestrel requires max_tokens because it will not guess how long you are willing to pay for. Jawab's app nearly always sends maxTokens, but a gateway must not break when it does not: send a default (1024) for Kestrel.

Tara has no assistant. A conversation with an earlier answer in it must call that turn model:

400 {"error":{"code":400,"message":"Please use a valid role: user, model.","status":"INVALID_ARGUMENT"}}

Three providers, three error shapes too. Chapter 3 is about that; for now, every one of them puts its message in error.message.

Adapters: one request in, three out

An adapter is a plain function: Darwaza's GatewayRequest in, and out everything fetch needs:

type Built = { url: string; headers: Record<string, string>; body: unknown };
toKestrel('kestrel-swift', req);
{
  "url": "http://127.0.0.1:11500/kestrel/v1/messages",
  "headers": { "content-type": "application/json", "x-api-key": "kes-...-key", "kestrel-version": "2026-01-15" },
  "body": {
    "model": "kestrel-swift",
    "max_tokens": 160,
    "system": "Summarise this support conversation for the person taking it over, ...",
    "messages": [{ "role": "user", "content": "Customer (Kavya): My order PS-54624 ..." }]
  }
}

(Trimmed, and the key masked.) The system message has moved out of messages into system. Jawab's requests have one system message; if one ever has two, join their texts. An adapter never sends a field the request did not have: no system when there is no system message, no temperature when the app gave none.

Keeping adapters as plain functions that only build a request has a payoff later: chapter 8 builds a request again for the next provider when one fails, and chapter 16 builds the streaming version. Only call touches the network.

Three answers, one shape

Here is what came back for that summary (real, trimmed):

nimbus   {"choices":[{"message":{"role":"assistant","content":"Kavya wrote about delivery ..."},"finish_reason":"stop"}],
          "usage":{"prompt_tokens":162,"completion_tokens":30,"prompt_tokens_details":{"cached_tokens":0}}}
kestrel  {"content":[{"type":"text","text":"Kavya wrote about delivery ..."}],"stop_reason":"end_turn",
          "usage":{"input_tokens":176,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":33}}
tara     {"candidates":[{"content":{"role":"model","parts":[{"text":"Kavya wrote about delivery ..."}]},"finishReason":"STOP"}],
          "usageMetadata":{"promptTokenCount":142,"candidatesTokenCount":27}}

Darwaza reads all three into one Reply:

type Reply = { text: string; finish: 'stop' | 'length'; usage: Usage };
  • text: Nimbus's first choice; Kestrel's content is a list of blocks, so join the text of every block of type text; Tara's first candidate's parts, joined.
  • finish: 'stop' when the model finished, 'length' when it hit the most tokens it was allowed and was cut off mid-sentence. Nimbus says length, Kestrel max_tokens, Tara MAX_TOKENS. A cut-off answer matters later: chapter 11 never caches one.
  • usage: next.

Usage that means the same thing

Each provider counted the same summary differently: 162 tokens in for Nimbus, 176 for Kestrel, 142 for Tara. That is not a bug. Every provider has its own tokenizer, and the lab's three are as different as real ones are (chapter 4 measures them). The provider's count is the one you are billed for, so Darwaza always takes the provider's own number.

There is a subtler difference. Both Nimbus and Kestrel have a prompt cache (chapter 13): when the start of a prompt is the same as a recent request's, they charge much less for those tokens. They report it differently. Here is a Thenga reply sent twice to nimbus-mini (real):

{"prompt_tokens":1392,"completion_tokens":41,"prompt_tokens_details":{"cached_tokens":0}}
{"prompt_tokens":1392,"completion_tokens":41,"prompt_tokens_details":{"cached_tokens":1280}}

The second time, 1,280 of the 1,392 prompt tokens came from Nimbus's cache. Nimbus's prompt_tokens includes the cached ones. Kestrel's input_tokens leaves them out: it reports cache reads and cache writes in their own fields. If Darwaza copied both numbers into usage.input, a cached Nimbus call would look ten times dearer than it was.

So Darwaza's Usage has one meaning, whoever answered:

UsagemeansNimbusKestrelTara
inputinput billed at the full priceprompt_tokens - cached_tokensinput_tokenspromptTokenCount
cachedread from the prompt cachecached_tokenscache_read_input_tokensnone
cacheWritewritten to the cachenonecache_creation_input_tokensnone
outputwritten by the modelcompletion_tokensoutput_tokenscandidatesTokenCount

That second Thenga call becomes { input: 112, output: 41, cached: 1280 }. Chapter 1's v0 used prompt_tokens as input; from now on Darwaza uses this table, and chapter 4 prices each part at its own rate.

call: any model by name

From here on, a model in Darwaza is a target: 'provider/model', like 'tara/tara-flash'. build(target, req) splits the name at the slash and picks the adapter; call(target, req) sends it and reads the answer:

type Attempt =
  | { ok: true; text: string; usage: Usage; finish: 'stop' | 'length'; model: string }
  | { ok: false; error: { provider: string; status: number; message: string } };

An Attempt is one try at one target. Like chapter 1's handle, call never throws: a provider's error status comes back as { ok: false, error } with the provider's own message, and a failed fetch (no answer at all) as status 502. Chapter 3 gives every error one shape and decides which are worth trying again; chapter 8 uses call to move to the next provider when one fails.

The same request through all three

Sent at the same moment to the three small models, the day's first summary (real):

nimbus/nimbus-mini: 162 in, 30 out, 168 ms
kestrel/kestrel-swift: 176 in, 33 out, 105 ms
tara/tara-flash: 142 in, 27 out, 283 ms

All three wrote the same words: "Kavya wrote about delivery for order PS-54624. They need it before Saturday, and the warehouse has not confirmed yet." The lab's providers answer the same request the same way every time, whoever is asked; a real model does not, which is why Jawab's traffic day checks for the words a right answer contains, not whole answers. What does differ is real: the token counts (each provider's own), and the speed: kestrel-swift starts writing in about 70 ms, tara-flash in about 240 ms. Chapter 5 uses exactly that to route.

To send them at the same moment, start all three calls and wait for all of them with Promise.all: the whole comparison takes as long as the slowest provider, not the sum of the three.

Trying a real provider

The lab has no internet, so its providers are stand-ins. On your own machine, with your own key, the same adapters talk to the real APIs: point toNimbus at https://api.openai.com, toKestrel at https://api.anthropic.com with the header anthropic-version: 2023-06-01, and toTara at https://generativelanguage.googleapis.com with x-goog-api-key. Never put a real key in a file you share; read it from the environment.

Your turn

Write the three request adapters in task 1 and the three answer readers in task 2. In task 3, put them together in build and call. In the interview question, send one request to all three providers at once and show that your adapters really are interchangeable.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    adapters.ts: one request, three shapes

    Write and export three adapters, each returning a Built ({ url, headers, body }) for fetch: toNimbus(model, req), toKestrel(model, req) and toTara(model, req), with model the name alone ('kestrel-swift') and the addresses and keys from PROVIDERS. Every request has content-type: application/json.

    Nimbus: ${baseUrl}/v1/chat/completions, authorization: Bearer key, body { model, messages, max_tokens, temperature }.

    Kestrel: ${baseUrl}/v1/messages, x-api-key and kestrel-version (process.env.KESTREL_VERSION) headers, body { model, max_tokens, system, messages, temperature }: the system messages' text in system, only the user and assistant turns in messages.

    Tara: ${baseUrl}/v1beta/models/${model}:generateContent, x-tara-api-key header, body { systemInstruction: { parts: [{ text }] }, contents: [{ role, parts: [{ text }] }], generationConfig: { maxOutputTokens, temperature } }.

    Trap: Kestrel requires max_tokens (send 1024 when the request has no maxTokens), and a system role in its messages is a 400; Tara's roles are user and model (an assistant turn is model). Leave out what the request does not have.

  2. Task 2

    replies.ts: three answers, one shape

    Write and export fromNimbus(body), fromKestrel(body) and fromTara(body), each turning a provider's 200 JSON into a Reply: { text, finish: 'stop' | 'length', usage }.

    text: Nimbus's choices[0].message.content; Kestrel's content blocks of type text, joined; Tara's candidates[0].content.parts texts, joined.

    finish: 'length' for Nimbus's length, Kestrel's max_tokens and Tara's MAX_TOKENS; otherwise 'stop'.

    usage: { input, output, cached, cacheWrite }, where input is the input billed at the full price, cached the tokens read from the prompt cache, cacheWrite those written to it.

    Trap: Nimbus's prompt_tokens includes its cached_tokens (prompt_tokens_details.cached_tokens, sometimes missing), so take them out of input; Kestrel's input_tokens already leaves out cache_read_input_tokens and cache_creation_input_tokens.

  3. Task 3

    call.ts: any model by its name

    Write and export build(target, req), which splits a target such as 'tara/tara-flash' at the slash and returns that provider's request from ./adapters.ts, and call(target, req): Promise<Attempt>, which fetches it (POST, the headers, the body as JSON).

    A 200 becomes { ok: true, text, usage, finish, model: target }, read with ./replies.ts.

    Any other status becomes { ok: false, error: { provider, status, message } } with the provider's own error.message.

    Trap: call never throws: when fetch itself fails, return status 502 and the error's message.

  4. Task 4 · interview question

    Interview: three-ways.ts, one question, three providers

    Write and export compare(req): Promise<Row[]>: send req with call to 'nimbus/nimbus-mini', 'kestrel/kestrel-swift' and 'tara/tara-flash', all three at the same time, and return one Row each, in that order: { target, ok, text, input, output, ms }, with the usage call gave and the ms from before that call to after it (a failed call: ok false, its message as text, 0 tokens).

    Under if (import.meta.main), compare the day's first triage request (day({ features: ['triage'] })[0].request) and print:

    nimbus/nimbus-mini: P in, C out, N ms

    kestrel/kestrel-swift: P in, C out, N ms

    tara/tara-flash: P in, C out, N ms

    same answer: yes (or same answer: no, when the three texts differ)

    Trap: one after another takes the sum of the three; at the same time takes as long as the slowest.

What you will learn

  • What each provider's API wants: the URL, the key, the system prompt, the roles, and the required fields.
  • Adapters: one request in, three requests out.
  • One answer shape: text, finish and usage, the same whoever answered.
  • Usage that means the same thing: why Nimbus's input tokens and Kestrel's input tokens are not the same number.
  • call(target, req): any model, by its provider/model name.

More in One API over three providers

Build "Three providers, three APIs" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.