Build an AI gateway · Chapter 2 of 21

Three providers, three APIs

By the end of this chapter you can call Nimbus, Kestrel and Tara through one function, call("provider/model", req), explain what each provider's API wants and why their answers look different, and turn all three answers, token counts included, into one shape that nothing after Darwaza has to think about. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: One API over three providers
  • 4 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

In chapter 1, Nimbus went down at 14:00 and 44 of Jawab's customers got a 503 after six seconds. Kestrel and Tara were up the whole time. Anjali asks the obvious question: why didn't Darwaza just send those requests to Kestrel?

Because Darwaza v0 can only speak to Nimbus. Its one route builds Nimbus's request and reads Nimbus's answer. Kestrel and Tara do the same work, but they want a different URL, a different key header, the system prompt in a different place, different role names, and they answer in a different JSON with token counts that mean slightly different things. Send Kestrel a request built for Nimbus and it says 400.

Every gateway starts here. LiteLLM, Portkey and OpenRouter all have one adapter per provider: a small piece of code that turns the gateway's one request into that provider's request, and that provider's answer back into the gateway's one answer. Farhan wants Darwaza to have three, so that every later chapter (routing, fallbacks, caching) can treat a model as just a name: "kestrel/kestrel-swift".

The lesson

Three APIs for one job

Here is the same summary request (the first of Jawab's day) as each provider wants it. Same work, three shapes:

NimbusKestrelTara
Real API it copiesOpenAI's Chat CompletionsAnthropic's MessagesGoogle's Gemini generateContent
URL/v1/chat/completions/v1/messages/v1beta/models/tara-flash:generateContent
Keyauthorization: Bearer ...x-api-key: ... and a kestrel-version headerx-tara-api-key: ...
System prompta message with role systema top-level system fieldsystemInstruction: {parts: [{text}]}
Turnsmessages: [{role, content}]messages, only user and assistantcontents: [{role, parts: [{text}]}], roles user and model
Most tokens outmax_tokens, optionalmax_tokens, requiredgenerationConfig.maxOutputTokens

Where the lab's names differ from the real ones, it is only the names: Anthropic's version header is anthropic-version, and Gemini's key header is x-goog-api-key. The shapes are the real ones, so an adapter you write here works against the real APIs with those two names changed.

Nimbus is the easy one: Darwaza's messages are already in OpenAI's shape, which is why so many gateways use it as their own format. The other two need translating.

What a provider says when the shape is wrong

Send Kestrel the Nimbus-shaped request, with the system prompt as a message, and this comes back (real output):

400 {"type":"error","error":{"type":"invalid_request_error","message":"messages: Unexpected role \"system\". The Messages API accepts a top-level `system` parameter, not \"system\" as an input message role."}}

Leave out max_tokens, or the version header, and Kestrel says which:

400 {"type":"error","error":{"type":"invalid_request_error","message":"max_tokens: Field required"}}
400 {"type":"error","error":{"type":"invalid_request_error","message":"kestrel-version: header is required"}}

Kestrel requires max_tokens because it will not guess how long you are willing to pay for. Jawab's app nearly always sends max_tokens, but a gateway must not break when it does not: send a default (1024) for Kestrel.

Tara has no assistant. A conversation with an earlier answer in it must call that turn model:

400 {"error":{"code":400,"message":"Please use a valid role: user, model.","status":"INVALID_ARGUMENT"}}

Three providers, three error shapes too. Chapter 3 is about that; for now, every one of them puts its message in body["error"]["message"].

Adapters: one request in, three out

An adapter is a plain function: Darwaza's GatewayRequest in, and out everything httpx needs to send it, in a small dataclass:

@dataclass
class Built:
    url: str
    headers: dict[str, str]
    body: dict

to_kestrel("kestrel-swift", req)
Built(url='http://127.0.0.1:11500/kestrel/v1/messages',
      headers={'x-api-key': 'kes-...-key', 'kestrel-version': '2026-01-15'},
      body={'model': 'kestrel-swift', 'max_tokens': 160,
            'system': 'Summarise this support conversation for the person taking it over, ...',
            'messages': [{'role': 'user', 'content': 'Customer (Kavya): My order PS-54624 ...'}],
            'temperature': 0.2})

(Trimmed, and the key masked.) The system message has moved out of messages into system. Jawab's requests have one system message; if one ever has two, join their texts. An adapter never sends a field the request did not have: no system when there is no system message, no temperature when the app gave none. req.get("temperature") is None when the key is missing, so test for that before you add it. There is no content-type header: client.post(url, json=body) adds content-type: application/json itself.

Keeping adapters as plain functions that only build a request has a payoff later: chapter 8 builds a request again for the next provider when one fails, and chapter 16 builds the streaming version. Only call touches the network.

Three answers, one shape

Here is what came back for that summary (real, trimmed):

nimbus   {"choices":[{"message":{"role":"assistant","content":"Kavya wrote about delivery ..."},"finish_reason":"stop"}],
          "usage":{"prompt_tokens":162,"completion_tokens":30,"prompt_tokens_details":{"cached_tokens":0}}}
kestrel  {"content":[{"type":"text","text":"Kavya wrote about delivery ..."}],"stop_reason":"end_turn",
          "usage":{"input_tokens":176,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":33}}
tara     {"candidates":[{"content":{"role":"model","parts":[{"text":"Kavya wrote about delivery ..."}]},"finishReason":"STOP"}],
          "usageMetadata":{"promptTokenCount":142,"candidatesTokenCount":27}}

res.json() turns each into a Python dictionary. Darwaza reads all three into one tuple, (text, finish, usage):

  • text: Nimbus's first choice; Kestrel's content is a list of blocks, so join the text of every block of type text; Tara's first candidate's parts, joined.
  • finish: "stop" when the model finished, "length" when it hit the most tokens it was allowed and was cut off mid-sentence. Nimbus says length, Kestrel max_tokens, Tara MAX_TOKENS. A cut-off answer matters later: chapter 11 never caches one.
  • usage: next.

A key that may be missing is read with .get: body["usage"]["x"] raises KeyError when there is no x, and a reader that crashes on a valid answer is worse than none.

Usage that means the same thing

Each provider counted the same summary differently: 162 tokens in for Nimbus, 176 for Kestrel, 142 for Tara. That is not a bug. Every provider has its own tokenizer, and the lab's three are as different as real ones are (chapter 4 measures them). The provider's count is the one you are billed for, so Darwaza always takes the provider's own number.

There is a subtler difference. Both Nimbus and Kestrel have a prompt cache (chapter 13): when the start of a prompt is the same as a recent request's, they charge much less for those tokens. They report it differently. Here is a Thenga reply sent twice to nimbus-mini (real):

{"prompt_tokens": 1392, "completion_tokens": 41, "total_tokens": 1433, "prompt_tokens_details": {"cached_tokens": 0}}
{"prompt_tokens": 1392, "completion_tokens": 41, "total_tokens": 1433, "prompt_tokens_details": {"cached_tokens": 1280}}

The second time, 1,280 of the 1,392 prompt tokens came from Nimbus's cache. Nimbus's prompt_tokens includes the cached ones. Kestrel's input_tokens leaves them out: it reports cache reads and cache writes in their own fields. If Darwaza copied both numbers into usage["input"], a cached Nimbus call would look ten times dearer than it was.

So Darwaza's usage has one meaning, whoever answered:

usage keymeansNimbusKestrelTara
inputinput billed at the full priceprompt_tokens - cached_tokensinput_tokenspromptTokenCount
cachedread from the prompt cachecached_tokenscache_read_input_tokensnone
cache_writewritten to the cachenonecache_creation_input_tokensnone
outputwritten by the modelcompletion_tokensoutput_tokenscandidatesTokenCount

That second Thenga call becomes {"input": 112, "output": 41, "cached": 1280}. Chapter 1's v0 used prompt_tokens as input; from now on Darwaza uses this table, and chapter 4 prices each part at its own rate.

call: any model by name

From here on, a model in Darwaza is a target: "provider/model", like "tara/tara-flash". build(target, req) splits the name at the slash and picks the adapter; call(target, req) sends it and reads the answer into an Attempt, one try at one target:

@dataclass
class Attempt:
    ok: bool
    model: str          # the target
    text: str = ""
    usage: dict = field(default_factory=lambda: {"input": 0, "output": 0})
    finish: str = "stop"
    error: dict | None = None   # {"provider", "status", "message"}

Like chapter 1's handle, call never raises: a provider's error status comes back as ok=False with the provider's own message, and a request that fails on the way (httpx.HTTPError: nothing answered) as status 502. call uses one httpx.AsyncClient made at the top of the file, as v0 does, so the connections to the providers stay open from one call to the next. Chapter 3 gives every error one shape and decides which are worth trying again; chapter 8 uses call to move to the next provider when one fails.

The same request through all three

Sent at the same moment to the three small models, the day's first summary (real):

nimbus/nimbus-mini: 162 in, 30 out, 152 ms
kestrel/kestrel-swift: 176 in, 33 out, 100 ms
tara/tara-flash: 142 in, 27 out, 333 ms

All three wrote the same words: "Kavya wrote about delivery for order PS-54624. They need it before Saturday, and the warehouse has not confirmed yet." The lab's providers answer the same request the same way every time, whoever is asked; a real model does not, which is why Jawab's traffic day checks for the words a right answer contains, not whole answers. What does differ is real: the token counts (each provider's own), and the speed: kestrel-swift starts writing in about 70 ms, tara-flash in about 240 ms. Chapter 5 uses exactly that to route.

To send them at the same moment, make the three coroutines first and hand them all to asyncio.gather, which runs them together and returns their results in the order you gave them, whichever finished first:

rows = await asyncio.gather(*(timed(t, req) for t in SMALL))

The whole comparison takes as long as the slowest provider. Awaiting them one by one in a for loop takes the sum of the three: each await waits for its call to finish before the next one starts.

Trying a real provider

The lab has no internet, so its providers are stand-ins. On your own machine, with your own key, the same adapters talk to the real APIs: point to_nimbus at https://api.openai.com, to_kestrel at https://api.anthropic.com with the header anthropic-version: 2023-06-01, and to_tara at https://generativelanguage.googleapis.com with x-goog-api-key. Never put a real key in a file you share; read it from the environment (os.environ["OPENAI_API_KEY"]).

Your turn

Write the three request adapters in task 1 and the three answer readers in task 2. In task 3, put them together in build and call. In the interview question, send one request to all three providers at once and show that your adapters really are interchangeable.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    adapters.py: one request, three shapes

    Write a dataclass Built(url, headers, body) and three plain functions that return one: to_nimbus(model, req), to_kestrel(model, req) and to_tara(model, req), with model the name alone ("kestrel-swift") and the addresses and keys from PROVIDERS.

    Nimbus: f"{base_url}/v1/chat/completions", authorization: Bearer key, body {model, messages, max_tokens, temperature}.

    Kestrel: f"{base_url}/v1/messages", x-api-key and kestrel-version (os.environ["KESTREL_VERSION"]) headers, body {model, max_tokens, system, messages, temperature}: the system messages' text in system, only the user and assistant turns in messages.

    Tara: f"{base_url}/v1beta/models/{model}:generateContent", x-tara-api-key header, body {"systemInstruction": {"parts": [{"text"}]}, "contents": [{"role", "parts": [{"text"}]}], "generationConfig": {"maxOutputTokens", "temperature"}}.

    Trap: Kestrel requires max_tokens (send 1024 when the request has none), and a system role in its messages is a 400; Tara's roles are user and model (an assistant turn is model). Leave out what the request does not have.

  2. Task 2

    replies.py: three answers, one shape

    Write from_nimbus(body), from_kestrel(body) and from_tara(body), plain functions each turning a provider's 200 JSON (a dict) into a tuple (text, finish, usage).

    text: Nimbus's choices[0]["message"]["content"]; Kestrel's content blocks of type text, joined; Tara's candidates[0]["content"]["parts"] texts, joined.

    finish: "length" for Nimbus's length, Kestrel's max_tokens and Tara's MAX_TOKENS; otherwise "stop".

    usage: {"input", "output", "cached", "cache_write"}, where input is the input billed at the full price, cached the tokens read from the prompt cache, cache_write those written to it.

    Trap: Nimbus's prompt_tokens includes its cached_tokens (in prompt_tokens_details, sometimes missing), so take them out of input; Kestrel's input_tokens already leaves out cache_read_input_tokens and cache_creation_input_tokens.

  3. Task 3

    call.py: any model by its name

    Write a dataclass Attempt(ok, model, text, usage, finish, error), def build(target, req), which splits a target such as "tara/tara-flash" at the slash and returns that provider's request from adapters.py, and async def call(target, req) -> Attempt, which posts it with one module-level httpx.AsyncClient (the url, the headers, json= the body).

    A 200 becomes Attempt(True, target, text, usage, finish), read with replies.py.

    Any other status becomes ok False with error {"provider", "status", "message"}, the message the provider's own body["error"]["message"].

    Trap: call never raises: when the request itself fails (httpx.HTTPError), return status 502 and the error's message.

  4. Task 4 · interview question

    Interview: three_ways.py, one question, three providers

    Write async def compare(req) -> list[dict]: send req with call to "nimbus/nimbus-mini", "kestrel/kestrel-swift" and "tara/tara-flash", all three at the same time, and return one row each, in that order: {"target", "ok", "text", "input", "output", "ms"}, with the usage call gave and the ms from before that call to after it (a failed call: ok False, its message as text, 0 tokens).

    Under if __name__ == "__main__":, compare the day's first triage request (day(features=["triage"])[0].request) and print:

    nimbus/nimbus-mini: P in, C out, N ms

    kestrel/kestrel-swift: P in, C out, N ms

    tara/tara-flash: P in, C out, N ms

    same answer: yes (or same answer: no, when the three texts differ)

    Trap: one after another takes the sum of the three; at the same time takes as long as the slowest.

What you will learn

  • What each provider's API wants: the URL, the key, the system prompt, the roles, and the required fields.
  • Adapters: one request in, three requests out.
  • One answer shape: text, finish and usage, the same whoever answered.
  • Usage that means the same thing: why Nimbus's input tokens and Kestrel's input tokens are not the same number.
  • call(target, req): any model, by its provider/model name.
  • asyncio.gather: three calls at once, as long as the slowest.

More in One API over three providers

Build "Three providers, three APIs" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.