Build an AI gateway (TypeScript) · Chapter 8 of 21

Fallbacks across providers

By the end of this chapter you can send a request that one provider failed to another provider, in that provider's own format, decide which errors are worth moving on for and which are not, keep models that cannot do a job out of the chain, and show what fallbacks did to Nimbus's forty-minute outage. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: When providers fail
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 1 ended with a question nobody could answer: during Tuesday's outage, Kestrel and Tara were up the whole time, idle, while Jawab's customers got nothing. Chapter 6 made the waiting shorter, and chapter 7 retried a provider's bad moments. Neither helps when a provider is down: retrying Nimbus during the outage only waits for Nimbus again.

Joseph measured it. His route table puts Nimbus first for everything (NIMBUS_FIRST, given in darwaza.ts), to see the outage at its worst. 13:50 to 14:50, with chapter 7's Darwaza, which only ever tries the first target:

No fallback, 13:50 to 14:50: 62 requests
  answered  18 (29%)   failed 44   wrong 0
  latency   p50 802 ms   p95 1,511 ms   p99 1,512 ms
  spent     ₹1.59   provider calls 62 (44 errors)

The timeouts did their job: nobody waited six seconds. But 44 customers still got an error, fast. Anjali is not interested in fast errors. "Never again forty minutes of nothing" means someone else answers.

This is the reason companies put a gateway in front of more than one provider at all, and it is the feature LiteLLM, Portkey and OpenRouter each lead with: fallbacks.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    chain.ts: only models that can do the job

    Write and export chain(req: GatewayRequest, routes = ROUTES): string[]: route(req, routes) in its order, without the targets that cannot do this job. For a translate request whose system prompt says "into Malayalam", leave out every 'kestrel/...' target (Kestrel's models cannot write Malayalam). For a reply, leave out every target in SMALL (small models drop a reply's second problem). Keep everything else.

    So a Malayalam translation that asks for model 'fast' gives ['tara/tara-flash', 'nimbus/nimbus-mini'], and a reply that asks for 'cheap' gives ['nimbus/nimbus-large', 'kestrel/kestrel-pro'].

  2. Task 2

    fallback.ts: Darwaza falls back

    Write and export handle(req: GatewayRequest, routes = ROUTES): Promise<GatewayResponse>. Call budget.onRequest(), note the start time, make a deadline of DEADLINE_MS, then go down chain(req, routes): for each target, const { r, calls } = await tryTarget(target, req, deadline, started), and add up the calls. Return respond(r, attempts) at the first answer (its model is the one that answered). Stop at an error that is not retryable, or when deadline.aborted. When nothing answered, return respond({ ok: false, error: last }).

    Under if (import.meta.main), replay 09:00 to 12:00 through handle with the label 'Darwaza with fallbacks' and print the report.

  3. Task 3

    outage.ts: the outage, with fallbacks

    Replay 13:50 to 14:50 twice with Joseph's NIMBUS_FIRST table and print both reports: first through the given handle from darwaza.ts (import it as firstOnly), as (req) => firstOnly(req, NIMBUS_FIRST), labelled 'No fallback'; then through your handle from ./fallback.ts, as (req) => handle(req, NIMBUS_FIRST), labelled 'With fallbacks'. Then print one line from the second report's results:

    answered by: nimbus N, kestrel N, tara N

    (the results whose model starts with nimbus/, kestrel/ and tara/).

  4. Task 4 · interview question

    Interview: fix-fallback.ts, two bugs

    oldFallback, given in the file, is Jawab's old reply service's fallback: during the outage it says every provider failed, and on a normal day it reports Nimbus whoever answered. Find its two bugs, then write and export fallback(req: GatewayRequest): Promise<GatewayResponse>: go down route(req) with the given call (each target with attemptSignal(TIMEOUT_MS[t], deadline) and one DEADLINE_MS deadline), return respond of the first answer, stop at an error that is not retryable, and return the last error when nothing answered.

    Under if (import.meta.main), with day({ features: ['reply'] })[0].request, print:

    old: X (oldFallback)

    fixed: X (fallback)

    X is the model that answered, or the status and the error. Try it after lab clock 14:10.

What you will learn

  • A fallback chain: the route's targets, in order, tried one by one.
  • Which errors move on, which stop, and why a 429 moves on at once.
  • Rebuilding the request for each provider's API.
  • Models that cannot do the job, kept out of the chain.
  • What the answer says: the model that really answered, and the calls.
  • The outage, measured, with and without fallbacks.

More in When providers fail

Build "Fallbacks across providers" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.