Build an AI gateway · Chapter 3 of 21
Errors in one shape
By the end of this chapter you can read any error from Nimbus, Kestrel or Tara and say in one word what kind of failure it is, whether trying again could help, and how long the provider asked Darwaza to wait. You can also decide what Jawab's app should be told, without the app ever learning a provider's private codes. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: One API over three providers
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.
Why you need this
Darwaza can now call three providers. Every later chapter has to decide what to do when a call fails: try the same provider again (chapter 7), move to the next one (chapter 8), stop sending to a provider that keeps failing (chapter 9). Those decisions need one question answered first: what kind of failure was this?
Today Darwaza cannot answer it. Here is what the three providers sent back when each was asked to slow down, after a 350 ms wait was due (real output, headers trimmed to the ones that matter):
nimbus 429 {"retry-after": "1", "retry-after-ms": "350"}
{"error":{"message":"Rate limit reached for nimbus-mini on requests per min (RPM/TPM): Limit 3000, Used 3000. Please try again in 350ms.","type":"rate_limit_exceeded","param":null,"code":"rate_limit_exceeded"}}
kestrel 429 {"retry-after": "1"}
{"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed the rate limit for your organization of 4800 requests per minute. Please wait and try again."}}
tara 429 {}
{"error":{"code":429,"message":"Resource has been exhausted (e.g. check quota). Quota exceeded for requests per minute.","status":"RESOURCE_EXHAUSTED","details":[{"@type":"type.googleapis.com/google.rpc.RetryInfo","retryDelay":"0.35s"}]}}
The same situation, three JSON shapes, and three different ways of saying "wait 350 ms". And the status codes do not line up either: Kestrel says 529 when it is overloaded, a code no other provider uses, and Tara says 400, not 401, when the key is wrong.
Joseph, on call, wants one thing from Darwaza's errors: at 2 am, a word he can act on. "rate_limited, wait 350 ms" or "auth: our key is wrong", not three vendors' JSON. This chapter gives every error one shape.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
retry_after.py: "try again in..." in three dialects
Write def retry_after_of(headers, body) -> int | None: how long the provider asked Darwaza to wait, in milliseconds, or None when it did not say.
Look in this order: the retry-after-ms header (milliseconds, Nimbus), the retry-after header (seconds, x 1,000: Nimbus and Kestrel), then Tara's body: the entry of error's details list whose "@type" ends in "google.rpc.RetryInfo", whose retryDelay is seconds as text ("0.35s" is 350).
Trap: Nimbus sends both headers, and retry-after is rounded up to a whole second, so read retry-after-ms first. The body may be None or have no details: never raise.
- Task 2
normalize.py: every error in one shape
Write a dataclass ProviderError(provider, status, kind, retryable, message, retry_after_ms=None) and def normalize_error(provider, status, headers, body) -> ProviderError.
kind from the status: 429 rate_limited, 529 overloaded, 503 unavailable, 504 or 408 timeout, any other 5xx server, 401 or 403 auth, 404 not_found, any other 4xx bad_request.
retryable is True for rate_limited, overloaded, server, unavailable, timeout, network and dropped, False for bad_request, auth and not_found. retry_after_ms is retry_after_of from retry_after.py. message is the provider's body["error"]["message"], or f"HTTP {status}" when there is none.
Trap: Tara answers a wrong key with a 400 whose message mentions the API key: that is auth, not bad_request.
- Task 3
call.py: call, with one error shape
Write async def call(target, req) -> Attempt again, with a failed Attempt's error now a ProviderError. Use build, client, Attempt and the readers from darwaza.py (given).
A 200 is Attempt(True, target, text, usage, finish). Any other status is Attempt(False, target, error=normalize_error(provider, res.status_code, res.headers, body)).
Trap: when the request itself fails (httpx.HTTPError) there is no status at all: return ProviderError(provider, 502, "network", True, message), with message str(e), or type(e).__name__ when that is empty. call never raises.
- Task 4 · interview question
Interview: gateway.py, what status does the app get?
Write async def handle(req) -> GatewayResponse: call req["model"] when the app names one, else "nimbus/nimbus-large".
An answer is {"ok": True, "text", "model", "usage", "finish"}. A failure is {"ok": False, "status", "error": the provider's message, "kind", "retry_after_ms" (only when there is one)}, with status by kind: bad_request 400; rate_limited 429; auth and not_found 502; timeout 504; overloaded, unavailable, server, network and dropped 503.
Trap: the app never sees a provider's own code (a 529, or Tara's 400 for a wrong key): a provider refusing Darwaza's own key is Darwaza's problem, not the app's request.
What you will learn
- Why providers fail, and the kinds of failure a gateway sees.
- Retryable or not: which failures can go away if you try again.
- "Try again in...": the three forms of retry-after.
- normalize_error: any provider's error as one ProviderError.
- What the app is told: one status per kind, never a provider's code.
More in One API over three providers
Build "Errors in one shape" yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.