Build an AI gateway · Chapter 6 of 21
Timeouts and deadlines
By the end of this chapter you can stop Darwaza from waiting on a provider that has stopped answering, size a timeout to the model it guards, give a whole request one deadline however many providers it tries, and say what a call you gave up on still costs. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: One API over three providers
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.
Why you need this
Routing (chapter 5) moved most of Jawab's traffic off Nimbus, but replies still go to nimbus-large, and at 14:00 Nimbus goes down again. It does not fail fast. Every request hangs for six seconds, then gets a 503. Here is 13:30 to 14:30 through Darwaza as it is (real):
Darwaza, routed, 13:30 to 14:30: 83 requests
answered 69 (83.1%) failed 14 wrong 0
latency p50 133 ms p95 6,004 ms p99 6,008 ms
spent ₹3.78 provider calls 83 (14 errors)
Fourteen customers waited six seconds each for nothing. Joseph's SLO says p95 at most 1,000 ms; this hour's p95 is six times that. And six seconds is the lab's outage: a real provider in trouble can hold a connection open for a minute or more before anything says no.
Darwaza cannot make Nimbus answer. It can decide how long it is willing to wait, and then stop. That is a timeout. Every later chapter needs it: a retry (chapter 7) or a move to another provider (chapter 8) is only possible once Darwaza has stopped waiting for the first one.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
timed_call.py: a call that stops waiting
Write async def call_within(target, req, timeout_ms) -> Attempt: call's steps (build, await client.post, res.json(), a reader or normalize_error, from darwaza.py) inside async with asyncio.timeout(timeout_ms / 1000). When the time runs out (TimeoutError), return Attempt(False, target, error=ProviderError(provider, 504, "timeout", True, message)); any other failure of the request (httpx.HTTPError) is a network error (502), as in call. Never raise.
Under if __name__ == "__main__":, with day(features=["reply"], teams=["nila"])[0].request: set_clock("14:10"), call_within "nimbus/nimbus-large" with 1000 ms, reset_clock(); then call_within "kestrel/kestrel-pro" with 150 ms. For each, print what happened ("answered" or the error's kind), how long it took, and what the providers billed meanwhile (ledger()["total"]["inr"] after minus before, 4 decimals):
nimbus/nimbus-large at 14:10: KIND after N ms, billed ₹X
kestrel/kestrel-pro, 150 ms timeout: KIND after N ms, billed ₹X
Trap: running out of time is not a network failure: catch TimeoutError on its own. And httpx's own timeout= is per phase, not for the whole call: use asyncio's.
- Task 2
gateway.py: a timeout for every call
Write TIMEOUT_MS, a timeout for each of the five targets, at least twice the model's p50 on Jawab's day (the starter has chapter 5's numbers) and at most 1,500 ms.
Write async def handle(req) -> GatewayResponse: call_within (timed_call.py) route(req)[0] with TIMEOUT_MS[target], then respond(attempt) (darwaza.py).
Under if __name__ == "__main__":, replay 13:50 to 14:50 through handle with the label "Darwaza with timeouts" and print the report.
Trap: a timeout below twice a model's p50 cuts off ordinary answers, which are billed anyway.
- Task 3 · interview question
Interview: deadline.py, two timeouts, one deadline
Write async def try_in_order(targets, req, deadline_ms) -> Attempt: set one deadline before the first attempt (loop.time() + deadline_ms / 1000); for each target in order, stop if the deadline has passed, otherwise call_within with the smaller of TIMEOUT_MS[target] (gateway.py) and the milliseconds left, and return the first answer. When none answers, return the last attempt's error.
Trap: with timeouts alone, two attempts can take the sum of their timeouts; the deadline must cut the second one short, and no attempt may start after it.
What you will learn
- Why a hung call is worse than a failed one, and what it costs Jawab.
- asyncio.timeout: a limit on a whole block of awaits.
- Why httpx's own timeout= is not a deadline: it is per phase.
- A timeout as an error: kind timeout, retryable.
- Sizing a timeout to the model: long enough for an ordinary answer, short enough to matter.
- A deadline for the whole request, across several attempts.
- What giving up costs: an abandoned answer is still billed.
More in One API over three providers
Build "Timeouts and deadlines" yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.