Build an AI gateway · Chapter 10 of 21
Hedging the slow tail
By the end of this chapter you can say what a slow tail is and why timeouts and fallbacks cannot fix it, build a hedge that races a second call against a slow first one and stops the loser, decide which requests are worth hedging, and measure what it does to the whole day's slow tail, and what it costs. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: When providers fail
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.
Why you need this
Darwaza now retries, falls back, and skips a provider that is down. On a normal day, it answers all but one request. Joseph looks at the last column of the report, the one his SLO does not mention yet, and frowns:
No hedging, 09:00 to 21:00: 626 requests
answered 625 (99.8%) failed 1 wrong 0
latency p50 263 ms p95 715 ms p99 1,919 ms
spent ₹40.37 provider calls 680 (37 errors)
Half of Jawab's customers wait 263 ms. One in twenty waits more than 700 ms, and one in a hundred almost two seconds. Those are not failures, and they are not an outage: they are the slow tail, and on a busy day one customer in twenty is a lot of customers. Anjali has read the reviews: "the chat is usually instant, but sometimes it just sits there".
Farhan explains where those seconds come from, and the trick Google made famous for them: hedged requests.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
hedge.py: race a slow call
Write async def hedge(first, second, after_ms) -> tuple[Attempt, bool], where first and second take no arguments and return a coroutine that gives an Attempt. Start first() as a task. If it has not finished after after_ms, start second() as a task too (hedged becomes True). The first good answer wins: return (answer, hedged). A failure is not an answer: wait for the other call if it is running, and when every started call has failed, return the last failure. If first fails before after_ms, return (that failure, False) at once and never start second. In a finally, cancel every task you started and await them, so the loser really stops.
- Task 2
hedged.py: Darwaza hedges small models
Write HEDGE_AFTER_MS ({"kestrel/kestrel-swift": 200, "nimbus/nimbus-mini": 300, "tara/tara-flash": 600}), stats = {"hedges": 0}, and async def hedged_attempt(target, nxt, req, deadline, started): when nxt is not None, target and nxt are both in SMALL and req.get("stream") is not True, hedge a call to target with a call to nxt after HEDGE_AFTER_MS[target] (each call is call(t, req, timeout_ms=TIMEOUT_MS[t], deadline=deadline)), count the hedge, and return (r, 2 if hedged else 1, hedged); otherwise return try_target's (r, calls) and False. Write async def handle(req, routes=ROUTES) returning await run_chain(req, routes, hedged_attempt).
Under if __name__ == "__main__":, replay "09:00" to "12:00" through handle with the label "Darwaza with hedging", print the report, then hedges: N.
- Task 3
tail.py: the whole day's slow tail
Replay the whole day (no start or end) twice, each after resetting the lab (httpx.post(f"{LAB_URL}/lab/reset"), what lab reset does): first through the given handle from darwaza (import it as no_hedge), labelled "No hedging"; then through your handle from hedged, labelled "With hedging". Print both reports, then:
hedges: N (X% of requests)
(stats["hedges"], and X as N over the second report's requests x 100, one decimal).
- Task 4 · interview question
Interview: race.py, why cancel the loser?
Write async def count_calls(h) -> int. Send the first 10 triage requests of the day one after another through await h(first, second, 250), where first() calls "kestrel/kestrel-swift" and second() "tara/tara-flash" with the given call. Count every call started (make first and second small async functions that add 1, then call). After the last one, await asyncio.sleep(1) so late calls are counted, then return the count.
Under if __name__ == "__main__":, print:
naive hedge: N calls (count_calls(naive_hedge), the given one)
hedge: N calls (count_calls(hedge), yours)
What you will learn
- The slow tail: a few calls in a hundred that take far longer.
- Why a timeout is the wrong tool for it, and what it costs.
- A hedge: start a second call when the first is slow; first answer wins.
- Cancelling the loser, and why it is still billed.
- What to hedge: small, cheap models only.
- The whole day's p95 and p99, measured, and the bill.
More in When providers fail
Build "Hedging the slow tail" yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.