Build an AI gateway · Chapter 10 of 21

Hedging the slow tail

By the end of this chapter you can say what a slow tail is and why timeouts and fallbacks cannot fix it, build a hedge that races a second call against a slow first one and stops the loser, decide which requests are worth hedging, and measure what it does to the whole day's slow tail, and what it costs. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: When providers fail
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.

Why you need this

Darwaza now retries, falls back, and skips a provider that is down. On a normal day, it answers all but one request. Joseph looks at the last column of the report, the one his SLO does not mention yet, and frowns:

No hedging, 09:00 to 21:00: 626 requests
  answered  625 (99.8%)   failed 1   wrong 0
  latency   p50 263 ms   p95 715 ms   p99 1,919 ms
  spent     ₹40.37   provider calls 680 (37 errors)

Half of Jawab's customers wait 263 ms. One in twenty waits more than 700 ms, and one in a hundred almost two seconds. Those are not failures, and they are not an outage: they are the slow tail, and on a busy day one customer in twenty is a lot of customers. Anjali has read the reviews: "the chat is usually instant, but sometimes it just sits there".

Farhan explains where those seconds come from, and the trick Google made famous for them: hedged requests.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    hedge.py: race a slow call

    Write async def hedge(first, second, after_ms) -> tuple[Attempt, bool], where first and second take no arguments and return a coroutine that gives an Attempt. Start first() as a task. If it has not finished after after_ms, start second() as a task too (hedged becomes True). The first good answer wins: return (answer, hedged). A failure is not an answer: wait for the other call if it is running, and when every started call has failed, return the last failure. If first fails before after_ms, return (that failure, False) at once and never start second. In a finally, cancel every task you started and await them, so the loser really stops.

  2. Task 2

    hedged.py: Darwaza hedges small models

    Write HEDGE_AFTER_MS ({"kestrel/kestrel-swift": 200, "nimbus/nimbus-mini": 300, "tara/tara-flash": 600}), stats = {"hedges": 0}, and async def hedged_attempt(target, nxt, req, deadline, started): when nxt is not None, target and nxt are both in SMALL and req.get("stream") is not True, hedge a call to target with a call to nxt after HEDGE_AFTER_MS[target] (each call is call(t, req, timeout_ms=TIMEOUT_MS[t], deadline=deadline)), count the hedge, and return (r, 2 if hedged else 1, hedged); otherwise return try_target's (r, calls) and False. Write async def handle(req, routes=ROUTES) returning await run_chain(req, routes, hedged_attempt).

    Under if __name__ == "__main__":, replay "09:00" to "12:00" through handle with the label "Darwaza with hedging", print the report, then hedges: N.

  3. Task 3

    tail.py: the whole day's slow tail

    Replay the whole day (no start or end) twice, each after resetting the lab (httpx.post(f"{LAB_URL}/lab/reset"), what lab reset does): first through the given handle from darwaza (import it as no_hedge), labelled "No hedging"; then through your handle from hedged, labelled "With hedging". Print both reports, then:

    hedges: N (X% of requests)

    (stats["hedges"], and X as N over the second report's requests x 100, one decimal).

  4. Task 4 · interview question

    Interview: race.py, why cancel the loser?

    Write async def count_calls(h) -> int. Send the first 10 triage requests of the day one after another through await h(first, second, 250), where first() calls "kestrel/kestrel-swift" and second() "tara/tara-flash" with the given call. Count every call started (make first and second small async functions that add 1, then call). After the last one, await asyncio.sleep(1) so late calls are counted, then return the count.

    Under if __name__ == "__main__":, print:

    naive hedge: N calls (count_calls(naive_hedge), the given one)

    hedge: N calls (count_calls(hedge), yours)

What you will learn

  • The slow tail: a few calls in a hundred that take far longer.
  • Why a timeout is the wrong tool for it, and what it costs.
  • A hedge: start a second call when the first is slow; first answer wins.
  • Cancelling the loser, and why it is still billed.
  • What to hedge: small, cheap models only.
  • The whole day's p95 and p99, measured, and the bill.

More in When providers fail

Build "Hedging the slow tail" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.