Build an AI gateway (TypeScript) · Chapter 10 of 21

Hedging the slow tail

By the end of this chapter you can say what a slow tail is and why timeouts and fallbacks cannot fix it, build a hedge that races a second call against a slow first one and stops the loser, decide which requests are worth hedging, and measure what it does to the whole day's slow tail, and what it costs. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: When providers fail
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Darwaza now retries, falls back, and skips a provider that is down. On a normal day, it answers all but one request. Joseph looks at the last column of the report, the one his SLO does not mention yet, and frowns:

No hedging, 09:00 to 21:00: 626 requests
  answered  625 (99.8%)   failed 1   wrong 0
  latency   p50 259 ms   p95 826 ms   p99 1,920 ms
  spent     ₹40.49   provider calls 681 (38 errors)

Half of Jawab's customers wait 259 ms. One in twenty waits more than 800 ms, and one in a hundred almost two seconds. Those are not failures, and they are not an outage: they are the slow tail, and on a busy day one customer in twenty is a lot of customers. Anjali has read the reviews: "the chat is usually instant, but sometimes it just sits there".

Farhan explains where those seconds come from, and the trick Google made famous for them: hedged requests.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    hedge.ts: race a slow call

    Write and export hedge(first, second, afterMs): Promise<Attempt & { hedged: boolean }>, where first and second are (signal: AbortSignal) => Promise<Attempt>. Start first with its own AbortController's signal. If it has not settled after afterMs, start second with another controller's signal (hedged becomes true). The first good answer wins: abort the other call and return { ...answer, hedged }. A failure is not an answer: wait for the other call if it is running, and when every started call has failed, return the last failure. If first fails before afterMs, return that failure at once and never start second. Clear the timer once there is an answer.

  2. Task 2

    hedged.ts: Darwaza hedges small models

    Export HEDGE_AFTER_MS ({ 'kestrel/kestrel-swift': 200, 'nimbus/nimbus-mini': 300, 'tara/tara-flash': 600 }), stats = { hedges: 0 }, and hedgedAttempt, a TryOne: when there is a next target, target and next are both in SMALL and req.stream is not true, hedge a call to target with a call to next after HEDGE_AFTER_MS[target] (each call's signal joins the hedge's signal with attemptSignal(TIMEOUT_MS[t], deadline) through AbortSignal.any), count the hedge, and return { r, calls: 2 if hedged else 1, usedNext: hedged }; otherwise return tryTarget(target, req, deadline, started). Export handle(req, routes = ROUTES) = runChain(req, routes, hedgedAttempt).

    Under if (import.meta.main), replay 09:00 to 12:00 through handle with the label 'Darwaza with hedging', print the report, then hedges: N.

  3. Task 3

    tail.ts: the whole day's slow tail

    Replay the whole day (no from or to) twice, each after resetting the lab (await fetch(${LAB_URL}/lab/reset, { method: 'POST' }), what lab reset does): first through the given handle from darwaza.ts (import it as noHedge), labelled 'No hedging'; then through your handle from ./hedged.ts, labelled 'With hedging'. Print both reports, then:

    hedges: N (X% of requests)

    (stats.hedges, and X as N over the second report's requests x 100, one decimal).

  4. Task 4 · interview question

    Interview: race.ts, why not Promise.race?

    Write and export countCalls(h): Promise<number>. Send the first 10 triage requests of the day one after another through h(first, second, 250), where first calls 'kestrel/kestrel-swift' and second 'tara/tara-flash' with the given call and the signal h passes in. Count every call started (wrap first and second). After the last one, sleep 1,000 ms so late calls are counted, then return the count.

    Under if (import.meta.main), print:

    Promise.race: N calls (countCalls(naiveHedge), the given one)

    hedge: N calls (countCalls(hedge), yours)

What you will learn

  • The slow tail: a few calls in a hundred that take far longer.
  • Why a timeout is the wrong tool for it, and what it costs.
  • A hedge: start a second call when the first is slow; first answer wins.
  • Aborting the loser, and why it is still billed.
  • What to hedge: small, cheap models only.
  • The whole day's p95 and p99, measured, and the bill.

More in When providers fail

Build "Hedging the slow tail" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.