Build an AI gateway (TypeScript) · Chapter 7 of 21

Retries, backoff and jitter

By the end of this chapter you can say which provider errors are worth trying again and which are not, how long to wait before the next try, why every client waiting the same time is a problem, how a deadline caps the waiting, and how a retry budget stops retries from making an outage worse. Darwaza will retry, and you will measure what that bought. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: When providers fail
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Darwaza now sends each feature to the right model (chapter 5) and gives up on a slow attempt (chapter 6). On a normal afternoon it still loses customers. Joseph replayed 15:00 to 18:00 through it:

Darwaza, chapter 6, 15:00 to 18:00: 104 requests
  answered  96 (92.3%)   failed 8   wrong 0
  latency   p50 278 ms   p95 502 ms   p99 702 ms
  spent     ₹7.42   provider calls 104 (6 errors)

Six of the eight failures were a provider's bad moment: four 529 Overloaded from Kestrel and two 500s, one from Nimbus and one from Tara. None of them was down. Each was busy or hiccupped for a moment, said so, and Darwaza passed the error straight to the customer (as a 503: chapter 3's statuses for the app). Anjali asks the obvious question: if Kestrel was fine a second later, why did nobody ask it again?

Farhan agrees, and adds the warning every platform engineer learns the hard way: a retry is the easiest reliability feature to write and the easiest to get wrong. Done carelessly, retries turn a provider's bad second into a bad hour. This chapter does it carefully.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    backoff.ts: how long to wait

    Write and export backoff(attempt: number, rand = Math.random): number, exponential backoff with full jitter: Math.round(rand() * Math.min(400, 50 * 2 ** attempt)), where attempt is the retry's number (1 for the first retry). So with rand () => 1, retry 1 waits 100 ms, retry 3 waits 400 ms, and retry 5 is capped at 400 ms; with Math.random, every wait falls anywhere from 0 to that ceiling.

    Then write and export waitFor(error: ProviderError, attempt: number, rand = Math.random): number: error.retryAfterMs when the provider gave one (exactly that), otherwise backoff(attempt, rand).

  2. Task 2

    budget.ts: a retry budget

    Write and export class RetryBudget with constructor(ratio = 0.2, max = 10) and a field tokens that starts at max. onRequest() adds ratio to tokens, never above max. tryRetry() spends one token and returns true when tokens is at least 1, and returns false otherwise.

    So a new budget allows 10 retries in a row; after that, every 5 requests earn one more.

  3. Task 3

    retry.ts: Darwaza retries

    Write and export handle(req: GatewayRequest, deadlineMs = DEADLINE_MS): Promise<GatewayResponse>: the given handle (the first target of route(req), one call with attemptSignal and the deadline), plus retries. Make one RetryBudget(0.2, 10) at the top of the file and call onRequest() for each request.

    After a failed attempt, retry at most 2 times, only when error.retryable is true and error.kind is not 'timeout'. Wait waitFor(error, retry) (retry is 1, then 2), but if the time so far plus the wait would reach deadlineMs, stop without sleeping; and stop when budget.tryRetry() says no. Return respond(r, attempts).

    Under if (import.meta.main), replay 09:00 to 12:00 through handle with the label 'Darwaza with retries' and print the report.

  4. Task 4 · interview question

    Interview: herd.ts, why add randomness?

    Write and export retrySpread(wait: (attempt: number) => number): Promise<number>. Take the first triage request of the day and send it 20 times at once (Promise.all) to 'kestrel/kestrel-swift' with the given call. Each one that fails with a retryable error waits wait(1), notes Date.now() just before it retries, and sends the request once more. Return the latest time noted minus the earliest, in ms.

    Under if (import.meta.main), print:

    no jitter: N ms (retrySpread(() => 100))

    full jitter: N ms (retrySpread((a) => backoff(a)), your backoff)

    Try it after lab fault kestrel 529 80 in the terminal.

What you will learn

  • Which errors to retry, and which never.
  • How long to wait: the provider's own retry-after, in three forms.
  • Exponential backoff, and why it needs jitter.
  • Deadlines: never sleep past the time the customer has.
  • A retry budget: retries capped at a share of the real requests.
  • What retries are worth, measured on Jawab's afternoon.

More in When providers fail

Build "Retries, backoff and jitter" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.