Build an AI gateway (TypeScript) · Chapter 6 of 21

Timeouts and deadlines

By the end of this chapter you can stop Darwaza from waiting on a provider that has stopped answering, size a timeout to the model it guards, give a whole request one deadline however many providers it tries, and say what a call you gave up on still costs. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: One API over three providers
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Routing (chapter 5) moved most of Jawab's traffic off Nimbus, but replies still go to nimbus-large, and at 14:00 Nimbus goes down again. It does not fail fast. Every request hangs for six seconds, then gets a 503. Here is 13:30 to 14:30 through Darwaza as it is (real):

Darwaza, routed, 13:30 to 14:30: 83 requests
  answered  69 (83.1%)   failed 14   wrong 0
  latency   p50 99 ms   p95 6,014 ms   p99 6,019 ms
  spent     ₹3.78   provider calls 83 (14 errors)

Fourteen customers waited six seconds each for nothing. Joseph's SLO says p95 at most 1,000 ms; this hour's p95 is six times that. And six seconds is the lab's outage: a real provider in trouble can hold a connection open for a minute or more before anything says no.

Darwaza cannot make Nimbus answer. It can decide how long it is willing to wait, and then stop. That is a timeout. Every later chapter needs it: a retry (chapter 7) or a move to another provider (chapter 8) is only possible once Darwaza has stopped waiting for the first one.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    timed-call.ts: a call that stops waiting

    Write and export callWithin(target, req, signal: AbortSignal): Promise<Attempt>: call's steps (build, fetch, the JSON, a reader or normalizeError, from ./darwaza.ts) with signal passed to fetch. When it throws because the signal aborted (signal.aborted), return { ok: false, error: { provider, status: 504, kind: 'timeout', retryable: true, message } }; any other failure of fetch is a network error (502), as in call. Never throw.

    Under if (import.meta.main), with day({ features: ['reply'], teams: ['nila'] })[0].request: setClock('14:10'), callWithin 'nimbus/nimbus-large' with AbortSignal.timeout(1000), resetClock(); then callWithin 'kestrel/kestrel-pro' with AbortSignal.timeout(150). For each, print what happened ('answered' or the error's kind), how long it took, and what the providers billed meanwhile (ledger().total.inr after minus before, 4 decimals):

    nimbus/nimbus-large at 14:10: KIND after N ms, billed ₹X

    kestrel/kestrel-pro, 150 ms timeout: KIND after N ms, billed ₹X

    Trap: an abort is not a network failure: look at signal.aborted, not at the error's message.

  2. Task 2

    gateway.ts: a timeout for every call

    Write and export TIMEOUT_MS: Record<string, number>, a timeout for each of the five targets, at least twice the model's p50 on Jawab's day (the starter has chapter 5's numbers) and at most 1,500 ms.

    Write and export handle(req): Promise<GatewayResponse>: callWithin (./timed-call.ts) route(req)[0] with AbortSignal.timeout(TIMEOUT_MS[target]), then respond(attempt) (./darwaza.ts).

    Under if (import.meta.main), replay 13:50 to 14:50 through handle with the label 'Darwaza with timeouts' and print the report.

    Trap: a timeout below twice a model's p50 cuts off ordinary answers, which are billed anyway.

  3. Task 3 · interview question

    Interview: deadline.ts, two timeouts, one deadline

    Write and export tryInOrder(targets: string[], req, deadlineMs: number): Promise<Attempt>: make one AbortSignal.timeout(deadlineMs) before the first attempt; for each target in order, stop if the deadline has passed (deadline.aborted), otherwise callWithin with AbortSignal.any([AbortSignal.timeout(TIMEOUT_MS[target]), deadline]) (TIMEOUT_MS from ./gateway.ts) and return the first answer. When none answers, return the last attempt's error.

    Trap: with timeouts alone, two attempts can take the sum of their timeouts; the deadline must cut the second one short, and no attempt may start after it.

What you will learn

  • Why a hung call is worse than a failed one, and what it costs Jawab.
  • AbortSignal.timeout and passing a signal to fetch.
  • A timeout as an error: kind timeout, retryable.
  • Sizing a timeout to the model: long enough for an ordinary answer, short enough to matter.
  • A deadline for the whole request, across several attempts, with AbortSignal.any.
  • What giving up costs: an abandoned answer is still billed.

More in One API over three providers

Build "Timeouts and deadlines" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.