Build an AI gateway (TypeScript) · Chapter 9 of 21
The circuit breaker
By the end of this chapter you can explain what a circuit breaker is and why every serious gateway has one, build one with its three states, decide which errors count against a provider, and measure what it saves during Nimbus's outage: calls to a dead provider, and the time customers spend waiting for them. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: When providers fail
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Chapter 8's fallbacks answered every customer during the outage. Joseph is happy with the answered column and not with the rest:
Fallbacks, no breaker, 13:50 to 14:50: 62 requests
answered 62 (100%) failed 0 wrong 0
latency p50 893 ms p95 1,956 ms p99 1,992 ms
spent ₹5.56 provider calls 108 (46 errors)
A p50 of 893 ms during the outage means the typical customer waited almost a second. Why? Every request still went to Nimbus first, waited out nimbus-mini's 800 ms timeout (or nimbus-large's 1,500 ms), and only then moved on. At 14:30, Nimbus had been down for half an hour and had failed dozens of times in a row, and Darwaza still sent it the next request as if nothing had happened. All 62 requests called Nimbus; 45 of those calls failed.
Farhan puts it simply: Darwaza knows Nimbus is down. It just does not remember. A circuit breaker is that memory.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
breaker.ts: closed, open, half-open
Write and export class Breaker with constructor(opts = {}): window (10), minCalls (4), failureRate (0.5), cooldownMs (500) and now (Date.now), and a field state, starting 'closed'.
allow(): boolean: closed gives true; open gives false until cooldownMs has passed since it opened, then it is half-open; half-open gives true for one probe and false while that probe is out.
record(ok: boolean): in half-open, the probe's result: ok closes it and forgets the old results, a failure opens it again (a fresh cooldown). Otherwise keep the last window results, and once there are at least minCalls, open it when failures / results reaches failureRate, noting now() as when it opened.
- Task 2
guarded.ts: a breaker for every target
Write and export handle(req: GatewayRequest, routes = ROUTES): Promise<GatewayResponse>: the given handle (down chain(req, routes) with tryTarget), with one Breaker per target, kept in a Map and made the first time a target is seen. Skip a target whose breaker's allow() says no. After tryTarget, record healthy (true) for an answer, an error that is not retryable, or a 429; record false for every other error. When every target was skipped, return { ok: false, status: 503, error, kind: 'unavailable' }.
Under if (import.meta.main), replay 09:00 to 12:00 through handle with the label 'Darwaza with breakers' and print the report.
- Task 3
outage-slow.ts: the outage at 250 ms a minute
Replay 13:50 to 14:50 at msPerMinute 250 twice with Joseph's NIMBUS_FIRST table, and print both reports: through the given handle (import it as noBreaker), (req) => noBreaker(req, NIMBUS_FIRST), labelled 'Fallbacks, no breaker'; then through your handle from ./guarded.ts the same way, labelled 'With the breaker'. Then print:
calls to Nimbus: N without, M with
(each report's byProvider.nimbus.calls).
- Task 4 · interview question
Interview: what-counts.ts, what is a failure?
Write and export isFailure(e: ProviderError): boolean: true for the kinds that say the provider is unhealthy (overloaded, server, unavailable, timeout, network, dropped), false for rate_limited, bad_request, auth and not_found.
Under if (import.meta.main), make a ProviderError for a 400 (kind 'bad_request', retryable false) and two new Breakers. Record it 20 times in each: record(false) in the first (every error counts), record(!isFailure(e)) in the second. Print:
every error counts: <the first breaker's state>
isFailure: <the second breaker's state>
What you will learn
- What a circuit breaker is: three states, and what moves it between them.
- Failure rate over a window, and why a minimum number of calls.
- The cooldown and the probe: how the breaker finds out a provider is back.
- What counts as a failure: not a 400, not a 429.
- Why this chapter replays the outage slower: the squeeze's one cost.
- What the breaker saves, measured.
More in When providers fail
Build The circuit breaker yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.