Build an AI gateway · Chapter 9 of 21

The circuit breaker

By the end of this chapter you can explain what a circuit breaker is and why every serious gateway has one, build one with its three states, decide which errors count against a provider, and measure what it saves during Nimbus's outage: calls to a dead provider, and the time customers spend waiting for them. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: When providers fail
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.

Why you need this

Chapter 8's fallbacks answered every customer during the outage. Joseph is happy with the answered column and not with the rest:

Fallbacks, no breaker, 13:50 to 14:50: 62 requests
  answered  62 (100%)   failed 0   wrong 0
  latency   p50 897 ms   p95 1,957 ms   p99 1,991 ms
  spent     ₹5.56   provider calls 108 (46 errors)

A p50 of 897 ms during the outage means the typical customer waited almost a second. Why? Every request still went to Nimbus first, waited out nimbus-mini's 800 ms timeout (or nimbus-large's 1,500 ms), and only then moved on. At 14:30, Nimbus had been down for half an hour and had failed dozens of times in a row, and Darwaza still sent it the next request as if nothing had happened. All 62 requests called Nimbus; 45 of those calls failed.

Farhan puts it simply: Darwaza knows Nimbus is down. It just does not remember. A circuit breaker is that memory.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    breaker.py: closed, open, half-open

    Write class Breaker with __init__(self, window=10, min_calls=4, failure_rate=0.5, cooldown_ms=500, now=now_ms), where now() gives the time in ms (the default: time.monotonic() * 1000), and a field state, starting "closed".

    allow(self) -> bool: closed gives True; open gives False until cooldown_ms has passed since it opened, then it is half-open; half-open gives True for one probe and False while that probe is out.

    record(self, ok: bool): in half-open, the probe's result: ok closes it and forgets the old results, a failure opens it again (a fresh cooldown). Otherwise keep the last window results, and once there are at least min_calls, open it when failures / results reaches failure_rate, noting now() as when it opened.

  2. Task 2

    guarded.py: a breaker for every target

    Write async def handle(req: GatewayRequest, routes=ROUTES) -> GatewayResponse: the given handle (down chain(req, routes) with try_target), with one Breaker per target, kept in a dictionary at the top of the file and made the first time a target is seen. Skip a target whose breaker's allow() says no. After try_target, record healthy (True) for an answer, an error that is not retryable, or a 429; record False for every other error. When every target was skipped, return {"ok": False, "status": 503, "error": ..., "kind": "unavailable"}.

    Under if __name__ == "__main__":, replay "09:00" to "12:00" through handle with the label "Darwaza with breakers" and print the report.

  3. Task 3

    outage_slow.py: the outage at 250 ms a minute

    Replay "13:50" to "14:50" at ms_per_minute=250 twice with Joseph's NIMBUS_FIRST table, and print both reports: through the given handle (import it as no_breaker), lambda req: no_breaker(req, NIMBUS_FIRST), labelled "Fallbacks, no breaker"; then through your handle from guarded the same way, labelled "With the breaker". Then print:

    calls to Nimbus: N without, M with

    (each report's by_provider["nimbus"]["calls"]).

  4. Task 4 · interview question

    Interview: what_counts.py, what is a failure?

    Write def is_failure(e: ProviderError) -> bool: True for the kinds that say the provider is unhealthy (overloaded, server, unavailable, timeout, network, dropped), False for rate_limited, bad_request, auth and not_found.

    Under if __name__ == "__main__":, make a ProviderError for a 400 (kind "bad_request", retryable False) and two new Breakers. Record it 20 times in each: record(False) in the first (every error counts), record(not is_failure(e)) in the second. Print:

    every error counts: <the first breaker's state>

    is_failure: <the second breaker's state>

What you will learn

  • What a circuit breaker is: three states, and what moves it between them.
  • Failure rate over a window, and why a minimum number of calls.
  • The cooldown and the probe: how the breaker finds out a provider is back.
  • What counts as a failure: not a 400, not a 429.
  • Why this chapter replays the outage slower: the squeeze's one cost.
  • What the breaker saves, measured.

More in When providers fail

Build The circuit breaker yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.