Build an AI gateway · Chapter 14 of 21
Rate limits per team
By the end of this chapter you can explain how a token bucket enforces "so many a minute", hold each of Jawab's brands to its own plan with Darwaza's own 429, keep Darwaza under a provider's limit before the provider has to say no, and explain why a counter that resets every minute lets twice the limit through. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Spend less
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.
Why you need this
At 18:00 every Tuesday, Payyoli Snacks runs a flash sale on banana chips. For half an hour its customers flood the support widget: 90 of the 125 requests between 18:00 and 18:30 are Payyoli's. In the replay, those thirty minutes of Jawab's day take 750 ms of real time, so Payyoli sends about 120 requests a second.
The providers' limits are per account: Kestrel's 4,800 requests a minute are for Jawab's key, shared by all six brands. On Tuesday the sale fit inside them. But Payyoli's plan with Jawab is 1,800 requests a minute, like every brand's, and its sale used four times that. Next month's sale may be bigger, or Kestrel's limit smaller, and then Kestrel's "too many requests" lands on whichever request comes next: a Kaapi Kottai customer asking about a refund, who has nothing to do with banana chips. That is the noisy neighbour.
Anjali wants every brand to get what it pays for, whatever another brand is doing. Joseph wants Darwaza to never be the reason a provider says no. Darwaza so far has no idea how fast anyone is sending.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
bucket.py: the token bucket
Write class TokenBucket: __init__(self, per_minute, capacity=None, now=a clock in ms) (capacity None means per_minute / 60; now defaults to a function returning time.monotonic() * 1000). It starts full, refills per_minute / 60,000 tokens per ms (worked out from now() at each take) and never holds more than capacity. take(n=1) takes n and returns (True, 0) when there are n; otherwise it takes nothing and returns (False, retry_after_ms), the ms until there are n, rounded up (math.ceil).
- Task 2
gateway.py: limits per team
Write async def handle(req): per team (TEAMS), one TokenBucket(team.rpm) for requests and one TokenBucket(team.tpm) for tokens. Take 1 request, then estimate_tokens(req["messages"]) + (req.get("max_tokens") or 256) tokens; if either says no, return {"ok": False, "status": 429, "kind": "team_limit", "retry_after_ms": ..., "error": ...} without calling a provider; otherwise return await darwaza.handle(req), with darwaza = create_darwaza() made once at the top.
Under if __name__ == "__main__":, replay "18:00" to "18:30" through handle, print the report, then one line per team of TEAMS:
payyoli: N requests, R refused
- Task 3
provider_limits.py: stay under the provider's limit
Write class ProviderLimits: __init__(self, rpm, share=0.9, now=a clock in ms) makes one TokenBucket per provider of the dictionary rpm, at share x rpm a minute (holding one second of that); allow(target) takes 1 from the bucket of the target's provider (the part before the slash) and returns whether it could; a provider not in rpm is always allowed; each False adds 1 to self.skipped (0 at the start).
- Task 4 · interview question
Interview: fixed_window.py, why not a counter per minute?
Write class FixedWindow(limit, window_ms, now=a clock in ms) whose take() allows while the count in the current window (math.floor(now() / window_ms)) is below limit, starting again at 0 in each new window, and def max_in_any_window(times, window_ms): the most times inside any span [t, t + window_ms).
Under if __name__ == "__main__":, on a fake clock, offer one request every ms from t = 900 to t = 1099 to a FixedWindow(60, 1000, now) and a TokenBucket(3600, 60, now), keep the times each allowed, and print:
fixed window: N allowed, at most M in one second
token bucket: N allowed, at most M in one second
What you will learn
- The token bucket: a capacity, a steady refill, and how long to wait.
- Limits per team: requests and tokens a minute, from each brand's plan.
- Darwaza's own 429: a refusal with "kind": "team_limit" and "retry_after_ms", not a failure.
- The noisy neighbour: what one brand's burst does to the others at a shared provider.
- Limits per provider: counting Darwaza's own calls and stopping short of the provider's limit.
- Fixed windows: why "reset the counter every minute" is the wrong design.
More in Spend less
Build "Rate limits per team" yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.