Build an AI gateway (TypeScript) · Chapter 14 of 21

Rate limits per team

By the end of this chapter you can explain how a token bucket enforces "so many a minute", hold each of Jawab's brands to its own plan with Darwaza's own 429, keep Darwaza under a provider's limit before the provider has to say no, and explain why a counter that resets every minute lets twice the limit through. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Spend less
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.

Why you need this

At 18:00 every Tuesday, Payyoli Snacks runs a flash sale on banana chips. For half an hour its customers flood the support widget: 90 of the 125 requests between 18:00 and 18:30 are Payyoli's. In the replay, those thirty minutes of Jawab's day take 750 ms of real time, so Payyoli sends about 120 requests a second.

The providers' limits are per account: Kestrel's 4,800 requests a minute are for Jawab's key, shared by all six brands. On Tuesday the sale fit inside them. But Payyoli's plan with Jawab is 1,800 requests a minute, like every brand's, and its sale used four times that. Next month's sale may be bigger, or Kestrel's limit smaller, and then Kestrel's "too many requests" lands on whichever request comes next: a Kaapi Kottai customer asking about a refund, who has nothing to do with banana chips. That is the noisy neighbour.

Anjali wants every brand to get what it pays for, whatever another brand is doing. Joseph wants Darwaza to never be the reason a provider says no. Darwaza so far has no idea how fast anyone is sending.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    bucket.ts: the token bucket

    Export class TokenBucket: constructor(perMinute, capacity = perMinute / 60, now = Date.now); it starts full, refills perMinute / 60,000 tokens per ms (worked out from now() at each take) and never holds more than capacity. take(n = 1) takes n and returns { ok: true } when there are n; otherwise it takes nothing and returns { ok: false, retryAfterMs }, the ms until there are n, rounded up.

  2. Task 2

    gateway.ts: limits per team

    Export handle(req): per team (TEAMS), one TokenBucket(team.rpm) for requests and one TokenBucket(team.tpm) for tokens. Take 1 request, then estimateTokens(req.messages) + (req.maxTokens ?? 256) tokens; if either says no, return { ok: false, status: 429, kind: 'team_limit', retryAfterMs, error } without calling a provider; otherwise return createDarwaza().handle(req) (one Darwaza, made once).

    Under if (import.meta.main): replay 18:00 to 18:30 through handle, print the report, then one line per team of TEAMS:

    payyoli: N requests, R refused

  3. Task 3

    provider-limits.ts: stay under the provider's limit

    Export class ProviderLimits: constructor(rpm: Record<provider, number>, share = 0.9, now = Date.now) makes one TokenBucket per provider at share x rpm a minute (holding one second of that); allow(target) takes 1 from the bucket of the target's provider (the part before the slash) and returns whether it could; a provider not in rpm is always allowed; each false adds 1 to skipped.

  4. Task 4 · interview question

    Interview: fixed-window.ts, why not a counter per minute?

    Export class FixedWindow(limit, windowMs, now = Date.now) whose take() allows while the count in the current window (Math.floor(now() / windowMs)) is below limit, starting again at 0 in each new window, and maxInAnyWindow(times, windowMs): the most times inside any span [t, t + windowMs).

    Under if (import.meta.main): on a fake clock, offer one request every ms from t = 900 to t = 1099 to a FixedWindow(60, 1000, now) and a TokenBucket(3600, 60, now), keep the times each allowed, and print:

    fixed window: N allowed, at most M in one second

    token bucket: N allowed, at most M in one second

What you will learn

  • The token bucket: a capacity, a steady refill, and how long to wait.
  • Limits per team: requests and tokens a minute, from each brand's plan.
  • Darwaza's own 429: a refusal with kind: 'team_limit' and retryAfterMs, not a failure.
  • The noisy neighbour: what one brand's burst does to the others at a shared provider.
  • Limits per provider: counting Darwaza's own calls and stopping short of the provider's limit.
  • Fixed windows: why "reset the counter every minute" is the wrong design.

More in Spend less

Build "Rate limits per team" yourself

Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.