APIs with FastAPI · Chapter 20 of 26
Limits and caching
By the end of this chapter you can limit how fast each client may call your API and tell it when to come back, explain a token bucket, let a client skip a download it already has with an ETag and a 304, keep a slow partner's answers for a while, and say where a limit must live when the API runs on more than one server. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 4 tasks
- FastAPI
- In the all labs plan
The first 2 chapters of APIs with FastAPI are free. This one comes with the all labs plan.
Why you need this
Last night one merchant's shop software had a bug: a loop that asked DakiyaGo for the same pincode 90,000 times in an hour. Nobody else's requests were refused, but every one was slow, and Imran spent the morning apologising to merchants who had done nothing wrong. The same week, Aman found that merchants' checkouts download the whole rate card on every page load, thousands of times a day, for a card that changes once a month; and every pincode lookup waits half a second for a partner's service that gives the same answer every time.
So two problems, one theme: doing less work. Refuse what is too much, and do not redo what is already done.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
ratelimit.py: a rate limit per API key
Make an app with GET /pincodes/{pincode} (the pincode's entry from /workspace/data/pincodes.json, 404 if unknown) behind a rate limit dependency. X-Api-Key must be a merchant's api_key from /workspace/data/merchants.json, or 401. Each key has a bucket of up to 5 tokens, full at its first request; a request takes 1; the bucket gains 1 token every 2 seconds, never above 5. A request that finds less than 1 token answers 429, takes nothing, and has Retry-After: the whole seconds until a token is back, rounded up. An allowed request has X-RateLimit-Remaining: the whole tokens left.
Read the time with clock(), where clock = time.monotonic at the top of the file: the check replaces clock and moves time by hand.
- Task 2
etag.py: the rate card, only when it changed
Make an app with GET /rates answering the rate card from /workspace/data/rate_card.json with an ETag header (a fingerprint of the card, in double quotes, the same every time for the same card) and Cache-Control: public, max-age=300.
When the request's If-None-Match matches (the ETag itself, the ETag with W/ in front, a comma-separated list holding either, or *), answer 304 with no body and the same two headers. Anything else gets the whole card with 200.
- Task 3
cached_lookup.py: a cached pincode lookup
The starter has lookup_partner (a slow partner call) and clock. Add GET /pincodes/{pincode}: answer what lookup_partner(pincode) gives, keeping it for 600 seconds and answering from your cache within that time; after 600 seconds ask again. Every answer has X-Cache: HIT (from the cache) or MISS (asked the partner). When the partner gives None, answer 404 and keep nothing.
Call lookup_partner by that name and read the time with clock(): the check replaces both, counts the partner's calls, and moves time by hand.
- Task 4 · interview question
Interview: shared_limit.py, a rate limit for four servers
Make an app with GET /pincodes/{pincode} (as in task 1) behind a limit of 3 requests per API key per minute, kept in Postgres: X-Api-Key must be a merchant's api_key, or 401; a minute starts at a multiple of 60 seconds of clock(); the 4th request and later in a minute answer 429 with Retry-After, the whole seconds until the next minute, rounded up. Keep the counts in a table in the database named by the environment variable DATABASE_URL, made when the app starts if it is missing. Use clock = time.time.
The check runs two copies of your app on one fresh database, and sends eight requests at once. The interviewer will ask: where does a rate limit live when there are four servers, and what goes wrong if two requests count at the same moment?
What you will learn
- Rate limits: 429 Too Many Requests and Retry-After.
- The token bucket: bursts allowed, a steady rate enforced.
- HTTP caching: Cache-Control, ETag, If-None-Match and 304.
- Caching in the server: keeping a slow answer, and for how long.
- Where state lives when there are several servers.
More in Production
Build "Limits and caching" yourself
DakiyaGo, a Jaipur courier startup, takes bookings on WhatsApp. Merchants want an API, and you build it. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of APIs with FastAPI are free. This one comes with the all labs plan.