Build an AI gateway · Chapter 11 of 21
The exact cache
By the end of this chapter you can say when an AI answer can be reused and when it must not be, build a cache key from everything that changes an answer, keep answers for a while without keeping too many, share one provider call between identical requests that arrive together, and show the bug that serves one brand's answer to another brand's customer. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Spend less
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.
Why you need this
Darwaza is reliable now: on a normal day it answers every request, and the slow tail is short. The next three chapters are for Lakshmi: what it costs, and how to make it cost less.
Lakshmi starts with the simplest question there is. Kaapi Kottai's FAQ widget asked "Is the coffee powder chicory free?" five times on Tuesday, word for word, with the same brand facts, at temperature 0. The answer was the same all five times:
09:07:54 kaapi faq | 11:11:30 kaapi faq | 13:25:21 kaapi faq | 16:57:05 kaapi faq | 19:39:55 kaapi faq
Jawab paid for it five times, and each customer waited for a model to work out what it had already worked out. Over the whole day, 75 requests are exact repeats of an earlier one: FAQs asked again, and triage tickets the app sent twice. A gateway is the natural place to remember answers, because it sees every request from every team. LiteLLM, Portkey and Cloudflare's AI Gateway all have a cache for this reason.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
cache_key.py: everything that changes the answer
Write def cache_key(req: GatewayRequest) -> str: the sha256 hex (hashlib, hexdigest()) of json.dumps([team, feature, model or None, messages as [role, content] pairs, max_tokens or None, temperature or None]), encoded. Never the id, and never json.dumps(req) (the order of a dictionary's keys would change the key).
Then write def cacheable(req: GatewayRequest) -> bool: True only when temperature is exactly 0 and stream is not True.
- Task 2
exact_cache.py: a TTL and a size cap
Write class ExactCache with __init__(self, ttl_ms=600_000, max=5000, now=now_ms) (now gives ms; the default is time.monotonic() * 1000), fields hits and misses (0), and an OrderedDict from cache_key(req) to (value, expires).
get(req): when there is no entry, or its expires is at or before now(), count a miss (delete a stale one) and return None; otherwise count a hit, move the entry to the end (move_to_end) and return its value.
set(req, value): never keep an answer whose "finish" is "length"; store (value, now() + ttl_ms) at the end; while there are more than max entries, drop the first one (popitem(last=False), the least recently used).
- Task 3
cached.py: Darwaza caches
Write cache = ExactCache() and async def handle(req: GatewayRequest, routes=ROUTES) -> GatewayResponse. A request that is not cacheable goes straight to the given handle (import it as darwaza). A hit returns {**hit, "cached": "exact", "cost_inr": 0, "attempts": 0}. On a miss, if an identical request (same cache_key) is on its way to a provider, await its task (keep a dictionary from key to asyncio.create_task(darwaza(req, routes)) while the call is out), and return its answer marked the same way when it is ok. Otherwise start the call that way, await it, and keep the answer in the cache when it is ok, never an error.
Under if __name__ == "__main__":, replay "17:00" to "20:00" through handle with the label "Darwaza with a cache", print the report, then hits: N, misses: M (cache.hits and cache.misses).
- Task 4 · interview question
Interview: shared_key.py, a key that mixes brands
Write async def nila_gets(key) -> str, where key(req) gives a cache key. Make a new dictionary from key to answer text. Ask Kaapi Kottai's FAQ "Do you ship outside India?" (the first kaapi faq in day() with that question), then Nila Handlooms' same question: for each, look up key(req), and on a miss send it through the given handle and keep its text when ok. Return the text Nila's customer got.
Under if __name__ == "__main__":, print:
naive key: <nila_gets(naive_key)> (the given key: the question alone)
cache_key: <nila_gets(cache_key)> (yours)
What you will learn
- When an answer can be reused: temperature 0, a whole answer.
- The cache key: everything that changes the answer, and nothing else.
- A TTL and a size cap, and which entry to drop.
- Requests in flight: one provider call for identical requests at once.
- The cross-brand bug, and why the key holds the team.
- What it saves, measured on the whole day.
More in Spend less
Build The exact cache yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.