Build an AI gateway (TypeScript) · Chapter 11 of 21
The exact cache
By the end of this chapter you can say when an AI answer can be reused and when it must not be, build a cache key from everything that changes an answer, keep answers for a while without keeping too many, share one provider call between identical requests that arrive together, and show the bug that serves one brand's answer to another brand's customer. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Spend less
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Darwaza is reliable now: on a normal day it answers every request, and the slow tail is short. The next three chapters are for Lakshmi: what it costs, and how to make it cost less.
Lakshmi starts with the simplest question there is. Kaapi Kottai's FAQ widget asked "Is the coffee powder chicory free?" five times on Tuesday, word for word, with the same brand facts, at temperature 0. The answer was the same all five times:
09:07:54 kaapi faq | 11:11:30 kaapi faq | 13:25:21 kaapi faq | 16:57:05 kaapi faq | 19:39:55 kaapi faq
Jawab paid for it five times, and each customer waited for a model to work out what it had already worked out. Over the whole day, 75 requests are exact repeats of an earlier one: FAQs asked again, and triage tickets the app sent twice. A gateway is the natural place to remember answers, because it sees every request from every team. LiteLLM, Portkey and Cloudflare's AI Gateway all have a cache for this reason.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
cache-key.ts: everything that changes the answer
Write and export cacheKey(req: GatewayRequest): string: the sha256 hex (node:crypto) of JSON.stringify([team, feature, model ?? null, messages as [role, content] pairs, maxTokens ?? null, temperature ?? null]). Never the id, and never JSON.stringify(req) (the order of an object's fields would change the key).
Then write and export cacheable(req: GatewayRequest): boolean: true only when temperature is exactly 0 and stream is not true.
- Task 2
exact-cache.ts: a TTL and a size cap
Write and export class ExactCache with constructor(ttlMs = 600_000, max = 5000, now = Date.now), fields hits and misses (0), and a Map from cacheKey(req) to { value, expires }.
get(req): when there is no entry, or its expires is at or before now(), count a miss (delete a stale one) and return undefined; otherwise count a hit, move the entry to the end of the Map (delete and set again) and return its value.
set(req, value): never keep an answer whose finish is 'length'; store { value, expires: now() + ttlMs } at the end of the Map; while there are more than max entries, delete the first one (the least recently used).
- Task 3
cached.ts: Darwaza caches
Write and export cache (an ExactCache) and handle(req: GatewayRequest, routes = ROUTES): Promise<GatewayResponse>. A request that is not cacheable goes straight to the given handle (import it as darwaza). A hit returns { ...hit, cached: 'exact', costInr: 0, attempts: 0 }. On a miss, if an identical request (same cacheKey) is on its way to a provider, wait for its promise (keep a Map from key to promise while the call is out), and return its answer marked the same way when it is ok. Otherwise call darwaza(req, routes) and keep the answer in the cache when it is ok, never an error.
Under if (import.meta.main), replay 17:00 to 20:00 through handle with the label 'Darwaza with a cache', print the report, then hits: N, misses: M (cache.hits and cache.misses).
- Task 4 · interview question
Interview: shared-key.ts, a key that mixes brands
Write and export nilaGets(key: (req: GatewayRequest) => string): Promise<string>. Make a new Map from key to answer text. Ask Kaapi Kottai's FAQ "Do you ship outside India?" (the first kaapi faq in day() with that question), then Nila Handlooms' same question: for each, look up key(req), and on a miss send it through the given handle and keep its text when ok. Return the text Nila's customer got.
Under if (import.meta.main), print:
naive key: <nilaGets(naiveKey)> (the given key: the question alone)
cacheKey: <nilaGets(cacheKey)> (yours)
What you will learn
- When an answer can be reused: temperature 0, a whole answer.
- The cache key: everything that changes the answer, and nothing else.
- A TTL and a size cap, and which entry to drop.
- Requests in flight: one provider call for identical requests at once.
- The cross-brand bug, and why the key holds the team.
- What it saves, measured on the whole day.
More in Spend less
Build The exact cache yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.