Build an AI gateway (TypeScript) · Chapter 13 of 21
Provider prompt caching
By the end of this chapter you can explain what a provider's prompt cache keeps and what it charges, read cached tokens out of three providers' usage, ask Kestrel to cache a brand's guide, order a prompt so the cache keeps working, and measure what all of it does to Jawab's bill. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Spend less
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Lakshmi has Kestrel's invoice for Tuesday open, and one line is most of it: kestrel-pro, ₹52.10, almost all of it input tokens. Every reply Jawab drafts sends the brand's guide first: the brand's voice, its facts and Jawab's support policy, about 1,500 tokens, the same for every one of that brand's customers. The customer's message after it is 30 tokens. Darwaza pays full price for the same 1,500 tokens 190 times a day.
The caches of chapters 11 and 12 cannot help: they reuse whole answers, and every reply is to a different customer about a different order, so no two replies have the same answer. What repeats is the start of the prompt. The providers have a cache for exactly that, and Darwaza has never asked for it.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
by-hand.ts: Kestrel's prompt cache, by hand
Send nila's first reply (day({ features: ['reply'], teams: ['nila'] })[0].request) to Kestrel's kestrel-pro twice with fetch, with the system prompt as one block marked for the cache: system: [{ type: 'text', text: SYSTEM, cache_control: { type: 'ephemeral' } }], and the messages without the system one. Print one line per call:
call N: written W, read R, input I, output O, ₹X
W is usage.cache_creation_input_tokens, R cache_read_input_tokens, I input_tokens, O output_tokens; ₹X prices each at its own rate in PRICES['kestrel/kestrel-pro'] (in, cacheWrite, cacheRead, out), over 1,000,000, 4 decimals.
- Task 2
kestrel-cache.ts: Kestrel's adapter, with the cache
Export toKestrelCached(model, req, stream = false): Built, the same request toKestrel (darwaza.ts) builds, except that system is an array of one block: [{ type: 'text', text: SYSTEM, cache_control: { type: 'ephemeral' } }], SYSTEM being the system messages' content joined with \n. With no system message, leave system out.
- Task 3
gateway.ts: Darwaza with prompt caching
Export gateway: createDarwaza({ adapters: { toKestrel: toKestrelCached } }). Under if (import.meta.main), take ledger() before and after replaying 12:00 to 15:00 through gateway, print the report, and print:
kestrel cache: R tokens read, W written
R and W add up what the replay added to cached and cacheWrite over the ledger's Kestrel models.
- Task 4 · interview question
Interview: stable-prefix.ts, the note that broke the cache
Export withNote(req, note): a new request (the original unchanged) with the note at the start of the last user message,
${note}\n\n${content}, and nothing else changed.Under if (import.meta.main): with req = day({ features: ['faq'], teams: ['malabar'] })[0].request and a fresh note
Local time: ${new Date().toISOString()}for each call, send req to 'nimbus/nimbus-mini' with call four times: twice with the given onTop(req, note), then twice with withNote(req, note). After each, print usage.cached:on top, call 1: N cached
on top, call 2: N cached
withNote, call 1: N cached
withNote, call 2: N cached
What you will learn
- What a prompt cache keeps: the prefix, for a few minutes, at the provider.
- Three providers, three rules: Nimbus caches on its own, Kestrel only when asked, Tara not at all.
- What it costs: cached reads at a quarter or a tenth, Kestrel's writes at a premium.
- cache_control: asking Kestrel to keep the brand guide.
- The order of a prompt: why one changing line at the top turns the cache off.
- Warm and cold: why fallbacks and spreading traffic cost cache hits.
- Measuring it: the whole day's bill, with and without.
More in Spend less
Build "Provider prompt caching" yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway (TypeScript) are free. This one comes with the all labs plan.