Build an AI gateway · Chapter 13 of 21
Provider prompt caching
By the end of this chapter you can explain what a provider's prompt cache keeps and what it charges, read cached tokens out of three providers' usage, ask Kestrel to cache a brand's guide, order a prompt so the cache keeps working, and measure what all of it does to Jawab's bill. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Spend less
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.
Why you need this
Lakshmi has Kestrel's invoice for Tuesday open, and one line is most of it: kestrel-pro, ₹52.10, almost all of it input tokens. Every reply Jawab drafts sends the brand's guide first: the brand's voice, its facts and Jawab's support policy, about 1,500 tokens, the same for every one of that brand's customers. The customer's message after it is 30 tokens. Darwaza pays full price for the same 1,500 tokens 190 times a day.
The caches of chapters 11 and 12 cannot help: they reuse whole answers, and every reply is to a different customer about a different order, so no two replies have the same answer. What repeats is the start of the prompt. The providers have a cache for exactly that, and Darwaza has never asked for it.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
by_hand.py: Kestrel's prompt cache, by hand
Send nila's first reply (day(features=["reply"], teams=["nila"])[0].request) to Kestrel's kestrel-pro twice with httpx.post, with the system prompt as one block marked for the cache: "system": [{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}], and the messages without the system one. Print one line per call:
call N: written W, read R, input I, output O, ₹X
W is usage's cache_creation_input_tokens, R cache_read_input_tokens, I input_tokens, O output_tokens; ₹X prices each at its own rate in PRICES["kestrel/kestrel-pro"] (.input, .cache_write, .cache_read, .output), over 1,000,000, 4 decimals.
- Task 2
kestrel_cache.py: Kestrel's adapter, with the cache
Write def to_kestrel_cached(model, req, stream=False) -> Built, the same request to_kestrel (darwaza.py) builds, except that "system" is a list of one block: [{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}], SYSTEM being the system messages' content joined with "\n". With no system message, leave "system" out.
- Task 3
gateway.py: Darwaza with prompt caching
At the top level: gateway = create_darwaza(adapters={"to_kestrel": to_kestrel_cached}). Under if __name__ == "__main__":, take ledger() before and after replaying "12:00" to "15:00" through gateway, print the report, and print:
kestrel cache: R tokens read, W written
R and W add up what the replay added to "cached" and "cacheWrite" over the ledger's models whose "provider" is "kestrel".
- Task 4 · interview question
Interview: stable_prefix.py, the note that broke the cache
Write def with_note(req, note) -> GatewayRequest: a new request (the original and its messages unchanged) with the note at the start of the last user message, f"{note}\n\n{content}", and nothing else changed.
Under if __name__ == "__main__":, with req = day(features=["faq"], teams=["malabar"])[0].request and a fresh note f"Local time: {datetime.now().isoformat()}" for each call, send req to "nimbus/nimbus-mini" with call four times: twice with the given on_top(req, note), then twice with with_note(req, note). After each, print the answer's usage["cached"]:
on top, call 1: N cached
on top, call 2: N cached
with_note, call 1: N cached
with_note, call 2: N cached
What you will learn
- What a prompt cache keeps: the prefix, for a few minutes, at the provider.
- Three providers, three rules: Nimbus caches on its own, Kestrel only when asked, Tara not at all.
- What it costs: cached reads at a quarter or a tenth, Kestrel's writes at a premium.
- cache_control: asking Kestrel to keep the brand guide.
- The order of a prompt: why one changing line at the top turns the cache off.
- Warm and cold: why fallbacks and spreading traffic cost cache hits.
- Measuring it: the whole day's bill, with and without.
More in Spend less
Build "Provider prompt caching" yourself
Jawab, a Kochi support desk, answers customers for six Kerala brands on three AI providers that fail, slow down and cost different amounts. You build Darwaza, its gateway, and replay a whole day of traffic against an SLO and a budget. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build an AI gateway are free. This one comes with the all labs plan.