Building with LLMs (TypeScript) · Chapter 21 of 24
Caching
By the end of this chapter you can read how much of a prompt the provider served from its cache, order a request so the cache works, keep your own cache of replies, and say when a cached answer is safe to give and when it must expire. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Building with LLMs
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Harpreet has the month's bill open. "Every WhatsApp question sends the whole policy. The question is ten tokens and the policy is twelve hundred, so we pay for the policy thousands of times a day." Lata adds the other half: "And half of the questions are the same five questions."
Everything so far sends the full request and pays for every token of it, every time. That is correct, and it is also slow and expensive when most of the request repeats. There are two caches to use, one at the provider and one in your own code, and each has rules about when it is safe.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
cache-hits.ts: see the cache at work
Harpreet wants to see what prompt caching saves. Ask lab-chat three questions, in this order, one call each, at temperature 0: Can I cancel my ticket on WhatsApp? then How early should I reach the boarding point? then Is GST included in the fare?
Every call sends the same system message first (an instruction to answer only from the policy, then the whole of /workspace/data/policy.md) and the question as the user message.
After each call print its prompt tokens and cached tokens from usage, as: 1290 prompt tokens, 1024 cached
Then print the three calls' cost together and their cost with no cache, both with four decimals, as: Cost: ₹0.1234, without the cache ₹0.5678
Prices in rupees per million tokens: 200 for input, 50 for cached input, 800 for output.
- Task 2
response-cache.ts: a response cache
Export cacheKey(request) and cachedReply(request, cache). A request is the object you pass to client.chat.completions.create (type ChatCompletionCreateParamsNonStreaming); cache is a Map<string, string> the caller keeps.
cacheKey returns a string: the same for the same request, even with its keys in another order (at the top or inside a message), and different when anything in it changes.
cachedReply returns a promise of the reply's text. At temperature 0 it returns the stored text when the cache holds the request's key, and otherwise calls lab-chat with the request and stores the text. At any other temperature, or with none, it always calls the model and stores nothing.
The check calls both with requests of its own and counts the calls the model receives.
- Task 3
ttl-cache.ts: answers that expire
Export class TTLCache<V>, made with new TTLCache(ttl, clock = () => performance.now()), both in milliseconds: put(key, value) stores the value with the time clock() gives; get(key) returns it, or undefined when the key is missing or was stored more than ttl milliseconds ago.
Then seatsReply(tripId, cache): a promise of the answer to How many seats are left on this bus? for that trip of /workspace/data/trips.json. It returns the cached reply for the trip id if there is one; otherwise it asks lab-chat at temperature 0 with a system message holding the trip's data (departure time, bus, from, to, date, seats left and fare), stores the reply, and returns it.
The check passes its own clock and moves time forward: a value 59 seconds old with a ttl of 60,000 is still there, 61 seconds old it is gone.
- Task 4 · interview question
Interview: fix-prefix.ts, why is nothing cached?
fix-prefix.ts sends the same policy three times and prints each call's cached tokens, but they are always 0. Change ask so that the second and third calls get cached tokens.
Every request must still send the time now in the same format (fresh for each call, to the millisecond), the instruction and the whole policy. Keep the questions and what the program prints.
What you will learn
- Prompt caching: the provider reuses the work on the start of a prompt it saw recently, and charges less for it.
- Stable first, changing last: why one timestamp in the wrong place turns the cache off.
- A response cache of your own: a Map keyed by the whole request, and why only some requests may use it.
- Expiry: a time to live for answers built on live data, with a clock you pass in.
More in Building with LLMs
Build "Caching" yourself
SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant, in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Building with LLMs (TypeScript) are free. This one comes with the all labs plan.