Building with LLMs · Chapter 21 of 24

Caching

By the end of this chapter you can read how much of a prompt the provider served from its cache, order a request so the cache works, keep your own cache of replies, and say when a cached answer is safe to give and when it must expire. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Building with LLMs
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.

Why you need this

Harpreet has the month's bill open. "Every WhatsApp question sends the whole policy. The question is ten tokens and the policy is twelve hundred, so we pay for the policy thousands of times a day." Lata adds the other half: "And half of the questions are the same five questions."

Everything so far sends the full request and pays for every token of it, every time. That is correct, and it is also slow and expensive when most of the request repeats. There are two caches to use, one at the provider and one in your own code, and each has rules about when it is safe.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    cache_hits.py: see the cache at work

    Harpreet wants to see what prompt caching saves. Ask lab-chat three questions, in this order, one call each, at temperature 0: Can I cancel my ticket on WhatsApp? then How early should I reach the boarding point? then Is GST included in the fare?

    Every call sends the same system message first (an instruction to answer only from the policy, then the whole of /workspace/data/policy.md) and the question as the user message.

    After each call print its prompt tokens and cached tokens from usage, as: 1290 prompt tokens, 1024 cached

    Then print the three calls' cost together and their cost with no cache, both with four decimals, as: Cost: ₹0.1234, without the cache ₹0.5678

    Prices in rupees per million tokens: 200 for input, 50 for cached input, 800 for output.

  2. Task 2

    response_cache.py: a response cache

    Write cache_key(request) and cached_reply(request, cache). A request is a dict of the arguments for chat.completions.create (model, messages, temperature and so on); cache is a dict the caller keeps.

    cache_key returns a string: the same for the same request, even with its keys in another order, and different when anything in it changes.

    cached_reply returns the reply's text. At temperature 0 it returns the stored text when the cache holds the request's key, and otherwise calls lab-chat with the request and stores the text. At any other temperature, or with none, it always calls the model and stores nothing.

    The check calls both with requests of its own and counts the calls the model receives.

  3. Task 3

    ttl_cache.py: answers that expire

    Write class TTLCache, made with TTLCache(ttl, clock=time.monotonic): put(key, value) stores the value with the time clock() gives; get(key) returns it, or None when the key is missing or was stored more than ttl seconds ago.

    Then seats_reply(trip_id, cache): the answer to How many seats are left on this bus? for that trip of /workspace/data/trips.json. It returns the cached reply for the trip id if there is one; otherwise it asks lab-chat at temperature 0 with a system message holding the trip's data (departure time, bus, from, to, date, seats left and fare), stores the reply, and returns it.

    The check passes its own clock and moves time forward: a value 59 seconds old with a ttl of 60 is still there, 61 seconds old it is gone.

  4. Task 4 · interview question

    Interview: fix_prefix.py, why is nothing cached?

    fix_prefix.py sends the same policy three times and prints each call's cached tokens, but they are always 0. Change ask so that the second and third calls get cached tokens.

    Every request must still send the time now in the same format (fresh for each call, to the microsecond), the instruction and the whole policy. Keep the questions and what the program prints.

What you will learn

  • Prompt caching: the provider reuses the work on the start of a prompt it saw recently, and charges less for it.
  • Stable first, changing last: why one timestamp in the wrong place turns the cache off.
  • A response cache of your own: a key for a whole request, and why only some requests may use it.
  • Expiry: a time to live for answers built on live data.

More in Building with LLMs

Build "Caching" yourself

SafarLine, an intercity bus company, drowns in WhatsApp messages. You build its first AI assistant. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Building with LLMs are free. This one comes with the all labs plan.