RAG in Python · Chapter 31 of 37
Answer faster, pay less
By the end of this chapter you can cache embeddings by the hash of their text, answer repeated questions from a semantic cache with a threshold you measured, and forget cached answers when a document changes. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.
Why you need this
Of Kabir's 600 questions a day, many are the same few asked in different words: when does my money arrive, what does the Scale plan cost. Every one of them runs the whole pipeline: an embedding request, a vector search, and a chat request with about 500 tokens of documents in it. Arjun wants the bill and the waiting time down before the assistant goes to every merchant.
A cache sounds like one decorator. It is not: a cache that answers the Growth plan question with the Scale plan's price, or keeps quoting last week's fee after the policy changed, is worse than no cache at all. Nothing you have built so far knows when two questions are "the same", or which answers a document edit makes wrong.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
caching.py: embed_cached, embed each text once
Arjun wants the bill down. Write cache_key(text): the SHA-256 hex digest of EMBED_MODEL, a new line, then the text. Write embed_cached(texts): one vector per text, in the order of texts. Texts whose key is in the embeddings dict come from it; the others are embedded with EMBED_MODEL, each distinct text once, in requests of at most 256 inputs, and stored in embeddings under their key.
embed_cached([a, b, a]) sends one request with 2 inputs and returns 3 vectors (the first and third the same); embed_cached([b, a]) afterwards sends no request at all. 300 texts with 3 already cached send 297 inputs in requests of 256 or fewer.
- Task 2
caching.py: choose_threshold, a threshold from measured similarities
Write choose_threshold(same, different): return Threshold(threshold, hit_rate). threshold is the smallest two-decimal value strictly above every different pair: the highest different similarity rounded down to two decimals, plus 0.01. hit_rate is the share of same pairs at or above it.
choose_threshold([0.95, 0.9, 0.8], [0.7, 0.853]) is Threshold(threshold=0.86, hit_rate=2/3); with a highest different pair of 0.88 the threshold is 0.89, and with 0.57 it is 0.58 (0.57 * 100 is 56.99999999999999 in floating point). Then set THRESHOLD to what it gives for MEASURED, the lab's measured cosines.
- Task 3
caching.py: cached_answer, answer repeated questions from the cache
Write cached_answer(conn, question, table="chunks"): embed the question with embed_cached. Among the entries of answers whose question has the same codes_of as this one, find the highest cosine with this question's vector. When it is at least THRESHOLD, return Answer(answer, cached=True, chunk_ids) of that entry, with no search and no chat request.
Otherwise retrieve 5 chunks with the vector from table, generate the answer, append an Entry (the question, its vector, the answer, and chunk_ids, the ids of the chunks in order) to answers, and return it with cached=False. "how much is the Scale plan per month?" after "what is the monthly fee of the Scale plan?" is cached (cosine 0.902); the Growth plan question is not (0.793); "what does a settlement hold SH-02 mean?" after the SH-04 one is not, because the codes differ, although their cosine is 0.926.
- Task 4 · interview question
Interview: caching.py, document_changed, forget what changed
Write document_changed(doc_id): a document was edited, so remove every entry of answers whose chunk_ids has a chunk of that document, and return how many were removed. Chunk ids are "<doc id>#<n>", so match the document id followed by "#": document_changed("help-plans") removes nothing, although help-plans-overview starts with those letters. Keep every other entry in order, in the same answers list (other code holds it).
After the Scale plan question, document_changed("help-plans-overview") is 1, the next Scale plan question goes to the model again, and the SH-04 answer, which used other documents, stays cached. Arjun will ask how long merchants see the old fee after it changes at 3 pm.
What you will learn
- Content hashes as keys: the same text is embedded once, a changed text is embedded again, automatically.
- A semantic cache: reuse an answer when a new question's vector is close enough to an old one.
- Choosing the threshold by measuring, and why exact codes need their own check.
- Invalidation: cached answers remember the chunks they used.
More in Production
Build "Answer faster, pay less" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.