RAG: search that answers · Chapter 31 of 37
Answer faster, pay less
By the end of this chapter you can cache embeddings by the hash of their text, answer repeated questions from a semantic cache with a threshold you measured, and forget cached answers when a document changes. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 4 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
Of Kabir's 600 questions a day, many are the same few asked in different words: when does my money arrive, what does the Scale plan cost. Every one of them runs the whole pipeline: an embedding request, a vector search, and a chat request with about 500 tokens of documents in it. Arjun wants the bill and the waiting time down before the assistant goes to every merchant.
A cache sounds like one line of code. It is not: a cache that answers the Growth plan question with the Scale plan's price, or keeps quoting last week's fee after the policy changed, is worse than no cache at all. Nothing you have built so far knows when two questions are "the same", or which answers a document edit makes wrong.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
embedCached: embed each text once
cacheKey(text) is the SHA-256 hex digest of EMBED_MODEL, a new line, then the text. embedCached(texts) returns one vector per text, in the order of texts: texts whose key is in the embeddings Map come from it; the others are embedded with EMBED_MODEL, each distinct text once, in requests of at most 256 inputs, and stored in embeddings under their key.
embedCached([a, b, a]) sends one request with 2 inputs and returns 3 vectors (the first and third the same); embedCached([b, a]) afterwards sends no request at all. 300 texts with 3 already cached send 297 inputs in requests of 256 or fewer.
- Task 2
chooseThreshold: a threshold from measured similarities
chooseThreshold(same, different) returns { threshold, hitRate }. threshold is the smallest two-decimal value strictly above every different pair: the highest different similarity rounded down to two decimals, plus 0.01. hitRate is the share of same pairs at or above it.
chooseThreshold([0.95, 0.9, 0.8], [0.7, 0.853]) is { threshold: 0.86, hitRate: 2/3 }, and with a highest different pair of 0.88 the threshold is 0.89. Then set THRESHOLD to what it gives for MEASURED, the lab's real cosines: 0.8, catching 9 of the 12 paraphrases.
- Task 3
cachedAnswer: answer repeated questions from the cache
Embed the question with embedCached. Among the entries of answers whose question has the same codesOf as this one, find the highest cosine with this question's vector. When it is at least THRESHOLD, return { answer, cached: true, chunkIds } of that entry, with no search and no chat request.
Otherwise retrieve 5 chunks with the vector, generate the answer, push an Entry (the question, its vector, the answer, and chunkIds, the ids of the chunks in order) onto answers, and return it with cached false. 'how much is the Scale plan per month?' after 'what is the monthly fee of the Scale plan?' is cached (cosine 0.882); the Growth plan question is not (0.772); 'what does a settlement hold SH-02 mean?' after the SH-04 one is not, because the codes differ, although their cosine is 0.939.
- Task 4 · interview question
Interview: documentChanged, forget what changed
A document was edited: remove every entry of answers whose chunkIds has a chunk of that document, and return how many were removed. Chunk ids are '<doc id>#<n>', so match the document id followed by '#': documentChanged('help-plans') removes nothing, although help-plans-overview starts with those letters. Keep every other entry in order, in the same answers array.
After the Scale plan question, documentChanged('help-plans-overview') is 1, the next Scale plan question goes to the model again, and the SH-04 answer, which used other documents, stays cached. Arjun will ask how long merchants see the old fee after it changes at 3 pm.
What you will learn
- Content hashes as keys: the same text is embedded once, a changed text is embedded again, automatically.
- A semantic cache: reuse an answer when a new question's vector is close enough to an old one.
- Choosing the threshold by measuring, and why exact codes need their own check.
- Invalidation: cached answers remember the chunks they used.
More in Production
Build "Answer faster, pay less" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.