RAG: search that answers · Chapter 36 of 37
The design review
By the end of this chapter you can answer a design review's trade-off questions (chunk size, k, hybrid or vector, rerank or not, cost) with numbers from small experiments, read those numbers honestly, and pick a configuration that fits a budget. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The interview
- 5 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
Before the assistant takes 600 questions a day, Arjun holds a design review. His questions are the classic ones: why five chunks, why chunks this size, why not hybrid search, why not a reranker, and what does it cost. Everyone in the room has an opinion. Kabir read that rerankers always help. Someone else wants whole documents in the prompt "so nothing is missed".
Each chapter so far measured one idea on its own. A design review needs them side by side: the same questions, the same measure, one knob changed at a time, with the time and the rupees next to the recall. And it needs a way to decide when the numbers disagree.
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
sweepK: why k = 5?
sweepK(ks) returns one row per k, in the order given: { k, recall, tokens, rupeesPerDay }. Over the labelled questions (labelled()), with vectorSearch: recall is the average recallOf of each question's top k, rounded to 3 decimals; tokens is the average total tokens of the top k chunks, rounded to a whole number; rupeesPerDay is rupeesPerDay(tokens), rounded to 2 decimals (round gives you both roundings).
Search once per question, for the biggest k, and slice for each k. sweepK([1, 5, 10]) gives recall 0.321, 0.69 and 0.814, tokens 94, 500 and 992, and rupeesPerDay 1.41, 7.5 and 14.88.
- Task 2
windows and chunkSizes: bigger chunks?
windows(chunks, n) cuts each document (the chunks of one doc_id, in their order) into groups of n neighbouring chunks, the last one perhaps shorter. Each window is a Unit: id is the doc_id, '#w' and its number in the document (help-refund-timelines#w0 to #w3 for its 7 chunks with n 2), text is the chunks' texts joined by a blank line, tokens is their sum, and doc_id, audience and title are the document's. windows(readChunks(), 2) has 472 windows, and n 1000 gives the 145 whole documents.
chunkSizes(sizes) returns { n, units, recall, tokens } for each n: load the windows with loadUnits and search them with keywordSearch (top 5, in the table loadUnits returns) for every labelled question; recall and tokens as in task 1. chunkSizes([1, 4]) gives units 866 and 275, recall 0.604 and 0.606, tokens 923 and 2494.
- Task 3
fuse and compareRetrievers: hybrid or vector?
fuse(lists, k) is reciprocal rank fusion: a row at rank r (1 for the first) in a list adds 1 / (60 + r) to its score, summed over the lists; return the k rows with the best scores, and on a tie the row seen first stays first. fuse([[a, b, c], [c, d, a]], 3) is [a, c, b].
compareRetrievers() takes, for every labelled question, vectorSearch top 20 and keywordSearch top 20. vector is the vector top 5, keyword the keyword top 5, and hybrid fuse of the two lists with k 5. For each, return { all, keyword, paraphrase }: the average recall over all the questions, over kind 'keyword' and over kind 'paraphrase', rounded to 3 decimals. vector is { all: 0.69, keyword: 0.625, paraphrase: 0.615 } and hybrid { all: 0.693, keyword: 0.781, paraphrase: 0.547 }.
- Task 4
rerankOrNot: rerank or not?
For each question in qs, one at a time: without is vectorSearch top 5; with is vectorSearch top n, then rerank, keeping the first 5. Time each with performance.now(): without is the search, with is the search and the rerank. Return { without: { recall, ms }, with: { recall, ms } }: the average recall (3 decimals) and the average ms (1 decimal).
On the first 8 paraphrase questions with n 10, without.recall is 0.521 and with.recall 0.708, and with.ms is far above without.ms, because the reranker reads every chunk with the question.
- Task 5 · interview question
Interview: pickConfig, pick a configuration
pickConfig(options, budget, tie = 0.01) keeps the options that fit: ms at most budget.maxMs and rupeesPerDay(tokens) at most budget.maxRupeesPerDay. Of those, every option whose recall is within tie of the best recall counts as just as good; return the one with the fewest tokens, then the fewest ms. Return null when nothing fits.
With the five measured options (vector k=5 0.69, vector k=10 0.814, hybrid k=5 0.693, hybrid k=10 0.843, rerank 10 to 5 0.682) and { maxMs: 200, maxRupeesPerDay: 10 }, the answer is vector k=5: hybrid k=5's 0.003 more is less than one question. With ₹25 a day it is hybrid k=10. With tie 0 the first budget picks hybrid k=5. Arjun will ask: 600 questions a day, 200 ms and ₹10 a day for retrieval, so which configuration, and why?
What you will learn
- An experiment: one knob, the labelled questions, a number.
- Reading a difference: when a gain is smaller than one question.
- Recall against cost: tokens, milliseconds and rupees for each gain.
- Choosing under a budget: a function that picks, so the decision is repeatable.
More in The interview
Build The design review yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.