RAG in Python · Chapter 36 of 37

The design review

By the end of this chapter you can answer a design review's trade-off questions (chunk size, k, hybrid or vector, rerank or not, cost) with numbers from small experiments, read those numbers honestly, and pick a configuration that fits a budget. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The interview
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Before the assistant takes 600 questions a day, Arjun holds a design review. His questions are the classic ones: why five chunks, why chunks this size, why not hybrid search, why not a reranker, and what does it cost. Everyone in the room has an opinion. Kabir read that rerankers always help. Someone else wants whole documents in the prompt "so nothing is missed".

Each chapter so far measured one idea on its own. A design review needs them side by side: the same questions, the same measure, one knob changed at a time, with the time and the rupees next to the recall. And it needs a way to decide when the numbers disagree.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    review.py: sweep_k, why k = 5?

    Arjun's first question: why five chunks? Write sweep_k(conn, ks, table="chunks") in review.py: one KRow(k, recall, tokens, rupees_per_day) per k, in the order given, over the labelled questions (labelled()) with vector_search. recall is the average recall_of of each question's top k, rounded to 3 decimals; tokens is the average total tokens of the top k chunks, rounded to a whole number; rupees_per_day is rupees_per_day(tokens), rounded to 2 decimals.

    Search once per question, for the biggest k, and slice that list for each k. sweep_k(conn, [1, 5, 10]) gives recall 0.321, 0.69 and 0.814, tokens 94, 500 and 992, and rupees_per_day 1.41, 7.5 and 14.88.

  2. Task 2

    review.py: windows and chunk_sizes, bigger chunks?

    Write windows(chunks, n): cut each document (the chunks of one doc_id, in their order) into groups of n neighbouring chunks, the last one perhaps shorter. Each group is a Unit: id is the doc_id, "#w" and its number in the document (help-refund-timelines#w0 to #w3 for its 7 chunks with n 2), text is the chunks' texts joined by a blank line, tokens is their sum, and doc_id, audience and title are the document's. windows(read_chunks(), 2) has 472 windows, and n 1000 gives the 145 whole documents.

    Write chunk_sizes(conn, sizes, table="units36"): one SizeRow(n, units, recall, tokens) per n: load the windows with load_units(conn, units, table) and search them with keyword_search (top 5, in the table load_units returns) for every labelled question; recall and tokens as in task 1, units is how many windows there are. chunk_sizes(conn, [1, 4]) gives units 866 and 275, recall 0.604 and 0.613, tokens 904 and 2492.

  3. Task 3

    review.py: fuse and compare_retrievers, hybrid or vector?

    Write fuse(lists, k), reciprocal rank fusion: a row at rank r (1 for the first) in a list adds 1 / (60 + r) to its score, summed over the lists it is in (rows are the same when their ids are); return the k rows with the best scores, and on a tie the row seen first stays first. fuse([[a, b, c], [c, d, a]], 3) is [a, c, b].

    Write compare_retrievers(conn, table="chunks"): for every labelled question take vector_search top 20 and keyword_search top 20. vector is the vector top 5, keyword the keyword top 5, and hybrid fuse of the two lists of 20 with k 5. For each, return ByKind(all, keyword, paraphrase): the average recall over all the questions, over kind "keyword" and over kind "paraphrase", rounded to 3 decimals, in a Compared. vector is ByKind(all=0.69, keyword=0.625, paraphrase=0.615) and hybrid ByKind(all=0.69, keyword=0.76, paraphrase=0.547).

  4. Task 4

    review.py: rerank_or_not, rerank or not?

    Kabir read that rerankers always help. Write rerank_or_not(conn, qs, n, table="chunks"): for each question in qs, one at a time, plain is vector_search top 5, and reranked is vector_search top n, then rerank, keeping the first 5. Time each with time.perf_counter in milliseconds: plain is the search, reranked is the search and the rerank. Return RerankOrNot(plain=Timed(recall, ms), reranked=Timed(recall, ms)): the average recall (3 decimals) and the average ms (1 decimal).

    On the first 8 paraphrase questions with n 10, plain.recall is 0.521 and reranked.recall 0.708, and reranked.ms is far above plain.ms, because the reranker reads every chunk with the question. One rerank request per question.

  5. Task 5 · interview question

    Interview: review.py, pick_config, pick a configuration

    Write pick_config(options, budget, tie=0.01): keep the options that fit, ms at most budget.max_ms and rupees_per_day(tokens) at most budget.max_rupees_per_day. Of those, every option whose recall is within tie of the best recall counts as just as good; return the one with the fewest tokens, then the fewest ms. Return None when nothing fits.

    With the five measured options (vector k=5 0.69, 1.3 ms, 500 tokens; vector k=10 0.814, 1.3 ms, 992; hybrid k=5 0.69, 2.0 ms, 638; hybrid k=10 0.852, 2.0 ms, 1337; rerank 10 to 5 0.682, 137.8 ms, 527) and Budget(max_ms=200, max_rupees_per_day=10), the answer is vector k=5: hybrid k=5 adds no recall for more tokens. With ₹25 a day it is hybrid k=10. Arjun will ask: 600 questions a day, 200 ms and ₹10 a day for retrieval, so which configuration, and why?

What you will learn

  • An experiment: one knob, the labelled questions, a number.
  • Reading a difference: when a gain is smaller than one question.
  • Recall against cost: tokens, milliseconds and rupees for each gain.
  • Choosing under a budget: a function that picks, so the decision is repeatable.

More in The interview

Build The design review yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.