RAG in Python · Chapter 8 of 37
Where each one wins
By the end of this chapter you can measure keyword search and vector search on Paisa Setu's labelled questions, kind by kind, explain where each one wins and why, and show with numbers why the answer to "which one do we ship?" is both. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Embeddings
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.
Why you need this
Kabir has two complaints on the same morning. The old keyword search found nothing useful for "I have to pay about 40 delivery boys on the same day, any faster way than one by one?" (the bulk payouts article never says "delivery boys"). The new vector search sends "can I use Paisa Setu for my online betting app?" to the wrong place, when the prohibited businesses list says "betting" in plain words.
Arjun has heard that embeddings "understand meaning" and wants to delete the keyword search. One example on each side proves nothing. What you need is both searches run on the same labelled questions, scored the same way, split by the kind of question, so the decision rests on measurements and not on the last complaint.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
compare.py: first_relevant, the rank of the first relevant chunk
Write first_relevant(results, relevant) in compare.py: the position, counting from 1, of the first result whose doc_id is one of the relevant document ids, or None when no result is. Any relevant document counts, wherever it is in relevant.
With results from help-fee-table, help-refund-timelines and policy-refund-policy, first_relevant(results, ["help-refund-timelines"]) is 2, first_relevant(results, ["help-two-factor-login"]) is None, and no results at all is None. Compare the doc_id, not the chunk id: ["help-refund-timelines#0"] matches nothing.
- Task 2
compare.py: score_by_kind, a score table by kind of question
Write score_by_kind(questions, search): for every question, call search(question.question, 5) and find its rank with first_relevant. Return a dict with one KindScore per kind and one for "all": questions (how many), top1 (found at rank 1) and top5 (found anywhere in the top 5).
Skip questions whose relevant list is empty (the 10 unanswerable ones): there is no "unanswerable" row, and "all" counts 70 questions. score_by_kind(read_questions(), vector_search)["all"] is KindScore(questions=70, top1=43, top5=64). The check also tries a small search of its own, where a question with no results is not found.
- Task 3
compare.py: where_they_differ, the questions only one search finds
Write where_they_differ(questions): for every answerable question (relevant not empty), check whether keyword_search's top 5 and vector_search's top 5 find a relevant document. Return Differ(keyword_only=..., vector_only=...): the ids of the questions found by keyword search and not vector search, and the other way round, in question order.
q62 (online betting app) is in keyword_only and q70 (40 delivery boys) is in vector_only.
- Task 4 · interview question
Interview: compare.py, both_search
Write both_search(question, k=5): keyword_search's top k, in its order, then vector_search's top k, in its order, leaving out the chunks keyword search already gave (the same chunk id). Up to 2k results, no chunk twice.
Over the 70 answerable questions, both_search(question, 5) finds a relevant document for 69, with fewer than 10 chunks a question on average: more than either search's own top 10 (67 and 68).
Arjun will ask: keyword search or vector search, which one do we ship?
What you will learn
- The rank of the first relevant chunk, and what "found" means.
- Scoring by kind: exact codes, paraphrases, plain questions, traps.
- Why each one wins: rare exact words against shared meaning.
- Disagreement: equal totals can hide different misses.
More in Embeddings
Build "Where each one wins" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.