RAG: search that answers · Chapter 8 of 37

Where each one wins

By the end of this chapter you can measure keyword search and vector search on Paisa Setu's labelled questions, kind by kind, explain where each one wins and why, and show with numbers why the answer to "which one do we ship?" is both. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Embeddings
  • 4 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

Kabir has two complaints on the same morning. The old keyword search could not find anything for "I have to pay about 40 delivery boys on the same day, any faster way than one by one?" (the bulk payouts article never says "delivery boys"). The new vector search sends "can I use Paisa Setu for my online betting app?" to the wrong place, when the prohibited businesses list says "betting" in plain words.

Arjun has heard that embeddings "understand meaning" and wants to delete the keyword search. One example on each side proves nothing. What you need is both searches run on the same labelled questions, scored the same way, split by the kind of question, so the decision rests on measurements and not on the last complaint.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    firstRelevant: the rank of the first relevant chunk

    Return the position, counting from 1, of the first result whose doc_id is one of the relevant document ids, or null when no result is. Any relevant document counts, whatever its place in relevant.

    With results from help-fee-table, help-refund-timelines and policy-refund-policy, firstRelevant(results, ['help-refund-timelines']) is 2, firstRelevant(results, ['help-two-factor-login']) is null, and no results at all is null. Compare doc_id, not the chunk id.

  2. Task 2

    scoreByKind: a score table by kind of question

    For every question, call search(question.question, 5) and find its first relevant rank with firstRelevant. Return a table with one row per kind and an 'all' row, each { questions, top1, top5 }: how many questions, how many found at rank 1, how many found anywhere in the top 5.

    Skip questions whose relevant list is empty (the 10 unanswerable ones): there must be no 'unanswerable' row, and 'all' has 70 questions. scoreByKind(readQuestions(), vectorSearch) has all { questions: 70, top1: 43, top5: 64 } and paraphrase { questions: 16, top1: 8, top5: 13 }. The Check also tries a small search of its own, where a question with no results is not found.

  3. Task 3

    whereTheyDiffer: the questions only one search finds

    For every answerable question (relevant not empty), check whether keywordSearch's top 5 and vectorSearch's top 5 find a relevant document. Return { keywordOnly, vectorOnly }: the ids of the questions found by keyword search and not vector search, and the other way round, in question order.

    q62 (online betting app) is in keywordOnly and q70 (40 delivery boys) is in vectorOnly.

  4. Task 4 · interview question

    Interview: bothSearch, ship both

    Return keywordSearch's top k, in its order, then vectorSearch's top k, in its order, leaving out the chunks keyword search already gave (the same chunk id): up to 2k results, no chunk twice.

    Over the 70 answerable questions, bothSearch(question, 5) finds a relevant document for 69, with fewer than 10 chunks per question on average: more than either search's own top 10 (67 and 68).

    Arjun will ask: keyword search or vector search, which one do we ship?

What you will learn

  • The rank of the first relevant chunk, and what "found" means.
  • Scoring by kind: exact codes, paraphrases, plain questions, traps.
  • Why each one wins: rare exact words against shared meaning.
  • Disagreement: equal totals can hide different misses.

More in Embeddings

Build "Where each one wins" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.