RAG: search that answers · Chapter 4 of 37

Measure before you improve

By the end of this chapter you can measure a search with numbers: hit@k, recall@k, precision@k and MRR on Paisa Setu's labelled questions, in a report by kind of question, and you know the BM25 baseline's numbers. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Why RAG
  • 4 tasks
  • In the all labs plan

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.

Why you need this

The rest of this lab makes search smarter: embeddings, hybrid search, reranking, rewritten queries. Each one costs something (milliseconds, rupees, complexity), and each one will look better on the two questions you try by hand. Arjun will not ship on that: "Better by how much, on which questions?"

Trying questions by hand cannot answer him. You remember the hits and forget the misses, and a change that fixes paraphrases can quietly break exact codes. Kabir's team did the slow part: 80 real merchant questions, each labelled with the documents that answer it. With those, "better" becomes a number you can compare before and after.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    rankedDocs: from chunks to documents

    A search returns chunk ids, best first: the document id, a #, and the chunk's number, like help-refund-timelines#4. Return the document ids, best first, each only once, where it first appears.

    rankedDocs(['a-doc#3', 'b-doc#0', 'a-doc#1', 'c-doc#2']) is ['a-doc', 'b-doc', 'c-doc'], and no chunks give [].

  2. Task 2

    hitAt and recallAt: did we find it, and how much of it

    docs are ranked document ids, relevant the ids that answer the question; look only at the first k docs. hitAt is 1 when any relevant document is in the first k, otherwise 0. recallAt is how many of the relevant documents are in the first k, divided by how many relevant documents there are.

    With docs ['d1', 'd2', 'd3', 'd4', 'd5', 'd6'] and relevant ['d2', 'd6', 'd9']: hitAt with k 1 is 0 and with k 2 is 1; recallAt with k 5 is 1/3 and with k 6 is 2/3. A k bigger than the list looks at all of it.

    With no relevant documents recall is 0 out of 0, which is neither 0 nor 1: recallAt must throw an Error.

  3. Task 3

    precisionAt and reciprocalRank: how much was right, and how high

    precisionAt is how many of the first k docs are relevant, divided by k (always k, even when fewer docs came back). reciprocalRank is 1 divided by the position of the first relevant document in all of docs (1 for first place, 1/2 for second), or 0 when none is there.

    With the docs and relevant of task 2, precisionAt with k 5 is 0.2 and reciprocalRank is 0.5; precisionAt(['d2'], relevant, 5) is 0.2.

  4. Task 4 · interview question

    Interview: evaluate, the report by kind

    For every question whose relevant list is not empty, call retrieve(question.question), turn the chunk ids into rankedDocs, and score hitAt, recallAt and precisionAt at k, and reciprocalRank. Do not call retrieve for the unanswerable questions: they cannot have a recall.

    Return one Row per kind in the order of KINDS, skipping kinds with no questions, then a row with kind 'all' for every question you scored. A row has kind, questions (how many), and the means hit, recall, precision and mrr. The 'all' row is the mean over its questions, not over the rows.

    For the 80 labelled questions and the given BM25 search at k 5 there are 7 rows; 'all' has 70 questions, recall 0.683 and MRR 0.751, and paraphrase recall is 0.469.

    Arjun will ask: a colleague reports recall 0.723 for the same search. What did they do with the 10 questions that have no answer?

What you will learn

  • A labelled set: questions with the documents that answer them, and a kind for each.
  • hit@k and recall@k: did the first k documents contain an answer, and how much of it.
  • precision@k and MRR: how much of what came back was right, and how high the first right one was.
  • A report by kind, with the questions that have no answer left out of recall.

More in Why RAG

Build "Measure before you improve" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.