RAG in Python · Chapter 4 of 37
Measure before you improve
By the end of this chapter you can measure a search with numbers: hit@k, recall@k, precision@k and MRR on Paisa Setu's labelled questions, in a report by kind of question, and you know the BM25 baseline's numbers. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Why RAG
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.
Why you need this
The rest of this lab makes search smarter: embeddings, hybrid search, reranking, rewritten queries. Each one costs something (milliseconds, rupees, complexity), and each one will look better on the two questions you try by hand. Arjun will not ship on that: "Better by how much, on which questions?"
Trying questions by hand cannot answer him. You remember the hits and forget the misses, and a change that fixes paraphrases can quietly break exact codes. Kabir's team did the slow part: 80 real merchant questions, each labelled with the documents that answer it. With those, "better" becomes a number you can compare before and after.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
metrics.py: ranked_docs, from chunks to documents
A search returns chunk ids, best first: the document id, a #, and the chunk's number, like help-refund-timelines#4. Write ranked_docs(chunk_ids) in metrics.py: return the document ids, best first, each only once, where it first appears.
ranked_docs(["a-doc#3", "b-doc#0", "a-doc#1", "c-doc#2"]) is ["a-doc", "b-doc", "c-doc"], and no chunks give [].
- Task 2
metrics.py: hit_at and recall_at, did we find it and how much
Write hit_at(docs, relevant, k) and recall_at(docs, relevant, k) in metrics.py. docs are ranked document ids, relevant the ids that answer the question; look only at the first k docs. hit_at is 1 when any relevant document is in the first k, otherwise 0. recall_at is how many of the relevant documents are in the first k, divided by how many relevant documents there are.
With docs ["d1", "d2", "d3", "d4", "d5", "d6"] and relevant ["d2", "d6", "d9"]: hit_at with k 1 is 0 and with k 2 is 1; recall_at with k 5 is 1/3 and with k 6 is 2/3. A k bigger than the list looks at all of it.
With no relevant documents recall is 0 out of 0, which is neither 0 nor 1: recall_at must raise ValueError (not ZeroDivisionError).
- Task 3
metrics.py: precision_at and reciprocal_rank, how much was right and how high
Write precision_at(docs, relevant, k): how many of the first k docs are relevant, divided by k (always k, even when fewer docs came back). And reciprocal_rank(docs, relevant): 1 divided by the position of the first relevant document in all of docs (1 for first place, 1/2 for second), or 0.0 when none is there.
With the docs and relevant of task 2, precision_at with k 5 is 0.2 and reciprocal_rank is 0.5; precision_at(["d2"], relevant, 5) is 0.2.
- Task 4 · interview question
Interview: metrics.py, evaluate by kind
Write evaluate(questions, retrieve, k) in metrics.py: for every question whose relevant list is not empty, call retrieve(question.question), turn the chunk ids into ranked_docs, and score hit_at, recall_at and precision_at at k, and reciprocal_rank. Do not call retrieve for the unanswerable questions: they cannot have a recall.
Return one Row per kind in the order of KINDS, skipping kinds with no questions, then a Row with kind "all" for every question you scored. A Row has kind, questions (how many), and the means hit, recall, precision and mrr. The "all" row is the mean over its questions, not over the rows.
For the 80 labelled questions and the given BM25 search at k 5 there are 7 rows; "all" has 70 questions, recall 0.683 and MRR 0.751, and paraphrase recall is 0.469.
Arjun will ask: a colleague reports recall 0.723 for the same search. What did they do with the 10 questions that have no answer?
What you will learn
- A labelled set: questions with the documents that answer them, and a kind for each.
- hit@k and recall@k: did the first k documents contain an answer, and how much of it.
- precision@k and MRR: how much of what came back was right, and how high the first right one was.
- A report by kind, with the questions that have no answer left out of recall.
More in Why RAG
Build "Measure before you improve" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.