RAG: search that answers · Chapter 34 of 37
Is the answer right
By the end of this chapter you can measure the answers, not only the retrieval: faithfulness with a judge, a cheap check with no model, answer relevance, a report over the labelled questions, and a gate that stops a change that makes answers worse. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 5 tasks
- In the all labs plan
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.
Why you need this
Kabir wants a shorter system prompt: the new one is 19 tokens instead of 38, and at 600 questions a day every token counts. Does it change the answers? Neha asks the question compliance always asks: is every claim in an answer really in the chunk it cites?
Recall@k cannot answer either of them. It says the right chunks were found; it says nothing about what the model did with them. You could read 80 answers by hand after every change, once. Nobody does it the second time, and the change that breaks the answers is always the small one nobody thought to check.
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
judge: a judge for faithfulness
Send one request to lab-chat with response_format { type: 'json_object' }. The system message gives the job in words: judge (or grade, or evaluate) whether the answer is faithful to (supported by) the context, and reply in JSON. The user message is <context>, every chunk as [id] text, </context>, then <answer> the answer </answer>.
The model replies { "faithful": bool, "unsupported": [sentences] }: parse it and return { faithful, unsupported }. With help-chargeback-ratio#0 as the context, 'Only card transactions count. UPI, net banking and wallet payments are not part of the ratio.' is { faithful: true, unsupported: [] }, and 'Refunds are paid in cash at our Pune office.' comes back as unsupported. Use the tags, not "Context:" labels: the chunk has a line "Example: ..." that would end a labelled context early.
- Task 2
quickCheck: a cheap check, with no model
Return { ok, unknownCitations, missingNumbers }. unknownCitations are the ids the answer cites (citedIds) that are not among the hits. missingNumbers are the numbers in the answer, with its ids taken out (withoutIds), that appear in none of the cited chunks, once each. ok is true when both lists are empty.
A number is the whole thing: 2,000 and 0.15 are one number each, and the 30 in U30 is not a number. 'Card refunds take 7 to 10 working days. [help-refund-timelines#1]' has missingNumbers ['10']. 'Instant refunds cost ₹5 per refund plus GST. [help-refund-timelines#4]' is ok: the 4 in the id is not a claim. An answer citing help-refund-timelines#9, which is not a hit, has it in unknownCitations, and its numbers are missing because no real chunk was cited.
- Task 3
relevance: does the answer answer the question?
Score the answer for the question with one rerankScores call: the question as the query, and the answer without its ids (withoutIds) as the only document. Return that one score.
An answer about U30 for 'what does UPI error U30 mean?' scores above 0.9; an answer about card refunds for the same question scores below 0.05.
- Task 4
evaluate: a report on the labelled questions
Run answerFn on every question and return a Report of five shares, each rounded to 3 decimals, and 0 when there is nothing to count: answered (answerable questions, relevant not empty, whose answer is not a refusal), refusals (unanswerable questions, relevant empty, whose answer is a refusal, by isRefusal), faithfulness (answers that are not refusals and that judge says are faithful), citations (answers that are not refusals and pass quickCheck) and relevance (the average relevance of those answers).
Never judge a refusal: the judge calls "I don't know" unsupported. With 4 questions (two answerable, both answered; two unanswerable, one refused) and 2 of the 3 answers faithful, faithfulness is 0.667 and refusals 0.5. On the 80 labelled questions the pipeline gives about answered 0.671, refusals 0.8, faithfulness 1 and relevance 0.856.
- Task 5 · interview question
Interview: gate, a regression gate
gate(now, baseline, tolerance = 0.02) compares a new report with the baseline. A metric drops when now is below baseline minus tolerance. Return { pass, drops }: pass is true when nothing dropped, and drops lists { metric, baseline, now } for each metric that dropped, in the order of the baseline's keys. A metric that goes up is never a failure.
gate(BASELINE, BASELINE) is { pass: true, drops: [] }. The short prompt's report (answered 1, faithfulness 0.625, citations 0.888, relevance 0.525, refusals 0) drops faithfulness, citations, relevance and refusals. A drop of 0.015 passes with the default tolerance and fails with 0.01. Arjun will ask: someone shortened the prompt to save 19 tokens, so how do you know it did not make the answers worse?
What you will learn
- A judge: a model call that grades another answer for faithfulness, and what a judge misses.
- Cheap checks: citations and numbers checked with plain code.
- Answer relevance: does the answer address the question at all.
- A regression gate: a report on the labelled questions, compared with a stored baseline, that fails when a number drops.
More in Production
Build "Is the answer right" yourself
Paisa Setu, a payments company, needs its help centre to answer merchants correctly. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of RAG: search that answers are free. This one comes with the all labs plan.