RAG in Python · Chapter 34 of 37

Is the answer right

By the end of this chapter you can measure the answers, not only the retrieval: faithfulness with a judge, a cheap check with no model, answer relevance, a report over the labelled questions, and a gate that stops a change that makes answers worse. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.

Why you need this

Kabir wants a shorter system prompt: the new one is 19 tokens instead of 38, and at 600 questions a day every token counts. Does it change the answers? Neha asks the question compliance always asks: is every claim in an answer really in the chunks it cites?

Recall@k cannot answer either of them. It says the right chunks were found; it says nothing about what the model did with them. You could read 80 answers by hand after every change, once. Nobody does it the second time, and the change that breaks the answers is always the small one nobody thought to check.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    evaluate.py: judge, a judge for faithfulness

    Neha asks whether every claim in an answer is really in the chunks it came from. Write judge(text, hits) in evaluate.py: send one request to MODEL with response_format={"type": "json_object"}. The system message gives the job in words: judge (or grade, or evaluate) whether the answer is faithful to (supported by) the context, and reply in JSON. The user message is <context>, every hit as [id] text with blank lines between, </context>, then <answer>, the answer, </answer>.

    The model replies {"faithful": bool, "unsupported": [sentences]}: return it as a Verdict. With help-chargeback-ratio#0 as the context, "Only card transactions count. UPI, net banking and wallet payments are not part of the ratio." is Verdict(faithful=True, unsupported=[]), and "Refunds are paid in cash at our Pune office." comes back unsupported. Use the tags, not "Context:" labels: the chunk has a line "Example: ..." that would end a labelled context early.

  2. Task 2

    evaluate.py: quick_check, a cheap check with no model

    Write quick_check(text, hits): return Quick(ok, unknown_citations, missing_numbers). unknown_citations are the ids the answer cites (cited_ids) that are not among the hits. missing_numbers are the numbers in the answer, with its ids taken out (without_ids), that appear in none of the cited chunks, once each, in the order they appear. ok is True when both lists are empty.

    A number is the whole thing: 2,000.50 and 0.15 are one number each, and the 69 in U69 is not a number. "Card refunds take 7 to 10 working days. [help-refund-timelines#1]" has missing_numbers ["10"]. "Instant refunds cost ₹5 per refund plus GST. [help-refund-timelines#4]" is ok: the 4 in the id is not a claim. An answer citing help-refund-timelines#9, which is not a hit, has it in unknown_citations, and all its numbers are missing because no real chunk was cited.

  3. Task 3

    evaluate.py: relevance, does the answer answer the question?

    Write relevance(question, text): score the answer for the question with one rerank_scores call, the question as the query and the answer without its ids (without_ids) as the only document. Return that one score.

    An answer about U30 for "what does UPI error U30 mean?" scores above 0.9; an answer about card refunds for the same question scores below 0.05.

  4. Task 4

    evaluate.py: evaluate, a report on the labelled questions

    Write evaluate(questions, answer_fn): run answer_fn(q) for every question and return a Report of five shares, each made with the given share (rounded to 3 decimals, and 0 when there is nothing to count). answered: answerable questions (relevant not empty) whose answer is not a refusal (is_refusal). refusals: unanswerable questions (relevant empty) whose answer is a refusal. faithfulness: answers that are not refusals and that judge says are faithful. citations: answers that are not refusals and pass quick_check. relevance: the average relevance of those answers.

    Never judge a refusal: the judge calls "I don't know" unsupported. With 4 questions (two answerable, both answered; two unanswerable, one refused), answered is 1 and refusals 0.5, and the other three are shares of the 3 real answers. On the 80 labelled questions the pipeline gives answered 0.671, faithfulness 1.0, citations 1.0, relevance 0.856 and refusals 0.8.

  5. Task 5 · interview question

    Interview: evaluate.py, gate, a regression gate

    Write gate(now, baseline, tolerance=0.02): compare a new report with the baseline. A metric drops when now is below baseline minus tolerance. Return Gate(passed=nothing dropped, drops=a Drop(metric, baseline, now) for each metric that dropped, in the order of the Report's fields). A metric that goes up is never a failure.

    gate(BASELINE, BASELINE) passes with no drops. Kabir's short prompt (answered 0.929, faithfulness 0.676, citations 0.878, relevance 0.567, refusals 0.1) drops faithfulness, citations, relevance and refusals. A drop of 0.015 passes with the default tolerance and fails with 0.01. Arjun will ask: someone shortened the prompt to save 19 tokens, so how do you know it did not make the answers worse?

What you will learn

  • A judge: a model call that grades another answer for faithfulness, and what a judge misses.
  • Cheap checks: citations and numbers checked with plain code.
  • Answer relevance: does the answer address the question at all.
  • A regression gate: a report on the labelled questions, compared with a stored baseline, that fails when a number drops, run by pytest.

More in Production

Build "Is the answer right" yourself

Paisa Setu, a payments company, needs its help centre to answer merchants correctly, with sources. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of RAG in Python are free. This one comes with the all labs plan.