Evals that catch regressions · Chapter 9 of 16

RAG evals

By the end of this chapter you can evaluate a retrieval-augmented assistant in two halves: whether retrieval found the right text (recall@k), whether the answer sticks to what was found (faithfulness), how often it abstains, and, for every wrong answer, which half to blame. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Evals in production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.

Why you need this

YatraDesk's website has a "search the policy" box. Behind it is a second bot, answer_rag: instead of pasting the whole policy into every request, it retrieves the three chunks of the policy and help centre that look most relevant to the question and answers from those. It is cheaper, and it scales to documents far bigger than one policy. It is what most teams build, and it is called retrieval-augmented generation, RAG.

Arif has a pile of complaints from it. Customers who copy the policy's words get good answers. Customers who write the way people write, "my trip is three days away and I can't go, do I get anything?", get "I don't know", or a confident answer about something else.

A RAG system fails in two places, and a score on the final answer cannot say which. Retrieval can miss the chunk that holds the answer; then no model can answer well. Or retrieval finds it and the model ignores it, or adds things that are in no chunk at all. The fixes are different (a better retriever, or a better prompt or model), so the eval has to measure each half on its own.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    recall.py: did retrieval find the gold chunks?

    Write recall_at_k(retrieved, gold): the share of the gold ids that are in the list of retrieved ids. Run as a program, for k = 1, 3 and 5, call retrieve(question, k) from yatradesk.rag for every question of /workspace/data/rag.jsonl and print one line per k with each kind's recall summed over its questions, out of its number of questions, kinds in the order they first appear, sums written with the g format:

    recall@1: direct 3/4, paraphrase 0.5/2

  2. Task 2

    faithfulness.py: does the answer stick to the context?

    Write Faith (Pydantic: faithful, a bool, and unsupported, a list of str); faithful(answer, context, model="lab-chat"), whose system message tells the model to check whether every sentence of the answer is supported by the context and to reply in JSON with faithful and unsupported, and whose user message is the line Context:, the context, then the line Answer: and the answer (parse, response_format=Faith, temperature=0); abstains(answer), True when it contains don't know, do not know, not sure or does not say, ignoring case; and rag_run(prompt="v1"), answer_rag for every question of rag.jsonl in order, 8 at a time with a ThreadPoolExecutor.

    Run as a program, for prompt v1 and then v2, judge every answer that does not abstain against its own context and print:

    prompt v1: answered 8, faithful 7/8, abstained 2

  3. Task 3

    diagnose.py: which half failed?

    Write sentences(text) (split after every full stop, question mark or exclamation mark followed by whitespace; stripped; no empty pieces), uses_gold(answer, chunks, gold) (some sentence of some chunk whose id is in gold appears in the answer exactly), and diagnose(row, result), which returns the first label that applies for prompt v1: retrieval miss (no gold chunk among the result's chunks), abstained, unfaithful (the judge says not faithful), wrong chunk (not uses_gold), else correct.

    Run as a program, label every question and print one line per label in the order correct, retrieval miss, abstained, unfaithful, wrong chunk, with the count and the ids in the file's order, or none:

    correct: 2 (r101, r104)

    abstained: 0 (none)

  4. Task 4 · interview question

    Interview: faithful_but_wrong.py, faithful and wrong

    Thomas wants prompt v2 for the policy bot because more of its answers are faithful. Change his script so that, for each prompt, it judges every answer (abstentions included) and splits the faithful ones into those that use a gold chunk's sentence (uses_gold from diagnose.py) and those that do not:

    prompt v1: faithful 7/10, of which right 5, faithful but wrong 2

    prompt v2: faithful 8/10, of which right 5, faithful but wrong 3

    Be ready to explain how an answer can be faithful and wrong, and which measurement catches it.

What you will learn

  • The two halves of a RAG system, and why they are graded apart.
  • Recall@k: did retrieval find the gold chunks?
  • Faithfulness: is every claim of the answer in the retrieved text?
  • Abstentions, counted on their own.
  • Diagnosing each wrong answer: retrieval, abstention, or generation.

More in Evals in production

Build "RAG evals" yourself

YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Evals that catch regressions are free. This one comes with the all labs plan.