Evals that catch regressions in TypeScript · Chapter 9 of 16
RAG evals
By the end of this chapter you can evaluate a retrieval-augmented assistant in two halves: whether retrieval found the right text (recall@k), whether the answer sticks to what was found (faithfulness), how often it abstains, and, for every wrong answer, which half to blame. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Evals in production
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.
Why you need this
YatraDesk's website has a "search the policy" box. Behind it is a second bot, answerRag: instead of pasting the whole policy into every request, it retrieves the three chunks of the policy and help centre that look most relevant to the question and answers from those. It is cheaper, and it scales to documents far bigger than one policy. It is what most teams build, and it is called retrieval-augmented generation, RAG.
Arif has a pile of complaints from it. Customers who copy the policy's words get good answers. Customers who write the way people write, "my trip is three days away and I can't go, do I get anything?", get "I don't know", or a confident answer about something else.
A RAG system fails in two places, and a score on the final answer cannot say which. Retrieval can miss the chunk that holds the answer; then no model can answer well. Or retrieval finds it and the model ignores it, or adds things that are in no chunk at all. The fixes are different (a better retriever, or a better prompt or model), so the eval has to measure each half on its own.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
recall.ts: did retrieval find the gold chunks?
Export recallAtK(retrieved, gold): the share of the gold ids that are in the array of retrieved ids. Run as a program, for k = 1, 3 and 5, call retrieve(question, k) from ../../yatradesk/rag.ts for every question of /workspace/data/rag.jsonl and print one line per k with each kind's recall summed over its questions, out of its number of questions, kinds in the order they first appear, sums written as JavaScript writes numbers (3, not 3.0; 1.5):
recall@1: direct 3/4, paraphrase 0.5/2
- Task 2
faithfulness.ts: does the answer stick to the context?
Export Faith (zod: faithful, a boolean, and unsupported, an array of strings); faithful(answer, context, { model = 'lab-chat' }), whose system message tells the model to check whether every sentence of the answer is supported by the context and to reply in JSON with faithful and unsupported, and whose user message is the line Context:, the context, then the line Answer: and the answer (parse, zodResponseFormat(Faith, 'faith'), temperature: 0); abstains(answer), true when it contains don't know, do not know, not sure or does not say, ignoring case; ROWS, the rows of rag.jsonl; and ragRun(prompt = 'v1'), answerRag for every question of rag.jsonl, asked together, resolving to the results in the file's order.
Run as a program, for prompt v1 and then v2, judge every answer that does not abstain against its own context and print:
prompt v1: answered 8, faithful 7/8, abstained 2
- Task 3
diagnose.ts: which half failed?
Export sentences(text) (split after every full stop, question mark or exclamation mark followed by whitespace; trimmed; no empty pieces), usesGold(answer, chunks, gold) (some sentence of some chunk whose id is in gold appears in the answer exactly), and diagnose(row, result), which resolves to the first label that applies for prompt v1: retrieval miss (no gold chunk among the result's chunks), abstained, unfaithful (the judge says not faithful), wrong chunk (not usesGold), else correct.
Run as a program, label every question and print one line per label in the order correct, retrieval miss, abstained, unfaithful, wrong chunk, with the count and the ids in the file's order, or none:
correct: 2 (r101, r104)
abstained: 0 (none)
- Task 4 · interview question
Interview: faithful-but-wrong.ts, faithful and wrong
Thomas wants prompt v2 for the policy bot because more of its answers are faithful. Change his script so that, for each prompt, it judges every answer (abstentions included) and splits the faithful ones into those that use a gold chunk's sentence (usesGold from ./diagnose.ts) and those that do not:
prompt v1: faithful 7/10, of which right 5, faithful but wrong 2
prompt v2: faithful 8/10, of which right 5, faithful but wrong 3
Be ready to explain how an answer can be faithful and wrong, and which measurement catches it.
What you will learn
- The two halves of a RAG system, and why they are graded apart.
- Recall@k: did retrieval find the gold chunks?
- Faithfulness: is every claim of the answer in the retrieved text?
- Abstentions, counted on their own.
- Diagnosing each wrong answer: retrieval, abstention, or generation.
More in Evals in production
Build "RAG evals" yourself
YatraDesk, a Kochi travel agency, needs proof that every prompt change makes its assistant better, not worse. You build its evals in TypeScript. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Evals that catch regressions in TypeScript are free. This one comes with the all labs plan.