AI support copilot project · Chapter 11 of 12
Evals
By the end of this chapter you can score the copilot on a fixed set of labelled questions: measure whether search found the right pages (recall at k), ask a model to judge whether each answer says only what its sources say (faithfulness), read what those two numbers cannot tell you, and write tests that would fail if the metrics themselves were wrong. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Ready for members
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.
Why you need this
In ten chapters the copilot changed a lot: chunk sizes, hybrid search, a reranker, new instructions, three guards, follow-up rewriting. Every change was checked by hand on a few questions. Farah's next question is simple: is it better than last week, and did anything get worse? Kunal wants to change the chunk size again; Bhavna wants to add twenty pages before Diwali. After each change, someone would have to read every answer again.
Sana has done the slow part already. She wrote down 34 questions members really ask, and for each one the help pages that answer it. Four of them the pages cannot answer, such as "Do you deliver dinner?", where the right reply is "I don't know". That file, data/evals/questions.jsonl, is an eval set:
{"id": "q01", "question": "How do I pause my subscription?", "pages": ["pause-subscription"]}
{"id": "q02", "question": "My dabba came 40 minutes late. Do I get my money back?", "pages": ["late-delivery-refunds"]}
...
{"id": "q31", "question": "Do you deliver dinner?", "pages": []}
What is missing is the scoring. Chapter 7's checks say whether one answer has real citations; nothing in the code says how good 34 answers are, as a number that can go up or down.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
metrics.py: recall at k, and a mean that skips None
In app/evals/metrics.py write recall_at_k(expected, retrieved, k): the share of the expected page ids (each counted once) that are among the first k retrieved page ids, or None when nothing is expected. Then mean(values): the average of the values that are not None, rounded to 3 places, or None when none are left.
The Check runs six recall cases (a page at place 2 with k=1 and k=2, a page retrieved twice, an expected page listed twice, nothing expected) and four mean cases.
- Task 2
metrics.py: a model as the faithfulness judge
Write judge_faithful(ai, answer, context) in app/evals/metrics.py. Send one chat request (settings.chat_model, temperature=0) with JUDGE_INSTRUCTIONS as the system message and, as the user's, "Context:\n<context>\n\nAnswer:\n<answer>". Return True only when the reply, stripped and upper cased, starts with "YES".
The Check reads the lab model's request log for a faithful answer about the ₹30 delivery charge and an answer the context does not support, then gives your function stand-in models that reply " yes\n", "NO" and "No, YES appears only later."
- Task 3 · interview question
Interview: test_metrics.py, tests that catch broken metrics
Write tests in tests/test_metrics.py (functions named test_...) for recall_at_k and judge_faithful. Use the fake_ai fixture tests/conftest.py gives: fake_ai("YES") is a model client whose every chat request is answered "YES", and its .calls holds the arguments of each request. Test at least that recall counts only the first k pages, that it is None when nothing is expected, and that the judge is True for YES, False for NO, and sent the answer and the context.
The Check runs your file on our code, where every test must pass, and on three broken copies of metrics.py, where at least one test must fail: a recall that ignores k, a recall that says 0.0 instead of None, and a judge that says True whatever the model replied.
Kunal will ask: your tests pass. How do you know they would fail if the metrics were wrong, and why does the eval need its own tests at all?
What you will learn
- An eval: labelled questions, metrics, a report you run after every change.
- Recall at k: did search put the right pages in the first k?
- A mean that skips None: why a question with no right pages has no recall at all.
- A model as a judge: faithfulness, the request, reading the verdict, and what a judge gets wrong.
- Faithful is not correct: real answers that pass every metric and still do not help.
- Tests for the eval: a fake model client as a fixture, and mutation testing.
More in Ready for members
Build "Evals" yourself
DabbaDrop answers the same customer questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of AI support copilot project are free. This one comes with the all labs plan.