AI support copilot project (TypeScript) · Chapter 11 of 12

Evals

By the end of this chapter you can score the copilot on a fixed set of labelled questions: measure whether search found the right pages (recall at k), ask a model to judge whether each answer says only what its sources say (faithfulness), read what those numbers cannot tell you, and write tests that would fail if the metrics themselves were wrong. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ready for members
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of AI support copilot project (TypeScript) are free. This one comes with the all labs plan.

Why you need this

In ten chapters the copilot changed a lot: chunk sizes, hybrid search, a reranker, structured answers, guards, follow-up rewriting. Every change was checked by hand on a few questions. Sayali now asks the simple question: is it better than last week, and did anything get worse? Nikhil wants to try another chunk size; Asha wants to add pages for the new venues before Diwali. After each change, someone would have to read every answer again.

Datta has done the slow part already. He wrote down 34 questions members really ask on WhatsApp, and for each one the help pages that answer it. Four of them the pages cannot answer, such as "Do you sell tickets for cricket matches?", where the right reply is "I don't know". That file, data/evals/questions.jsonl, is an eval set:

{"id": "q01", "question": "My show was cancelled. Do I get my money back?", "pages": ["cancelled-shows"]}
{"id": "q02", "question": "Is the booking fee refunded when I cancel my booking myself?", "pages": ["cancel-a-booking"]}
...
{"id": "q31", "question": "Do you sell tickets for cricket matches?", "pages": []}

What is missing is the scoring. Nothing in the code says how good 34 answers are, as a number that can go up or down.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    metrics.ts: recall at k, and a mean that skips null

    In src/evals/metrics.ts write recallAtK(expected, retrieved, k): the share of the expected page ids (each counted once) that are among the first k retrieved page ids, or null when nothing is expected. Then mean(values): the average of the values that are not null, rounded to 3 places (1/3 gives 0.333, a number), or null when none are left.

    The Check runs six recall cases (a page at place 2 with k 1 and k 2, a page retrieved twice, an expected page listed twice, nothing expected, four pages with k 3) and four mean cases ([1, null, 0.5], [null, null], [], [1, 0, 0]).

  2. Task 2

    metrics.ts: a model as the faithfulness judge

    Write judgeFaithful(ai, answer, context) in src/evals/metrics.ts. Send one request with ai.chat.completions.create (model config.chatModel, temperature 0): JUDGE_INSTRUCTIONS as the system message, then as the user's message "Context:\n" + context + "\n\nAnswer:\n" + answer. Return true only when the reply, trimmed and in upper case, starts with YES; false otherwise, and for a null reply.

    The Check gives your function stand-in models (it reads the request you sent; replies "YES", " yes\n", "YES, every sentence is supported.", "NO", "No, YES appears only later." and null), then asks the lab model about a faithful answer on the booking fee and one it does not support, then calls GET /evals/report?limit=3, which needs Task 1 too.

  3. Task 3 · interview question

    Interview: metrics.test.ts, tests that catch broken metrics

    Write Vitest tests in tests/metrics.test.ts (it(...) blocks, as in tests/chunking.test.ts) for recallAtK and judgeFaithful. Use fakeAi from tests/helpers.ts: fakeAi('YES') is a model client whose every chat request is answered "YES", and its calls hold what each request was sent. Test at least that recall counts only the first k pages, that it is null when nothing is expected, and that the judge is true for YES (and sent the answer and the context) and false for NO. Every test must pass (npm test).

    The Check runs your file on our metrics.ts, where every test must pass, and on three broken copies of it, where at least one test must fail: a recallAtK that ignores k, a recallAtK that gives 0 instead of null, and a judgeFaithful that says true whatever the model replied.

    Nikhil will ask: your tests pass. How do you know they would fail if the metrics were wrong, and why does the eval need its own tests at all?

What you will learn

  • An eval: labelled questions, metrics, a report you run after every change.
  • Recall at k: did search put the right pages in the first k?
  • A mean that skips null: why a question with no right pages has no recall at all.
  • A model as a judge: faithfulness, the request, reading the verdict, and what a judge gets wrong.
  • Faithful is not correct: real answers that pass every metric and still do not help.
  • Tests for the eval: a fake model client, Vitest, and mutation testing.

More in Ready for members

Build "Evals" yourself

Parda answers the same audience questions all day. You build the copilot that answers them from its own help pages, with sources, and says so when it does not know. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI support copilot project (TypeScript) are free. This one comes with the all labs plan.