AI engineering traps · Chapter 16 of 16

Evaluating on the examples

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Tools
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.py: score the labeller the way the team did

    Write reproduce() in repro.py. Run evaluate from evaluate.py on the team's labelled complaints, LABELLED from complaints.py, as they are, and return the accuracy it reports as a whole percentage: round(report.accuracy * 100).

    The check runs your reproduce() twice: beside the original evaluate.py it must return 100 (the score in the ticket), and beside a fixed evaluate.py 67, the honest score. evaluate uses the lab model when you leave out its client; reproduce() must finish within 10 seconds.

  2. Task 2

    evaluate.py: score the labeller on complaints it has not seen

    Write split(data) in evaluate.py: it returns (train, test), two lists of labelled complaints. First drop repeats: a complaint whose text, in lower case with every run of spaces made one space and the ends trimmed, was seen earlier (keep the first). Of what is left, every 4th one (positions 3, 7, 11 and so on, counting from 0) goes in test, the rest in train, in their order. Then make evaluate show the model the train complaints as its examples and score it only on the test complaints, one request each.

    Keep build_prompt, label, Report and the printed line eval: <right>/<tested> right, shown <n> examples as they are; the report's examples and tested are the two counts. The check uses its own complaints, repeats with other capitals and spaces included, and reads every request (no complaint scored may be among its examples). Holding out every 4th row without dropping repeats is not a fix. complaints.py is not to be changed.

  3. Task 3

    test_evaluate.py: a test that would have caught it

    Write a pytest file, test_evaluate.py, that would have caught this before the prompt shipped. The check runs your file three times: beside the original evaluate.py, where at least one test must fail; beside the fixed evaluate.py, where every test must pass; and beside a wrong fix that holds out every 4th row but keeps repeats, where at least one test must fail.

    So give evaluate a fake client that remembers every request body and answers a label, with labelled data of your own in which one complaint is repeated (one copy at a held-out position, the other not), and assert that in every request the complaint scored (the user message after 'Complaint: ') is not in that request's examples (the system message). Each test must finish within 10 seconds.

What you will learn

  • repro.py: score the labeller the way the team did
  • evaluate.py: score the labeller on complaints it has not seen
  • test_evaluate.py: a test that would have caught it

More in Tools

Build "Evaluating on the examples" yourself

Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.