Harness engineering · Chapter 20 of 23

A task suite

By the end of this chapter you can run a harness on a whole suite of tasks the fair way, each in a fresh checkout, and turn the runs into numbers that mean something: a success rate, steps, tokens, rupees per task and per success, a rate per kind of task, and repeat runs counted as pass@k and pass^k. You can also test the suite runner itself for leaks. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Scale and measure
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.

Why you need this

Farah has a meeting with Hisaab's investors on Monday. "We built a coding agent" is not a slide; "it fixes 12 of our 12 test bugs, at ₹2.57 each" is. Nandini, the QA lead, has a rule for every claim like that: if it is not measured, it did not happen. And Kabir wants something more practical: the next time anyone touches the harness, a single command that says whether it got better.

Chapter 1's scorecard was a start: three tasks, a fresh checkout each, a line per task. It cannot answer Farah. It runs three tasks of twelve, so "3 of 3" says nothing about the eight kinds of task it never tried. It prints and forgets: nothing is saved to compare with next week. It counts requests, not rupees. And it ran once, on a model that, in real life, does not answer the same way twice. This chapter turns it into a suite runner, the tool every later harness change is judged by.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    suite.py: the suite runner

    Nandini's rule: if it is not measured, it did not happen. In suite.py, write run_suite(harness, task_ids=None). For each task id (every task of workbench.TASKS, in order, when task_ids is None) it makes a fresh checkout, runs harness(root, task.issue), which returns a Run (harnesses.py), and grades the checkout with workbench.grade(root, task_id).

    It returns one row per task, in order: a dictionary with task, tags (the task's tags, as a list), passed and reason (from the grade), steps, stop and usage (from the Run) and checkout (the folder, as text).

    Under if __name__ == "__main__":, run the naive harness on the whole suite and print one line per task, two spaces apart, where tokens are every prompt and completion token in the row's usage:

    T01 passed 5 steps 6902 tokens

    Then print the success rate, R rounded to a whole number:

    Success: P/12 (R%)

    Trap: passed comes from the grade, never from the stop. A run that stopped with an error can still have fixed the bug.

  2. Task 2

    report.py: what a suite run costs

    Farah wants rupees per task and per task fixed. In report.py, write prices(): every model in /v1/models that has a price, as a dictionary model id -> (input, output), from its price_inr_per_million (rupees per million tokens). Write cost(usage, prices): for each model in a run's usage, its prompt tokens at the input price plus its completion tokens at the output price. Write summary(rows, prices): a dictionary with tasks, passed, rate (passed / tasks), steps, tokens (prompt and completion, every model), cost, cost_per_task and cost_per_success (cost / passed, or None when nothing passed).

    Under if __name__ == "__main__":, run run_suite with naive, then with full. For each, add cost_inr (the row's cost) to every row, write the rows as JSON to results/naive.json or results/full.json, and print one line, two spaces apart, money with two decimals:

    naive 4/12 (33%) 57 steps 122136 tokens ₹30.05 ₹2.50 per task ₹7.51 per success

    With nothing passed, the end reads "no per success".

    Trap: the full harness's helper runs on lab-chat-mini, which has its own price.

  3. Task 3

    per_tag.py: rates per tag

    One number hides where a harness is weak. In per_tag.py, write by_tag(rows): a dictionary tag -> (passed, total), its tags sorted by name, where a task counts in every one of its tags.

    Under if __name__ == "__main__":, read the results files named on the command line (results/naive.json and results/full.json when none are given) and print a table: a first line with tag and each file's name without .json, then one line per tag (every tag of any file, sorted) with P/N for each file, or - when that file has no task with the tag. Separate the columns with spaces:

    tag naive full

    basic 2/2 2/2

    Trap: the tags add up to more than twelve tasks, and two files need not have the same tags.

  4. Task 4

    repeats.py: repeat runs

    A real model samples, so a suite is run more than once. In repeats.py, write repeat(harness, task_ids, k): run the tasks k times, every run in a fresh checkout, and return a dictionary task id -> its k results (True or False), in run order. Then pass_at_1(results), the share of all runs that passed; pass_at_k(results), the share of tasks that passed at least once; and pass_all_k(results), the share of tasks that passed every run (pass^k).

    Under if __name__ == "__main__":, repeat the naive harness on T01, T07 and T09, three times, and print one line per task, then the three numbers as whole percentages:

    T01 passed passed passed

    pass@1 P% pass@3 P% pass^3 P%

  5. Task 5 · interview question

    Interview: leaks.py, a suite that leaks

    "How do you know your eval runs are isolated?" shared_suite (given in leaks.py: do not change it) makes one checkout for the whole suite. Write leaks(runner, task_ids=None): it calls runner(spy, task_ids) with a spy harness that calls no model. Each time the spy runs, it notes whether its folder already holds a file an earlier spy run left, then leaves one of its own (a name starting with a dot), and returns Run("I changed nothing.", 1, "answered"). leaks returns the ids of the tasks whose run found an earlier run's file, in order, or [] when none did.

    Under if __name__ == "__main__":, print for run_suite and for shared_suite:

    run_suite: no leaks

    shared_suite: 11 tasks saw an earlier run's files (T02 T03 ...)

    (the count and the ids it found, space-separated), then run the naive harness under each runner and print:

    naive under run_suite: P/12

    naive under shared_suite: P/12

What you will learn

  • What a task is: an issue, a checkout with its bug, and a grader.
  • The suite runner: every task in a fresh checkout, one row per task, kept as a results file.
  • Cost: rupees from /v1/models prices, per task and per success.
  • Rates per tag: which kind of task a harness fails.
  • Repeat runs: pass@1, pass@k and pass^k, and why a real model needs them.
  • Isolation: how to prove one run never sees another's work.

More in Scale and measure

Build "A task suite" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.