Harness engineering · Chapter 21 of 23

Comparing harnesses

By the end of this chapter you can compare two versions of a harness on the same model, task by task (fixed, broken, same), find out which feature carries which task by removing one at a time (an ablation), and put a regression gate in front of every harness change, so that "it scores higher" is never the whole argument. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Scale and measure
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.

Why you need this

Hisaab runs a harness Nandini calls V1: most of chapters 3 to 17, but no todo tool, bare edit errors and no deny rules. It passes 10 of 12. A colleague sends V2: the todo tool and helpful errors added, and one line trimmed from the system prompt to keep it short. It passes 11 of 12. Farah reads the two numbers and says ship it.

Chapter 20's suite gives each harness a rate, a cost and a results file. It does not say what changed. Two rates cannot tell "V2 fixed one task" from "V2 fixed two and broke one", and the second is a very different change to ship: the broken task could be the one that matters most. Nor can a rate say which part of a harness earns its place, or stop the next change from quietly undoing this one. This chapter compares harnesses the way Nandini compares test runs: per task, one change at a time, with a rule at the end.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    compare.py: two harnesses, task by task

    A rate says how many tasks pass, not which. In compare.py, write compare(before, after): two lists of suite rows, matched by task id. Return a dictionary of four lists of task ids, in the before rows' order: fixed (failed before, passes after), broken (passed before, fails after), still_passing and still_failing. A task missing from either list is left out.

    Under if __name__ == "__main__":, run the suite with naive, then with full (run_suite and save are given in suite.py), save them as results/naive.json and results/full.json, and print one line per task, two spaces apart, ending in fixed, broken or same, then the counts:

    T03 failed -> passed fixed

    Fixed N, broken N, still passing N, still failing N

    Trap: the two lists need not be in the same order: match by id, never by position.

  2. Task 2

    ablation.py: one feature at a time

    Which of full's features does each task need? In ablation.py, write ablate(features, task_ids=None, base=FULL): for each feature, in order, run run_suite on task_ids with a harness that is run_harness with base minus that one feature. Return a dictionary feature -> its rows.

    Under if __name__ == "__main__":, ablate the features named on the command line (python ablation.py todo search), or every one of FEATURES when none are given, and print one line per feature, two spaces apart (summary is in suite.py; money with two decimals, "no per success" when nothing passed), with the failing tasks at the end when there are any:

    without search 12/12 ₹2.51 per success

    without edit_tool 10/12 ₹4.63 per success fails T03 T04

    Trap: a harness takes only (root, issue): the feature to leave out has to travel inside it.

  3. Task 3

    gate.py: a regression gate

    Every harness change is checked by a rule before it ships. In gate.py, write gate(before, after, max_cost_rise=0.10): it returns a pair (ok, reasons), where ok is True when reasons is empty. The reasons, in this order: for each task that passed before and fails after (in the before rows' order), "T09 passed before and fails now: " and the after row's reason; then, when nothing passes after, "nothing passes now"; else, when the cost per success (summary, in suite.py) rose by more than max_cost_rise (0.10 is 10%), with whole percentages and two-decimal rupees:

    cost per success rose 14% (₹2.57 to ₹2.92), more than 10%

    When nothing passed before, there is no cost to compare.

    Under if __name__ == "__main__":, gate the two results files named on the command line (results/naive.json and results/full.json when none are given; load is in suite.py) and print Gate: pass or Gate: fail, then each reason on a line of its own, indented two spaces.

  4. Task 4 · interview question

    Interview: better.py, a "better" harness

    A colleague's V2 adds the todo tool and helpful edit errors to V1, and trims a line from the system prompt (both are in harnesses.py). "It scores higher, ship it." In better.py, run the suite with V1, then with V2 (run_harness with each set of features), save them as results/v1.json and results/v2.json, and print for each, two spaces apart:

    v1 10/12 (83%) ₹2.96 per success

    Then gate V2 against V1 with gate (from gate.py), print its verdict and reasons the way gate.py does, and finish with Ship v2: yes or Ship v2: no.

    Trap: read the reasons, not only the rates.

What you will learn

  • An A/B of harnesses: the same model, the same tasks, two harnesses.
  • Per-task changes: fixed, broken, still passing, still failing.
  • Ablations: remove one feature at a time, and see which task needed it.
  • What an ablation cannot see: features that back each other up, and tasks the suite does not have.
  • A regression gate: no passing task may fail, and the cost per success may not rise past a limit.

More in Scale and measure

Build "Comparing harnesses" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.