Harness engineering · Chapter 16 of 23
Verification loops
By the end of this chapter your harness checks the model's work itself: after the model says it is done, the harness runs the tests, sends any failures back as a new message, and lets the model try again, with a retry budget and a stop when nothing changes. You will measure it on three tasks that fail without it and pass with it, and on one where the loop must know when to give up. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Control
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.
Why you need this
Farah reads the agent's answers, and on T04 this is what it says:
Done: gst_amount now rounds half away from zero with round_half_up,
in integer paise (hisaab/gst.py).
It sounds right. The code it wrote is int(base_paise * rate / 100 + 0.5), which rounds credit notes the wrong way, and the grade says the tests fail. The answer is a claim, not a check.
So far your harness has asked the model to check itself: the written prompt says "run the tests with python -m pytest -q before you finish", and the lab model does, reads the failure and fixes its first try. That works while the prompt says so and the model obeys. Under a one-line prompt, or a long task where the instruction is far behind, the model never runs the tests, and three tasks of the suite fail that way: T04 (its first try is wrong without the project file's rounding rule), T11 (its first split does not add up) and T12 (two trailing spaces).
Nandini's rule is that a task is done when the tests say so, and the harness can run the tests as easily as the model can. It just never has.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
verify.py: the harness runs the tests
Nandini's rule: a task is done when the tests say so. In verify.py, write a dataclass TestRun (passed, failed, summary, output) and run_tests(root): run python -m pytest -q in the checkout with the Python that runs this file, stdout and stderr together, and return passed (exit code 0), failed (the test id of every FAILED or ERROR line, like tests/test_money.py::test_split_evenly_adds_up), summary (the output's last line that is not blank, like 1 failed, 44 passed in 0.05s) and output (the output, shaped with shape from agent).
Write feedback(run): the user message that sends a failing run back: The tests fail (SUMMARY):, a newline, the output, a newline and Fix the code; do not edit the tests.
Under if __name__ == "__main__":, run run_tests on a checkout of T11, then on a clean checkout, and print for each its passed and failed ids, then its summary indented by two spaces:
T11: passed False, failed tests/test_money.py::test_split_evenly_adds_up
1 failed, 44 passed in 0.05s
clean: passed True, failed none
45 passed in 0.04s
Careful: the test's message is in the middle of the output. Shape it; do not keep only its first lines.
- Task 2
with_verify.py: a verify loop
Farah wants "done" to mean the tests pass. In with_verify.py, write a dataclass Verified (answer, rounds, stop, steps, messages) and run_with_verify(issue, ws, system=None, budget=3). It runs run_agent(issue, ws, system=system), then again and again: if that run did not answer, stop with its own stop; run run_tests (from verify) on ws.root; if they pass, stop with passed; if budget rounds of feedback were sent, stop with budget; else add feedback(run) as a user message at the end of the run's messages, run run_agent(issue, ws, messages=...) and count a round. Return the last answer, the rounds, the stop, every run's steps added up and the last run's messages.
Under if __name__ == "__main__":, for T04, T11 and T12, run the task under system=NAIVE_PROMPT on two fresh checkouts, with run_agent alone and with run_with_verify, grade both and print:
T04 without: failed with: passed (1 round, 6 steps)
Careful: the loop decides from the tests it runs itself, never from what the answer says.
- Task 3
no_progress.py: no progress
Amit does not want to pay for rounds that change nothing. In no_progress.py, write verify_loop(issue, ws, system=None, budget=3): run_with_verify's loop, with one more stop: after a failing test run, if the same set of test ids fails as in the round before, stop with no_progress before sending anything. The order of the stops: the run's own, passed, no_progress, budget. Return a Verified (from with_verify).
Under if __name__ == "__main__":, run, each on a fresh checkout and under system=NAIVE_PROMPT, T07 with run_with_verify, T07 with verify_loop, and T04 with verify_loop, and print the stop, the rounds and the steps:
T07 budget only: budget after 3 rounds, 7 steps
T07 no progress: ...
T04 no progress: ...
Careful: a round whose failing tests are different is progress, even if the tests still fail.
- Task 4 · interview question
Interview: never_ends.py, a verify loop that never ends
"Our verify loop has been running on T07 all night." teammate_verify.py (given) has the teammate's teammate_verify(issue, ws, system=None, key=None, give_up_after=None): no budget; it stops when the tests pass, or when key(message) is the same two rounds in a row (key is the whole feedback message unless you pass one); give_up_after is a safety cap. In never_ends.py, write failure_key(message): messages whose failing tests are the same (the ids of their FAILED and ERROR lines, in any order) get the same key, whatever else differs; a different set of failing tests gets a different key.
Under if __name__ == "__main__":, run teammate_verify on a fresh checkout of T07 with system=NAIVE_PROMPT, once with give_up_after=5 and once with key=failure_key. It returns (stop, rounds, steps). Print:
teammate: gave_up after 5 rounds, 9 steps
with failure_key: ...
Careful: read the message the teammate builds before deciding what to compare.
What you will learn
- Who verifies: the model when told to, or the harness every time.
- Running the tests from the harness: passed or not, the failing test ids, the summary line, the output.
- The verify loop: tests after the answer, the failures sent back as a user message, the model tries again.
- When to stop: a pass, a retry budget, and no progress (the same failure twice).
- What "the same failure" means, and the loop that never ends without it.
More in Control
Build "Verification loops" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.