Harness engineering (TypeScript) · Chapter 16 of 23
Verification loops
By the end of this chapter your harness checks the model's work itself: after the model says it is done, the harness runs the tests, sends any failures back as a new message, and lets the model try again, with a retry budget and a stop when nothing changes. You will measure it on three tasks that fail without it and pass with it, and on one where the loop must know when to give up. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Control
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Farah reads the agent's answers, and on T04 this is what it says:
Done: gstAmount now rounds half away from zero with roundHalfUp,
in integer paise (src/gst.ts).
It sounds right. The code it wrote is Math.floor((basePaise * rate) / 100 + 0.5), which rounds credit notes the wrong way, and the grade says the tests fail. The answer is a claim, not a check.
So far your harness has asked the model to check itself: the written prompt says "run the tests with npm test before you finish", and the lab model does, reads the failure and fixes its first try. That works while the prompt says so and the model obeys. Under a one-line prompt, or a long task where the instruction is far behind, the model never runs the tests, and three tasks of the suite fail that way: T04 (its first try is wrong without the project file's rounding rule), T11 (its first split does not add up) and T12 (two trailing spaces).
Nandini's rule is that a task is done when the tests say so, and the harness can run the tests as easily as the model can. It just never has.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
verify.ts: the harness runs the tests
Nandini's rule: a task is done when the tests say so. In verify.ts, export type TestRun = { passed, failed, summary, output } and runTests(root): run npm test in the checkout, stdout and stderr together, and return passed (exit code 0), failed (the names of the ✖ lines after the line ✖ failing tests:, without their time, like splitEvenly adds up), summary (the ℹ pass and ℹ fail lines joined, like ℹ pass 44, ℹ fail 1) and output (the output, shaped with shape from agent.ts).
Export feedback(run): the user message that sends a failing run back: The tests fail (SUMMARY):, a newline, the output, a newline and Fix the code; do not edit the tests.
Under if (import.meta.main), run runTests on a checkout of T11, then on a clean checkout, and print for each its passed and failed tests, then its summary indented by two spaces:
T11: passed false, failed splitEvenly adds up
ℹ pass 44, ℹ fail 1
clean: passed true, failed none
ℹ pass 45, ℹ fail 0
Careful: the spec reporter lists every test, passing ones too. Only the ✖ lines under ✖ failing tests: are failures.
- Task 2
with-verify.ts: a verify loop
Farah wants "done" to mean the tests pass. In with-verify.ts, export type Verified = { answer, rounds, stop, steps, messages } and async function runWithVerify(issue, ws, { system, budget = 3 } = {}). It runs runAgent(issue, ws, { system }), then again and again: if that run did not answer, stop with its own stop; run runTests (from ./verify.ts) on ws.root; if they pass, stop with passed; if budget rounds of feedback were sent, stop with budget; else add feedback(run) as a user message at the end of the run's messages, run runAgent(issue, ws, { messages }) and count a round. Return the last answer, the rounds, the stop, every run's steps added up and the last run's messages.
Under if (import.meta.main), for T04, T11 and T12, run the task under { system: NAIVE_PROMPT } on two fresh checkouts, with runAgent alone and with runWithVerify, grade both and print:
T04 without: failed with: passed (1 round, 6 steps)
Careful: the loop decides from the tests it runs itself, never from what the answer says.
- Task 3
no-progress.ts: no progress
Amit does not want to pay for rounds that change nothing. In no-progress.ts, export async function verifyLoop(issue, ws, { system, budget = 3 } = {}): runWithVerify's loop, with one more stop: after a failing test run, if the same set of failing tests fails as in the round before, stop with no_progress before sending anything. The order of the stops: the run's own, passed, no_progress, budget. Return a Verified (from ./with-verify.ts).
Under if (import.meta.main), run, each on a fresh checkout and under { system: NAIVE_PROMPT }, T07 with runWithVerify, T07 with verifyLoop, and T04 with verifyLoop, and print the stop, the rounds and the steps:
T07 budget only: budget after 3 rounds, 7 steps
T07 no progress: ...
T04 no progress: ...
Careful: a round whose failing tests are different is progress, even if the tests still fail.
- Task 4 · interview question
Interview: never-ends.ts, a verify loop that never ends
"Our verify loop has been running on T07 all night." teammate-verify.ts (given) has the teammate's teammateVerify(issue, ws, { system, key, giveUpAfter }): no budget; it stops when the tests pass, or when key(message) is the same (===) two rounds in a row (key is the whole feedback message unless you pass one); giveUpAfter is a safety cap. In never-ends.ts, export failureKey(message): messages whose failing tests are the same (the ✖ names under ✖ failing tests:, in any order) get the same key, whatever else differs; a different set gets a different key. Return a string, since the teammate compares with ===.
Under if (import.meta.main), run teammateVerify on a fresh checkout of T07 with { system: NAIVE_PROMPT }, once with giveUpAfter: 5 and once with key: failureKey. It returns [stop, rounds, steps]. Print:
teammate: gave_up after 5 rounds, 9 steps
with failureKey: ...
Careful: read the message the teammate builds before deciding what to compare.
What you will learn
- Who verifies: the model when told to, or the harness every time.
- Running the tests from the harness: passed or not, the failing test ids, the summary line, the output.
- The verify loop: tests after the answer, the failures sent back as a user message, the model tries again.
- When to stop: a pass, a retry budget, and no progress (the same failure twice).
- What "the same failure" means, and the loop that never ends without it.
More in Control
Build "Verification loops" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.