Harness engineering · Chapter 15 of 23

Hooks

By the end of this chapter you can give a harness hooks: your own code that runs before a tool call, after it and at the end of a run, without touching the tools or the loop. You will build a hook bus, a hook that blocks a dangerous command with a reason the model reads, and a formatter that cleans every file the model edits, and you will measure the formatter on the task where the model always leaves a mess. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Control
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.

Why you need this

Nandini has a task the agent keeps failing in a silly way. T12 says UPI amounts need two decimals. The model finds upi_link, writes the right fix, and leaves two spaces at the end of the line. Hisaab's style test fails on trailing whitespace, so the task fails. Real coding models do this kind of thing all the time: a stray space, a missing final newline, an import out of order. Hisaab's own developers never think about it, because their editor formats the file when they save.

Under the written prompt the agent gets there anyway: it runs the tests, reads trailing whitespace in hisaab/upi.py line 13, and edits the line a second time. That is two extra steps and a second edit, every time, for something a program could fix in a millisecond. Under a one-line prompt the model never runs the tests, and T12 fails.

Amit has the opposite problem. Chapter 13's rules say yes or no by pattern, and he wants code in the path of every call: one place to say "git push never runs", whatever the rules say, and to say it in words the model can act on.

What you have cannot do either. The loop calls the tools directly, so anything new means changing the loop, the tool, or both, for every project that wants something different.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    hooks.py: a hook bus

    Kabir wants one place for the team's own checks, outside the tools and the loop. In hooks.py, write class HookBus with before(pattern, fn), after(pattern, fn), on_stop(fn), call(ws, name, args), stop(ws, answer) and errors (a list). A before-hook fn(ws, name, args) returns None to allow the call or a reason to block it; an after-hook fn(ws, name, args, result) returns None to keep the result or a new one; a stop hook fn(ws, answer) runs at the end. Patterns match tool names with fnmatch.

    call runs the matching before-hooks in the order added; at the first reason it returns {"error": "blocked by a hook: REASON"} as JSON, and neither the tool nor the later hooks run. Else it runs call_tool (from agent), then the matching after-hooks in order, each one seeing the result so far, and returns it. stop runs every stop hook. A hook that raises is recorded in errors as EVENT HOOKNAME on TOOL: TYPE: MESSAGE (for example after note_line_count on list_files: KeyError: 'path'), and the bus carries on as if it returned None.

    Under if __name__ == "__main__":, add hooks on "*" that note before NAME and after NAME, and a stop hook that notes stop; run run_agent on a checkout of T01 with hooks=bus, print the notes one per line, then T01: passed or T01: failed.

    Careful: an after-hook that returns None leaves the result as it was.

  2. Task 2

    block.py: a hook that blocks

    Amit wants rm -r, rm -f and git push never to run, whatever the model was told. In block.py, write block_dangerous(ws, name, args), a before-hook for run_command: when args["command"] runs rm with a flag that has r or f in it (rm -rf, rm -fr, rm -r, rm -f), or git push, anywhere in the command, return a reason saying why and what to do instead; for any other command, return None.

    Under if __name__ == "__main__":, run run_agent on a checkout of T09 with system=NAIVE_PROMPT and no hooks, and print no hooks: T09 and the grade's reason. Then run it again with a HookBus whose before-hook on run_command is block_dangerous, and print the error the model read for the blocked call and the grade:

    no hooks: T09 the tests were changed: tests/ (deleted)

    blocked: blocked by a hook: ...

    with block_dangerous: T09 passed

    Careful: cd hisaab && rm -rf ../tests is the same command with something in front of it.

  3. Task 3

    format_hook.py: a formatter hook

    Nandini wants T12 to stop failing on two stray spaces. In format_hook.py, write format_after_edit(ws, name, args, result), an after-hook for edit_file: when the result is "ok", read the edited file with ws.text, remove the spaces at the end of every line, make it end with exactly one newline, and if that changed the text, save it with ws.save and return a result that starts with ok and says what it did, like ok (formatted hisaab/upi.py: trailing spaces removed from 1 line). After a failed edit, or on a clean file, return None.

    Under if __name__ == "__main__":, run T12 four ways, each on a fresh checkout: NAIVE_PROMPT and the written prompt (system=None), each without and with a HookBus whose after-hook on edit_file is format_after_edit, and print the grade, the run's steps and the edit_file calls:

    one-line prompt: T12 failed, 4 steps, 1 edit

    written prompt: T12 ...

    one-line prompt + formatter: T12 ...

    written prompt + formatter: T12 ...

    Careful: format only the file the edit changed, never the whole checkout.

  4. Task 4 · interview question

    Interview: quiet_failure.py, a formatter that never formats

    "Our formatter hook is on, and T12 still fails. Why?" teammate_hooks.py (given) has the teammate's teammate_format, an after-hook for edit_file. In quiet_failure.py, write fixed_format(ws, name, args, result): the same hook, fixed, so that T12 under the one-line prompt passes in one edit (it may return None).

    Under if __name__ == "__main__":, for the teammate's hook and then yours, run run_agent on a fresh checkout of T12 with system=NAIVE_PROMPT and a HookBus whose after-hook on edit_file is that hook. Print the grade and how many hook errors the bus recorded, then each error indented by two spaces:

    teammate: T12 failed, hook errors: 1

    after teammate_format on edit_file: ...

    fixed: T12 passed, hook errors: 0

    Careful: read the recorded error before changing anything.

What you will learn

  • What a hook is: code the harness runs at fixed events (before a tool, after it, at the end of a run).
  • A hook bus: hooks by tool name, in order, with a block that stops the call and an after-hook that changes the result.
  • A blocking hook whose reason the model reads, measured on T09.
  • A formatter hook, measured on T12: one edit instead of two, and a pass under a one-line prompt.
  • Why a hook that fails must not fail quietly.

More in Control

Build "Hooks" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.