Harness engineering (TypeScript) · Chapter 15 of 23
Hooks
By the end of this chapter you can give a harness hooks: your own code that runs before a tool call, after it and at the end of a run, without touching the tools or the loop. You will build a hook bus, a hook that blocks a dangerous command with a reason the model reads, and a formatter that cleans every file the model edits, and you will measure the formatter on the task where the model always leaves a mess. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Control
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Nandini has a task the agent keeps failing in a silly way. T12 says UPI amounts need two decimals. The model finds upiLink, writes the right fix, and leaves two spaces at the end of the line. Hisaab's style test fails on trailing whitespace, so the task fails. Real coding models do this kind of thing all the time: a stray space, a missing final newline, an import out of order. Hisaab's own developers never think about it, because their editor formats the file when they save.
Under the written prompt the agent gets there anyway: it runs the tests, reads trailing whitespace in src/upi.ts line 9: remove it, and edits the line a second time. That is two extra steps and a second edit, every time, for something a program could fix in a millisecond. Under a one-line prompt the model never runs the tests, and T12 fails.
Amit has the opposite problem. Chapter 13's rules say yes or no by pattern, and he wants code in the path of every call: one place to say "git push never runs", whatever the rules say, and to say it in words the model can act on.
What you have cannot do either. The loop calls the tools directly, so anything new means changing the loop, the tool, or both, for every project that wants something different.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
hooks.ts: a hook bus
Kabir wants one place for the team's own checks, outside the tools and the loop. In hooks.ts, write class HookBus with before(pattern, fn), after(pattern, fn), onStop(fn), call(ws, name, args), stop(ws, answer) and errors (a list). A before-hook fn(ws, name, args) returns nothing (null or undefined) to allow the call or a reason to block it; an after-hook fn(ws, name, args, result) returns nothing to keep the result or a new one; a stop hook fn(ws, answer) runs at the end. Patterns match tool names, '*' standing for any run of characters; a hook may be async.
call (async) runs the matching before-hooks in the order added; at the first reason it returns {"error": "blocked by a hook: REASON"} as JSON, and neither the tool nor the later hooks run. Else it runs callTool (from agent.ts), then the matching after-hooks in order, each one seeing the result so far, and returns it. stop runs every stop hook. A hook that throws is recorded in errors as EVENT HOOKNAME on TOOL: NAME: MESSAGE (for example after noteLineCount on list_files: TypeError: The "paths[1]" argument must be of type string. Received undefined), and the bus carries on as if it returned nothing.
Under if (import.meta.main),, add hooks on "*" that note before NAME and after NAME, and a stop hook that notes stop; run runAgent on a checkout of T01 with { hooks: bus }, print the notes one per line, then T01: passed or T01: failed.
Careful: an after-hook that returns nothing leaves the result as it was.
- Task 2
block.ts: a hook that blocks
Amit wants rm -r, rm -f and git push never to run, whatever the model was told. In block.ts, export blockDangerous(ws, name, args), a before-hook for run_command: when args.command runs rm with a flag that has r or f in it (rm -rf, rm -fr, rm -r, rm -f), or git push, anywhere in the command, return a reason saying why and what to do instead; for any other command, return null.
Under if (import.meta.main),, run runAgent on a checkout of T09 with { system: NAIVE_PROMPT } and no hooks, and print no hooks: T09 and the grade's reason. Then run it again with a HookBus whose before-hook on run_command is blockDangerous, and print the error the model read for the blocked call and the grade:
no hooks: T09 the tests were changed: test/ (deleted)
blocked: blocked by a hook: ...
with blockDangerous: T09 passed
Careful: cd src && rm -rf ../test is the same command with something in front of it.
- Task 3
format-hook.ts: a formatter hook
Nandini wants T12 to stop failing on two stray spaces. In format-hook.ts, export formatAfterEdit(ws, name, args, result), an after-hook for edit_file: when the result is "ok", read the edited file with ws.text, remove the spaces at the end of every line, make it end with exactly one newline, and if that changed the text, save it with ws.save and return a result that starts with ok and says what it did, like ok (formatted src/upi.ts: trailing spaces removed from 1 line). After a failed edit, or on a clean file, return null.
Under if (import.meta.main),, run T12 four ways, each on a fresh checkout: NAIVE_PROMPT and the written prompt (no system option), each without and with a HookBus whose after-hook on edit_file is formatAfterEdit, and print the grade, the run's steps and the edit_file calls:
one-line prompt: T12 failed, 4 steps, 1 edit
written prompt: T12 ...
one-line prompt + formatter: T12 ...
written prompt + formatter: T12 ...
Careful: format only the file the edit changed, never the whole checkout.
- Task 4 · interview question
Interview: quiet-failure.ts, a formatter that never formats
"Our formatter hook is on, and T12 still fails. Why?" teammate-hooks.ts (given) has the teammate's teammateFormat, an after-hook for edit_file. In quiet-failure.ts, export fixedFormat(ws, name, args, result): the same hook, fixed, so that T12 under the one-line prompt passes in one edit (it may return null).
Under if (import.meta.main),, for the teammate's hook and then yours, run runAgent on a fresh checkout of T12 with { system: NAIVE_PROMPT } and a HookBus whose after-hook on edit_file is that hook. Print the grade and how many hook errors the bus recorded, then each error indented by two spaces:
teammate: T12 failed, hook errors: 1
after teammateFormat on edit_file: ...
fixed: T12 passed, hook errors: 0
Careful: read the recorded error before changing anything.
What you will learn
- What a hook is: code the harness runs at fixed events (before a tool, after it, at the end of a run).
- A hook bus: hooks by tool name, in order, with a block that stops the call and an after-hook that changes the result.
- A blocking hook whose reason the model reads, measured on T09.
- A formatter hook, measured on T12: one edit instead of two, and a pass under a one-line prompt.
- Why a hook that fails must not fail quietly.
More in Control
Build "Hooks" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.