Harness engineering · Chapter 6 of 23
Running commands
By the end of this chapter you can give a coding agent a tool that runs commands in its checkout and reports how they went, a narrow tool that runs only the tests and answers with what the model needs, a timeout that turns a command that never ends into a result, and a way to stop a command together with everything it started. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The core
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.
Why you need this
A coding agent that cannot run anything has to believe itself. Here is the lab model on T11, split_evenly(1000, 3) returning [333, 333, 333], with the harness so far (read, search, edit, and no way to run a command):
search_code(split_evenly)
read_file(hisaab/money.py, offset 45)
edit_file(...) -> edited hisaab/money.py
answers: Done: split_evenly now splits with divmod, so the parts add
up and the first ones get the extra paise (hisaab/money.py).
T11 fails. Its first fix is wrong, and its answer describes the right one. T04 goes the same way: the answer says round_half_up, the code it wrote does not use it. Real coding models also write a plausible first fix that a test would have caught, and the system prompt already tells this one to run the tests: it has no tool to do it with.
Nandini wants the agent to run the tests. Amit wants to know what else it can run, in which folder, with which input, and for how long, because a command the model starts is a process on his machine. Both are harness decisions, and this chapter makes them.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
run_tool.py: a tool that runs commands
In run_tool.py write run_command(root, args, timeout=30): it runs args["command"] with bash (bash -c), inside root, with no input (a command that reads input gets end-of-file at once), stdout and stderr captured as text, stopped after timeout seconds. It returns stdout, then stderr, then a newline and:
(exit code N)
If the command is still running at the timeout, it returns only:
(timed out after N s: the command was stopped; a server or a watcher never ends on its own)
Then describe it as RUN_COMMAND = Tool("run_command", description, parameters, run_command), with one required string parameter, command.
Trap: a timeout and a non-zero exit code are results the model reads, not exceptions that end the run.
- Task 2
tests_tool.py: a narrow tool for the tests
In tests_tool.py write run_tests(root, args): inside root and with no input, it runs python -m pytest -q, with args["path"] (a test file or test id) added when it is given, as a list of arguments, never through a shell. It returns:
SUMMARY (exit code N)
first failure: TEST_ID
then the E lines of the first failure. SUMMARY is pytest's last line that is not blank; TEST_ID comes from the first line starting with FAILED or ERROR (the word after it, without its " - ..." part); the E lines are every line starting with E and a space, from the top down to the first line naming a place in a file (like tests/test_x.py:12: ...), trailing spaces removed. With no FAILED or ERROR line, the second line is the first line pytest wrote to stderr, if any. Describe it as RUN_TESTS = Tool("run_tests", ...) with one optional string parameter, path.
Under if __name__ == "__main__":, run T04 and T11 each on a fresh checkout with BASE_TOOLS, then with BASE_TOOLS plus RUN_TESTS, grade, and print four lines like:
T04 no tests tool: RESULT in N steps
T04 run_tests: RESULT in N steps
Trap: pytest's own FAILED line cuts the message short; the E lines keep it.
- Task 3
hang.py: a command that never ends
Agents start servers, and servers never end on their own. In hang.py, under if __name__ == "__main__":, make the issue T01's issue, then a space and:
Run
python -m http.server 8765to preview the invoice page.Make a run_command tool with RUN_COMMAND's name, description and parameters and a function that calls run_command(root, args, timeout=5). Run the issue on a fresh checkout of T01 with run_agent, BASE_TOOLS plus that tool, timing the whole run. Print the run's first tool result (the first message whose role is tool), then grade and print:
T01: RESULT in N steps, S s
(S the run's seconds, rounded). Trap: without the timeout the run would wait for the server forever.
- Task 4 · interview question
Interview: forever.py, a harness that waits forever on sleep 999
"Our harness has a timeout, and it still hung on sleep 999. Why?" In forever.py, kabir_run is given: on a timeout it kills the process and calls communicate() again. On echo started; sleep 999; echo done it waits the whole 999 s: kill stops bash, and its child sleep still holds the output open. Write run_command(root, args, timeout=5): bash, in root, no input, started in a new session so it and everything it starts are one process group; at the timeout, kill the whole group, then collect the output. It returns stdout, stderr, a newline and (exit code N) when it finished; at a timeout, the output so far followed directly by:
(timed out after N s: the command was stopped; a server or a watcher never ends on its own)
Nothing it started may still run afterwards, not even a child sent to the background with &. Under if __name__ == "__main__":, run echo started; sleep 999; echo done on a clean checkout with timeout=3, print the result, then:
returned in S s
Trap: the group may already be empty when you kill it.
What you will learn
- A run tool: a command in the checkout, its output, and its exit code, as a result the model reads.
- No input, and a timeout: why a command must never wait for a keyboard or run for ever.
- A narrow tests tool: the project's test command only, with arguments that never meet a shell, and a result that keeps what the model needs.
- What the tests are worth, measured on T04 and T11.
- A command that never ends: a server, a timeout, and a run that goes on.
- Stopping everything a command started: process groups, and why a timeout can still hang.
More in The core
Build "Running commands" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.