Harness engineering (TypeScript) · Chapter 6 of 23

Running commands

By the end of this chapter you can give a coding agent a tool that runs commands in its checkout and reports how they went, a narrow tool that runs only the tests and answers with what the model needs, a timeout that turns a command that never ends into a result, and a way to stop a command together with everything it started. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The core
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.

Why you need this

A coding agent that cannot run anything has to believe itself. Here is the lab model on T11, splitEvenly(1000, 3) returning [333, 333, 333], with the harness so far (read, search, edit, and no way to run a command):

search_code(splitEvenly)
read_file(src/money.ts, offset 49)
edit_file(...)  -> edited src/money.ts
answers: Done: splitEvenly now splits with a quotient and a remainder,
so the parts add up and the first ones get the extra paise (src/money.ts).

T11 fails. Its first fix is wrong, and its answer describes the right one. T04 goes the same way: the answer says roundHalfUp, the code it wrote does not use it. Real coding models also write a plausible first fix that a test would have caught, and the system prompt already tells this one to run the tests: it has no tool to do it with.

Nandini wants the agent to run the tests. Amit wants to know what else it can run, in which folder, with which input, and for how long, because a command the model starts is a process on his machine. Both are harness decisions, and this chapter makes them.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    run-tool.ts: a tool that runs commands

    In run-tool.ts export runCommand(root: string, args: Args, timeout = 30): string: it runs args.command with bash (bash -c), inside root, with no input (a command that reads its input gets end-of-file at once), stdout and stderr captured as text, stopped after timeout seconds. It returns stdout, then stderr, then a newline and:

    (exit code N)

    If the command is still running at the timeout, it returns only:

    (timed out after N s: the command was stopped; a server or a watcher never ends on its own)

    Then export RUN_COMMAND: Tool, named run_command, with fn runCommand and one required string parameter, command.

    Trap: a timeout and a non-zero exit code are results the model reads, not exceptions; and the timeout is in seconds, while Node counts milliseconds.

  2. Task 2

    tests-tool.ts: a narrow tool for the tests

    In tests-tool.ts export runTests(root: string, args: Args): string: inside root and with no input, it runs node --test --test-reporter=spec, with args.path added when it is given, as a list of arguments, never through a shell. It returns the output's lines that start with "ℹ pass " or "ℹ fail ", then (exit code N), then:

    first failure: TEST NAME

    and the lines of the first failure. TEST NAME comes from the first line starting with "✖ " after the line "✖ failing tests:" (without the "✖ " and its time, " (1.2ms)"); the failure lines are the ones after it, up to the first whose text starts with "at ", trailing spaces removed, blank ones left out. With no failing tests section, the line after the exit code is the first line node wrote to stderr, if any. Export RUN_TESTS: Tool, named run_tests, with one optional string parameter, path.

    Under if (import.meta.main), run T04 and T11 each on a fresh checkout with BASE_TOOLS, then with BASE_TOOLS plus RUN_TESTS, grade, and print four lines like:

    T04 no tests tool: RESULT in N steps

    T04 run_tests: RESULT in N steps

    Trap: the path is the model's text; it reaches node as one argument, never inside a shell line.

  3. Task 3

    hang.ts: a command that never ends

    Agents start servers and watchers, and they never end on their own. In hang.ts export ISSUE: T01's issue, then a space and:

    Run node --watch src/reminders.ts to watch the fees while you work.

    Under if (import.meta.main), make a run_command tool from RUN_COMMAND with a fn that calls runCommand(root, args, 5), run ISSUE on a fresh checkout of T01 with runAgent and BASE_TOOLS plus that tool, timing the whole run. Print the run's first tool result (the content of the first message whose role is tool), then grade and print:

    T01: RESULT in N steps, S s

    (S the run's seconds, rounded). Trap: without the timeout the run would wait for the watcher forever.

  4. Task 4 · interview question

    Interview: forever.ts, a harness that waits forever on sleep 999

    "Our harness has a timeout, and it still hung on sleep 999. Why?" In forever.ts, kabirRun is given: at the timeout it kills the child and resolves on its 'close' event. On echo started; sleep 999; echo done it waits the whole 999 s: the kill stops bash, and its child sleep still holds the output pipes open. Export runCommand(root: string, args: Args, timeout = 5): Promise<string>: bash, in root, no input, spawned detached so it and everything it starts are one process group; at the timeout, kill the whole group, then resolve on 'close'. It resolves to stdout, stderr, a newline and (exit code N) when it finished; at a timeout, the output so far followed directly by:

    (timed out after N s: the command was stopped; a server or a watcher never ends on its own)

    Nothing it started may still run afterwards, not even a child sent to the background with &. Under if (import.meta.main), run echo started; sleep 999; echo done on a clean checkout with a timeout of 3, print the result, then:

    returned in S s

    Trap: the group may already be gone when you kill it.

What you will learn

  • A run tool: a command in the checkout, its output, and its exit code, as a result the model reads.
  • No input, and a timeout: why a command must never wait for a keyboard or run for ever.
  • A narrow tests tool: the project's test command only, with arguments that never meet a shell, and a result that keeps what the model needs.
  • What the tests are worth, measured on T04 and T11.
  • A command that never ends: a watcher, a timeout, and a run that goes on.
  • Stopping everything a command started: process groups, and why a timeout can still hang.

More in The core

Build "Running commands" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.