Harness engineering (TypeScript) · Chapter 14 of 23

Sandbox and dry runs

By the end of this chapter you can say what a command the model runs may touch and build it that way: the checkout as its folder, an environment with nothing secret in it, a timeout that stops everything the command started, and a cap on what comes back. You can also run the agent in a dry run, where nothing in the checkout changes and the run ends with a diff, and apply that diff later, only if the files have not moved on. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Control
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Amit looks after Hisaab's servers, the payment gateway's secret and the bill. Chapter 13 gave him rules: rm * denied, test/** read-only, .env never read. He likes the rules, and he still has two worries.

The first is what a command can reach after a rule lets it through. npm test is allowed, and it is a program like any other: it runs with everything your harness has. Your harness has the lab model's key in OPENAI_API_KEY, and like many apps it reads .env for its own settings, so it also has HISAAB_GATEWAY_KEY_SECRET. Every command it starts inherits both. A test file, a package's install script or a line the model was tricked into running can print them, or send them anywhere.

The second is Farah's request. Before the agent touches the real repository she wants to see what it would change, read the diff, and only then say yes. The harness you have cannot do that: every edit_file writes the file the moment the model asks, and every command runs for real. A rule says yes or no to one call; it cannot say "do all of it, but on paper".

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    clean-env.ts: what a command may touch

    Amit wants every command the model runs kept away from the harness's secrets. In clean-env.ts, export sandboxEnv(root): a new object with only PATH, LANG and TERM from process.env (those it has) and HOME set to root. Export runSandboxed(root, command, { timeout = 30, cap = 4000 }), resolving to the text the model reads: run the command with bash -c in the checkout, with that environment, stdout and stderr together; past timeout seconds, stop it and everything it started and resolve to a text saying it timed out; else shape(output, cap) (from agent.ts) and then "\n(exit code N)". Export class Sandboxed extends Workspace, whose runCommand uses runSandboxed.

    Under if (import.meta.main), make a checkout of T08, load its .env into process.env, run env | cut -d= -f1 with new Workspace(root).runCommand and with new Sandboxed(root).runCommand, and print the names with KEY, SECRET, TOKEN or PASSWORD in them, sorted and comma-separated (or none):

    plain: HISAAB_GATEWAY_KEY_ID, ...

    sandboxed: none

    Then run runAgent on T08's issue with new Sandboxed(root), grade it and print T08: passed (5 steps), with the run's steps.

    Careful: bash starts the command as a second process, and a timeout that kills only bash (spawnSync's does) leaves it running.

  2. Task 2

    dry-run.ts: changes on paper

    Farah wants to see what the agent would change before it changes anything. In dry-run.ts, export isReadOnly(command): true when the command is, or starts with followed by a space, one of npm test, node --test, ls, cat, head, tail, grep, wc, git diff, git status, git log, git show, and has none of ; & | < > ` $ or a newline in it.

    Export class DryRun extends Workspace with overlay (a Map: path relative to the checkout to text) and refused (an array). text(path) returns the overlay's text if the run changed the file, else the file's; save(path, text) puts the text in the overlay and never writes the checkout; files() also lists the overlay's new files. runCommand(command): not read-only, push it to refused and return {"error": "dry run: ..."} as JSON, saying why; read-only, run it in a temporary copy of the checkout with the overlay written into it. diff(): unifiedDiff (from agent.ts) of every overlay file against the checkout, named a/PATH and b/PATH.

    Under if (import.meta.main), make a checkout of T09, print checkout: PATH, run runAgent on T09's issue in a DryRun with { system: NAIVE_PROMPT }, then print refused: COMMAND for each refused command, the diff, and last checkout diff: N lines (the lines of diff(root)).

    Careful: key the overlay by the resolved path relative to the checkout, so ./src/upi.ts and src/upi.ts are one file.

  3. Task 3

    apply.ts: apply a dry run later

    In apply.ts, export savePlan(dry, file): write a JSON file with one entry per file whose text changed in the dry run, { "PATH": { "before": the checkout's text, or null for a new file, "after": the overlay's text } }, and return the array of those paths. Export applyPlan(root, file): read the plan and check every file first; if any file's text in root is not its "before", write nothing and throw an Error naming those files. Else write every "after" and return the array of paths written.

    Under if (import.meta.main), make a checkout of T01, run runAgent on T01's issue in a DryRun (from ./dry-run.ts), save the plan to a file next to the checkout (not inside it), and print, with the grades from grade(root):

    plan: src/money.ts

    before apply: T01 failed

    applied: src/money.ts

    after apply: T01 passed

    Careful: a plan written over a file someone changed since the dry run undoes their work. Check every file before writing any.

  4. Task 4 · interview question

    Interview: leak.ts, a dry run that still writes

    "Your dry run passed, and the checkout changed. How?" teammate.ts (given) has a teammate's dry run: edits in an overlay, commands with rm, mv or > refused, every other command run in the checkout. In leak.ts, export LEAKS: at least three commands with no rm, mv or > in them, each of which changes a file in the checkout when TeammateDryRun runs it.

    Under if (import.meta.main), for each command in order, run it with new TeammateDryRun(root).runCommand on a fresh clean checkout (checkout(null)) and with your DryRun on another, and say whether each checkout changed (diff(root) is not empty). Print the teammate's result, yours and the command, two spaces apart:

    leaked held touch notes.txt

    then the totals:

    TeammateDryRun: 3 of 3 leaked, DryRun: 0 of 3

    Careful: one fresh checkout per command and per dry run, or one command's change shows up as the next one's.

What you will learn

  • What a command may touch: its folder, its environment, its time and its output.
  • Why an allowlist beats a filter for the environment, measured on the lab's own variables.
  • A timeout that stops a command's children too, not just the shell.
  • A dry run: an overlay the tools read and write, commands that could change files refused, a diff at the end.
  • Applying a dry run later, and refusing when the files changed since.

More in Control

Build "Sandbox and dry runs" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.