Harness engineering (TypeScript) · Chapter 5 of 23

Tool errors the model can act on

By the end of this chapter you can write a tool's errors as results the model can act on (what went wrong and what to do next), give an edit tool the four errors it needs, measure on T05 what one sentence of error text is worth, and keep a run alive when a tool throws. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The core
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Hisaab's invoices are adding up wrong for a printing press in Chowk. Its issue is T05: invoiceTotal ignores shipping, so an invoice with ₹150 of shipping comes out ₹150 short. The fix is one line. The agent finds it, reads it, and sends an edit whose old text is return total;.

Kabir looks at the file: src/invoice.ts has that text in two places. An exact-match edit tool must refuse it, because it cannot know which one the model means (chapter 4). The question is what the refusal says. Your edit tool so far says something like {"error":"error"}, and Nandini has measured what the model does with that: it tries the same edit again, gets the same error, and gives up. T05 fails on a bug the model had already found.

The model is not the problem. It is the harness's sentence that went back to it. A coding agent spends a good part of its run being told no: a path that does not exist, a text that is not there, a command that failed. Every one of those results is the model's only clue for its next step, and this chapter writes them.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    errors.ts: messages the model can act on

    A tool's error is a result the model reads before its next step, so it should say what went wrong and what to do. In errors.ts export five functions that return strings:

    error(message): the JSON text {"error": message}.

    notFound(path): old_text not found in PATH: read the file again and copy the text exactly, spaces included

    matchesMany(path, text, oldText), when oldText is in text N times: old_text matches N places in PATH (lines A, B, C): add the lines around it so it matches one

    with the line each match starts on, in order, each line named once.

    noSuchFile(root, path): no file PATH in the repository. Did you mean: X, Y, Z?

    naming up to three of the checkout's files (filesOf(root) from ./harness.ts) at an edit distance of at most 3 from PATH, nearest first, ties in alphabetical order (the edit distance counts the one-character inserts, deletes and changes that turn one text into the other); with none that near: no file PATH in the repository: list_files shows every file

    outside(path): PATH is outside the repository: use a path relative to it, like src/money.ts

    Trap: a match inside a longer line counts, and two matches on one line are two places but one line.

  2. Task 2

    edit-errors.ts: an edit tool with helpful errors

    In edit-errors.ts export editFile(root: string, args: Args): string, args being path, old_text and new_text. Check in this order, and return the error as error(...) from ./errors.ts: the path, resolved, leads outside root (outside); there is no such file, or it is a folder (noSuchFile); old_text is not in it (notFound); old_text is in it more than once (matchesMany). Otherwise replace that one place, write the file, and return:

    edited PATH at line N

    with N the line old_text started on. Then export EDIT_FILE: Tool, named edit_file, with fn editFile: the description says old_text must appear exactly once; path, old_text and new_text are required strings.

    Trap: on every error the file is unchanged; an edit never makes a new file, never changes a file outside root, even through "..", and writes new_text exactly as it is.

  3. Task 3

    t05-both.ts: a bare error against a helpful one

    Nandini wants the difference measured. In t05-both.ts export bareEdit(root: string, args: Args): string: the same edit, but every problem returns just {"error":"error"} (as JSON text), and a good edit returns ok.

    Under if (import.meta.main), run T05's issue twice with runAgent from ./harness.ts, each on a fresh checkout, with BASE_TOOLS plus an edit tool named edit_file: bare (bareEdit, with EDIT_FILE's description and parameters), then helpful (EDIT_FILE). After each run, grade the checkout and print:

    bare: RESULT in N steps

    (RESULT is passed or failed, N the run's steps), then the run's answer on the next line, indented two spaces.

    Trap: the two runs differ only in the error messages: same tool name, description and parameters.

  4. Task 4 · interview question

    Interview: crash.ts, a tool whose exception ends the run

    "A tool threw and the whole run died. Fix it." In crash.ts, strictEdit is given: it throws an Error on a problem, and runAgent does not catch exceptions. Export guarded(fn: Tool['fn']): Tool['fn']: it returns a tool function (root, args) that returns fn(root, args), or, if fn throws, the JSON text {"error":"Error: the message"} (the error's name, a colon and a space, its message).

    Under if (import.meta.main), run T05 on a fresh checkout with BASE_TOOLS plus editTool(strictEdit), catch what runAgent throws and print:

    unguarded: the run died (Error: the message)

    Then run it on a fresh checkout with editTool(guarded(strictEdit)), grade it and print:

    guarded: T05 passed in N steps

    (or failed). Trap: an error that hides the message keeps the run alive and still fails T05.

What you will learn

  • An error is a result: the model reads it the way it reads a file, and acts on it.
  • What a good error says: what happened, then what to do next, in words the model can follow.
  • The four errors of an edit tool: the text is not there, it is there more than once, the file does not exist, the path leads outside the repository.
  • What one sentence is worth, measured on T05: a bare error against a helpful one.
  • When a tool throws: why an exception ends the run, and why the error that replaces it must keep its message.

More in The core

Build "Tool errors the model can act on" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.