Harness engineering (TypeScript) · Chapter 2 of 23

The agent loop

By the end of this chapter you can turn chapter 1's loop into the core of a harness: tools described as data and run by a registry that never throws, a loop that stops for a reason it can name, a result object that says what happened, and a trail of the run that a person can read in ten seconds. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The core
  • 4 tasks
  • TypeScript
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Chapter 1's harness fixed T01 with three tools and a loop. Every chapter after this one adds to it: a paged reader, a search, an edit tool, a test runner, rules, hooks, a verify step. Kabir has seen what happens to a loop that grows by pasting another if (name === ...) into it: by the tenth tool nobody can tell which tools exist, and one bad call takes the whole run down.

Chapter 1's loop has three weaknesses for that:

  • The tools are an if chain. Adding one means editing the loop, and the array the model reads and the code that runs can drift apart.
  • One bad call ends everything. A path that does not exist throws ENOENT, the promise rejects, and the conversation, the steps and what was already changed are gone with it. Real models send bad calls: a misspelled path, a tool from another harness, arguments cut off when they ran out of output tokens.
  • It cannot say why it stopped. It returns an answer or null. null could mean "ran out of steps" or "the API refused the request", and Nandini cannot count failures she cannot tell apart.

The lesson

A tool is data

A tool has two halves: what the model reads (its name, a sentence and its parameters as JSON Schema) and what your code runs. Chapter 1 kept them in two places, the TOOLS array and the if chain. Keep them in one object instead, a Tool with four fields: name, description, parameters and fn. The function takes the checkout's folder and the model's arguments as one object, and returns text (or a promise of text). Here is one that is not in the tasks:

const countLines: Tool = {
  name: 'count_lines',
  description: 'How many lines a file of the repository has.',
  parameters: { type: 'object', properties: { path: { type: 'string' } }, required: ['path'] },
  fn: (root, args: { path: string }) => String(readFileSync(join(root, args.path), 'utf8').split('\n').length - 1),
};

A registry holds the tools by name. It does two jobs for the loop: schemas() gives the array the API takes as tools (each one wrapped as { type: 'function', function: {...} }, typed ChatCompletionTool[]), and run(root, name, args) runs a call. Adding a tool is one add, and the loop never changes again.

A registry never throws

run gets the call as the model wrote it: a name and the arguments as JSON text. Three things can go wrong before your tool even starts, and the registry answers each one with a result instead of an exception:

> await registry.run(root, 'delete_file', '{"path": "src/upi.ts"}')
'error: there is no tool called delete_file. The tools are: list_files, read_file, write_file.'
> await registry.run(root, 'read_file', '{"path": "src/upi.ts"')
'error: the arguments of read_file are not a JSON object: {"path": "src/upi.ts"'
> await registry.run(root, 'read_file', '{"path": "src/upi.tss"}')
"error: read_file failed: Error: ENOENT: no such file or directory, open '/tmp/hisaab-runs/clean-ez4mJ6/src/upi.tss'"
  • An unknown tool says which tools there are.
  • Arguments that are not a JSON object (cut off, null, or an array) say so, with the text that came in. Empty text means no arguments.
  • Anything fn throws comes back as the error's name and message.

Each of these goes back to the model as the call's result, and the run goes on: the model can try another path, and the harness keeps everything it has done so far. This is the most important rule of a tool layer: an error is a result. Chapter 5 makes those results helpful enough that the model fixes its call; this chapter makes sure the run survives them.

The loop and its stop reasons

The loop is chapter 1's: send the messages and registry.schemas(), push the model's message, run every call through the registry, push one tool message per call, and ask again. What changes is how it ends. It always ends for one of three reasons, and it says which:

StopWhenanswer
'answered'the model replied with text and no tool callsthe text
'max_steps'it used every step it was allowed and is still asking for toolsnull
'error'the API refused a request (the openai package throws an OpenAI.APIError)null, the error's message in error

A step is one request to the model, the one that failed included. The run resolves to a RunResult: answer, steps, stop, messages and error. Type stop as the union of the three strings, and TypeScript holds every caller to them. Keep the whole conversation in messages (ChatCompletionMessageParam[]), the answering message last, so the result is the whole run and the next chapters can read, count and replay it.

Catching the API's error inside the loop matters as much as the registry's: a run that dies on its twelfth request still changed files, and the harness needs its messages and its steps to say what happened. Catch only around the request, and only OpenAI.APIError: anything else is a bug in the harness, and should be loud.

A trail of the run

A RunResult holds everything, and its messages are hard to read: one write_file call carries the whole file. A trail is the run in one line per step: the step, the tool and its arguments, and the size of what came back. Here is T02's, under this chapter's loop and chapter 1's three tools:

1  list_files()  -> 24 lines
2  read_file(path="src/words.ts")  -> 54 lines
3  write_file(path="src/words.ts", content=<1937 chars>)  -> ok
4  answer: Done: numberInWords now treats exactly one crore as a crore ...
stop: answered after 4 steps

Long string arguments are shown by their size, results by their line count (a one-line result, such as ok or an error, as itself), and the answer by its first line. Every coding assistant keeps something like this, because the first question about a failed run is always "what did it do, step by step?". Chapter 19 turns it into a trace with tokens, seconds and cost.

Measured: the same loop on four tasks

Nandini ran the loop with chapter 1's three tools on four tasks, each in a fresh checkout:

TaskmaxStepsStopStepsLast promptGrade
T0110answered52,551 tokenspassed
T0210answered41,825passed
T0310answered44,245failed (gst.ts written back with 39 of its 234 lines)
T1010max_steps107,458failed
T1020error127,819, then refusedpassed
  • 'answered' is not "fixed": T03 answered "Done" and failed the grade.
  • T10 renames a function used in three modules. With no search, the model reads file after file. At ten steps it stops mid-way. At twenty, its twelfth request is over the 8,192-token window, and the API says This model's maximum context length is 8192 tokens. However, your messages resulted in 8240 tokens.
  • The run that died passed: all three callers were already fixed when the request was refused. Only the grade knows; the stop reason says how the run ended, never whether the work is right.

That is why the harness reports both, and why every later chapter judges a change by the grade on fresh checkouts. Chapter 3 gives the model a search, and T10 stops reading every file.

The order the API checks

The API checks the shape of every conversation before the model sees it. An assistant message with tool_calls must be followed by one tool message for each of its call ids, and a tool message must answer a call made before it. Break either rule and the request is a 400:

400 Invalid parameter: messages with role 'tool' must be a response to a
preceeding message with 'tool_calls'.

The usual way to break it is a loop that pushes the tool results first and the model's message after them, or one that skips a call it could not run. A harness that resends the whole conversation on every request breaks every request after that one, so it is worth checking the order yourself, before sending, with an error that names the message. The interview question writes that check and tries it on a real run.

Your turn

Build the registry with chapter 1's three tools in task 1. Write the loop with its stop reasons and its RunResult in task 2, and run it on T02. Print T10's trail in task 3 and read it. In the interview question, check the order of a conversation the way the API does.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    registry.ts: tools as data, a registry that never throws

    Kabir wants every tool in one place. In registry.ts, export a type Tool, an object with four fields: name, description, parameters (the JSON Schema of its arguments) and fn (the function: fn(root, args) returns text, or a promise of text; args is the object the model sent).

    Then export a class Registry: new Registry(tools) takes an array of Tools; add(tool) adds one; schemas() returns them in order as the API takes them, each one { type: 'function', function: { name, description, parameters } }; async run(root, name, args) runs a call, where args is the JSON text the model sent, and resolves to the tool's text. run never throws. Instead it returns:

    error: there is no tool called NAME. The tools are: A, B, C.

    for an unknown tool (the names in order, a comma and a space between them);

    error: the arguments of NAME are not a JSON object: TEXT

    when the text is not JSON, or is JSON but not an object (null and arrays are not objects; an empty text means no arguments);

    error: NAME failed: ERROR NAME: MESSAGE

    when fn throws (the error's name, like TypeError, and its message).

    Last, export hisaabRegistry(): a Registry of chapter 1's three tools, from the given repo-tools.ts: list_files (no parameters), read_file (path, required) and write_file (path and content, both required).

  2. Task 2

    loop.ts: the loop, its stop reasons and a RunResult

    In loop.ts, export a type RunResult, { answer, steps, stop, messages, error }, and export async function runAgent(root, issue, registry, maxSteps = 10, model = 'lab-chat').

    runAgent sends model a system message saying it is Hisaab's coding agent and the issue as the user's message, with registry.schemas() as the tools; pushes the model's message; runs every call with await registry.run(root, name, args); pushes one tool message per call; and asks again. A step is one request. It resolves to a RunResult with stop:

    'answered': the model replied with no tool calls (answer is its text);

    'max_steps': maxSteps requests and still no answer (answer null, steps maxSteps);

    'error': a request threw an OpenAI.APIError (answer null, error the error's message, steps counting that request).

    messages is the whole conversation, with the answering message last; error is null unless the stop is 'error'.

    Under if (import.meta.main), run it on a checkout of T02 with hisaabRegistry() and print:

    T02: STOP after N steps, passed

    (or failed, from the checkout's grade).

  3. Task 3

    log.ts: a trail of the run

    In log.ts, export trail(result): an array of lines, one per tool call and one for the answer, read from result.messages, then a last line. Number the steps from 1 (one per assistant message; every call of a message gets that message's number). Two spaces separate the parts:

    STEP NAME(ARGS) -> RESULT

    ARGS: each argument as key=value, value as JSON.stringify writes it, a comma and a space between them; a string value longer than 40 characters is shown as key=<N chars>. RESULT: a result of exactly one line is that line, cut to its first 60 characters with ... after when longer; any other result is N lines (a final newline does not start another line, and an empty result is 0 lines).

    STEP answer: FIRST LINE OF THE ANSWER (cut to 60 characters the same way)

    When result.stop is 'error', a line STEP error: THE ERROR (cut to 80 characters), STEP being result.steps. The last line is:

    stop: STOP after N steps

    Under if (import.meta.main), run runAgent on a checkout of T10 with hisaabRegistry() and maxSteps 20, print the trail, then T10: passed or T10: failed.

  4. Task 4 · interview question

    Interview: wrong-order.ts, the order the API checks

    "Why does my agent get a 400 on its second request?" In wrong-order.ts, export checkOrder(messages): null when the API will accept the messages' order, else the first problem, naming the message by its index in the array:

    message I: a tool result with no call before it

    for a tool message whose tool_call_id is not a call still waiting for its result;

    message I: calls with no result: ID, ID

    when an assistant message's calls are not all answered before the next message that is not a tool message, or before the end (I is that assistant message, the ids that are still waiting in the order it asked for them).

    Under if (import.meta.main), run runAgent (from ./loop.ts) on a checkout of T02 with hisaabRegistry(), and print the run's messages: ok (or checkOrder's text). Then take the run's first four messages, swap the last two (the tool result now comes before the call), and print the wrong order: followed by checkOrder's text. Send those four messages to lab-chat with the registry's tools and print the API says: and the status code (200 when it answers, else the error's status).

What you will learn

  • A tool as data: a name, a description, a JSON Schema and a function, in one object.
  • A registry that offers the tools, runs a call from its JSON text and turns every failure into a result the model reads.
  • Stop reasons: answered, max_steps and error, and a RunResult that carries them.
  • A trail: one line per step, the first thing to read when a run goes wrong.
  • The order the API checks: why a tool result before its call is a 400.

More in The core

Build The agent loop yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.