Harness engineering · Chapter 2 of 23

The agent loop

By the end of this chapter you can turn chapter 1's loop into the core of a harness: tools described as data and run by a registry that never crashes, a loop that stops for a reason it can name, a result object that says what happened, and a trail of the run that a person can read in ten seconds. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: The core
  • 4 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

Chapter 1's harness fixed T01 with three tools and a loop. Every chapter after this one adds to it: a paged reader, a search, an edit tool, a test runner, rules, hooks, a verify step. Kabir has seen what happens to a loop that grows by pasting another if name == ... into it: by the tenth tool nobody can tell which tools exist, and one bad call takes the whole run down.

Chapter 1's loop has three weaknesses for that:

  • The tools are an if chain. Adding one means editing the loop, and the list the model reads and the code that runs can drift apart.
  • One bad call ends everything. A path that does not exist raises FileNotFoundError, the run crashes, and the conversation, the steps and what was already changed are gone with it. Real models send bad calls: a misspelled path, a tool from another harness, arguments cut off when they ran out of output tokens.
  • It cannot say why it stopped. It returns an answer or None. None could mean "ran out of steps" or "the API refused the request", and Nandini cannot count failures she cannot tell apart.

The lesson

A tool is data

A tool has two halves: what the model reads (its name, a sentence and its parameters as JSON Schema) and what your code runs. Chapter 1 kept them in two places, the TOOLS list and the if chain. Keep them in one object instead, a Tool with four fields: name, description, parameters and fn. The function takes the checkout's folder first, then the model's arguments by name, and returns text. Here is one that is not in the tasks:

def count_lines(root, path):
    return str(len((Path(root) / path).read_text().splitlines()))

Tool("count_lines", "How many lines a file of the repository has.",
     {"type": "object", "properties": {"path": {"type": "string"}},
      "required": ["path"]}, count_lines)

A registry holds the tools by name. It does two jobs for the loop: schemas() gives the list the API takes as tools= (each one wrapped as {"type": "function", "function": {...}}), and run(root, name, arguments) runs a call. Adding a tool is one add, and the loop never changes again.

A registry never raises

run gets the call as the model wrote it: a name and the arguments as JSON text. Three things can go wrong before your tool even starts, and the registry answers each one with a result instead of an exception:

>>> registry.run(root, "delete_file", '{"path": "hisaab/upi.py"}')
'error: there is no tool called delete_file. The tools are: list_files, read_file, write_file.'
>>> registry.run(root, "read_file", '{"path": "hisaab/upi.py"')
'error: the arguments of read_file are not a JSON object: {"path": "hisaab/upi.py"'
>>> registry.run(root, "read_file", '{"path": "hisaab/upi.pyy"}')
"error: read_file failed: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/hisaab-runs/clean-a2yyzhct/hisaab/upi.pyy'"
  • An unknown tool says which tools there are.
  • Arguments that are not a JSON object (cut off, or a list) say so, with the text that came in. Empty text means no arguments.
  • Anything the tool raises comes back as its type and message.

Each of these goes back to the model as the call's result, and the run goes on: the model can try another path, and the harness keeps everything it has done so far. This is the most important rule of a tool layer: an error is a result. Chapter 5 makes those results helpful enough that the model fixes its call; this chapter makes sure the run survives them.

The loop and its stop reasons

The loop is chapter 1's: send the messages and registry.schemas(), append the model's message, run every call through the registry, append one tool message per call, and ask again. What changes is how it ends. It always ends for one of three reasons, and it says which:

StopWhenanswer
answeredthe model replied with text and no tool callsthe text
max_stepsit used every step it was allowed and is still asking for toolsNone
errorthe API refused a request (the openai package raises an openai.APIError)None, the error's text in error

A step is one request to the model, the one that failed included. The run returns a RunResult: the answer, the steps, the stop, the messages and the error. Keep the messages as plain dictionaries (the model's own message can dump itself: message.model_dump(exclude_none=True)), with the answering message last, so the result is the whole run and the next chapters can read, count and replay it.

Catching the API's error inside the loop matters as much as the registry's: a run that dies on its fourteenth request still changed files, and the harness needs its messages and its steps to say what happened.

A trail of the run

A RunResult holds everything, and its messages are hard to read: one write_file call carries the whole file. A trail is the run in one line per step: the step, the tool and its arguments, and the size of what came back. Here is T02's, under this chapter's loop and chapter 1's three tools:

1  list_files()  -> 26 lines
2  read_file(path="hisaab/words.py")  -> 54 lines
3  write_file(path="hisaab/words.py", content=<1789 chars>)  -> ok
4  answer: Done: number_in_words now treats exactly one crore as a cror...
stop: answered after 4 steps

Long text arguments are shown by their size, results by their line count (a one-line result, such as ok or an error, as itself), and the answer by its first line. Every coding assistant keeps something like this, because the first question about a failed run is always "what did it do, step by step?". Chapter 19 turns it into a trace with tokens, seconds and cost.

Measured: the same loop on four tasks

Nandini ran the loop with chapter 1's three tools on four tasks, each in a fresh checkout:

Taskmax_stepsStopStepsLast promptGrade
T0110answered52,491 tokenspassed
T0210answered41,857passed
T0310answered44,275failed (gst.py written back with 37 of its 230 lines)
T1010max_steps107,173failed
T1020error148,071, then refusedpassed
  • "answered" is not "fixed": T03 answered "Done" and failed the grade.
  • T10 renames a function used in three modules. With no search, the model reads file after file. At ten steps it stops mid-way. At twenty, its fourteenth request is over the 8,192-token window, and the API says This model's maximum context length is 8192 tokens. However, your messages resulted in 8684 tokens.
  • The run that died passed: all three callers were already fixed when the request was refused. Only the grade knows; the stop reason says how the run ended, never whether the work is right.

That is why the harness reports both, and why every later chapter judges a change by the grade on fresh checkouts. Chapter 3 gives the model a search, and T10 stops reading every file.

The order the API checks

The API checks the shape of every conversation before the model sees it. An assistant message with tool_calls must be followed by one tool message for each of its call ids, and a tool message must answer a call made before it. Break either rule and the request is a 400:

Invalid parameter: messages with role 'tool' must be a response to a
preceeding message with 'tool_calls'.

The usual way to break it is a loop that appends the tool results first and the model's message after them, or one that skips a call it could not run. A harness that resends the whole conversation on every request breaks every request after that one, so it is worth checking the order yourself, before sending, with an error that names the message. The interview question writes that check and tries it on a real run.

Your turn

Build the registry with chapter 1's three tools in task 1. Write the loop with its stop reasons and its RunResult in task 2, and run it on T02. Print T10's trail in task 3 and read it. In the interview question, check the order of a conversation the way the API does.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    registry.py: tools as data, a registry that never raises

    Kabir wants every tool in one place. In registry.py, write a dataclass Tool with four fields, in this order: name, description, parameters (the JSON Schema of its arguments) and fn (the function: it takes the checkout's folder, then the arguments by name, and returns text).

    Then a class Registry: Registry(tools) takes a list of Tools; add(tool) adds one; schemas() returns them in order as the API takes them, each one {"type": "function", "function": {"name": ..., "description": ..., "parameters": ...}}; run(root, name, arguments) runs a call, where arguments is the JSON text the model sent, and returns the tool's text. run never raises. Instead it returns:

    error: there is no tool called NAME. The tools are: A, B, C.

    for an unknown tool (the names in order, a comma and a space between them);

    error: the arguments of NAME are not a JSON object: TEXT

    when the text is not JSON, or is JSON but not an object (an empty text means no arguments);

    error: NAME failed: TYPE: MESSAGE

    when the function raises (the exception's class name and its text).

    Last, hisaab_registry() returns a Registry of chapter 1's three tools, from the given repo_tools.py: list_files (no parameters), read_file (path, required) and write_file (path and content, both required).

  2. Task 2

    loop.py: the loop, its stop reasons and a RunResult

    In loop.py, write a dataclass RunResult with the fields answer, steps, stop, messages and error (error defaults to None), and run_agent(root, issue, registry, max_steps=10, model="lab-chat").

    run_agent sends model a system message saying it is Hisaab's coding agent and the issue as the user's message, with registry.schemas() as the tools; appends the model's message; runs every call with registry.run(root, name, arguments); appends one tool message per call; and asks again. A step is one request. It returns a RunResult with stop:

    answered: the model replied with no tool calls (answer is its text);

    max_steps: max_steps requests and still no answer (answer None, steps max_steps);

    error: a request raised openai.APIError (answer None, error the exception's text, steps counting that request).

    messages is the whole conversation as dictionaries (the model's message as message.model_dump(exclude_none=True)), with the answering message last.

    Under if __name__ == "__main__":, run it on a checkout of T02 with hisaab_registry() and print:

    T02: STOP after N steps, passed

    (or failed, from the checkout's grade).

  3. Task 3

    log.py: a trail of the run

    In log.py, write trail(result): a list of lines, one per tool call and one for the answer, read from result.messages, then a last line. Number the steps from 1 (one per assistant message; every call of a message gets that message's number). Two spaces separate the parts:

    STEP NAME(ARGS) -> RESULT

    ARGS: each argument as key=value, value written as JSON (json.dumps), a comma and a space between them; a text value longer than 40 characters is shown as key=<N chars>. RESULT: a result of exactly one line is that line, cut to its first 60 characters with ... after when longer; any other result is N lines.

    STEP answer: FIRST LINE OF THE ANSWER (cut to 60 characters the same way)

    When result.stop is error, a line STEP error: THE ERROR (cut to 80 characters), STEP being result.steps. The last line is:

    stop: STOP after N steps

    Under if __name__ == "__main__":, run run_agent on a checkout of T10 with hisaab_registry() and max_steps=20, print the trail, then T10: passed or T10: failed.

  4. Task 4 · interview question

    Interview: wrong_order.py, the order the API checks

    "Why does my agent get a 400 on its second request?" In wrong_order.py, write check_order(messages): None when the API will accept the messages' order, else the first problem, naming the message by its index in the list:

    message I: a tool result with no call before it

    for a tool message whose tool_call_id is not a call still waiting for its result;

    message I: calls with no result: ID, ID

    when an assistant message's calls are not all answered before the next message that is not a tool message, or before the end (I is that assistant message, the ids that are still waiting in the order it asked for them).

    Under if __name__ == "__main__":, run run_agent (from loop.py) on a checkout of T02 with hisaab_registry(), and print the run's messages: ok (or check_order's text). Then take the run's first four messages, swap the last two (the tool result now comes before the call), and print the wrong order: followed by check_order's text. Send those four messages to lab-chat with the registry's tools and print the API says: and the status code (200 when it answers, else the error's status_code).

What you will learn

  • A tool as data: a name, a description, a JSON Schema and a function, in one object.
  • A registry that offers the tools, runs a call from its JSON text and turns every failure into a result the model reads.
  • Stop reasons: answered, max_steps and error, and a RunResult that carries them.
  • A trail: one line per step, the first thing to read when a run goes wrong.
  • The order the API checks: why a tool result before its call is a 400.

More in The core

Build The agent loop yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.