Harness engineering · Chapter 7 of 23
Shaping results
By the end of this chapter you can give every tool result a budget, count it in tokens the way the API does, cut a long output to its head and tail with a marker that says what is missing, summarise test output down to what failed and why, and measure on T08 the difference between a run that dies and one that passes. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The core
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.
Why you need this
Some shop owners pay late, and Hisaab charges them for it: 18% a year, for every day late. T08 says the fees are a day short, and its issue says how to see them: "Run python -m hisaab.reminders to see the fee for every day." That command prints a fee for every day, for three sample invoices: 1,200 lines.
The lab model does what the issue asks first. With chapter 6's run tool, which sends back everything a command prints, the run ends at its second request:
raw: the run stopped at step 2: Error code: 400 - This model's maximum
context length is 8192 tokens. However, your messages resulted in 27350
tokens. ... 'code': 'context_length_exceeded'
No test failed and no edit went wrong. One tool result, 26,899 tokens, was more than three times the whole context window, and every later request would have carried it. Amit adds that even with a bigger window the harness would pay for those tokens on every request for the rest of the run (chapter 1: each request resends everything). The model needed perhaps five of those lines.
A coding agent's tools print far more than it can read: build logs, test runs, a long file, a search that matched everywhere. What goes back to the model is the harness's choice, and this chapter makes it on purpose.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
shape.py: a budget for every result
In shape.py write count_tokens(text) (o200k tokens, with tiktoken) and shape(text, budget=1000). A text within the budget comes back unchanged. Otherwise split it at "\n" into lines and keep a head and a tail of whole lines: the head takes lines from the top while their tokens (each line counted alone) stay within budget // 2; the tail does the same from the bottom, among the lines after the head. Between them goes one marker line:
[... N lines cut (T tokens): narrow the command with grep, head or tail to see them ...]
with N the number of cut lines and T the tokens of the cut lines joined with "\n". Join head, marker and tail with "\n".
Trap: stop at the first line that does not fit; never skip it for a shorter one further on.
- Task 2
test_summary.py: a summary of test output
In test_summary.py write summarise_tests(output) for pytest's output. It returns, joined with "\n": the summary (the last line that is not blank, with the = signs and spaces around it stripped), then for each failure in order: FAILED TEST_ID, and that failure's E lines (lines starting with E and a space), trailing spaces removed. A failure starts at a header line of underscores around its test's name (___ test_x ___) and ends at the next header or a line starting with =; TEST_ID is the id from pytest's FAILED lines that ends with ::NAME (the name alone if none). A passing run is just its summary line.
Under if __name__ == "__main__":, put the model's first try into a T11 checkout (replace " return [share] * parts\n" with " return [share] * (parts - 1) + [paise - share * (parts - 1)]\n" in hisaab/money.py), run python -m pytest -v there, print the summary, then:
raw: N tokens, summary: M tokens
Trap: the passing tests are most of the output and none of what the model needs.
- Task 3
t08.py: T08, raw and shaped
T08's issue says to run python -m hisaab.reminders, which prints 1,200 lines. In t08.py write shaped_run(root, args): run args["command"] with run(root, command) from harness.py; the body is summarise_tests(output) if the command has pytest in it, else shape(output); return the body, a newline and (exit code N). Describe it as SHAPED_RUN, with RAW_RUN's name, description and parameters.
Under if __name__ == "__main__":, run T08's issue on a fresh checkout with BASE_TOOLS plus RAW_RUN (raw), then with BASE_TOOLS plus SHAPED_RUN (shaped). For a run whose stop is "error", print:
LABEL: the run stopped at step N: MESSAGE
(MESSAGE is its answer, the API's error). Otherwise grade it and print:
LABEL: RESULT in N steps, largest prompt T tokens
Trap: the raw run fails on no test: one result is bigger than the whole context window.
- Task 4 · interview question
Interview: head_only.py, a shaper that loses the FAILED line
"We cut every output to its first 2,000 characters, and now the agent cannot fix failing tests. Why?" In head_only.py, kabir_shape, VERBOSE (the system prompt with python -m pytest -v) and run_tool(shaper) are given. Write shape_keeping(text, budget=300): the same as shape (head, marker, tail, the same rule and marker), except that the lines in the cut middle that start with "FAILED ", "ERROR " or E and a space are kept, right after the marker, in order, before the tail. A text within the budget comes back unchanged.
Under if __name__ == "__main__":, run T11 three times on fresh checkouts with run_agent(..., system=VERBOSE) and BASE_TOOLS plus run_tool(SHAPER): head only (kabir_shape), head and tail (shape(text, 300)), keeping failures (shape_keeping). Grade each and print lines like:
head only: RESULT in N steps
Trap: the kept lines go past the budget, and should: a budget is for noise, never for the reason a test failed.
What you will learn
- A budget per result, counted in tokens with tiktoken (o200k, as the lab model counts).
- Head and tail with a marker: what to keep of a long output, and how to say what was cut.
- Summaries of test output: the summary line first, then each failure and its reason.
- What shaping is worth, measured on T08: raw, the run dies; shaped, it passes.
- What not to cut: a shaper that keeps only the head, and the failure it loses.
More in The core
Build "Shaping results" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.