Harness engineering (TypeScript) · Chapter 7 of 23
Shaping results
By the end of this chapter you can give every tool result a budget, count it in tokens the way the API does, cut a long output to its head and tail with a marker that says what is missing, summarise test output down to what failed and why, and measure on T08 the difference between a run that dies and one that passes. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The core
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Some shop owners pay late, and Hisaab charges them for it: 18% a year, for every day late. T08 says the fees are a day short, and its issue says how to see them: "Run node src/reminders.ts to see the fee for every day." That command prints a fee for every day, for three sample invoices: 1,200 lines.
The lab model does what the issue asks first. With chapter 6's run tool, which sends back everything a command prints, the run ends at its second request:
raw: the run stopped at step 2: 400 This model's maximum context length
is 8192 tokens. However, your messages resulted in 27342 tokens. Please
reduce the length of the messages.
No test failed and no edit went wrong. One tool result, 26,899 tokens, was more than three times the whole context window, and every later request would have carried it. Amit adds that even with a bigger window the harness would pay for those tokens on every request for the rest of the run (chapter 1: each request resends everything). The model needed perhaps five of those lines.
A coding agent's tools print far more than it can read: build logs, test runs, a long file, a search that matched everywhere. What goes back to the model is the harness's choice, and this chapter makes it on purpose.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
shape.ts: a budget for every result
In shape.ts export countTokens(text: string): number (o200k tokens, with js-tiktoken) and shape(text: string, budget = 1000): string. A text within the budget comes back unchanged. Otherwise split it at '\n' into lines and keep a head and a tail of whole lines: the head takes lines from the top while their tokens (each line counted alone) stay within Math.floor(budget / 2); the tail does the same from the bottom, among the lines after the head. Between them goes one marker line:
[... N lines cut (T tokens): narrow the command with grep, head or tail to see them ...]
with N the number of cut lines and T the tokens of the cut lines joined with '\n'. Join head, marker and tail with '\n'.
Trap: stop at the first line that does not fit; never skip it for a shorter one further on.
- Task 2
test-summary.ts: a summary of test output
In test-summary.ts export summariseTests(output: string): string for the spec reporter's output. It returns, joined with '\n': the lines starting with "ℹ pass " or "ℹ fail ", then, for each failing test in the section after the line "✖ failing tests:", its line starting with "✖ " without the time at its end (" (1.2ms)"), and the lines after it up to the first whose text starts with "at " (the stack), trailing spaces removed, blank ones left out. A passing run is just its two ℹ lines.
Under if (import.meta.main), put the model's first try into a T11 checkout (replace " return Array(parts).fill(share);\n" with " return [...Array(parts - 1).fill(share), paise - share * (parts - 1)];\n" in src/money.ts), run node --test --test-reporter=spec there, print the summary, then:
raw: N tokens, summary: M tokens
Trap: in that section each failure starts with a "test at ..." line; it is not part of the message before it.
- Task 3
t08.ts: T08, raw and shaped
T08's issue says to run node src/reminders.ts, which prints 1,200 lines. In t08.ts export shapedRun(root: string, args: Args): string: run args.command with run(root, command) from ./harness.ts; the body is summariseTests(output) if the command has npm test or node --test in it, else shape(output); return the body, a newline and (exit code N). Export SHAPED_RUN: RAW_RUN with fn shapedRun.
Under if (import.meta.main), run T08's issue on a fresh checkout with BASE_TOOLS plus RAW_RUN (raw), then with BASE_TOOLS plus SHAPED_RUN (shaped). For a run whose stop is 'error', print:
LABEL: the run stopped at step N: MESSAGE
(MESSAGE is its answer, the API's error). Otherwise grade it and print:
LABEL: RESULT in N steps, largest prompt T tokens
Trap: the raw run fails on no test: one result is bigger than the whole context window.
- Task 4 · interview question
Interview: head-only.ts, a shaper that loses the failure
"We cut every output to its first 2,000 characters, and now the agent cannot fix failing tests. Why?" In head-only.ts, kabirShape, TAP (the system prompt with node --test --test-reporter=tap) and runTool(shaper) are given. Export shapeKeeping(text: string, budget = 1000): string: the same as shape, except that from the cut middle it keeps every failing test's report (from a line starting with "not ok " up to, not including, the first line after it whose text starts with "code:" or "stack:" or is "...") and every line starting with "✖ ", right after the marker, in order, before the tail. A text within the budget comes back unchanged.
Under if (import.meta.main), run T11 four times on fresh checkouts with BASE_TOOLS plus runTool(SHAPER): head only, tap (kabirShape, system TAP); head and tail, tap (shape, TAP); head and tail, spec (shape, SYSTEM); keeping failures, tap (shapeKeeping, TAP). Grade each and print lines like:
head only, tap: RESULT in N steps
Trap: TAP reports a failure where the test ran, in the middle; the kept lines may go past the budget, and should.
What you will learn
- A budget per result, counted in tokens with js-tiktoken (o200k, as the lab model counts).
- Head and tail with a marker: what to keep of a long output, and how to say what was cut.
- Summaries of test output: the counts first, then each failure and its reason.
- What shaping is worth, measured on T08: raw, the run dies; shaped, it passes.
- What not to cut: a shaper that keeps only the head, a reporter that puts the failure in the middle, and the failure they lose.
More in The core
Build "Shaping results" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.