Harness engineering · Chapter 19 of 23
Tracing and cost
By the end of this chapter your harness writes a trace of every run (one line of JSON per model call, per tool call, and a grade at the end), turns the trace of a whole suite run into rupees per task and per success, and finds in a trace the step that broke a run. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Scale and measure
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.
Why you need this
Nandini runs the suite with this chapter's harness every morning. Today it says 10 of 12. Three questions follow, and the grade answers none of them:
- Farah: what did that cost, and what does each fix cost?
- Amit: why is the bill what it is? Which tasks, which requests?
- Kabir: why did T02 and T07 fail?
the tests fail: FAILED tests/test_words.py::test_croresays what is wrong with the checkout, not what the agent did.
In chapter 1 you watched one run step by step and saw gst.py come back 37 lines long. A suite run is 12 runs, 71 requests and 59 tool calls, every morning, and every chapter from here compares suite runs. Nobody watches that. What is not written down while it happens is gone when the run ends: the conversation lives in memory, and the checkout keeps only the last state of the files.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
tracer.py: a trace of a coding agent's run
Nandini wants to see inside every run, not just its grade. In tracer.py, write a class Tracer(path, run_id=None) (run_id: a new uuid4().hex[:8] when None) whose span(kind, task, **fields) adds one line of JSON, {"run", "task", "kind", ...fields}, to the end of the file and returns it; and read_spans(path, run_id=None), the file's spans in order (that run's only when run_id is given, [] with no file).
Then traced_run(root, task_id, tools, tracer, system=SYSTEM, model="lab-chat", max_steps=20): harness.py's loop on the task's issue, returning the same RunResult. After every request, a model span: step, model, prompt_tokens, cached_tokens (0 when usage has none), completion_tokens, total_tokens, seconds, asked (the tool names) and error None; a refused request gets a model span with every token count 0, asked [], error the error's text, and the run stops with stop error. After every tool call, a tool span: step, tool, args (a dictionary), result_chars, error (the result's "error" when it is JSON with one, else None), file (args["path"] or None), lines_before and lines_after (that file's line count in root around the call, None when there is no such file), seconds. At the end, a grade span: passed and reason from workbench.grade(root), steps and stop.
Under if __name__ == "__main__":, trace T05 into trace.jsonl with every repo tool and print its spans, one line each.
- Task 2
cost.py: rupees per task, and per success
Amit pays the bill. In cost.py, write prices(): {model: price_inr_per_million} for every model the lab model lists that has one (client.models.list(), in model_extra). Then rupees(span, table): (prompt_tokens - cached_tokens) at input, cached_tokens at cached_input and completion_tokens at output, per million tokens, for the span's model. Then report(spans, table): {"rows": one {"task", "rupees", "passed", "steps"} per task in the order they ran, "total", "passed", "per_success": total / passed, or None with no pass}.
Under if __name__ == "__main__":, run every task of workbench.TASKS in a fresh checkout with traced_run, every repo tool and one Tracer on suite.jsonl, then report that run's spans and print a line per task, then the totals:
T01 passed 5 steps ₹1.2345
total ₹15.0000, 11/12 passed, ₹1.3636 per success
(rupees with 4 decimals; those numbers only show the format).
Trap: suite.jsonl keeps every run you make: report this run's only.
- Task 3
where_it_broke.py: the step that broke a run
T03 fails under the naive harness, and the grade only says the tests fail. In where_it_broke.py, write first_break(spans): the first tool span, in order, that left a file with less than half its lines (lines_after * 2 < lines_before), or None; a span with no lines_before or lines_after is skipped.
Under if __name__ == "__main__":, trace T03 into t03.jsonl with traced_run, every tool of naive_tools(root) (naive.py, given) and system=NAIVE_SYSTEM. From that run's spans print:
T03: failed (the grade's reason)
broke at step S: TOOL FILE, B lines -> A
or no step shrank a file when first_break finds nothing.
- Task 4 · interview question
Interview: bill.py, a cost report that is too high
last_run.jsonl (given) is the trace of one run of the suite. Amit's bill for it, worked out from the usage the lab model reported, is far lower than what bill.py's run_cost says. Find what run_cost gets wrong and fix it, so that it returns what the run really cost in rupees: for every model span, its prompt tokens that were not cached at the input price, its cached tokens at the cached_input price and its completion tokens at the output price, per million tokens, by the span's model.
Keep run_cost(spans, table) and prices(), and what each returns.
What you will learn
- A span per step: what a model span and a tool span record, and what a coding agent's tool span adds (the file and its lines, before and after).
- One trace file for many runs: a run id and a task id on every span.
- Rupees from the models list (/v1/models): input, cached and output prices, per million tokens.
- Cost per task and per success over the suite, failures included.
- Reading a trace: why T02 failed, and the step that broke T03.
More in Scale and measure
Build "Tracing and cost" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.