Harness engineering (TypeScript) · Chapter 19 of 23

Tracing and cost

By the end of this chapter your harness writes a trace of every run (one line of JSON per model call, per tool call, and a grade at the end), turns the trace of a whole suite run into rupees per task and per success, and finds in a trace the step that broke a run. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Scale and measure
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Nandini runs the suite with this chapter's harness every morning. Today it says 10 of 12. Three questions follow, and the grade answers none of them:

  • Farah: what did that cost, and what does each fix cost?
  • Amit: why is the bill what it is? Which tasks, which requests?
  • Kabir: why did T02 and T07 fail? the tests fail: crore says what is wrong with the checkout, not what the agent did.

In chapter 1 you watched one run step by step and saw gst.ts come back 39 lines long. A suite run is 12 runs, 71 requests and 59 tool calls, every morning, and every chapter from here compares suite runs. Nobody watches that. What is not written down while it happens is gone when the run ends: the conversation lives in memory, and the checkout keeps only the last state of the files.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    tracer.ts: a trace of a coding agent's run

    Nandini wants to see inside every run, not just its grade. In tracer.ts, export a class Tracer, made with new Tracer(path, runId?) (runId: the first 8 characters of a randomUUID() without dashes when not given), whose span(kind, task, fields = {}) adds one line of JSON, { run, task, kind, ...fields }, to the end of the file and returns it; and readSpans(path, runId?), the file's spans in order (that run's only when runId is given, [] with no file).

    Then export async function tracedRun(root, taskId, tools, tracer, { system = SYSTEM, model = 'lab-chat', maxSteps = 20 } = {}): harness.ts's loop on the task's issue, resolving to a RunResult. After every request, a model span: step, model, promptTokens, cachedTokens (0 when usage has none), completionTokens, totalTokens, seconds, asked (the tool names) and error null; a refused request gets a model span with every token count 0, asked [], error the error's message, and the run stops with stop error. After every tool call, a tool span: step, tool, args (an object), resultChars, error (the result's error when it is JSON with one, else null), file (args.path or null), linesBefore and linesAfter (that file's line count in root around the call, null when there is no such file), seconds. At the end, a grade span: passed and reason from grade(root), steps and stop.

    Under if (import.meta.main), trace T05 into trace.jsonl with every repo tool and print its spans, one line each.

  2. Task 2

    cost.ts: rupees per task, and per success

    Amit pays the bill. In cost.ts, export async function prices(): { model: price_inr_per_million } for every model the lab model lists that has one (for await over client.models.list(); the field is not in the Model type). Then rupees(span, table): (promptTokens - cachedTokens) at input, cachedTokens at cached_input and completionTokens at output, per million tokens, for the span's model. Then report(spans, table): { rows: one { task, rupees, passed, steps } per task in the order they ran, total, passed, perSuccess: total / passed, or null with no pass }.

    Under if (import.meta.main), run every task of TASKS in a fresh checkout with tracedRun, every repo tool and one Tracer on suite.jsonl, then report that run's spans and print a line per task, then the totals:

    T01 passed 5 steps ₹1.2345

    total ₹15.0000, 11/12 passed, ₹1.3636 per success

    (rupees with 4 decimals; those numbers only show the format).

    Trap: suite.jsonl keeps every run you make: report this run's only.

  3. Task 3

    where-it-broke.ts: the step that broke a run

    T03 fails under the naive harness, and the grade only says the tests fail. In where-it-broke.ts, export firstBreak(spans): the first tool span, in order, that left a file with less than half its lines (linesAfter * 2 < linesBefore), or null; a span with no linesBefore or linesAfter is skipped.

    Under if (import.meta.main), trace T03 into t03.jsonl with tracedRun, every tool of naiveTools(root) (naive.ts, given) and { system: NAIVE_SYSTEM }. From that run's spans print:

    T03: failed (the grade's reason)

    broke at step S: TOOL FILE, B lines -> A

    or no step shrank a file when firstBreak finds nothing.

  4. Task 4 · interview question

    Interview: bill.ts, a cost report that is too high

    last-run.jsonl (given) is the trace of one run of the suite. Amit's bill for it, worked out from the usage the lab model reported, is far lower than what bill.ts's runCost says. Find what runCost gets wrong and fix it, so that it returns what the run really cost in rupees: for every model span, its prompt tokens that were not cached at the input price, its cached tokens at the cached_input price and its completion tokens at the output price, per million tokens, by the span's model.

    Keep runCost(spans, table) and prices() exported, and what each returns.

What you will learn

  • A span per step: what a model span and a tool span record, and what a coding agent's tool span adds (the file and its lines, before and after).
  • One trace file for many runs: a run id and a task id on every span.
  • Rupees from the models list (/v1/models): input, cached and output prices, per million tokens.
  • Cost per task and per success over the suite, failures included.
  • Reading a trace: why T02 failed, and the step that broke T03.

More in Scale and measure

Build "Tracing and cost" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.