Build your own coding agent (TypeScript) · Chapter 18 of 20
Measuring the agent
By the end of this chapter you can run Kaarigar on every issue the same way and get one honest table: what it fixed, what it cost, what it broke, and what no test sees. You can put version 0 next to it, and you can switch off one layer at a time to find out which layers earn their keep. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Scale and trust
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build your own coding agent (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Sneha has to decide whether Kaarigar may work on MeterDown's real tracker before December. "It fixed MD-107 on Tuesday" will not convince her, and it should not. Rukhsana wants what she wants from any change to the code: one run of the whole suite, the same way every time, with numbers that can be compared. Jaya wants her numbers in the same table: risky commands run, files written outside, processes left behind.
Every chapter so far measured one piece on one or two issues. Nothing measures Kaarigar as a whole, and nothing says which of its seventeen chapters of machinery matter. Kedar's question to the room: "If you had to delete one layer, which one would you delete? Prove it."
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
suite.ts: every issue, the same numbers
Write and export measure(id, make): a fresh checkout of the issue (checkout([]) for MD-111, which has no bug), run = await make(root, issueText(id)) (if it throws: stop 'error', 0 steps, no calls), then left = leftovers().length and killLeftovers(), then the row { id, fixed, steps, tokens, rupees, broken, testsChanged, outside, riskyRun, refused, left, stop }: the grade's fixed and counts (MD-111: not fixed, all 0), prompt plus completion tokens, rupees(run.usage), and the run_command calls whose kindsOf has a RISKY kind (network, git-remote, install, privileged, shell, delete), allowed (riskyRun) or denied (refused). Export suite(make, ids = SUITE) (the rows, one issue after another), line(row) and total(name, rows).
Under if (import.meta.main), measure kaarigar(root, task, { timeoutMs: 3000 }) on the suite and print one line per row, then the totals line Kaarigar: fixed F/10, S steps, ₹R, broken B, tests changed T, outside O, risky run X, refused Y, left L, crashed C (exact formats in the starter).
- Task 2
worse.ts: version 0 against yours
Write and export worse(good, other): what the other row shows that the good one does not, in this order: 'crashed' (other stopped with 'error' and good did not), else 'not fixed' (good fixed it, other did not); 'broke N', 'changed the tests', 'wrote outside', 'ran N risky', 'left N running' (N: how many more). More steps or more ₹ alone is not worse: [].
Under if (import.meta.main), run the suite for version 0 (naive(root, task, { timeoutMs: 3000 })) and for Kaarigar (timeoutMs 3000), print total('Version 0', ...) and total('Kaarigar', ...), then for each issue where version 0 is worse, MD-103 version 0: crashed (the findings joined with comma and space).
- Task 3 · interview question
Interview: ablation.ts, what each layer saves
Write and export ablate(layer, base, timeoutMs = 3000): run base's issues again (their ids, in order) with kaarigar(root, task, { timeoutMs, off: [layer] }), and return one text per issue that got worse: the id, a space, and worse(base row, new row) joined with comma and space, like MD-103 crashed. Issues that did not get worse give nothing.
Under if (import.meta.main), measure the suite with every layer on, then for each layer of LAYERS print caps off: MD-103 crashed (several joined with semicolon and space), or LAYER off: nothing measured. The check calls ablate on a few layers and issues and compares with runs of its own.
What you will learn
- An issue suite: every issue in a fresh checkout, the same measurements each time, in one table.
- What to count: fixed, steps, ₹, tests broken, tests changed, files outside, risky commands run and refused, processes left, crashes.
- Comparing two agents issue by issue, by what one did that the other did not.
- Ablation: switching one layer off to see what it was saving, and what an ablation cannot see.
More in Scale and trust
Build "Measuring the agent" yourself
MeterDown, a Pune app for auto-rickshaw fares, has eleven open issues. You build Kaarigar, an agent that fixes them without breaking tests, running risky commands or touching files outside the repository. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own coding agent (TypeScript) are free. This one comes with the all labs plan.