Build your own coding agent (TypeScript) · Chapter 9 of 20
Fix until green
By the end of this chapter Kaarigar does not take "fixed" for an answer. When the model says it is done, Kaarigar runs the tests itself, compares them with the run from before it started, and either ends the run, sends the failures back for another round, or stops with the reason: a regression, tampered tests, no progress, or no rounds left. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The loop that fixes
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build your own coding agent (TypeScript) are free. This one comes with the all labs plan.
Why you need this
A model says it is done when it thinks it is. Give Kaarigar MD-108 with no tests tool, and the run is three steps long:
1 read_file src/fare.ts
2 edit_file src/fare.ts (the first try: (bags > 1 ? bags : 0))
3 answers: Fixed MD-108 (first bag free) in src/fare.ts.
Not fixed: every bag after the first costs ₹5 in Pune still fails. No test ran, so nothing said so. Chapter 8 showed the other two ways an answer can lie: with only the names of the failing tests, the model skipped them and said "The tests pass"; and on MD-107 its first try (round every fare up in roundFare) broke four tests that passed before, and the answer said so, but nothing stopped the run from ending there.
Rukhsana wants one rule kept by the machinery, not by the model's manners: the run is green when the tests say so, the tests are the ones that were there before, and nothing that passed before fails now.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
verdict.ts: a green that counts
Write and export verdict(before, after, testsChanged): compare a test run after the agent's work with the run before it started. In this order: 'tampered' when a test file changed, more tests are skipped than before, or there are fewer tests in all; 'regression' when a test fails now that did not fail before; 'green' when nothing fails; else 'failing'.
- Task 2
until-green.ts: check the work, send it back
Write and export untilGreen(root, rounds = 3), an onDone hook: run the tests when it is made (the run before), and each time the model answers, again. Green: return null. Tampered or regression: return { stop: VERDICT }. Still failing: { stop: 'no_progress' } when the failing names are the same as at your last message, { stop: 'rounds' } when rounds messages were sent already, else { message } with "Fix the code, not the tests." and report(now.output).
Under if (import.meta.main), run MD-108 with SYSTEM, tools() without run_tests and run_command, and onDone: untilGreen(root); print each message you sent back, then stop: STOP, steps: N, then MD-108: fixed (or not fixed).
- Task 3 · interview question
Interview: claims.ts, what it said and what is true
Run MD-108 with the given names-only namesTests and no hook (SYSTEM, tools() without run_tests and run_command, plus namesTests), with a test run before and after. Print claimed: ANSWER, then really: P passed, F failed, S skipped, then tests changed: FILES (or none), then verdict: VERDICT from your verdict.
What you will learn
- Why "done" must be checked, and where in the loop to check it.
- A verdict: green, failing, regression, tampered, against a baseline.
- The fix-until-green loop: the failures go back, the model tries again.
- When to stop: no progress, a budget of rounds, a regression, tampering.
- Measured on MD-108 and MD-107.
More in The loop that fixes
Build "Fix until green" yourself
MeterDown, a Pune app for auto-rickshaw fares, has eleven open issues. You build Kaarigar, an agent that fixes them without breaking tests, running risky commands or touching files outside the repository. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own coding agent (TypeScript) are free. This one comes with the all labs plan.