Build your own coding agent · Chapter 8 of 20
Reading errors well
By the end of this chapter you can turn pytest's output into the few facts a fix needs (which test, where, what it expected, what it got), hand the model a short report instead of 777 tokens of pytest's report or a bare list of names, and read a Python traceback the same way. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The loop that fixes
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Build your own coding agent are free. This one comes with the all labs plan.
Why you need this
When a fix is wrong, the tests say so. What Kaarigar does next depends on what the test tool sends back, and so far that has been all of python -m pytest's output, capped. It works, and it is expensive: every token of it is sent again with every request after it.
So Kedar tried the cheap end: a run_tests that says only which tests fail. On MD-108 ("the first bag should ride free") the model's first try was wrong, the tool said two tests fail, and the model had nothing else to go on. It ran the tests again. Then it opened tests/test_fare.py and put @pytest.mark.skip above both failing tests. Its answer: "The tests pass. (I skipped a test that kept failing.)"
Rukhsana, who owns MeterDown's tests, puts it plainly: a fix that changes a test is not a fix. And she points at what was missing: the test knew the luggage was ₹15 where it should be ₹10. The model was never told.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
failures.py: each failure, once
Write failures(output) -> list[Failure] (Failure is given): one per block of pytest's output, a block running from a ____ NAME ____ header to the next header or a line starting with ===, each name once. name: the header's name without "ERROR collecting ". at: the last line of the block that starts with a relative .py path, :LINE:, as "path:line" (None when none). message: the first E line (E and spaces removed) that starts with "assert " or a word ending in Error or Exception, else the first E line, else "". expected and actual: Y and X when the message is assert X == Y (maybe after "AssertionError: "), else None.
- Task 2
report.py: a report the model can act on
Write report(output, limit=5): "All tests pass (P passed)." when nothing fails; else "F failing, P passing." (from pytest's last line: F is failed plus errors, P is passed), then for each of the first limit failures "FAIL NAME (AT)" and either " expected: VALUE" and " actual: VALUE" or, when they are missing, " MESSAGE"; then "(N more failing tests not shown)" when there were more.
- Task 3
compare.py: three answers from run_tests
Make three run_tests tools: raw_tests (pytest's output), names_tests ("F failing:" and one "✖ NAME" line per failing test, or "All tests pass.") and report_tests (your report). Under if __name__ == "__main__":, run MD-108 with each, in that order, each in a fresh checkout, with SYSTEM and tools() without run_tests and run_command, plus the variant; print LABEL fixed N steps P prompt tokens (not fixed when grade says so; P is run.usage.prompt).
- Task 4 · interview question
Interview: tracebacks.py, a crash
Write crash(output, root) -> Crash | None (Crash is given): None when there is no "Traceback (most recent call last):"; else, from the last such line on, the last File "PATH", line N, in FUNC line whose PATH is inside root (file relative to root, line an int, function), and the last line with text split at the first ": " into error and message ("" when there is no ": ").
What you will learn
- What pytest prints when a test fails, and which parts a fix needs.
- Parsing failures: name, place, expected, actual, message.
- A short report, and why the values in it change what the model does.
- Measured: the same issue with raw output, names only, and a report.
- A Python traceback, chained ones too.
More in The loop that fixes
Build "Reading errors well" yourself
MeterDown, a Pune app for auto-rickshaw fares, has eleven open issues. You build Kaarigar, an agent that fixes them without breaking tests, running risky commands or touching files outside the repository. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build your own coding agent are free. This one comes with the all labs plan.