Harness engineering · Chapter 3 of 23
Reading and searching
By the end of this chapter you can give a coding agent tools that find code without flooding its context: a read that shows numbered lines a page at a time and says how to read on, a listing without the noise, and a search that answers "where is this?" in one step. You will measure each on T03 and T10, in steps and prompt tokens, and prove a pager never drops a line. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The core
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.
Why you need this
Chapter 2 measured the cost of whole files. T10 renames a function that three modules still call, and with only list_files and a whole-file read_file, the model read file after file until its fourteenth request was refused at 8,684 tokens. T03's bug sits at line 227 of gst.py, below a 192-row table of HSN codes: to see four lines of gst_split, the model read 3,193 tokens of table, and every later request carried them again.
Hisaab is 26 files. Kabir points out that the repositories coding assistants work in have thousands, and files of thousands of lines. A harness that can only hand over whole files cannot work there at all: the first big file fills the window. What the model needs is what a person does: open a file at a line, read a screenful, read on if needed, and search for a name instead of opening files to look for it.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
read_tool.py: a paged read with line numbers and a footer
A whole-file read of gst.py is 230 lines in every later request. In read_tool.py, write read_file(root, path, offset=1, footer=True): the file's lines from line offset (1 is the first; an offset below 1 reads from 1), at most 200 of them, each one as its number right-aligned in 4 characters, a tab, then the line:
1 """GST for Hisaab invoices: ...
When lines are left after the page and footer is True, add a last line:
(lines A-B of N: read on with offset=C)
(A and B the first and last line shown, N the file's lines, C the next line to read). An offset past the end of a file that has lines returns:
error: offset X is past the end of PATH (N lines)
Then READ, a Tool (from the given agent.py) named read_file, with a description and the parameters path (a string, required) and offset (an integer, not required), and read_file as its function.
- Task 2
search_tool.py: list_files and search_code
In search_tool.py, write list_files(root): every file of the checkout, relative to it, sorted, one per line, leaving out any file inside a .git or __pycache__ folder. Then search_code(root, pattern): every line of every file list_files lists that contains pattern as plain text (not a regular expression), as
path:line: text
in the files' order, then the lines' order; text is the line with the spaces at both ends stripped. Skip files that are not UTF-8 text. With no match, return no matches for PATTERN. With more than 50, return the first 50, then a last line:
(N more matches: search for something longer)
Then LIST (named list_files, no parameters) and SEARCH (named search_code, a required string parameter pattern), Tools from agent.py.
- Task 3
find_it.py: T03 three ways
Measure what reading on and searching do for T03. In find_it.py, run run_agent (from agent.py, with its default max_steps) on a fresh checkout of T03 with three registries, each with LIST and agent.py's WRITE:
no footer: a read_file tool with READ's name, description and parameters whose function calls read_file with footer=False
footer: READ
footer and search: READ and SEARCH
Print one line per harness, in that order:
LABEL: S steps, T prompt tokens, gst.py unchanged
or, when the checkout's diff touches hisaab/gst.py, gst.py rewritten (A lines) with the file's lines after the run. S is the run's steps, T its prompt_tokens.
- Task 4 · interview question
Interview: pager.py, prove a pager drops no line
"How do you know your pager never loses a line?" pager.py has a colleague's pager, colleague_read, given. Write walk(read, root, path): read the file the way the model does, from offset 1, following each footer: a line that is not numbered and has offset=N in it means call read(root, path, offset=N) next. Stop when a result has no footer, starts with error, or asks for an offset already read. Return every line number it showed, in order (a numbered line starts with its number, after any spaces, then a tab or a colon and a space).
Under if __name__ == "__main__":, on a clean checkout, print for colleague_read and then for your read_file (from read_tool.py):
NAME: saw S of N lines, missing A, B
(NAME colleague or read_tool, S the distinct lines seen, N hisaab/gst.py's lines, the missing line numbers in order, or missing none).
What you will learn
- A paged read: numbered lines, an offset, 200 lines a page.
- A footer that says the page is cut and how to read on, and what a page without one does.
- A listing without noise: .git and __pycache__ left out.
- A search that answers with path:line: text, capped so it cannot flood the context.
- The numbers: T03 and T10 in steps and prompt tokens, with and without each.
More in The core
Build "Reading and searching" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.