Harness engineering · Chapter 10 of 23

Loading what is needed

By the end of this chapter you can say, with numbers, why a harness does not paste the whole repository into the prompt, build a repository map (every function's signature, from ast, in about 700 tokens), measure a map against search and against both on T01, T03 and T10, and keep a map from walking into folders that are not the project's code. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Context
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.

Why you need this

Farah has a question that sounds like a shortcut. The agent spends its first steps finding code: a search, a read from the hit, sometimes a wrong file first. Hisaab's repository is only 26 files. Why not give the model all of it at the start, and let it skip the finding?

Chapter 9 showed one side of the answer: a project file that grew to 20,897 tokens killed the run on its first request. Pasting the repository is the same mistake on a larger scale, and it hides a second one. The harness you have finds code on demand, one search and one page at a time, and keeps the window for what the task needs. This chapter measures the difference, and the useful middle: a map that says what is where without the code itself.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    stuff_it.py: paste the whole repository in

    Farah asks why the agent does not get every file at the start. In stuff_it.py, make a T01 checkout and count the tokens (o200k_base) of the text of every file of it (files(root) from agent.py), each file on its own. Print one line per top-level folder in sorted order, then the files at the root, then the total:

    docs/: N tokens

    hisaab/: N tokens

    tests/: N tokens

    root files: N tokens

    everything: N tokens

    Then run run_agent on the checkout with T01's issue and a system prompt of SYSTEM followed by every file (a blank line, # PATH relative to the checkout, a new line and its text), and print the run's answer as lab-chat: ANSWER.

  2. Task 2

    repo_map.py: a repository map from ast

    Kabir wants the model told what is where, not handed the code. In repo_map.py, write repo_map(root, paths): for each Python file in paths, sorted by path, and each top-level statement of it in order, one line: a function as PATH:LINE: def NAME(ARGS) -> RETURNS (async def for an async one, no -> part without a return annotation), a class as PATH:LINE: class NAME, and an assignment whose names are all upper case as PATH:LINE: NAME = .... PATH is relative to root, LINE the statement's line, ARGS and RETURNS as ast.unparse writes the node's args and returns. Methods, nested functions and lowercase variables are left out. Return the lines joined with new lines.

    Under if __name__ == "__main__":, print the map of a clean checkout's hisaab/*.py, then N lines, N tokens.

  3. Task 3

    map_vs_search.py: a map, search, or both

    Nandini wants the three ways compared. In map_vs_search.py, for T01, then T03, then T10, run three harnesses in this order, each on a fresh checkout: search (SYSTEM and TOOLS), map (SYSTEM, a blank line, # Repository map, a new line and repo_map of the checkout's hisaab/*.py, with tools_without("search_code")), and map+search (that prompt with TOOLS). Grade each and print:

    T01 map passed 4 steps 7000 tokens

    (the task, the harness, passed or failed, result.steps and result.prompt_tokens, two spaces apart): nine lines.

  4. Task 4 · interview question

    Interview: map_walk.py, a map that walks too far

    Amit's map_files gives repo_map every .py file under the checkout, and with a virtual environment in the repository the map is bigger than the window. In map_walk.py, rewrite map_files(root): every .py file under root, sorted, except the ones under a hidden folder (a name starting with a dot, like .git or .venv), __pycache__, docs/ or tests/. Every other folder counts.

    Under if __name__ == "__main__":, twice (Amit's walk, kept as amits_map_files, then yours): make a T03 checkout, copy Python's email package (Path(email.__file__).parent) into it as .venv/lib/email without __pycache__, map the walk's files, run T03 with SYSTEM, a blank line, # Repository map, a new line and the map, and print Amit's walk: N files, N tokens, T03 failed (error, 1 steps), then the same for map_files (the files, the map's tokens, the grade, result.stop and result.steps).

What you will learn

  • What the whole repository costs, folder by folder, against the window.
  • A repository map: every top-level name, its signature and its line, built with ast.
  • Map, search, or both, measured on T01, T03 and T10: steps, tokens and grades.
  • What belongs on a map, and what a careless walk drags in.

More in Context

Build "Loading what is needed" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.