Harness engineering · Chapter 18 of 23
Memory
By the end of this chapter your harness keeps what the agent learns between sessions: a memory file loaded into the system prompt, a remember tool the model saves lessons with, repeats left out, a cap in tokens, and a review that throws out lessons that are no longer true. You will measure the second session of T11 against the first, and see what a year of memory does to the window. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Scale and measure
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.
Why you need this
T11 says split_evenly(1000, 3) returns [333, 333, 333], and the parts must add up to the amount. Under the harness so far, the agent finds the function, reads it, and makes the same first try every time:
search_code(split_evenly)
read_file(hisaab/money.py, offset=45)
edit_file(...) first try: the last part gets the extra paisa
run_command(python -m pytest -q) 1 failed: "split with divmod: ..."
edit_file(...) divmod: the first parts get it
run_command(python -m pytest -q) 45 passed
The test's message taught it the house rule (Hisaab splits with divmod, and the first parts get the extra paise), and the run passed. Tomorrow a new session starts from nothing: the same wrong first try, the same failing test, the same two extra steps. Farah asks the obvious question: why is the agent not getting better at Hisaab's code, when a new engineer would after one day?
Nothing it learns survives the session, because the context is the only memory it has and every session starts a new one. A project file (chapter 9) holds what the team wrote down in advance. This chapter is about what the agent finds out on the job.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
memory.py: a memory file and a remember tool
Kabir wants the agent to keep what it learns. In memory.py, write a class Memory(path): path is a JSON file holding {"lessons": [...]}, oldest first, read into self.lessons when it exists ([] when it does not).
add(lesson) adds the lesson at the end, writes the file again in the same shape and returns saved; a lesson memory already has (the same words ignoring case, extra spaces and a final full stop) returns already in memory and changes nothing. section() returns "" with no lessons, else the line # Lessons from earlier sessions and then one line "- lesson" per lesson, in order. tool() returns a Tool named remember, with one required parameter, lesson, described as saving a lesson for future sessions; its function calls add and returns what add returned.
Trap: "Use divmod." and "use divmod" are the same lesson.
- Task 2
two_sessions.py: two sessions, one memory
Nandini wants to see the memory pay off. In two_sessions.py, write session(memory, task_id="T11"): a fresh checkout of the task, run_agent with its issue, every tool of repo_tools(root) and memory.tool(), and a system prompt of SYSTEM followed (only when memory has lessons) by a blank line and memory.section(). It returns {"passed": the grade, "steps": the requests sent, "prompt_tokens": their prompt tokens added up, "saved": how many lessons memory gained}.
Under if __name__ == "__main__":, make a Memory on a new file in a fresh temporary folder, run two sessions of T11, and print one line per session like:
session 1: passed, 9 steps, 9000 prompt tokens, 2 saved
(those numbers only show the format), then every lesson in memory as - lesson.
- Task 3
cap.py: review and cap a memory
Amit says the team's memory (team_memory.json, given) is a year old and huge. In cap.py, write fit(lessons, root, budget=1500, review=True): the lessons to put in the prompt, in their order, chosen from the newest back. With review, a lesson is left out when it repeats a newer lesson already kept (same words ignoring case, extra spaces and a final full stop) or when it is stale: it names a path with a folder in it, like hisaab/splits.py, that is not a file in root. Every other lesson is kept while its line "- lesson" (tokens counted with tiktoken's o200k_base) fits in what is left of budget; the first that does not fit stops the choosing. With review=False nothing is left out as a repeat or stale.
Under if __name__ == "__main__":, with a clean checkout as root, print team memory: N lessons, T tokens and fitted: n lessons, t tokens (the tokens of their lines added up). Then run T11 three times, each in a fresh checkout, with run_agent, every repo tool and a system prompt of SYSTEM, a blank line, # Lessons from earlier sessions and one "- lesson" line per lesson, and print:
T11 with all of it: passed, stop answered, steps 6
then T11 with the newest, not reviewed: ... (fit with review=False) and T11 with the fitted memory: ... (the numbers above only show the format; passed or failed is the grade).
- Task 4 · interview question
Interview: crowded.py, memory that crowds out the task
Someone saw the system prompt growing and capped the memory: memory_section keeps only the newest MAX_LESSONS lessons. With the team's memory, T11 still fails on its very first request. Run crowded.py, find what is wrong with the cap, and fix memory_section so that it keeps the newest lessons whose "- lesson" lines (counted with tiktoken's o200k_base) add up to at most BUDGET tokens, in their order; a memory of many short lessons that fit is kept whole; and T11 passes with the team's memory.
Keep BUDGET, memory_section(lessons) (the header line, then one "- lesson" line each) and run_session(root, issue, lessons), and what each returns.
What you will learn
- A memory file loaded into the system prompt, and a remember tool the model saves lessons with.
- Two sessions, measured: what the second one saves on T11.
- Repeats: why the same lesson must not be saved twice.
- The cost of memory: it is resent with every request, and a year of it does not fit.
- A cap in tokens, newest first, and a review that throws out stale lessons, which do harm.
More in Scale and measure
Build "Memory" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.