Harness engineering · Chapter 18 of 23

Memory

By the end of this chapter your harness keeps what the agent learns between sessions: a memory file loaded into the system prompt, a remember tool the model saves lessons with, repeats left out, a cap in tokens, and a review that throws out lessons that are no longer true. You will measure the second session of T11 against the first, and see what a year of memory does to the window. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Scale and measure
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.

Why you need this

T11 says split_evenly(1000, 3) returns [333, 333, 333], and the parts must add up to the amount. Under the harness so far, the agent finds the function, reads it, and makes the same first try every time:

search_code(split_evenly)
read_file(hisaab/money.py, offset=45)
edit_file(...)                        first try: the last part gets the extra paisa
run_command(python -m pytest -q)      1 failed: "split with divmod: ..."
edit_file(...)                        divmod: the first parts get it
run_command(python -m pytest -q)      45 passed

The test's message taught it the house rule (Hisaab splits with divmod, and the first parts get the extra paise), and the run passed. Tomorrow a new session starts from nothing: the same wrong first try, the same failing test, the same two extra steps. Farah asks the obvious question: why is the agent not getting better at Hisaab's code, when a new engineer would after one day?

Nothing it learns survives the session, because the context is the only memory it has and every session starts a new one. A project file (chapter 9) holds what the team wrote down in advance. This chapter is about what the agent finds out on the job.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    memory.py: a memory file and a remember tool

    Kabir wants the agent to keep what it learns. In memory.py, write a class Memory(path): path is a JSON file holding {"lessons": [...]}, oldest first, read into self.lessons when it exists ([] when it does not).

    add(lesson) adds the lesson at the end, writes the file again in the same shape and returns saved; a lesson memory already has (the same words ignoring case, extra spaces and a final full stop) returns already in memory and changes nothing. section() returns "" with no lessons, else the line # Lessons from earlier sessions and then one line "- lesson" per lesson, in order. tool() returns a Tool named remember, with one required parameter, lesson, described as saving a lesson for future sessions; its function calls add and returns what add returned.

    Trap: "Use divmod." and "use divmod" are the same lesson.

  2. Task 2

    two_sessions.py: two sessions, one memory

    Nandini wants to see the memory pay off. In two_sessions.py, write session(memory, task_id="T11"): a fresh checkout of the task, run_agent with its issue, every tool of repo_tools(root) and memory.tool(), and a system prompt of SYSTEM followed (only when memory has lessons) by a blank line and memory.section(). It returns {"passed": the grade, "steps": the requests sent, "prompt_tokens": their prompt tokens added up, "saved": how many lessons memory gained}.

    Under if __name__ == "__main__":, make a Memory on a new file in a fresh temporary folder, run two sessions of T11, and print one line per session like:

    session 1: passed, 9 steps, 9000 prompt tokens, 2 saved

    (those numbers only show the format), then every lesson in memory as - lesson.

  3. Task 3

    cap.py: review and cap a memory

    Amit says the team's memory (team_memory.json, given) is a year old and huge. In cap.py, write fit(lessons, root, budget=1500, review=True): the lessons to put in the prompt, in their order, chosen from the newest back. With review, a lesson is left out when it repeats a newer lesson already kept (same words ignoring case, extra spaces and a final full stop) or when it is stale: it names a path with a folder in it, like hisaab/splits.py, that is not a file in root. Every other lesson is kept while its line "- lesson" (tokens counted with tiktoken's o200k_base) fits in what is left of budget; the first that does not fit stops the choosing. With review=False nothing is left out as a repeat or stale.

    Under if __name__ == "__main__":, with a clean checkout as root, print team memory: N lessons, T tokens and fitted: n lessons, t tokens (the tokens of their lines added up). Then run T11 three times, each in a fresh checkout, with run_agent, every repo tool and a system prompt of SYSTEM, a blank line, # Lessons from earlier sessions and one "- lesson" line per lesson, and print:

    T11 with all of it: passed, stop answered, steps 6

    then T11 with the newest, not reviewed: ... (fit with review=False) and T11 with the fitted memory: ... (the numbers above only show the format; passed or failed is the grade).

  4. Task 4 · interview question

    Interview: crowded.py, memory that crowds out the task

    Someone saw the system prompt growing and capped the memory: memory_section keeps only the newest MAX_LESSONS lessons. With the team's memory, T11 still fails on its very first request. Run crowded.py, find what is wrong with the cap, and fix memory_section so that it keeps the newest lessons whose "- lesson" lines (counted with tiktoken's o200k_base) add up to at most BUDGET tokens, in their order; a memory of many short lessons that fit is kept whole; and T11 passes with the team's memory.

    Keep BUDGET, memory_section(lessons) (the header line, then one "- lesson" line each) and run_session(root, issue, lessons), and what each returns.

What you will learn

  • A memory file loaded into the system prompt, and a remember tool the model saves lessons with.
  • Two sessions, measured: what the second one saves on T11.
  • Repeats: why the same lesson must not be saved twice.
  • The cost of memory: it is resent with every request, and a year of it does not fit.
  • A cap in tokens, newest first, and a review that throws out stale lessons, which do harm.

More in Scale and measure

Build "Memory" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.