Harness engineering (TypeScript) · Chapter 18 of 23

Memory

By the end of this chapter your harness keeps what the agent learns between sessions: a memory file loaded into the system prompt, a remember tool the model saves lessons with, repeats left out, a cap in tokens, and a review that throws out lessons that are no longer true. You will measure the second session of T11 against the first, and see what a year of memory does to the window. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Scale and measure
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.

Why you need this

T11 says splitEvenly(1000, 3) returns [333, 333, 333], and the parts must add up to the amount. Under the harness so far, the agent finds the function, reads it, and makes the same first try every time:

search_code(splitEvenly)
read_file(src/money.ts, offset=49)
edit_file(...)               first try: the last part gets the extra paisa
run_command(npm test)        ℹ fail 1: "split with a quotient and a remainder: ..."
edit_file(...)               a quotient and a remainder: the first parts get it
run_command(npm test)        ℹ pass 45

The test's message taught it the house rule (Hisaab splits with a quotient and a remainder, and the first parts get the extra paise), and the run passed. Tomorrow a new session starts from nothing: the same wrong first try, the same failing test, the same two extra steps. Farah asks the obvious question: why is the agent not getting better at Hisaab's code, when a new engineer would after one day?

Nothing it learns survives the session, because the context is the only memory it has and every session starts a new one. A project file (chapter 9) holds what the team wrote down in advance. This chapter is about what the agent finds out on the job.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    memory.ts: a memory file and a remember tool

    Kabir wants the agent to keep what it learns. In memory.ts, export a class Memory, made with new Memory(path): path is a JSON file holding { lessons: [...] }, oldest first, read into this.lessons when it exists ([] when it does not).

    add(lesson) adds the lesson at the end, writes the file again in the same shape and returns saved; a lesson memory already has (the same words ignoring case, extra spaces and a final full stop) returns already in memory and changes nothing. section() returns "" with no lessons, else the line # Lessons from earlier sessions and then one line "- lesson" per lesson, in order. tool() returns a Tool (the type in ./harness.ts) named remember, with one required parameter, lesson, described as saving a lesson for future sessions; its fn calls add and returns what add returned.

    Trap: "Use divmod." and "use divmod" are the same lesson.

  2. Task 2

    two-sessions.ts: two sessions, one memory

    Nandini wants to see the memory pay off. In two-sessions.ts, export async function session(memory, taskId = 'T11'): a fresh checkout of the task, runAgent with its issue, every tool of repoTools(root) and memory.tool(), and a system prompt of SYSTEM followed (only when memory has lessons) by a blank line and memory.section(). It resolves to { passed: the grade, steps: the requests sent, promptTokens: their prompt tokens added up, saved: how many lessons memory gained } (export the type as Session).

    Under if (import.meta.main), make a Memory on a new file in a fresh temporary folder (mkdtempSync), run two sessions of T11, and print one line per session like:

    session 1: passed, 9 steps, 9000 prompt tokens, 2 saved

    (those numbers only show the format), then every lesson in memory as - lesson.

  3. Task 3

    cap.ts: review and cap a memory

    Amit says the team's memory (team-memory.json, given) is a year old and huge. In cap.ts, export fit(lessons, root, { budget = 1500, review = true } = {}): the lessons to put in the prompt, in their order, chosen from the newest back. With review, a lesson is left out when it repeats a newer lesson already kept (same words ignoring case, extra spaces and a final full stop) or when it is stale: it names a path with a folder in it, like src/splits.ts, that is not a file in root. Every other lesson is kept while its line "- lesson" (tokens counted with js-tiktoken's o200k_base) fits in what is left of budget; the first that does not fit stops the choosing. With review false nothing is left out as a repeat or stale (root may be null).

    Under if (import.meta.main), with a clean checkout as root, print team memory: N lessons, T tokens and fitted: n lessons, t tokens (the tokens of their lines added up). Then run T11 three times, each in a fresh checkout, with runAgent, every repo tool and a system prompt of SYSTEM, a blank line, # Lessons from earlier sessions and one "- lesson" line per lesson, and print:

    T11 with all of it: passed, stop answered, steps 6

    then T11 with the newest, not reviewed: ... (fit with review false) and T11 with the fitted memory: ... (the numbers above only show the format; passed or failed is the grade).

  4. Task 4 · interview question

    Interview: crowded.ts, memory that crowds out the task

    Someone saw the system prompt growing and capped the memory: memorySection keeps only the newest MAX_LESSONS lessons. With the team's memory, T11 still fails on its very first request. Run crowded.ts, find what is wrong with the cap, and fix memorySection so that it keeps the newest lessons whose "- lesson" lines (counted with js-tiktoken's o200k_base) add up to at most BUDGET tokens, in their order; a memory of many short lessons that fit is kept whole; and T11 passes with the team's memory.

    Keep HEADER, BUDGET, memorySection(lessons) (the header line, then one "- lesson" line each) and runSession(root, issue, lessons) exported, and what each returns.

What you will learn

  • A memory file loaded into the system prompt, and a remember tool the model saves lessons with.
  • Two sessions, measured: what the second one saves on T11.
  • Repeats: why the same lesson must not be saved twice.
  • The cost of memory: it is resent with every request, and a year of it does not fit.
  • A cap in tokens, newest first, and a review that throws out stale lessons, which do harm.

More in Scale and measure

Build "Memory" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.