Harness engineering · Chapter 11 of 23

The context window

By the end of this chapter you can count a conversation's tokens the way the API counts them, before you send it; keep a run inside the window by clearing old tool results; summarise old turns with the model and know what a summary loses; and spot a compactor that throws away the one message the model cannot work without. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Context
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.

Why you need this

T10 is a rename: gst.rate_for became gst.hsn_rate last week, and three modules still call the old name. Farah has seen the agent fix one-file bugs. She wants it to handle the ones that touch half the repository.

Run T10 under a harness with whole-file reads and no search (chapter 1's tools, with a run_command), and it does the right things: it lists the files, reads them one by one, rewrites invoice.py, quote.py and reports.py as it finds the old name. Then, on its 14th request, the API refuses it:

Error code: 400 - This model's maximum context length is 8192 tokens.
However, your messages resulted in 8718 tokens.

The run ends there: no answer, and the tests it was told to run never ran. (The writes it made before the crash happen to be right, so the grade passes; the harness cannot know that, and on a larger rename the last file would never have been reached.) Every file it read is still in the conversation, resent with every request, long after the model finished with it. Kabir puts it simply: a run grows, the window does not, and the harness is the only thing that can decide what stays. What you have so far only appends.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    count.py: count it as the API does

    Kabir wants the harness to know how full the window is before it sends a request, not after it is refused. In count.py, write count_tokens(messages, tools=None) with tiktoken's o200k_base encoding: 3 for the reply's start; for each message 3, plus the tokens of its content (None counts as nothing) followed by its tool_calls as compact JSON, plus 1 if it has a name; plus the tools as compact JSON. Compact JSON is json.dumps(value, separators=(",", ":"), ensure_ascii=False).

    Under if __name__ == "__main__":, run base.run_agent on a checkout of T10 with a prepare function that counts every request with count_tokens(messages, base.TOOLS) and sends the messages unchanged, then print one line per request:

    request 1: counted 272, usage 272

    and for the refused one: request 14: counted 8718, refused

    Trap: every count must equal the API's usage exactly; an assistant message with tool calls has no content.

  2. Task 2

    compact.py: clear old tool results

    In compact.py, write compact(messages, keep_last=4, min_tokens=200): a new list of the same length in which every tool message with more than min_tokens tokens, except in the first two messages and the last keep_last, has its content replaced by STUB, keeping its role and tool_call_id. Everything else stays as it was, and the list it was given does not change.

    Write prepare(messages): compact(messages) when count_tokens(messages, base.TOOLS) is over BUDGET (6000), else the messages as they are.

    Under if __name__ == "__main__":, run T10 with base.run_agent without prepare and then with it, and print:

    plain: error after 14 requests, largest prompt 8104 tokens

    compacted: STOP after N requests, largest prompt P tokens

    Answer: TEXT

    Trap: removing a tool message instead of clearing it leaves its call unanswered, and the API refuses the request.

  3. Task 3

    summarise.py: a summary from the model

    In summarise.py, write summarise(messages): send lab-chat the messages and then one user message asking it to summarise this coding session so far, with no tools, and return the reply's text.

    Write compact_with_summary(messages, keep_last=4): the system prompt, then a user message "Summary of the session so far:\n" followed by the summary of the messages before the tail, then the tail (the last keep_last messages, started earlier at the assistant call if it would begin with a tool message).

    Under if __name__ == "__main__":, run T10 with a prepare that compacts with a summary when the messages are over BUDGET tokens. Print the first summary message, then summaries: N, STOP after S requests, then T10: passed or T10: failed.

    Trap: a tail that starts with a tool message is refused (its call is in the summary, not in the messages).

  4. Task 4 · interview question

    Interview: keep_task.py, a compactor that loses the task

    A teammate's trim() keeps T10 inside the window, and the run answers, but T10 fails. Run keep_task.py, read what the model was sent after the first trim, and fix trim(messages, keep_last=4) so that the system prompt and every user message are kept as they are, old tool results over 1,000 characters become STUB, everything else stays, and the last keep_last messages are as they were. Keep the prints: fixed, it ends with T10: passed.

    Trap: the run does not crash. It answers, so only reading the messages shows what went missing.

What you will learn

  • Counting a conversation as the API does: message by message, tool calls and tools included, checked against usage.prompt_tokens.
  • A budget below the window, checked before every request.
  • Clearing old tool results: the call stays, its result becomes a short stub; the system prompt, the task and the last steps stay as they are.
  • Summarising old turns with the model, and what a summary keeps and loses.
  • The one rule of every compactor: never lose the task.

More in Context

Build The context window yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.