Harness engineering (TypeScript) · Chapter 11 of 23
The context window
By the end of this chapter you can count a conversation's tokens the way the API counts them, before you send it; keep a run inside the window by clearing old tool results; summarise old turns with the model and know what a summary loses; and spot a compactor that throws away the one message the model cannot work without. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Context
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.
Why you need this
T10 is a rename: rateFor in src/gst.ts became hsnRate last week, and three modules still call the old name. Farah has seen the agent fix one-file bugs. She wants it to handle the ones that touch half the repository.
Run T10 under a harness with whole-file reads and no search (chapter 1's tools, with a run_command), and it does the right things: it lists the files, reads them one by one, rewrites invoice.ts, quote.ts and reports.ts as it finds the old name. Then, on its 12th request, the API refuses it:
Error code: 400 - This model's maximum context length is 8192 tokens.
However, your messages resulted in 8277 tokens.
The run ends there: no answer, and the tests it was told to run never ran. (The writes it made before the crash happen to be right, so the grade passes; the harness cannot know that, and on a larger rename the last file would never have been reached.) Every file it read is still in the conversation, resent with every request, long after the model finished with it. Kabir puts it simply: a run grows, the window does not, and the harness is the only thing that can decide what stays. What you have so far only appends.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
count.ts: count it as the API does
Kabir wants the harness to know how full the window is before it sends a request, not after it is refused. In count.ts, export countTokens(messages, tools?) with js-tiktoken's o200k ranks: 3 for the reply's start; for each message 3, plus the tokens of its content (null counts as nothing) followed by its tool_calls as JSON, plus 1 if it has a name; plus the tools as JSON. JSON.stringify(value) writes the compact JSON the request carries.
Under if (import.meta.main), run runAgent (base.ts) on a checkout of T10 with a prepare function that counts every request with countTokens(messages, TOOLS) and returns the messages unchanged, then print one line per request:
request 1: counted 271, usage 271
and for the refused one: request 12: counted 8277, refused
Trap: every count must equal the API's usage exactly; an assistant message with tool calls has content null.
- Task 2
compact.ts: clear old tool results
In compact.ts, export compact(messages, keepLast = 4, minTokens = 200): a new array of the same length in which every tool message with more than minTokens tokens, except in the first two messages and the last keepLast, has its content replaced by STUB, keeping its role and tool_call_id. Everything else stays as it was, and the array it was given does not change.
Export prepare(messages): compact(messages) when countTokens(messages, TOOLS) is over BUDGET (6000), else the messages as they are.
Under if (import.meta.main), run T10 with runAgent without prepare and then with it, and print:
plain: error after 12 requests, largest prompt 7859 tokens
compacted: STOP after N requests, largest prompt P tokens
Answer: TEXT
Trap: removing a tool message instead of clearing it leaves its call unanswered, and the API refuses the request.
- Task 3
summarise.ts: a summary from the model
In summarise.ts, export async function summarise(messages): send lab-chat the messages and then one user message asking it to summarise this coding session so far, with no tools, and return the reply's text.
Export async function compactWithSummary(messages, keepLast = 4): the system prompt, then a user message "Summary of the session so far:\n" followed by the summary of the messages before the tail, then the tail (the last keepLast messages, started earlier at the assistant call if it would begin with a tool message).
Under if (import.meta.main), run T10 with a prepare that compacts with a summary when the messages are over BUDGET tokens. Print the first summary message, then summaries: N, STOP after S requests, then T10: passed or T10: failed.
Trap: a tail that starts with a tool message is refused (its call is in the summary, not in the messages).
- Task 4 · interview question
Interview: keep-task.ts, a compactor that loses the task
A teammate's trim() keeps T10 inside the window, and the run answers, but T10 fails. Run keep-task.ts, read what the model was sent after the first trim, and fix trim(messages, keepLast = 4) so that the system prompt and every user message are kept as they are, old tool results over 1,000 characters become STUB, everything else stays, and the last keepLast messages are as they were. Keep the prints: fixed, it ends with T10: passed.
Trap: the run does not crash. It answers, so only reading the messages shows what went missing.
What you will learn
- Counting a conversation as the API does: message by message, tool calls and tools included, checked against usage.prompt_tokens.
- A budget below the window, checked before every request.
- Clearing old tool results: the call stays, its result becomes a short stub; the system prompt, the task and the last steps stay as they are.
- Summarising old turns with the model, and what a summary keeps and loses.
- The one rule of every compactor: never lose the task.
More in Context
Build The context window yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Harness engineering (TypeScript) are free. This one comes with the all labs plan.