Harness engineering (TypeScript) · Chapter 1 of 23
What a harness is
By the end of this chapter you can say what a harness is and why it decides most of what a coding agent can do, build the smallest one that works (three tools and a loop), watch the same harness fail on a longer file, and put a number on how good a harness is. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: The core
- 4 tasks
- TypeScript
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
Hisaab, in Lucknow, makes GST invoices, UPI payment links and payment reminders for small businesses: kirana stores, a furniture shop in Aminabad, a printing press, a tiffin service. Its code is small and well tested, and its list of small bugs is long: a rupee amount grouped the Western way, a due date that lands on a Sunday, a UPI link that breaks when the shop's name has a space in it.
Farah, the founder, wants a coding agent to clear that list: read the issue, find the code, fix it, prove it works. She also wants to know how often it gets one right, because an agent that is right 4 times in 12 is a liability and one that is right 12 times in 12 is a colleague.
Kabir, Hisaab's staff engineer, says the model is the smaller part of that answer. The same model can be right 4 times in 12 or 12 times in 12, depending on what is built around it: which tools it gets, what they show it, what it is told, what it is not allowed to do, and what checks its work. That is the harness, and building and measuring it is this lab.
The lesson
A model only writes text
Hisaab's repository is in /workspace/hisaab, read-only: TypeScript run by Node, with no build step and Node's own test runner. The lab's workbench (/workspace/shared/workbench.ts) makes a checkout: a fresh copy of it with one task's bug in it, committed with git, in a folder of its own:
import { checkout, task } from '../../shared/workbench.ts';
const root = checkout('T02');
console.log(root);
console.log(task('T02').issue);
/tmp/hisaab-runs/T02-k3r1x0
`amountInWords(1000000000)` throws a TypeError. Exactly one crore rupees should read 'Rupees One Crore Only'.
lab tasks lists all twelve. Send an issue to lab-chat with no tools, and the reply is confident:
Done! I fixed `amountInWords` in src/money.ts, so it now gives 'Rupees
One Crore Only'. The tests pass now.
Nothing changed. amountInWords is not even in money.ts (it is in words.ts), and diff(root), the checkout's git diff, is empty: the model has no hands. It read a description of a bug and wrote the kind of text that usually follows one. A real model does better than this stand-in at guessing the fix, and it can show you a patch, but it still cannot open a file, run a test or know whether its patch applies. Task 1 sends T01's issue and looks at the diff.
(lab-chat is this lab's own model, with the same API as OpenAI's, so the official openai npm package works unchanged. It is a predictable stand-in for a coding model, not a neural network: it acts on what your harness shows it, the same way every time, which is what lets this lab measure harnesses. OPENAI_BASE_URL and OPENAI_API_KEY point the package at it.)
A harness is everything around the model
A coding agent is a model in a loop: the model asks for a tool, your code runs it and sends the result back, and the model asks again, until it answers in text. Everything that is not the model is the harness:
| Part | What it decides | Chapters |
|---|---|---|
| the loop | when to ask again, when to stop | 1, 2 |
| the tools | what the model can do: read, search, edit, run | 3 to 6 |
| the results | what the model sees: a whole file, a page, the last 40 lines | 7 |
| the context | what it knows: the system prompt, the project's rules, what to leave out | 8 to 12 |
| the controls | what it may not do: permissions, a sandbox, hooks | 13 to 15 |
| the checks | whether the work is done: tests after every change | 16 |
| the scale | helpers, memory, traces and cost | 17 to 19 |
| the measurement | how good all of it is, on a suite of tasks | 20, 21 |
The model only ever writes requests. Which of them run, what the results look like, and whether "done" is believed are your code's decisions. That is why the same model behaves so differently in two harnesses.
The smallest harness: three tools and a loop
A coding agent needs, at the least, to find a file, read it and change it. Three tools do that:
- list_files: every file in the repository, one path per line, relative to it.
- read_file(path): a file's text.
- write_file(path, content): the whole file, replaced.
A tool is a description the model reads, a name, a sentence and its parameters as JSON Schema, and a function your code runs. Here is one that is not in the tasks, to show the shape:
const countLines: ChatCompletionTool = {
type: 'function',
function: {
name: 'count_lines',
description: 'How many lines a file of the repository has.',
parameters: { type: 'object', properties: { path: { type: 'string' } }, required: ['path'] },
},
};
Typed as ChatCompletionTool (from openai/resources/chat/completions), type: 'function' stays a literal, which the tools parameter needs.
Every tool runs inside the checkout: the model gives paths like src/money.ts, relative to the repository, and your code joins them to the checkout's folder. The loop around them is the one any agent has: send the messages and the tools, run every call the model asks for, append the model's own message and one tool message per call, ask again (a tool call is a union, so check call.type === 'function' before reading call.function), and stop when it answers in text or after a fixed number of requests.
One run, step by step
T01's issue: formatInr(12345650) returns '₹123,456.50', and Indian invoices need '₹1,23,456.50'. Under the three-tool harness, the requests go like this:
request 1 205 prompt tokens asks list_files()
request 2 371 asks read_file(src/invoice.ts)
request 3 1013 asks read_file(src/money.ts)
request 4 1725 asks write_file(src/money.ts, ...the whole file...)
request 5 2543 answers: Done: formatInr now groups lakhs and
thousands with groupIndian (src/money.ts).
- With no way to search, it listed the files and guessed:
invoice.tsfirst (the issue says "invoices"), thenmoney.ts. - It wrote the whole file back, with one line changed. That reply was 811 completion tokens: every line of the file, written out again.
- Each request resends everything: the prompt grows from 205 to 2,543 tokens, because the files it read are now part of the conversation.
grade(root) then runs the repository's 45 tests (node --test) in the checkout: T01 passes. Task 2 builds this harness.
The same harness, a longer file
T03 is a bug in gstSplit, at the bottom of src/gst.ts, below a 192-row table of HSN codes and their GST rates. The same harness, the same model: it lists, reads gst.ts, and writes it back. The file it writes starts like this:
*/
// ... rest of the table unchanged ...
export function hsnRate(hsn: string): number {
The bug is fixed, and the file is broken: the table and the line that declares it are gone. Asked to write a long file out again, a model shortens it: the table it did not need to change became a comment. Real models do exactly this with long whole-file rewrites, which is why coding assistants edit by replacing a piece of text instead (chapter 4). Nothing in the harness noticed: it ran the write, sent back "ok" and believed the model's "Done". Task 3 runs it and counts the lines.
A harness is measured
"It worked on T01" says little. Nandini, Hisaab's QA lead, wants a number that means something: run the harness on a set of tasks, each in a fresh checkout, grade each one the same way, and count. That is a success rate, and it is how every chapter of this lab will judge a change to the harness: did more tasks pass, at what cost, and did any that passed before now fail?
Hisaab's suite has twelve tasks, T01 to T12, each built to need one part of a good harness. With the lab model, a naive harness (whole files in and out, raw command output, a one-line prompt) passes 4 of 12. The harness you build by chapter 22 passes 12 of 12, with the same model. The interview task scores the three-tool harness on T01, T02 and T03.
Your turn
Send T01's issue with no tools and look at the diff in task 1. Build the three-tool harness and fix T01 in task 2. Run it on T03 in task 3, and in the interview question, give it a score.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
just-ask.ts: a model with no harness
Farah wants to see what a model does with a real bug report on its own. Make a fresh checkout of the repository with task T01's bug in it (checkout, from ../../shared/workbench.ts), and send lab-chat two messages: a system message saying it is Hisaab's coding agent, then T01's issue, exactly as task('T01').issue has it, as the user's message.
Send no tools. Print the reply's text. Then print how many lines the checkout's diff has now (diff), as:
Diff lines: N
- Task 2
tiny-harness.ts: the smallest harness
Kabir's smallest harness: three tools and a loop. In tiny-harness.ts, export TOOLS, three tool descriptions: list_files (no parameters: every file in the repository, one path per line, relative to it), read_file (path: the file's text) and write_file (path and content: writes the whole file, and returns ok).
Then export async function runHarness(root, issue, maxRequests = 10). It sends lab-chat a system message saying it is Hisaab's coding agent and the issue as the user's message, with TOOLS. Each time the model asks for tools, it prints each tool's name, runs it inside root (paths are relative to root) and sends the results back. When the model answers in text, it returns { answer: the text, requests: how many requests it sent }; after maxRequests requests with no answer, { answer: null, requests: maxRequests }.
Under if (import.meta.main), run it on a checkout of T01, print the answer, then grade the checkout and print T01: passed or T01: failed.
- Task 3
big-file.ts: the same harness, a longer file
Nandini has a bug in a longer file. In big-file.ts, make a checkout of T03 and print its path as checkout: PATH. Count the lines of src/gst.ts in it, run runHarness (import it from ./tiny-harness.ts) on T03's issue, and count them again. Print:
gst.ts: B lines before, A after
Then grade the checkout and print T03: passed or T03: failed.
- Task 4 · interview question
Interview: scorecard.ts, a harness's success rate
"How good is your agent?" needs a number. In scorecard.ts, run runHarness on a fresh checkout of each of T01, T02 and T03, grade each one, and print one line per task, in that order:
T01 passed 5 requests
(the task, passed or failed, and the requests the run sent, two spaces apart). Then print the success rate as:
Success: P/3 (R%)
with R rounded to a whole number.
What you will learn
- What a model does on its own with a bug report, and why that is not work.
- What a harness is: the loop, the tools, the context, the rules and the checks around a model.
- The smallest harness: three tools and a loop, fixing a real bug.
- Why a harness is measured: the same harness on a longer file, and a success rate.
More in The core
Build "What a harness is" yourself
Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness in TypeScript and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.