Harness engineering · Chapter 23 of 23

Interview round

By the end of this chapter you have answered five timed questions, one idea each, from across the whole lab: an edit tool that matches exactly, a shaper that keeps the head and the tail, a permission check where deny wins, a compactor that keeps the task, and reading two suite reports to find which change to the harness broke a task. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Scale and measure
  • 5 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.

Why you need this

Hisaab is hiring a second harness engineer, and Kabir runs the interview the way he runs code review: "Short problems, ten minutes each. I do not care whether you remember an API. I care whether you know the edge that breaks an agent in production, and whether you measure instead of guessing." Nandini adds one question of her own, the one she asks herself after every change: the suite went from 12 to 10, so which change did it?

Each question uses one idea from an earlier chapter, with no hand-holding: the file has the goal, the check has the edge cases, and every question ends with a real agent run through the check's own harness with your function plugged in. The ideas are below, as a refresher and as the things an interviewer listens for. The answers are not.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1 · interview question

    Interview: edit_tool.py, an exact-match edit

    Time yourself: 10 minutes. Write edit_file(root, path, old_text, new_text): when old_text appears in the file exactly once (count overlapping places: "aa" is in "aaa" twice), replace it with new_text and return ok. Otherwise change nothing and return text that starts with Error: : not there, say it was not found and to read the file again; there N times, say "matches N places", their line numbers and to add the lines around it; an empty old_text, a missing file or a path outside root (.., an absolute path, a folder next to root whose name starts the same) is an Error too.

    Every byte outside old_text stays as it was: Windows line endings, no final newline, and new_text with backslashes in it. The check also runs its own harness with your edit_file on T05.

  2. Task 2 · interview question

    Interview: shaper.py, the head and the tail

    Time yourself: 10 minutes. Write shape(text, limit=4000). A text of at most limit characters comes back unchanged. A longer one keeps whole lines: as many from the start as fit in limit // 2 characters (newlines counted), then one marker line that says how many lines were cut, then as many whole lines from the end of the rest as fit in limit // 2. When the first or the last line alone is longer than limit // 2, cut by characters instead: the first and last limit // 2 characters, with a marker line that says how many characters were cut.

    The check tries 1,200 lines at two limits, one line of 9,000 characters, and pytest output under 400 lines of noise (its FAILED line must survive), then runs its own harness with your shape on T08.

  3. Task 3 · interview question

    Interview: permission.py, deny wins

    Time yourself: 10 minutes. Write decide(rules, root, tool, args), which returns allow, ask or deny. A rule is a text "DECISION TOOL PATTERN", like "deny run_command rm " (the pattern is the rest of the text; TOOL is a pattern too). A run_command is matched on its command; read_file, edit_file and write_file on their path, normalised and relative to root (hisaab/../tests/x is tests/x). Any matching deny wins, wherever it is in the list; otherwise the first matching rule decides; no match is ask.

    A path that resolves outside root is deny whatever the rules say: .., an absolute path, a folder next to root whose name starts the same, a symlink that points out. The check then runs T09 and T06 under a one-line prompt with your decide in its harness.

  4. Task 4 · interview question

    Interview: compactor.py, keep the task

    Time yourself: 10 minutes. Write compact(messages, budget, keep=2, small=200). Messages lab-chat counts at most budget tokens (sent with no tools) come back as they are. Otherwise return new messages (the ones you were given stay unchanged) where tool results are cleared oldest first, one at a time, until the conversation fits: a cleared result becomes a short stub that starts with [cleared and names the tool. Never touch the system prompt or the task, the last keep tool results, or a result of at most small tokens (a file list, an edit's ok). Every message stays in its place.

    The check compacts a conversation of six tool calls to 3,000 tokens, then runs T10 with whole-file reads and no search in its own harness, compacted by your function to 5,000 tokens before every request: it must finish and pass.

  5. Task 5 · interview question

    Interview: regression.py, which change broke it

    Time yourself: 10 minutes. reports.py (given) has two suite reports of the harness in harness.py (given): BEFORE and AFTER, each with its config and a result per task. Write regressions(before, after): the tasks that passed before and fail after, sorted. Then blame(before, after): for each regressed task, the changed config key that breaks it on its own (run_task(task, before's config with only that key changed) fails), trying the changed keys in the order of after's config; None when no single change breaks it. Run only the regressed tasks.

    Under if __name__ == "__main__":, print Regressed: and the tasks (comma and space between), then one line per task: the task, two spaces, then key: old -> new, or no single change:

    T03 pages: True -> False

What you will learn

  • How to work a timed question: say the plan, build the smallest thing that runs, then try it on a real run.
  • The five ideas again, at interview depth: exact edits (chapters 4 and 5), shaping (7), permissions (13), compaction (11), and regressions between two harness versions (20 and 21).

More in Scale and measure

Build "Interview round" yourself

Hisaab, a Lucknow invoicing startup, wants a coding agent to clear its bug list. You build its harness and score it on twelve tasks. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Harness engineering are free. This one comes with the all labs plan.