AI engineering traps · Chapter 10 of 16

A tool loop with no step limit

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Tools
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.py: keep Sahayak checking the tanker on purpose

    Write reproduce() in repro.py. Build a fake tanker API, an object whose status(booking_id) always returns {"status": "on_the_way"} and counts how many times it was asked; on its 40th check it raises an error, after counting it. Ask ask_sahayak from sahayak.py 'Where is my water tanker TK-5521?' with your fake, catch the error if it is raised, and return how many times the fake was asked.

    The check runs your reproduce() twice: beside the original sahayak.py it must return 40 (your fake stopped the loop), and beside a fixed sahayak.py 6 (the step limit stopped it). So the fake must really be asked. ask_sahayak uses the lab model when you leave out its client; reproduce() must finish within 10 seconds.

  2. Task 2

    sahayak.py: give the loop a step limit and a way out

    Fix ask_sahayak in sahayak.py. Add a fourth parameter, max_steps, 6 when it is left out. One step is one request to the model: make at most max_steps of them. While the model answers with text, return it as today; while it asks for a tool, run it as today and go round again. When max_steps requests are made and it still wants tools, print sahayak: stopped after <max_steps> steps and return HANDOFF.

    Keep HANDOFF, the tool and the printed line sahayak: answered in <n> steps as they are. The check uses its own fake tankers with the lab model and counts the requests: a tanker that never arrives must take exactly 6 requests (or max_steps), and one delivered on the second check must still get the model's answer, in 3. A time limit is not a step limit.

  3. Task 3

    test_sahayak.py: a test that would have caught it

    Write a pytest file, test_sahayak.py, that would have caught this loop before the bill. The check runs your file three times: beside the original sahayak.py, where at least one test must fail; beside the fixed sahayak.py, where every test must pass; and beside a wrong fix with a 30-second time limit, where at least one test must fail.

    So give ask_sahayak a fake tanker that stays on the way and counts its checks (and raises after, say, 20, so the test also ends beside the original code), and assert HANDOFF comes back after at most 6 checks. Each test must finish within 10 seconds.

What you will learn

  • repro.py: keep Sahayak checking the tanker on purpose
  • sahayak.py: give the loop a step limit and a way out
  • test_sahayak.py: a test that would have caught it

More in Tools

Build "A tool loop with no step limit" yourself

Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.