AI engineering traps · Chapter 11 of 16

Tool calls left unanswered

Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Tools
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    repro.py: ask two things at once, on purpose

    Write reproduce() in repro.py. Build fake services, an object whose dues(flat) returns 4500 and next_tanker(society) 'Saturday at 9 am'. Ask ask_sahayak from assistant.py, for the resident Resident("B-304", "Gomti Heights"), the ticket's question exactly: 'What are my dues and when is the next water tanker?'. Return the answer; if ask_sahayak raises an openai.APIStatusError, catch it and return 'error ' followed by its status (e.status_code).

    The check runs your reproduce() twice: beside the original assistant.py it must return 'error 400', and beside a fixed assistant.py 'Your dues are ₹4,500. The next water tanker comes on Saturday at 9 am.' ask_sahayak uses the lab model when you leave out its client; reproduce() must finish within 10 seconds.

  2. Task 2

    assistant.py: answer every tool call the model made

    Fix ask_sahayak in assistant.py. When the model asks for tools, run every call it made, in order, and send each result back as its own tool message whose tool_call_id is that call's id: one call, one tool message; two calls, two. Keep the assistant message with every call in it, the tools, run_tool and the printed line sahayak: the model asked for <n> tool calls as they are.

    The check uses its own resident and services with the lab model: a question that needs both tools must get the answer with both the dues and the tanker, after a request that answers each call by its id, and a question that needs one tool still works. Dropping the second call from the assistant message is not a fix.

  3. Task 3

    test_assistant.py: a test that would have caught it

    Write a pytest file, test_assistant.py, that would have caught this bug before residents met it. The check runs your file three times: beside the original assistant.py, where at least one test must fail; beside the fixed assistant.py, where every test must pass; and beside a wrong fix that drops the second tool call, where at least one test must fail.

    So ask a question that needs both tools, with fake services of your own, and assert the answer contains both the dues and the tanker time. Each test must finish within 10 seconds.

What you will learn

  • repro.py: ask two things at once, on purpose
  • assistant.py: answer every tool call the model made
  • test_assistant.py: a test that would have caught it

More in Tools

Build "Tool calls left unanswered" yourself

Sunwai, a Lucknow app for housing societies, runs on a model, and Farah, its CTO, keeps assigning you tickets. Each is a trap AI engineering interviews love to ask about. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI engineering traps are free. This one comes with the all labs plan.