AI ordering agent project · Chapter 5 of 12

Errors from tools

By the end of this chapter your agent survives everything a tool call can run into: the API saying no, the model writing a wrong argument, a bug in a tool, an API that is down for a moment. Each one becomes a result the model reads and acts on, a bug leaves its traceback in the server log and nothing private in the model's hands, and a call is tried again only when trying again may work. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: When things go wrong
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI ordering agent project are free. This one comes with the all labs plan.

Why you need this

Thane's 12:30 slot has 30 places, and in launch week it will fill. When it does, the API answers 409 slot_full, "The 12:30 slot is full". Tara should hear exactly that, and be able to pick 13:00. Today she gets a 500 "Something went wrong on our side", because run_tool raises and the exception ends her whole request.

It is the same for the model's own mistakes. In chapter 4, your parse_args found lab-chat-confused writing zone_name instead of zone, said so in one clear sentence, and that sentence ended up in the server log while Tara got a 500. The model, which could have fixed it in one more step, never saw it.

Rafiq adds one more worry from the pilot. A payment was taken twice because a member's screen hung and she pressed Pay again. An agent that "tries again" on its own can do the same, so retries need rules.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    run.py: refusals become results

    Write run_tool(ctx, name, arguments, sleep=time.sleep) in agent/tools/run.py so that it never raises. args = parse_args(name, arguments); on BadArguments e return {"error": str(e)}. Then run FUNCTIONSname through call_with_retries (Task 3) with agent_settings.tool_retries and sleep; on an ApiError e return {"error": e.message, "code": e.code}; otherwise return the tool's result.

    The Check fills slot 80 and closes slot 79, then runs place_order as Tara in the full slot (slot_full), the closed one (slot_closed) and with Andheri Kitchen's Dal Tadka in a Thane slot (bad_dishes), find_dish for Jain Thali (the tool's own list), find_slots with zone_name instead of zone, and a tool called book_dabba (parse_args's sentences). Then it runs Tara's booking with lab-chat-confused, a model that gets one argument wrong in its first call: the run must go find_slots (wrong), find_slots (right), find_dish, place_order, and end answered.

  2. Task 2

    run.py: a bug in a tool must not end the run

    Make run_tool catch any other exception too: print its traceback for the server log (traceback.print_exc()) and return {"error": "The tool <name> failed (<the exception's class name>). Tell the member it did not work."}, never the exception's own message.

    The Check swaps find_dish for a function that raises ValueError("password=dabba-secret-123 ...") and wants an error that names find_dish and ValueError but not the password, and the traceback (with the password) on the server log.

  3. Task 3 · interview question

    Interview: run.py, try again only when it may work

    Write call_with_retries(fn, retries, sleep=time.sleep): return fn(); when it raises an ApiError whose status is in RETRY_STATUSES (502, 503, 504), call sleep(0.5 * 2 ** attempt) (0.5, then 1.0, ...) and try again, up to retries more times, then raise the error. Any other error is raised at once, without a sleep.

    The Check calls it with two 503s then success (it wants "done" after sleeping 0.5 and 1.0), three 504s (the 504 raised after 0.5 and 1.0), a 409 and a ValueError (raised at once, no sleep). Then it makes list_my_orders time out once and wants run_tool to retry it after sleeping 0.5 through the sleep it was given.

    Kunal will ask: place_order timed out (a 504). Should run_tool try it again? What could happen to Tara?

What you will learn

  • An error is a result: why every tool call needs its tool message, and the 400 when one is missing.
  • What the model does with an error: reads it, and answers or asks again.
  • Three kinds of trouble: the model's arguments, the API's refusals, a bug in your code.
  • Bugs: traceback.print_exc() for the log, and why the exception's message never goes to the model.
  • Retries: which statuses are worth another try, waits that double, and a sleep you can replace in tests.
  • What a timeout does not tell you: whether the request happened.

More in When things go wrong

Build "Errors from tools" yourself

DabbaDrop members want to order by chat. You build the agent that books their tiffins through DabbaDrop's own API, and asks before it pays. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI ordering agent project are free. This one comes with the all labs plan.