Memory and multi-agent systems · Chapter 14 of 16
When agents fail
By the end of this chapter you can stop two agents handing a conversation back and forth for ever, stop a model that calls the same tool again and again, give a conversation a budget in tokens and rupees, and let a referee choose between two specialists who disagree. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Many agents
- 5 tasks
- Python
- In the all labs plan
The first 2 chapters of Memory and multi-agent systems are free. This one comes with the all labs plan.
Why you need this
Last chapter's team works when every specialist knows its job. Rekha brought the morning after it did not. A family asked "How much will the Jain counter add to our budget?". Catering handed it to budget, budget handed it back to catering, and the logs showed them doing it all night. Sameer brought the bill for that night. And Pooja brought a worse one: a cousin had asked about the budget, and two specialists had given two different numbers, one of them January's.
None of these is a bug in one line of code. Each is what agents do when nothing stops them: a model has no sense of going round in circles, of time, or of money. Karan: "Every loop gets a limit, every limit has a fallback, and the fallback is a person."
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
pingpong.py: a hop limit
Write run(text, wedding_id, start, max_hops=3) in pingpong.py. The specialist start gets the text: a request to lab-chat at temperature 0 with its system(wedding_id), the text as the user message, and its transfer_tools from tools.py. When it calls transfer_to_ a specialist, that specialist gets the same text the same way. The first reply without a tool call is the answer. When a specialist asks for a handover after max_hops handovers, a planner takes it: the path ends with planner and the answer is PLANNER.
run returns the answer and the path: every agent the conversation reached, in order. Under if __name__ == "__main__":, run R091 from its route specialist and print the path joined with a space, an arrow made of a hyphen and a greater-than sign, and a space, then the answer.
- Task 2
loopy.py: a tool called again and again
Write answer(question, model="lab-chat-loopy", max_steps=5) in loopy.py: send SYSTEM and the question with tools=[RECALL] at temperature 0; while the reply has tool calls, run each with recall, add the assistant message and a tool message with each call's JSON result, and ask again. A call with the same name and arguments (compared as data) as one already run is not run: send the messages once more with tool_choice none and return that reply. Do the same after max_steps requests.
answer returns the reply and how many tool calls ran. Under if __name__ == "__main__":, answer What is the budget for WD1063? and print tools run: and the number, then the reply.
- Task 3
budget.py: a budget per conversation
Write prices(model), the model's price_inr_per_million from the API's list of models, and cost(usage, model), a request's cost in rupees: cached prompt tokens (prompt_tokens_details, which may be missing) at cached_input, the other prompt tokens at input and completion tokens at output, per million. Write BudgetSpent and Conversation(max_tokens, max_rupees, model="lab-chat") with history, tokens and rupees spent, and ask(message): before sending the history plus the message, raise BudgetSpent if the tokens spent plus the request's tokens as baraat's tokens counts them pass max_tokens, or the rupees spent are max_rupees or more; else send it at temperature 0, keep the message and reply in the history, add usage total_tokens and the cost, and return the reply.
Under if __name__ == "__main__":, send WD1063's thread messages as baraat's as_message, one by one, to Conversation(max_tokens=20000, max_rupees=3); after each print turn, its number, a colon, the tokens spent, tokens, a comma and the rupees spent with ₹ and 4 decimals; when it stops print stopped before turn, the number, a colon, the tokens, tokens and, the rupees the same way and spent.
- Task 4
referee.py: a referee between two specialists
Write judge(question, a, b, reference, model="lab-chat"): ask at temperature 0, with an instruction to compare the answers against the reference and reply A, B or tie, and the lines Question:, Answer A:, Answer B: and Reference:, each with its text; return A or B when the reply is that letter (ignoring case, spaces and a full stop), else tie. Write referee(question, first, second, reference, model="lab-chat"): judge with first as A, then with second as A; return the answer both verdicts pick, or None.
Under if __name__ == "__main__":, ask What is the budget? of two budget specialists for WD1057 at temperature 0, one with notes holding its budget as booked, then one with the budget in memory, and print from the booking: and from memory: with their replies; then lab-chat picks: and lab-chat-biased picks: with the referee's choice (reference: budget: and the budget in memory), or a planner.
- Task 5 · interview question
Interview: fix_me.py, a loop detector that stops the wrong conversations
first_repeat(handoffs) in fix_me.py should return the index of the first handover that repeats an earlier one, or None. Each handover is a dict with from, to and reason; it repeats when an earlier one went from the same agent to the same agent for the same reason, the reasons compared without case and without spaces at the ends. Run it: a client who comes back to catering with a new question is stopped as a loop.
Fix first_repeat. Keep its name and parameters. The check tries real loops and conversations that only look like them.
What you will learn
- Handoff loops between agents, and a hop limit (policy 6.4).
- Tool loops: a model that repeats a call, and how to notice.
- A budget per conversation, in tokens and in rupees, from the price list.
- A referee: a judge that compares two answers against what memory knows.
More in Many agents
Build "When agents fail" yourself
Baraat Bliss, a Jaipur wedding planner, must remember months of family decisions across many agents. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Memory and multi-agent systems are free. This one comes with the all labs plan.