AI ordering agent project · Chapter 12 of 12

Budgets, timeouts and streaming

By the end of this chapter you can limit what one conversation may cost in tokens, pay from the wallet and take in time; stop a model that does not answer; and send the member each step as it happens, over server-sent events. Then you take the whole agent home. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Ready for members
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of AI ordering agent project are free. This one comes with the all labs plan.

Why you need this

Three people, three numbers, the week before launch.

Rafiq, the token bill. Every step resends the whole conversation, so each step costs more than the one before. One booking, from its trace:

model  lab-chat     743 59 888
tool   find_slots   None None 9
model  lab-chat     867 60 911
tool   find_dish    None None 11
model  lab-chat     964 53 801
tool   place_order  None None 12
model  lab-chat     1066 31 470
{"model_calls": 4, "tool_calls": 3, "tool_errors": 0, "prompt_tokens": 3640, "completion_tokens": 203, "ms": 3102}

(Kind, name, prompt tokens, completion tokens, ms.) A conversation where Tara booked and paid twice reached 20,881 prompt tokens in 14 model calls. A loop that goes wrong, as lab-chat-loopy did in chapter 3, keeps paying until the step limit, and nothing adds the steps of a conversation together.

Rafiq again, the wallet. The agent pays whatever the member says yes to. A cap per conversation keeps one bad afternoon small.

Farah, the wait. A booking takes 3 seconds, and the member sees a spinner until the very end. With lab-chat-slow it is 4 seconds a step, four steps a booking, and a model that never answers would hold the request until the client gives up.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    budget.py: tokens and time

    In agent/budget.py, write Budget.charge(usage): add usage's prompt_tokens and completion_tokens to self.used_tokens; over self.max_tokens, raise BudgetExceeded("tokens", f"This conversation has used its {self.max_tokens:,} tokens. Please start a new conversation."). Then Budget.check_time(): when self.clock() - self.started is more than self.max_seconds, raise BudgetExceeded("time", f"That took longer than {self.max_seconds:g} seconds, so I stopped. Please try again.").

    The Check charges a budget of 2,000 tokens that has used 300 (1,000 more is 1,300; 700 more is exactly the limit; 1 more raises), then moves a fake clock to 30 s (fine) and 30.5 s (raises). Then Tara books through the app with agent_settings.max_tokens at 1,200: the second step goes past it, and the answer must be status over_budget with your sentence.

  2. Task 2

    budget.py: money

    In agent/budget.py, write Budget.allow_payment(amount): None counts as 0; when self.spent + amount is more than self.max_spend, raise BudgetExceeded("money", f"Paying ₹{amount} would take this conversation past its ₹{self.max_spend} limit (₹{self.spent} paid so far). Pay in the DabbaDrop app instead."); otherwise add amount to self.spent. Then budget_for(db, conversation_id): a Budget with agent_settings' max_tokens, max_spend and run_timeout (as max_seconds), whose used_tokens is the sum of prompt_tokens + completion_tokens of the conversation's model steps in agent_steps and whose spent is the sum of output["paid"] of its pay_order tool steps with no error (0 when there are none).

    The Check allows 550 on top of 300 (850), allows None, refuses 200 (and does not count it), then writes a trace of its own (two model steps of 850 and 920 tokens, a pay_order that paid 550, a failed one and another tool) and wants budget_for to read 1,770 tokens and ₹550, with the limits from the settings, and 0 and 0 for a new conversation. Then, with max_spend at 500, Tara books 2 Jain Thali (₹550) and approves paying: the result must be your refusal and the order still pending.

  3. Task 3 · interview question

    Interview: stream.py, stream the agent's steps

    In agent/stream.py, write sse(event, data): "event: <event>\ndata: <json.dumps(data, ensure_ascii=False)>\n\n". Then StreamHooks, which puts each step on its queue as an (event, data) pair: model_step gives ("model", {"tool_calls": [{"name", "arguments": arguments_of(c)}], "ms"}) for calls or ("model", {"text", "ms"}) for an answer; tool_step gives ("tool", {"name", "arguments", "result", "ms"}); waiting gives ("approval", {"question"}). Then stream_events(start): run start(StreamHooks(events)) in a daemon thread that puts ("answer", what start returned), or ("error", {"message": "The agent stopped: <the exception's class name>"}) when it raised, then None; meanwhile yield sse(*item) for each item until the None, then sse("done", {}).

    The Check compares your frame, the four kinds of steps on the queue, then streams a start that waits until the first frame has been read (it must arrive within 2 s), then the answer and done, and a start that raises (error, then done). Then POST /agent/chat/stream as Tara must send model, tool, model, tool, model, tool, model, answer, done.

    Kunal will ask: why run the turn in a thread and pass the steps through a queue, instead of yielding from inside the loop?

What you will learn

  • Tokens: what a model step costs, and a budget per conversation counted after every step.
  • Time: a model timeout and a run timeout, and a clock you can pass in.
  • Money: refuse before it moves, and count only what was allowed.
  • A budget that survives requests: read from the trace with SUM, COALESCE and a JSONB key.
  • Server-sent events: text/event-stream, frames of event: and data: lines.
  • A thread and a queue: steps sent the moment they happen.

More in Ready for members

Build "Budgets, timeouts and streaming" yourself

DabbaDrop members want to order by chat. You build the agent that books their tiffins through DabbaDrop's own API, and asks before it pays. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of AI ordering agent project are free. This one comes with the all labs plan.