Build a voice agent · Chapter 11 of 21

Keeping the conversation true

By the end of this chapter you can work out exactly which words of a reply the caller heard before cutting in, put only those words in the conversation the model reads, and explain the bug every voice agent with barge-in has until it does this: the model believing it said things the caller never heard. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Real time
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.

Why you need this

Chapter 10 taught Bolo to stop when Ritesh cuts in. He asked what a service includes and what it costs, interrupted six words into the answer to ask about pickup, got his pickup answer, and hung up. He never heard the price. The call failed.

Nothing crashed, and every part did its job. The bug is in what Bolo remembers. The conversation it sends to the model on every turn still says Bolo gave the whole answer:

assistant: Scooter ki general service mein engine oil badla jaata hai, air filter aur
           spark plug saaf hote hain, aur brake aur chain check hote hain. Scooter ki
           general service ₹899 ki hai.

Ritesh heard "scooter ki general service mein engine oil badla jaata hai". The model reads its own message, sees the price there, and does what any model does: it does not repeat what it has already said. On the next turn it answers only the new question. Tejas, who cares most about records matching reality, put it simply: "The history is a record. A record of things that did not happen is worse than no record."

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    heard.py: the words the caller heard

    Write heard_words(words, played_to): the words whose "end" is at or before played_to, joined with spaces, with "..." straight after the last one when some words were not heard; "" when no word was heard (or there are no words).

  2. Task 2

    history.py: keep the conversation true

    Write cut_reply(history, heard): change the list it is given; replace the last assistant message with {"role": "assistant", "content": heard}, or remove it when heard is ""; leave every other message alone. Through the given agent (which calls it in the reply's except asyncio.CancelledError): after Ritesh cuts in, the history holds what he heard (ending in "...", the same words the line says he heard, at most two words short), he hears his price later in the call, and Mr Pandey's call (never interrupted) keeps every reply whole and goes well.

  3. Task 3 · interview question

    Interview: as_i_said.py, the "as I said" bug

    Call "ritesh-hurry" twice at once, with run_call(..., truncate=False) then truncate=True. For each in that order, with NAME "whole reply kept" then "cut to what was heard", print NAME, in the history: TEXT (the second assistant message), NAME, Ritesh heard: TEXT (rec["replies"][1]["text"] of the record), NAME, next reply: TEXT (the third assistant message) and NAME: price heard yes or no (the goal "price").

What you will learn

  • What was heard: the speech's word times and the line's marks, together.
  • Where the cut happens: in the reply task's except asyncio.CancelledError.
  • Cutting the reply: replace Bolo's last message with what the caller heard.
  • Why it matters to the model: a model treats its own messages as said.
  • The "as I said" bug: measured, side by side with the fix, on the same caller.

More in Real time

Build "Keeping the conversation true" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.