Build a voice agent · Chapter 10 of 21
Barge-in: when the caller talks over Bolo
By the end of this chapter you can let Bolo listen while it talks, tell a real interruption from a "haan", a pressure cooker or a half-heard guess, stop a reply everywhere it lives at once, say why a barge-in on loudness alone fails, and take apart the time between a caller cutting in and Bolo going quiet. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Real time
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.
Why you need this
Ritesh is in a hurry. He asks what a service includes and what it costs, and six words into Bolo's answer he cuts in: "achha achha, pickup aaj ho jayega kya?" (OK, OK, can you pick it up today?). People do this all the time on the phone, and they expect the other side to stop.
Bolo so far does not. It talks to the end of its reply whatever the caller does. Here is Ritesh's call, from the line's record:
10.06- 15.92 Bolo scooter ki general service mein engine oil badla jaata hai air
filter aur spark plug saaf hote hain aur brake aur chain check hote hain
11.40- 13.07 interrupt achha achha, pickup aaj ho jayega kya?
16.20- 18.40 Bolo scooter ki general service aath sau ninyaanve rupaye ki hai
Bolo talked over him for 4,520 ms: through his whole question, and on for almost three seconds after it, then on to the price he had stopped waiting for. Kavita has a word for agents like that: "a recording". Callers hang up on recordings.
But "stop when the caller makes a sound" is not the answer either. Meena says "haan" while Bolo explains the process: she is listening, not asking to speak. Mrs Iyer's pressure cooker whistles for 1.3 seconds in the middle of her answer. Stop for either, and those callers lose the answer they asked for.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
interruption.py: is the caller interrupting?
Write def is_interruption(words) -> bool: True when at least one word has a confidence of 0.5 or more and is not in BACKCHANNELS (given in the file), False otherwise (no words, only listening noises like "haan" and "achha achha", or only a guessed word below 0.5). Each word is a dict with "word", "start", "end" and "confidence".
- Task 2
barge_in.py: stop talking
Write def stop_talking(ctx, task, speaker, playback): playback.clear(), speaker.cancel(ctx) and task.cancel() (task is the reply's asyncio task). Through the given agent: Bolo stops within 1,200 ms of Ritesh talking over it, with a clear, and the reply's speech context is cancelled; Mrs Iyer's pressure cooker does not stop Bolo and her call goes well; Meena's "haan" and "achha" do not stop Bolo and her call goes well.
- Task 3
loud.py: a barge-in on loudness
Write class LoudSound with push(frame) -> bool: True once 3 frames in a row have an RMS above 0.05, and while they stay loud; a quieter frame starts the count again. Under if __name__ == "__main__":, run an async main() that imports run_call from agent, calls "kitchen-whistle" twice at once with barge_in="words" and barge_in="energy" (both with loud_sound=LoudSound), and for each in that order prints WAY: Bolo said "TEXT"; the list heard: yes or no (TEXT is rec["replies"][1]["text"] of the call's record; yes when its goal "brakes" was met).
- Task 4 · interview question
Interview: talk_over.py, where does the talk-over go?
Run run_call("ritesh-hurry"), read its record, and print Ritesh started talking over Bolo at T s (the interrupt's "at"), the first real word, WORD, ended at T s (run.stops[0]), Bolo stopped at T s ("agentStoppedAt"), then talk-over: N ms: A waiting for a real word, B to stop (N is "talkOverMs"; A from his start to the word's end, B from the word's end to the stop, in whole milliseconds; times with 2 decimals). The file ends with asyncio.run(main()).
What you will learn
- Full duplex: Bolo listens while it talks, through the same speech-to-text.
- What counts as an interruption: sure words that are not listening noises.
- Stopping everywhere at once: the line's queue, the speech still being made, the reply's task.
- Cancelling a task: task.cancel() and asyncio.CancelledError.
- Why loudness alone fails: a barge-in on energy, measured.
- Talk-over: where the time between "he cut in" and "Bolo stopped" goes.
More in Real time
Build "Barge-in: when the caller talks over Bolo" yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.