Build a voice agent · Chapter 7 of 21
When has the caller finished?
By the end of this chapter you can say why silence alone cannot tell you that a caller has finished, use the words they stopped on to decide how long to wait, take back an answer that started too soon, measure the trade between cutting people off and making them wait, and ask the speech-to-text for its final the moment you know the turn is over. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Sound and the pipeline
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.
Why you need this
On a call, nobody says "over". People stop talking for all kinds of reasons: the sentence is finished, they are taking a breath, they are thinking of the next word, they are looking for a booking number. Kavita hears it all day at the desk: "Callers stop in the middle of a sentence. Mr Gupta says 'meri pulsar ki...' and then thinks. If you jump in there, he loses his place and gets annoyed."
Every voice agent has to guess, many times a call, whether the caller has finished. Guess too early and Bolo talks over people (a cut-in). Guess too late and every caller sits through extra silence after every sentence. Version 0 waited for one full second of quiet, and still cut Mr Gupta off twice, because he thinks for longer than a second.
The speech-to-text from chapter 5 already has a silence rule of its own, endpointing: after 300 ms of quiet it sends a final. That is fast, and it is wrong in both directions: 300 ms is long enough to end a turn in the middle of Mr Gupta's thinking, and Mr Pandey's breath after "mere paas activa hai." is 390 ms. This chapter makes Bolo decide like Kavita does: by the silence, and by the words.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
turns.py: does this turn sound finished?
Write sounds_unfinished(words) (True when the last word is in OPEN_ENDINGS; False for no words) and turn_wait(words) (LONG_WAIT when the words sound unfinished, 0 when they sound finished). OPEN_ENDINGS and LONG_WAIT are given in the file. Only the last word counts: "kitne ki hai" ends on "hai" and sounds finished.
- Task 2
turn_taker.py: when has the caller finished?
Write class TurnTaker(wait_for, on_turn) with final(f), speech_started(at) and answered(). final keeps the final's text and words and starts the timer again (loop.call_later) for wait_for(every word of the turn so far) ms; when it fires, on_turn gets the turn's finals joined with spaces and gives back the reply's task, which you keep. A final whose words ended before the caller started again (f["end"] earlier than the last speech_started at) starts no timer. speech_started stops the timer and takes back a turn given but not yet answered aloud: it cancels that turn's task and puts its words in front of the next turn's. answered() means Bolo has started to speak, so the last turn stays. On Mr Gupta's, Mr Pandey's and Shabnam's calls through the given agent: no cut-ins, every call goes well, and the typical wait (p50) is under 1.5 s.
- Task 3
trade.py: the trade, measured
After await add_thinker() (given), run "gupta-thinks-longer" (who stops to think for 1.6 s) and "pandey-price" with run_call under three settings, all six calls at once with asyncio.gather: "endpointing 1000" (Opts(endpointing=1000, semantic_turns=False)), "endpointing 300" (Opts(endpointing=300, semantic_turns=False)) and "endpointing 300 + words" (Opts(endpointing=300, semantic_turns=True)). For each setting in that order print CALLER, SETTING: ID, cut-ins N, latency A/B/C ms for each call (from its record), then SETTING: cut-ins N, p50 P ms (the cut-ins added, and the lower middle of both calls' latency turns sorted).
- Task 4 · interview question
Interview: finalize.py, a final without the wait
Write async def final_after(text, use_finalize): stream await say(text) and 1.5 s of silence to listen(endpointing=1000) in real time (a frame every 20 ms), keeping the latest partial's words; with use_finalize, at the 10th quiet frame in a row (rms 0.02 or less) call stt.finalize() if turn_wait(the partial's words) is 0. Give how many ms after the last word the final came (from when frame math.ceil(f["end"] / 0.02) was sent). Print endpointing 1000, general service kitne ki hai ?: final N ms after the last word (without finalize), then the same with finalize as with finalize, general service kitne ki hai ?: ..., then with finalize, meri pulsar ki: ...
What you will learn
- Endpointing: how long the speech-to-text waits for silence, and the trade it makes.
- Semantic end of turn: a turn that ends on "ki", "aur" or "my number is" is not over.
- Two clocks: audio time on every message, and why it matters which came first.
- Taking a turn back: answering is a guess, and a wrong guess can be undone if Bolo has not spoken yet, by cancelling the reply's task.
- finalize: asking for the final now, instead of waiting for the silence rule.
More in Sound and the pipeline
Build "When has the caller finished?" yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.