Build a voice agent · Chapter 16 of 21
When Bolo mishears
By the end of this chapter Bolo knows which words it did not hear well, tells the model so, reads an unsure booking number back before acting on it, and confirms before it does anything that cannot be undone. On a noisy line, it cancels the right booking, or none. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Production
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.
Why you need this
Sunita calls to cancel tomorrow's service. Her TV is on, loud, and it covers the last digit of her booking number just as she says it. Here is her call with Bolo as it would be without this chapter (tools, filler, digits for speech, everything but what this chapter adds):
5.70- 9.98 caller namaste, mera booking number k s chaar chaar ek saat hai. meri service cancel kar dijiye.
11.68- 17.02 Bolo ek second main check kar rahi hoon aapki booking k s chaar chaar ek aath cancel ho gayi... (1699 ms after the caller)
17.48- 19.35 caller nahi, chaar chaar ek saat. saat.
20.52- 23.16 Bolo aapki booking k s chaar chaar ek aath cancel ho gayi hai (1169 ms after the caller)
outcome: caller_hung_up, not a success
✖ KS-4417 (hers) cancelled
✖ KS-4418 (Farhan's) untouched
She said saat (seven). The line heard aath (eight). KS-4418 is a real booking: Farhan's scooter pickup, tomorrow at nine. Bolo looked it up, cancelled it, and told Sunita it was done. She corrected the digit, and it was too late: the cancel had gone through, and Bolo said so again. Tomorrow Farhan waits for a mechanic who is not coming, and Sunita's mechanic turns up at a house where nobody wants him.
Tejas has said it since chapter 1: "move the right booking, or none." This is the call he meant.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
unsure.py: marking what Bolo did not hear well
Write mark_unsure(words: list[Word], min_confidence: float = 0.6) -> str. It joins the words ("word" of each) with spaces and puts "(?)" right after each word whose "confidence" is under min_confidence: "k s chaar chaar ek aath(?) hai". No words give "". A word at exactly min_confidence is not marked.
The given agent.py makes each final the speech-to-text sends into the caller's message with it.
- Task 2
careful.py: telling the model what (?) means
Write CAREFUL, one or two sentences for Bolo's system prompt (the given agent.py adds them after its own): what a word marked (?) means, and what to do about it (read an unsure booking number back and wait for a yes before acting on it; confirm before cancelling anything).
Sunita (sunita-mishear) calls over a TV to cancel her service, KS-4417. Her call must succeed, and Farhan's KS-4418 must not be touched.
- Task 3 · interview question
Interview: wrong_booking.py, one noisy digit
Take Sunita's call twice, one after the other, each time after await reset_bookings(): first with run_call("sunita-mishear", careful=False) (no marks and no CAREFUL), then with run_call("sunita-mishear"). After each, print without confirming: cancelled IDS, success BOOL and then with confirming: cancelled IDS, success BOOL. IDS are the "booking_id" of every booking whose "status" is "cancelled" after that call (from await bookings()), joined with ", ", and BOOL is "success" from the call's record.
What you will learn
- Why speech-to-text mishears: noise, near neighbours, and the confidence it gives each word.
- Marking unsure words in the text the model reads: aath(?).
- Telling the model what the mark means: a line in the system prompt.
- Read back and confirm before anything you cannot undo.
- Measuring the fix: the same noisy call, with and without it.
More in Production
Build "When Bolo mishears" yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.