Build a voice agent (TypeScript) · Chapter 16 of 21

When Bolo mishears

By the end of this chapter Bolo knows which words it did not hear well, tells the model so, reads an unsure booking number back before acting on it, and confirms before it does anything that cannot be undone. On a noisy line, it cancels the right booking, or none. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Sunita calls to cancel tomorrow's service. Her TV is on, loud, and it covers the last digit of her booking number just as she says it. Here is her call with Bolo as it would be without this chapter (tools, filler, digits for speech, everything but what this chapter adds):

  5.72- 10.00  caller  namaste, mera booking number k s chaar chaar ek saat hai. meri service cancel kar dijiye.
 11.72- 17.08  Bolo    ek second main check kar rahi hoon aapki booking k s chaar chaar ek aath cancel ho gayi...
 17.54- 19.41  caller  nahi, chaar chaar ek saat. saat.
 20.58- 23.22  Bolo    aapki booking k s chaar chaar ek aath cancel ho gayi hai
outcome: caller_hung_up, not a success
  ✖ KS-4417 (hers) cancelled
  ✖ KS-4418 (Farhan's) untouched

She said saat (seven). The line heard aath (eight). KS-4418 is a real booking: Farhan's scooter pickup, tomorrow at nine. Bolo looked it up, cancelled it, and told Sunita it was done. She corrected the digit, and it was too late: the cancel had gone through, and Bolo said so again. Tomorrow Farhan waits for a mechanic who is not coming, and Sunita's mechanic turns up at a house where nobody wants him.

Tejas has said it since chapter 1: "move the right booking, or none." This is the call he meant.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    unsure.ts: marking what Bolo did not hear well

    Write and export markUnsure(words: Word[], min = 0.6): string. It joins the words with spaces and puts "(?)" right after each word whose confidence is under min: "k s chaar chaar ek aath(?) hai". No words give "". A word at exactly min is not marked.

    The given agent.ts makes each final the speech-to-text sends into the caller's message with it.

  2. Task 2

    careful.ts: telling the model what (?) means

    Write and export CAREFUL, one or two sentences for Bolo's system prompt (the given agent.ts adds them after its own): what a word marked (?) means, and what to do about it (read an unsure booking number back and wait for a yes before acting on it; confirm before cancelling anything).

    Sunita (sunita-mishear) calls over a TV to cancel her service, KS-4417. Her call must succeed, and Farhan's KS-4418 must not be touched.

  3. Task 3 · interview question

    Interview: wrong-booking.ts, one noisy digit

    Take Sunita's call twice, one after the other, each time after resetBookings(): first with runCall('sunita-mishear', { careful: false }) (no marks and no CAREFUL), then with runCall('sunita-mishear'). After each, print without confirming: cancelled IDS, success BOOL and then with confirming: cancelled IDS, success BOOL. IDS are the booking_id of every booking whose status is 'cancelled' after that call (from bookings()), joined with ", ", and BOOL is success from the call's record.

What you will learn

  • Why speech-to-text mishears: noise, near neighbours, and the confidence it gives each word.
  • Marking unsure words in the text the model reads: aath(?).
  • Telling the model what the mark means: a line in the system prompt.
  • Read back and confirm before anything you cannot undo.
  • Measuring the fix: the same noisy call, with and without it.

More in Production

Build "When Bolo mishears" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.