Build a voice agent (TypeScript) · Chapter 10 of 21

Barge-in: when the caller talks over Bolo

By the end of this chapter you can let Bolo listen while it talks, tell a real interruption from a "haan", a pressure cooker or a half-heard guess, stop a reply everywhere it lives at once, say why a barge-in on loudness alone fails, and take apart the time between a caller cutting in and Bolo going quiet. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Real time
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Ritesh is in a hurry. He asks what a service includes and what it costs, and six words into Bolo's answer he cuts in: "achha achha, pickup aaj ho jayega kya?" (OK, OK, can you pick it up today?). People do this all the time on the phone, and they expect the other side to stop.

Bolo so far does not. It talks to the end of its reply whatever the caller does. Here is Ritesh's call, from the line's record:

10.08-15.94  Bolo       scooter ki general service mein engine oil badla jaata hai air
                        filter aur spark plug saaf hote hain aur brake aur chain check hote hain
11.42-13.09  interrupt  achha achha, pickup aaj ho jayega kya?

Bolo talked over him for 4,520 ms: through his whole question, and on for almost three seconds after it. Kavita has a word for agents like that: "a recording". Callers hang up on recordings.

But "stop when the caller makes a sound" is not the answer either. Meena says "haan" while Bolo explains the process: she is listening, not asking to speak. Mrs Iyer's pressure cooker whistles for 1.3 seconds in the middle of her answer. Stop for either, and those callers lose the answer they asked for.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    interruption.ts: is the caller interrupting?

    Write and export isInterruption(words): true when at least one word has a confidence of 0.5 or more and is not in BACKCHANNELS (given in the file), false otherwise (no words, only listening noises like "haan" and "achha achha", or only a guessed word below 0.5).

  2. Task 2

    barge-in.ts: stop talking

    Write and export stopTalking(ctx, abort, speaker, playback): playback.clear(), speaker.cancel(ctx) and abort.abort(). Through the given agent: Bolo stops within 1,200 ms of Ritesh talking over it, with a clear, and the reply's speech context is cancelled; Mrs Iyer's pressure cooker does not stop Bolo and her call goes well; Meena's "haan" and "achha" do not stop Bolo and her call goes well.

  3. Task 3

    loud.ts: a barge-in on loudness

    Write and export class LoudSound with push(frame): true once 3 frames in a row have an RMS above 0.05, and while they stay loud; a quieter frame starts the count again. Under if (import.meta.main), import runCall from './agent.ts' with await import(...), call 'kitchen-whistle' twice at once with { bargeIn: 'words', loudSound: LoudSound } and { bargeIn: 'energy', loudSound: LoudSound }, and for each in that order print WAY: Bolo said "TEXT"; the list heard: yes or no (TEXT is replies[1].text of the call's record; yes when its goal 'brakes' was met).

  4. Task 4 · interview question

    Interview: talk-over.ts, where does the talk-over go?

    Run runCall('ritesh-hurry'), read its record, and print Ritesh started talking over Bolo at T s (the interrupt's at), the first real word, WORD, ended at T s (stops[0]), Bolo stopped at T s (agentStoppedAt), then talk-over: N ms: A waiting for a real word, B to stop (N is talkOverMs; A from his start to the word's end, B from the word's end to the stop, in whole milliseconds; times with 2 decimals).

What you will learn

  • Full duplex: Bolo listens while it talks, through the same speech-to-text.
  • What counts as an interruption: sure words that are not listening noises.
  • Stopping everywhere at once: the line's queue, the speech still being made, the model still writing.
  • Why loudness alone fails: a barge-in on energy, measured.
  • Talk-over: where the time between "he cut in" and "Bolo stopped" goes.

More in Real time

Build "Barge-in: when the caller talks over Bolo" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.