Build a voice agent (TypeScript) · Chapter 7 of 21

When has the caller finished?

By the end of this chapter you can say why silence alone cannot tell you that a caller has finished, use the words they stopped on to decide how long to wait, take back an answer that started too soon, measure the trade between cutting people off and making them wait, and ask the speech-to-text for its final the moment you know the turn is over. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Sound and the pipeline
  • 4 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.

Why you need this

On a call, nobody says "over". People stop talking for all kinds of reasons: the sentence is finished, they are taking a breath, they are thinking of the next word, they are looking for a booking number. Kavita hears it all day at the desk: "Callers stop in the middle of a sentence. Mr Gupta says 'meri pulsar ki...' and then thinks. If you jump in there, he loses his place and gets annoyed."

Every voice agent has to guess, many times a call, whether the caller has finished. Guess too early and Bolo talks over people (a cut-in). Guess too late and every caller sits through extra silence after every sentence. Version 0 waited for one full second of quiet, and still cut Mr Gupta off twice, because he thinks for longer than a second.

The speech-to-text from chapter 5 already has a silence rule of its own, endpointing: after 300 ms of quiet it sends a final. That is fast, and it is wrong in both directions: 300 ms is long enough to end a turn in the middle of Mr Gupta's thinking, and Mr Pandey's breath after "mere paas activa hai." is 390 ms. This chapter makes Bolo decide like Kavita does: by the silence, and by the words.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    turns.ts: does this turn sound finished?

    Write and export soundsUnfinished(words) (true when the last word is in OPEN_ENDINGS; false for no words) and turnWait(words) (LONG_WAIT when the words sound unfinished, 0 when they sound finished). OPEN_ENDINGS and LONG_WAIT are given in the file. Only the last word counts: "kitne ki hai" ends on "hai" and sounds finished.

  2. Task 2

    turn-taker.ts: when has the caller finished?

    Write and export class TurnTaker(waitFor, onTurn) with final(f), speechStarted(at) and answered(). final keeps the final's text and words and starts the timer again for waitFor(every word of the turn so far) ms; when it fires, onTurn gets the turn's finals joined with spaces and the signal of a new AbortController. A final whose words ended before the caller started again (f.end earlier than the last speechStarted at) starts no timer. speechStarted stops the timer and takes back a turn given but not yet answered aloud: it aborts that turn's signal and puts its words in front of the next turn's. answered() means Bolo has started to speak, so the last turn stays. On Mr Gupta's, Mr Pandey's and Shabnam's calls through the given agent: no cut-ins, every call goes well, and the typical wait (p50) is under 1.5 s.

  3. Task 3

    trade.ts: the trade, measured

    Run 'gupta-thinks-longer' (the caller given in the file, who stops to think for 1.6 s) and 'pandey-price' with runCall under three settings, all six calls at once: 'endpointing 1000' ({ endpointing: 1000, semanticTurns: false }), 'endpointing 300' ({ endpointing: 300, semanticTurns: false }) and 'endpointing 300 + words' ({ endpointing: 300, semanticTurns: true }). For each setting in that order print CALLER, SETTING: ID, cut-ins N, latency A/B/C ms for each call (from its record), then SETTING: cut-ins N, p50 P ms (the cut-ins added, and the lower middle of both calls' latency.turns sorted).

  4. Task 4 · interview question

    Interview: finalize.ts, a final without the wait

    Write finalAfter(text, useFinalize): stream say(text) and 1.5 s of silence to listen({ endpointing: 1000 }) in real time (a frame every 20 ms), keeping the latest partial's words; with useFinalize, at the 10th quiet frame in a row (RMS 0.02 or less) send stt.finalize() if turnWait(the partial's words) is 0. Give how many ms after the last word the final came (from when frame Math.ceil(f.end / 0.02) was sent). Print endpointing 1000, general service kitne ki hai ?: final N ms after the last word (without finalize), then the same with finalize as with finalize, general service kitne ki hai ?: ..., then with finalize, meri pulsar ki: ...

What you will learn

  • Endpointing: how long the speech-to-text waits for silence, and the trade it makes.
  • Semantic end of turn: a turn that ends on "ki", "aur" or "my number is" is not over.
  • Two clocks: audio time on every message, and why it matters which came first.
  • Taking a turn back: answering is a guess, and a wrong guess can be undone if Bolo has not spoken yet.
  • finalize: asking for the final now, instead of waiting for the silence rule.

More in Sound and the pipeline

Build "When has the caller finished?" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.