Build a voice agent (TypeScript) · Chapter 6 of 21

Voice activity detection

By the end of this chapter you can build an energy voice activity detector (VAD) frame by frame, give it a minimum length so a bang is not speech and a hangover so a comma is not the end, set its threshold from the line's own noise floor, and say why an energy VAD cannot tell a pressure cooker from a voice, and what can. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Sound and the pipeline
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Every voice agent keeps asking one question, fifty times a second: is anyone talking? Bolo needs the answer to know when a caller has started (so it can listen, or stop talking: chapter 10), when a caller has gone quiet (chapter 7), and when nobody has said anything for too long (chapter 12).

Version 0 answers it with isSpeech(frame): one frame, louder than a fixed threshold. Kavita has already listed what that gets wrong: "Callers ring from kitchens, from the road, from shops. A door slams. The TV is on. A pressure cooker whistles in the middle of your sentence. And people pause between words, at commas, while they think." A single frame cannot tell any of that from speech.

Imran wants the question answered properly before anything is built on it: "If Bolo stops talking every time a cooker whistles, Mrs Iyer never hears the end of a sentence."

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    vad.ts: a voice activity detector

    Write and export class Vad (options threshold 0.02, minSpeechMs 80, hangoverMs 300, frameMs 20) whose push(frame) gives { type: 'speech_start', at } once minSpeechMs of loud frames (rms >= threshold) come in a row, at the start of the first of them, and { type: 'speech_end', at } once hangoverMs of quiet frames come in a row, at the start of the first quiet one (3 decimals), else null. Export segments(frames, opts) giving { start, end } for each start and end, closing an open one at the end of the frames.

    Under if (import.meta.main), print hangover 20 ms: N segments: S-E ... and hangover 300 ms: N segments: S-E ... for ../../calls/joshi-hold-1.wav (2 decimals).

  2. Task 2

    noise-floor.ts: a threshold over the line's own quiet

    Write and export noiseFloor(frames) (the 10th percentile of the frames' rms, nearest rank) and threshold(floor, marginDb = 12) (floor x 10^(marginDb / 20)).

    Under if (import.meta.main), for the samples of ../../calls/farhan-status-1.wav and the same samples with the 100 Hz hum from the starter, print NAME: floor F (D dBFS), threshold T (D dBFS), N segments (F and T 4 decimals, D 1; segments with your threshold), then with a hum, fixed 0.02: N segments.

  3. Task 3 · interview question

    Interview: whistle.ts, the pressure cooker

    For ../../calls/whistle.wav and then ../../calls/joshi-hold-1.wav: segments of its frames, and a new listen() fed one frame every 20 ms (then 25 frames of quiet, 200 ms, close), keeping the first speech_started's at and the words of every final. Print NAME: the VAD hears S-E ...; the STT: speech at T s, W words (or the STT: no speech, W words; none for no segments), then verdict: loud, but not speech, speech, or quiet.

What you will learn

  • An energy VAD: loud frames become speech, quiet frames end it.
  • A minimum length, so a 60 ms door bang is not speech.
  • A hangover, so a comma or a thinking pause does not end it too soon.
  • Segments: the stretches of speech in a recording.
  • A threshold over the noise floor, so a noisy line still works.
  • The whistle problem, and the STT's own voice detector as a smarter VAD.

More in Sound and the pipeline

Build "Voice activity detection" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.