Build a voice agent · Chapter 6 of 21

Voice activity detection

By the end of this chapter you can build an energy voice activity detector (VAD) frame by frame, give it a minimum length so a bang is not speech and a hangover so a comma is not the end, set its threshold from the line's own noise floor, and say why an energy VAD cannot tell a pressure cooker from a voice, and what can. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Sound and the pipeline
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.

Why you need this

Every voice agent keeps asking one question, fifty times a second: is anyone talking? Bolo needs the answer to know when a caller has started (so it can listen, or stop talking: chapter 10), when a caller has gone quiet (chapter 7), and when nobody has said anything for too long (chapter 12).

Version 0 answers it with is_speech(frame): one frame, louder than a fixed threshold. Kavita has already listed what that gets wrong: "Callers ring from kitchens, from the road, from shops. A door slams. The TV is on. A pressure cooker whistles in the middle of your sentence. And people pause between words, at commas, while they think." A single frame cannot tell any of that from speech.

Imran wants the question answered properly before anything is built on it: "If Bolo stops talking every time a cooker whistles, Mrs Iyer never hears the end of a sentence."

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    vad.py: a voice activity detector

    Write class Vad (threshold=0.02, min_speech_ms=80, hangover_ms=300, frame_ms=20) whose push(frame) gives ("speech_start", at) once min_speech_ms of loud frames (rms >= threshold) come in a row, at the start of the first of them, and ("speech_end", at) once hangover_ms of quiet frames come in a row, at the start of the first quiet one (3 decimals), else None. Write segments(frames, **opts) giving a Segment(start, end) (given) for each start and end, closing an open one at the end of the frames.

    Under if __name__ == "__main__":, print hangover 20 ms: N segments: S-E ... and hangover 300 ms: N segments: S-E ... for CALLS / "joshi-hold-1.wav" (2 decimals).

  2. Task 2

    noise_floor.py: a threshold over the line's own quiet

    Write noise_floor(frames) (the 10th percentile of the frames' rms, nearest rank) and threshold(floor, margin_db=12) (floor * 10 ** (margin_db / 20)).

    Under if __name__ == "__main__":, for the samples of CALLS / "farhan-status-1.wav" and the same samples with_hum (given), print NAME: floor F (D dBFS), threshold T (D dBFS), N segments (F and T 4 decimals, D 1; segments with your threshold), then with a hum, fixed 0.02: N segments.

  3. Task 3 · interview question

    Interview: whistle.py, the pressure cooker

    In an async def main(), for CALLS / "whistle.wav" and then CALLS / "joshi-hold-1.wav": segments of its frames, and a new listen() fed one frame every 20 ms (then 25 frames of quiet, 0.2 s, close, await wait_closed), keeping the first speech_started's at and the words of every final. Print NAME: the VAD hears S-E ...; the STT: speech at T s, W words (or the STT: no speech, W words; none for no segments), then verdict: loud, but not speech, speech, or quiet.

What you will learn

  • An energy VAD: loud frames become speech, quiet frames end it.
  • A minimum length, so a 60 ms door bang is not speech.
  • A hangover, so a comma or a thinking pause does not end it too soon.
  • Segments: the stretches of speech in a recording.
  • A threshold over the noise floor, so a noisy line still works.
  • The whistle problem, and the STT's own voice detector as a smarter VAD.

More in Sound and the pipeline

Build "Voice activity detection" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.