Build a voice agent · Chapter 5 of 21

Streaming speech-to-text

By the end of this chapter you can stream a call's audio to a speech-to-text service as it arrives, tell a partial from a final and know which one to act on, read word times, confidence and language, force a final with finalize, and give Bolo ears that hear a turn about 400 ms after the caller stops instead of a second and a half. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Sound and the pipeline
  • 3 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.

Why you need this

Chapter 4's budget said where version 0's time goes. Before Bolo even thinks about a reply, every turn spends:

  • a full second of silence, because Bolo cannot tell that the caller has finished, and
  • another 300 to 400 ms of batch speech-to-text, because the STT gets the audio only after that second, all at once.

Both waits come from one habit: version 0 treats a turn as a recording. It collects the whole thing, then hands it over. Imran's line for this lab is "a call is a stream, not a request", and this is the first place it pays: the STT can listen while the caller talks. By the time she stops, it has already heard every word but the last.

Kavita adds the human side: "Callers do not wait for a beep. When they stop, they expect you to be there."

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    stt.py: a streaming speech-to-text client

    Write async def listen(sample_rate=16000, encoding="linear16", endpointing=300, utterance_end_ms=1000) and the Listener class. listen connects to WS + /v1/listen with the four settings in the query and returns a Listener. The Listener's reader task (async for raw in ws) calls on_partial(text, words), on_final(final), on_speech_started(at) and on_utterance_end(last_word_end) for the STT's messages; send(audio) sends an int16 array's bytes or bytes as one binary message; finalize() and close() send {"type": "finalize"} and {"type": "close"}. The three are plain methods: they queue, and a writer task awaits ws.send in order.

    Under if __name__ == "__main__":, send CALLS / "fernandes-english-1.wav" in 20 ms frames and a second of quiet, print final: TEXT (START to END s, LANGUAGE) for each final and utterance end: last word at T s, then close.

  2. Task 2

    live_words.py: words while she is still talking

    In an async def main(), listen(endpointing=300), start a clock, and send CALLS / "meena-first-time-1.wav" one 20 ms frame every 20 ms, then frames of quiet until the final comes. Print N ms partial: TEXT for each partial and N ms final: TEXT for the final (N since the clock started), then partials: P, F of them not the start of the final, and her last word ended at E ms; the final came D ms later (E from the final's last word, D your final's time minus E).

  3. Task 3 · interview question

    Interview: streaming_ears.py, hear while the caller talks

    Write async def streaming_ears(line): listen(endpointing=300), send every frame from line.on_audio, give each final with speech_final and some text as the next turn (agent.py's Turns), and when line.ended is done close the stream and give None. Never use partials.

    Under if __name__ == "__main__":, run run_call("shabnam-sunday") and run_call("shabnam-sunday", ears=streaming_ears) at once, and print version 0's ears: A, B, C ms (p50 P), streaming ears: A, B, C ms (p50 P) from each record's latency turns, then saved at p50: N ms.

What you will learn

  • Streaming STT: audio in over a WebSocket, words out while the caller talks.
  • Partials and finals: a guess that changes, and a result that does not.
  • Words with times, confidence and language, in every final.
  • Endpointing: how the STT decides the speech is over, and finalize.
  • Ears for Bolo: turns from finals, plugged into the agent, measured on a call.

More in Sound and the pipeline

Build "Streaming speech-to-text" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.