Build a voice agent (TypeScript) · Chapter 5 of 21

Streaming speech-to-text

By the end of this chapter you can stream a call's audio to a speech-to-text service as it arrives, tell a partial from a final and know which one to act on, read word times, confidence and language, force a final with finalize, and give Bolo ears that hear a turn about 400 ms after the caller stops instead of a second and a half. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Sound and the pipeline
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Chapter 4's budget said where version 0's time goes. Before Bolo even thinks about a reply, every turn spends:

  • a full second of silence, because Bolo cannot tell that the caller has finished, and
  • another 300 to 400 ms of batch speech-to-text, because the STT gets the audio only after that second, all at once.

Both waits come from one habit: version 0 treats a turn as a recording. It collects the whole thing, then hands it over. Imran's line for this lab is "a call is a stream, not a request", and this is the first place it pays: the STT can listen while the caller talks. By the time she stops, it has already heard every word but the last.

Kavita adds the human side: "Callers do not wait for a beep. When they stop, they expect you to be there."

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    stt.ts: a streaming speech-to-text client

    Write and export listen(opts) and the Listener class. listen opens WS + /v1/listen with sample_rate (default 16000), encoding (linear16), endpointing (300) and utterance_end_ms (1000) from opts, and resolves with a Listener when it opens. The Listener calls onPartial(text, words), onFinal(final), onSpeechStarted(at) and onUtteranceEnd(last_word_end) for the STT's messages; send(audio) sends an Int16Array's own bytes or a Buffer as one binary message; finalize() and close() send {"type":"finalize"} and {"type":"close"}.

    Under if (import.meta.main), send ../../calls/fernandes-english-1.wav in 20 ms frames and a second of quiet, print final: TEXT (START to END s, LANGUAGE) for each final and utterance end: last word at T s, then close.

  2. Task 2

    live-words.ts: words while she is still talking

    listen({ endpointing: 300 }), start a clock, and send ../../calls/meena-first-time-1.wav one 20 ms frame every 20 ms, then frames of quiet until the final comes. Print N ms partial: TEXT for each partial and N ms final: TEXT for the final (N since the clock started), then partials: P, F of them not the start of the final, and her last word ended at E ms; the final came D ms later (E from the final's last word, D your final's time minus E).

  3. Task 3 · interview question

    Interview: streaming-ears.ts, hear while the caller talks

    Write and export streamingEars(line): listen({ endpointing: 300 }), send every frame from line.onAudio, give each final with speech_final and some text as the next turn through next(), and when line.ended resolves close the stream and give null. Never use partials.

    Under if (import.meta.main), run runCall('shabnam-sunday') and runCall('shabnam-sunday', { ears: streamingEars }) at once, and print version 0's ears: A, B, C ms (p50 P), streaming ears: A, B, C ms (p50 P) from each record's latency.turns, then saved at p50: N ms.

What you will learn

  • Streaming STT: audio in over a WebSocket, words out while the caller talks.
  • Partials and finals: a guess that changes, and a result that does not.
  • Words with times, confidence and language, in every final.
  • Endpointing: how the STT decides the speech is over, and finalize.
  • Ears for Bolo: turns from finals, plugged into the agent, measured on a call.

More in Sound and the pipeline

Build "Streaming speech-to-text" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.