Build a voice agent (TypeScript) · Chapter 9 of 21
Streaming text-to-speech
By the end of this chapter you can drive a streaming text-to-speech service over a WebSocket, keep each reply in its own context, send its audio to the line chunk by chunk as it is made, know at every moment how much of a reply the caller has actually heard, and explain why a fast voice is made far ahead of the caller. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Real time
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Chapter 8 hands Bolo's reply over a sentence at a time. Now look at what happens to a sentence after that. The way Imran first wired it, each sentence went to POST /v1/speak, which makes all of the sentence's audio and then sends it back: about 280 ms, plus 12 ms a word, before the first sound. A twelve-word sentence costs over 400 ms of silence before the caller hears its first word.
Real speech services (ElevenLabs, Cartesia, Sarvam, OpenAI) offer a streaming way: a WebSocket that takes text as you have it and sends audio back in small chunks while it is still making the rest. The first chunk of a sentence comes about 120 ms after the sentence is complete.
There is a second reason, and the next two chapters depend on it. Once audio goes to the line, Bolo loses sight of it: the line plays it in real time, seconds after it was sent. When Ritesh interrupts (chapter 10), Bolo has to know what he actually heard and what he never did (chapter 11). Rashmi asked the question plainly after listening to a test call: "How do we know what the customer heard? Not what we sent. What they heard."
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
tts.ts: streaming speech
Write and export type Timed, class Speaker and speaker(). speaker() opens the WebSocket
${WS}/v1/speak/stream?sample_rate=16000and resolves with a Speaker once it is open. say(ctx, text, language = 'hi-IN') sends { type: 'text', context_id, text, language }; flush(ctx, language) sends { type: 'flush', context_id, language }; cancel(ctx) sends { type: 'cancel', context_id }; close() closes the socket. Each 'audio' message calls onAudio(ctx, the audio decoded to an Int16Array, its words); each 'done' calls onDone(ctx, seconds). Chunks are 3200 samples (200 ms at 16 kHz); contexts stay apart; a complete sentence comes back before any flush; the language reaches the service (en-IN reads ₹899 as "eight hundred ninety nine rupees"); a cancelled context sends no more audio and no done. - Task 2
playback.ts: what has been played
Write and export class Playback(line) with send(ctx, pcm) (line.play(pcm), then line.mark(
${ctx}:${seconds sent for ctx so far, 3 decimals})), marked(name) (ctx played up to those seconds), finish(ctx), playedTo(ctx) (0 to begin with), played(ctx) (resolves once ctx is finished and all its marks are back, at once if it sent nothing) and clear() (line.clear(); every context not played to the end is over and its played() resolves; marks that come back after the clear do not move its playedTo). On Mr Pandey's, Shabnam's and Meena's calls through the given agent: every call goes well, a mark comes back for every one sent, no silence inside a reply reaches 400 ms, the typical wait (p50) is at most 1,300 ms and more than 100 ms below the same calls with each sentence made all at once. - Task 3 · interview question
Interview: ahead.ts, how far ahead is the audio?
Speak Meena's whole reply (the three sentences in the file) in one context with your Speaker: say() then flush(). Time from just before say() the first and the last chunk, keep the seconds from onDone, and close the speaker. Print first audio: N ms, then made S s of audio in MAKING ms: X times faster than it plays (MAKING = last minus first chunk, S with 2 decimals, X = S / (MAKING / 1000) with 1 decimal), then when the last chunk came, R s of it had not been played yet (R = S - MAKING / 1000, 2 decimals).
What you will learn
- Streaming text-to-speech: contexts, text in, audio chunks and word times out, flush and done.
- Audio to the line as it comes: no waiting for a sentence's whole audio.
- Marks: bookmarks in the audio that come back when the caller has heard up to them.
- Played, not sent: how far the caller has heard, at any moment.
- Faster than real time: why most of a reply is made long before it is played.
More in Real time
Build "Streaming text-to-speech" yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.