Build a voice agent · Chapter 9 of 21
Streaming text-to-speech
By the end of this chapter you can drive a streaming text-to-speech service over a WebSocket, keep each reply in its own context, send its audio to the line chunk by chunk as it is made, know at every moment how much of a reply the caller has actually heard, and explain why a fast voice is made far ahead of the caller. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Real time
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.
Why you need this
Chapter 8 hands Bolo's reply over a sentence at a time. Now look at what happens to a sentence after that. The way Imran first wired it, each sentence went to POST /v1/speak, which makes all of the sentence's audio and then sends it back: about 280 ms, plus 12 ms a word, before the first sound. A twelve-word sentence costs over 400 ms of silence before the caller hears its first word.
Real speech services (ElevenLabs, Cartesia, Sarvam, OpenAI) offer a streaming way: a WebSocket that takes text as you have it and sends audio back in small chunks while it is still making the rest. The first chunk of a sentence comes about 120 ms after the sentence is complete.
There is a second reason, and the next two chapters depend on it. Once audio goes to the line, Bolo loses sight of it: the line plays it in real time, seconds after it was sent. When Ritesh interrupts (chapter 10), Bolo has to know what he actually heard and what he never did (chapter 11). Rashmi asked the question plainly after listening to a test call: "How do we know what the customer heard? Not what we sent. What they heard."
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
tts.py: streaming speech
Write class Speaker(ws) and async def speaker(). speaker() connects to f"{WS}/v1/speak/stream?sample_rate=16000" and gives a Speaker (RuntimeError if it cannot connect). A task reads the socket with async for: each "audio" message calls on_audio(ctx, the audio as a numpy int16 array, its words), each "done" calls on_done(ctx, seconds). say(ctx, text, language="hi-IN") sends {"type": "text", "context_id", "text", "language"}; flush(ctx, language="hi-IN") sends {"type": "flush", "context_id", "language"}; cancel(ctx) sends {"type": "cancel", "context_id"}; all three return at once (a queue and one writer task); await close() closes the socket. Chunks are 3200 samples (200 ms at 16 kHz); contexts stay apart; a complete sentence comes back before any flush; the language reaches the service (en-IN reads ₹899 as "eight hundred ninety nine rupees"); a cancelled context sends no more audio and no done.
- Task 2
playback.py: what has been played
Write class Playback(line) with send(ctx, pcm) (line.play(pcm), then line.mark(f"{ctx}:{seconds sent for ctx so far:.3f}")), marked(name) (ctx played up to those seconds), finish(ctx), played_to(ctx) (0 to begin with), played(ctx) (a future, done once ctx is finished and all its marks are back, at once if it sent nothing) and clear() (line.clear(); every context not played to the end is over and its played() is done; marks that come back after the clear do not move its played_to). On Mr Pandey's, Shabnam's and Meena's calls through the given agent: every call goes well, a mark comes back for every one sent, no silence inside a reply reaches 400 ms, the typical wait (p50) is at most 1,300 ms and more than 100 ms below the same calls with each sentence made all at once.
- Task 3 · interview question
Interview: ahead.py, how far ahead is the audio?
Speak Meena's whole reply (the three sentences in the file) in one context with your Speaker: say() then flush(). Time from just before say() the first and the last chunk, keep the seconds from on_done, and close the speaker. Print first audio: N ms, then made S s of audio in MAKING ms: X times faster than it plays (MAKING = last minus first chunk, S with 2 decimals, X = S / (MAKING / 1000) with 1 decimal), then when the last chunk came, R s of it had not been played yet (R = S - MAKING / 1000, 2 decimals). The file ends with asyncio.run(main()).
What you will learn
- Streaming text-to-speech: contexts, text in, audio chunks and word times out, flush and done.
- A WebSocket client in asyncio: a reader task with async for, and a queue with one writer task.
- Audio to the line as it comes: no waiting for a sentence's whole audio.
- Marks: bookmarks in the audio that come back when the caller has heard up to them.
- Played, not sent: how far the caller has heard, at any moment.
- Faster than real time: why most of a reply is made long before it is played.
More in Real time
Build "Streaming text-to-speech" yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.