Build a voice agent (TypeScript) · Chapter 8 of 21
Streaming the model's reply
By the end of this chapter you can stream a model's reply, cut the stream into sentences the moment each one is complete, send the first sentence to speech while the model is still writing the rest, make the model write for the ear instead of the eye, and say how much of the model's time the caller no longer waits for. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Real time
- 4 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Meena is calling Kickstart for the first time: "poora process bataiye na", tell me how it all works. Bolo's answer is three sentences: how booking and pickup work, when the scooter comes back, how to pay. The model takes about 1.6 seconds to write all three. Until this chapter, Bolo waited for all of it before saying a word.
That is how you read a chat reply on a screen: it arrives, you read it. On a phone the caller hears one word at a time, at the speed of speech, so she does not need the third sentence until she has heard the first two, which takes about seven seconds. Every millisecond spent waiting for the rest of the reply is silence she sits through for nothing. Imran puts it the way he puts everything: "A call is a stream, not a request. The reply is a stream too."
There is a second problem hiding in the same place. The model writes the way models write for screens: prices in bold, lists with bullets. Text-to-speech reads every character it is given. Mr Pandey asked the price on a test call with a plain prompt, and heard this:
scooter ki general service star star aath sau ninyaanve rupaye star star ki haiWhat you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
speech.ts: sentences as they stream
Write and export class Sentences with push(delta) and rest(). push adds delta to the text kept and gives back every sentence now complete, trimmed, with its punctuation: a sentence ends at . ? ! or । followed by white space (a space or a line break), except a full stop right after a word in ABBREVIATIONS (given: Dr, Mr, Mrs, Ms, Rs, No, St). A full stop with nothing after it yet ("hai.") is not complete. rest() gives what is left, trimmed, and keeps nothing. The result must be the same whether the text comes a character at a time or in big pieces.
- Task 2
for-the-ear.ts: write for the ear
Export SYSTEM, Bolo's system prompt, saying Bolo is the voice agent on Kickstart's phone line in Nagpur (scooter and motorbike service at the customer's door), on a phone call, answering in short spoken sentences in the caller's language: with it, the model's replies to the check's questions have no * and no line starting with - or #. Export forSpeech(text): take out every * and #, the list markers at the start of a line (- * • and 1.), turn line breaks into single spaces, trim, and keep everything else. On Mrs Iyer's call with a prompt that never says "phone call", the caller must not hear "star"; Mr Pandey's call with your SYSTEM must go well.
- Task 3
stream-reply.ts: speak while the model writes
Write and export streamReply(messages, speak, signal?): one 'lab-chat' request with stream: true and { signal } as create()'s second argument; push each chunk's chunk.choices[0]?.delta?.content into a Sentences and speak every sentence it gives at once; speak rest() when the stream ends (if not empty); return the whole text. For Meena's question the three sentences are spoken one by one, the first long before the stream ends; an aborted signal stops it. On Mr Pandey's, Shabnam's and Meena's calls through the given agent, callers wait less (p50) than with the whole reply first, and Meena's first answer comes more than 250 ms sooner.
- Task 4 · interview question
Interview: first-sentence.ts, the first sentence, not the whole reply
Stream one 'lab-chat' reply to Meena's question (SYSTEM, GREETING as the assistant, then 'namaste main pehli baar service karwa rahi hoon poora process bataiye na'), timing from just before create() the first delta with text, the first sentence push() gives, and the end of the stream. Print first token: N ms, first sentence: N ms: SENTENCE, whole reply: N ms, K sentences, and speech can start N ms sooner (whole reply minus first sentence).
What you will learn
- Streaming a model's reply: stream: true, deltas, and for await.
- Sentences from a stream: where a sentence ends, and the full stops that do not end one.
- Speaking while the model writes: the first sentence goes to speech at once.
- Writing for the ear: the system prompt, and markdown taken out in code.
- Time to first sentence: what the caller waits for now, instead of the whole reply.
More in Real time
Build "Streaming the model's reply" yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.