Build a voice agent (TypeScript) · Chapter 4 of 21
The latency budget
By the end of this chapter you can put a stopwatch on every stage of every turn of a live call, check that your stages add up to what the caller lived through, report a typical turn and a bad turn (p50 and p95), measure a model's time to first token and first sentence, and write a latency budget: how long each stage may take for Bolo to answer in about a second. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Sound and the pipeline
- 3 tasks
- TypeScript
- In the all labs plan
The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.
Why you need this
Rashmi listened to a morning of version 0's calls. "It answers everything right," she said, "and every caller sounds unsure whether anyone is there." Chapter 1 timed one turn and found 2.9 seconds. One turn is a story; a budget needs every turn.
Kavita knows the cost without a stopwatch. In a conversation between people, the gap between one person stopping and the other starting is usually a few hundred milliseconds. Past a second, a caller starts to wonder. Past a few seconds, they say "hello?", and some hang up.
Imran has watched teams fix the wrong thing: "Someone spends a week on a faster voice and saves 100 ms, while the bot still waits a full second before it does anything. Time every stage on real calls first. Then decide."
Version 0 cannot tell you where its time goes: it has no clocks, and one turn's numbers say nothing about the next. In this chapter you add them.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
turn-clock.ts: a stopwatch on every turn
Write and export timedCall(caller): version 0's loop from chapter 1, giving { callId, turns }, each turn { heard, said, ms: { silence, stt, model, tts } } in whole milliseconds. silence runs from when the turn's last speech frame arrived to when the turn ended (50 quiet frames later); stt, model and tts time transcribe, the 'lab-chat' request and speak. The four add up to what the line measured for that turn.
Under if (import.meta.main), run timedCall('shabnam-sunday') and print, per turn, turn N: silence S + stt T + model M + tts P = SUM ms; the line: L ms (L from latency.turns of callRecord(callId)).
- Task 2
first-token.ts: the first token, the first sentence, the whole reply
Write and export timeReply(model, messages): a streamed request (stream: true, read with for await), giving { firstTokenMs, firstSentenceMs, wholeMs, chunks, text }: the milliseconds from the start to the first chunk with text, to the first chunk containing . ? or !, and to the end; the number of chunks with text; and the whole text.
Under if (import.meta.main), time SYSTEM, GREETING and Meena's question ('namaste main pehli baar service karwa rahi hoon poora process bataiye na') with 'lab-chat' and 'lab-chat-mini', print MODEL: first token A ms, first sentence B ms, whole reply C ms (N chunks) for each, then the same words: yes or no.
- Task 3 · interview question
Interview: budget.ts, a latency budget
Write and export pct(xs, p): nearest rank (sort a copy; the value at rank ceil(p / 100 x n), from 1).
Under if (import.meta.main), run timedCall('pandey-price') and timedCall('meena-first-time') and, at the same time, timeReply('lab-chat', ...) on the three first questions in the starter. Print end of turn, speech to text, model and text to speech as NAME: V ms -> S ms (V: the p50 of that stage over every turn of both calls; S: 300, 100, the p50 of firstSentenceMs, 120), then total: V ms -> S ms, then the line: p50 A ms, p95 B ms over N turns from both records' latency.turns.
What you will learn
- A stopwatch on a live turn: where each clock starts and stops.
- Checking yourself against the line: your stages must add up to its latency.
- p50 and p95: the typical turn and the bad turn, and why not the average.
- Time to first token and time to first sentence, against the whole reply.
- A smaller model: what it saves, and what it may cost.
- The latency budget: where about a second can go.
More in Sound and the pipeline
Build The latency budget yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.