Build a voice agent · Chapter 4 of 21
The latency budget
By the end of this chapter you can put a stopwatch on every stage of every turn of a live call, check that your stages add up to what the caller lived through, report a typical turn and a bad turn (p50 and p95), measure a model's time to first token and first sentence, and write a latency budget: how long each stage may take for Bolo to answer in about a second. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Sound and the pipeline
- 3 tasks
- Python
- In the all labs plan
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.
Why you need this
Rashmi listened to a morning of version 0's calls. "It answers everything right," she said, "and every caller sounds unsure whether anyone is there." Chapter 1 timed one turn and found 2.9 seconds. One turn is a story; a budget needs every turn.
Kavita knows the cost without a stopwatch. In a conversation between people, the gap between one person stopping and the other starting is usually a few hundred milliseconds. Past a second, a caller starts to wonder. Past a few seconds, they say "hello?", and some hang up.
Imran has watched teams fix the wrong thing: "Someone spends a week on a faster voice and saves 100 ms, while the bot still waits a full second before it does anything. Time every stage on real calls first. Then decide."
Version 0 cannot tell you where its time goes: it has no clocks, and one turn's numbers say nothing about the next. In this chapter you add them.
What you build: 3 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
turn_clock.py: a stopwatch on every turn
Write async def timed_call(caller): version 0's loop from chapter 1, giving ClockedCall(call_id, turns), each ClockedTurn(heard, said, Stages(silence, stt, model, tts)) in whole milliseconds (all given). silence runs from when the turn's last speech frame arrived to when the turn ended (50 quiet frames later); stt, model and tts time transcribe, the "lab-chat" request and speak, with time.perf_counter(). The four add up to what the line measured for that turn.
Under if __name__ == "__main__":, run timed_call("shabnam-sunday") and print, per turn, turn N: silence S + stt T + model M + tts P = SUM ms; the line: L ms (L from rec["latency"]["turns"] of await call_record(call_id)).
- Task 2
first_token.py: the first token, the first sentence, the whole reply
Write async def time_reply(model, messages): a streamed request (stream=True, read with async for), giving Timing(first_token_ms, first_sentence_ms, whole_ms, chunks, text) (given): the milliseconds from the start to the first chunk with text, to the first chunk containing . ? or !, and to the end; the number of chunks with text; and the whole text.
Under if __name__ == "__main__":, time SYSTEM, GREETING and Meena's question ("namaste main pehli baar service karwa rahi hoon poora process bataiye na") with "lab-chat" and "lab-chat-mini", print MODEL: first token A ms, first sentence B ms, whole reply C ms (N chunks) for each, then the same words: yes or no.
- Task 3 · interview question
Interview: budget.py, a latency budget
Write pct(xs, p): nearest rank (sort a copy; the value at rank ceil(p / 100 x n), from 1).
Under if __name__ == "__main__":, run timed_call("pandey-price") and timed_call("meena-first-time") and, at the same time (asyncio.gather), time_reply("lab-chat", ...) on the three QUESTIONS (given). Print end of turn, speech to text, model and text to speech as NAME: V ms -> S ms (V: the p50 of that stage over every turn of both calls; S: 300, 100, the p50 of first_sentence_ms, 120), then total: V ms -> S ms, then the line: p50 A ms, p95 B ms over N turns from both records' latency turns.
What you will learn
- A stopwatch on a live turn: where each clock starts and stops.
- Checking yourself against the line: your stages must add up to its latency.
- p50 and p95: the typical turn and the bad turn, and why not the average.
- Time to first token and time to first sentence, against the whole reply.
- A smaller model: what it saves, and what it may cost.
- The latency budget: where about a second can go.
More in Sound and the pipeline
Build The latency budget yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.