Build a voice agent (TypeScript) · Chapter 4 of 21

The latency budget

By the end of this chapter you can put a stopwatch on every stage of every turn of a live call, check that your stages add up to what the caller lived through, report a typical turn and a bad turn (p50 and p95), measure a model's time to first token and first sentence, and write a latency budget: how long each stage may take for Bolo to answer in about a second. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Sound and the pipeline
  • 3 tasks
  • TypeScript
  • In the all labs plan

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.

Why you need this

Rashmi listened to a morning of version 0's calls. "It answers everything right," she said, "and every caller sounds unsure whether anyone is there." Chapter 1 timed one turn and found 2.9 seconds. One turn is a story; a budget needs every turn.

Kavita knows the cost without a stopwatch. In a conversation between people, the gap between one person stopping and the other starting is usually a few hundred milliseconds. Past a second, a caller starts to wonder. Past a few seconds, they say "hello?", and some hang up.

Imran has watched teams fix the wrong thing: "Someone spends a week on a faster voice and saves 100 ms, while the bot still waits a full second before it does anything. Time every stage on real calls first. Then decide."

Version 0 cannot tell you where its time goes: it has no clocks, and one turn's numbers say nothing about the next. In this chapter you add them.

What you build: 3 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    turn-clock.ts: a stopwatch on every turn

    Write and export timedCall(caller): version 0's loop from chapter 1, giving { callId, turns }, each turn { heard, said, ms: { silence, stt, model, tts } } in whole milliseconds. silence runs from when the turn's last speech frame arrived to when the turn ended (50 quiet frames later); stt, model and tts time transcribe, the 'lab-chat' request and speak. The four add up to what the line measured for that turn.

    Under if (import.meta.main), run timedCall('shabnam-sunday') and print, per turn, turn N: silence S + stt T + model M + tts P = SUM ms; the line: L ms (L from latency.turns of callRecord(callId)).

  2. Task 2

    first-token.ts: the first token, the first sentence, the whole reply

    Write and export timeReply(model, messages): a streamed request (stream: true, read with for await), giving { firstTokenMs, firstSentenceMs, wholeMs, chunks, text }: the milliseconds from the start to the first chunk with text, to the first chunk containing . ? or !, and to the end; the number of chunks with text; and the whole text.

    Under if (import.meta.main), time SYSTEM, GREETING and Meena's question ('namaste main pehli baar service karwa rahi hoon poora process bataiye na') with 'lab-chat' and 'lab-chat-mini', print MODEL: first token A ms, first sentence B ms, whole reply C ms (N chunks) for each, then the same words: yes or no.

  3. Task 3 · interview question

    Interview: budget.ts, a latency budget

    Write and export pct(xs, p): nearest rank (sort a copy; the value at rank ceil(p / 100 x n), from 1).

    Under if (import.meta.main), run timedCall('pandey-price') and timedCall('meena-first-time') and, at the same time, timeReply('lab-chat', ...) on the three first questions in the starter. Print end of turn, speech to text, model and text to speech as NAME: V ms -> S ms (V: the p50 of that stage over every turn of both calls; S: 300, 100, the p50 of firstSentenceMs, 120), then total: V ms -> S ms, then the line: p50 A ms, p95 B ms over N turns from both records' latency.turns.

What you will learn

  • A stopwatch on a live turn: where each clock starts and stops.
  • Checking yourself against the line: your stages must add up to its latency.
  • p50 and p95: the typical turn and the bad turn, and why not the average.
  • Time to first token and time to first sentence, against the whole reply.
  • A smaller model: what it saves, and what it may cost.
  • The latency budget: where about a second can go.

More in Sound and the pipeline

Build The latency budget yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent (TypeScript) are free. This one comes with the all labs plan.