Build a voice agent · Chapter 18 of 21

Measuring latency

By the end of this chapter you can run Bolo on every caller a few at a time, turn the line's records into the numbers voice teams watch (calls that went well, p50 and p95 of the caller's wait, cut-ins, talk-over), compare two versions of Bolo fairly, and find out what each layer is worth by switching it off. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Production
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.

Why you need this

Rashmi has one question before Bolo goes on Kickstart's number: "Is it ready?" Each chapter so far measured one piece on the two or three callers it was built for. That says the piece works. It does not say Bolo works, because a change that fixes Mr Gupta can slow down everybody else, and nobody would see it.

Kavita knows what "ready" means from the desk: every caller gets what they rang for, nobody waits long after a sentence, nobody is cut off or talked over. Imran turns that into numbers: run every caller, read what the line measured, and look at the typical turn and the bad one. Teams that run voice agents for a living watch exactly these numbers, per turn, after every change.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    stats.py: the numbers of a suite

    Write pct(xs, p) (nearest rank: sort a copy, the value at rank ceil(p / 100 x n) counting from 1; math.nan for none) and summarize(recs) giving a Summary (given) of calls, ok, turns, p50, p95, max, cut_ins, nudges and talk_over: p50, p95 and max of every call's ["latency"]["turns"] in one list, cut-ins and nudges added up, talk_over the longest "talkOverMs" of the interrupt barge-ins (0 if none).

  2. Task 2

    suite.py: every caller, a few at a time

    Write async def run_suite(run, ids, par=6): reset the bookings, keep at most par calls going (start the next as soon as one ends), read each call_record(call_id, 15), and return the records in the order of ids.

    Under if __name__ == "__main__":, run every caller from callers() with run_call from agent.py, 7 at a time, and print CALLER ok|FAIL OUTCOME A/B/C ms cut-ins N per call, then OK/N calls, T turns: p50 A ms, p95 B ms, max C ms, then cut-ins N, nudges N, longest talk-over N ms.

  3. Task 3

    versus.py: version 0 against Bolo

    Run run_suite with version 0 (naive.py) and with Bolo (agent.py) on pandey-price, gupta-pauses and fernandes-english at the same time. Print NAME CALLER: ok|FAIL, cut-ins N, turns A/B/C ms for each call (version 0's first), then NAME: OK/N calls, p50 A ms, p95 B ms, cut-ins C for each side, then the typical wait is N ms shorter.

  4. Task 4 · interview question

    Interview: ablation.py, what is each layer worth?

    Write async def ablate(off, callers): run_suite on the callers with every layer on and with run_call(caller, **off), at the same time; {"full": ..., "off": ...}, a summarize of each. Under if __name__ == "__main__":, run the three CASES in the starter at once (semantic_turns on gupta-pauses, barge_in on ritesh-hurry, language on farhan-status) and print LAYER off, CALLER: FULL -> OFF, each side as OK/N ok, cut-ins C, talk-over T ms.

What you will learn

  • The suite: every caller, a few calls at a time, in about a minute and a half.
  • A pool of asyncio workers that starts the next call as soon as any one ends.
  • p50 and p95 of every turn, and why the mean hides the bad turn.
  • Cut-ins and talk-over: the two ways a voice agent is rude.
  • A fair comparison: version 0 against Bolo, same callers, same time.
  • Ablation: switch one layer off and measure what it was worth.

More in Production

Build "Measuring latency" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Build a voice agent are free. This one comes with the all labs plan.