Build a voice agent (TypeScript) · Chapter 1 of 21
Meet Kickstart, and Bolo's first call
By the end of this chapter you can say what a voice agent is and what happens inside one during a call, take a call and look at the sound as it arrives, build the simplest agent that answers a whole call, read the call the way the caller lived it, and say where every second of waiting went. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Sound and the pipeline
- 4 tasks
- TypeScript
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
Kickstart, in Nagpur, services scooters and motorbikes at your door: a mechanic picks the bike up in the morning, services it, and brings it back by evening. Most customers book on the phone, and the phone line gets about 600 calls a day: what does a service cost, do you come on Sunday, move my service to Friday, is my Activa ready. Callers speak Hindi, English, and mostly both in one sentence.
Kavita runs the call desk with one colleague. At nine in the morning half the calls ring out. Rashmi, Kickstart's founder, wants a voice agent on the line that answers every call, in the caller's own language, and books, moves and cancels services the way Kavita does. Imran, the voice engineer, will build it with you, and he has named it Bolo.
Voice agents like this are answering phones across India right now: for banks, clinics, delivery apps and service companies. They are built from three services and a loop that runs in real time. Tejas, who looks after Kickstart's bookings and its data, has one condition: an agent that moves a booking must move the right one. Kavita has another: "Callers hang up after a few seconds of silence. They pause to find a number. They talk while you talk. Bolo has to handle people, not sentences."
Here is why that is hard. Imran wrote a first Bolo in an afternoon. On Kickstart's thirteen test callers, 6 of 13 calls went well. After every sentence, callers waited 2.5 seconds for an answer (3.5 seconds on a bad turn). It answered a caller halfway through his sentence. It kept a silent line open for the full 90 seconds. It read prices to an English-speaking caller in Hindi numbers. The Bolo you finish in chapter 20 gets 13 of 13, answers in about 1.1 seconds, never cuts a caller off, stops talking when interrupted, and never touches the wrong booking.
The lesson
The phone line in the lab
Nobody can speak into a lab, so Kickstart's callers are simulated, and the lab has its own phone line. lab callers lists the thirteen of them:
pandey-price hi An Activa owner asks what a service costs, and whether pickup is extra.
shabnam-sunday hi Asks whether Kickstart picks up on Sunday, then about Saturday evening.
gupta-pauses hi Thinks while he talks: pauses for a second in the middle of his sentences.
kitchen-whistle hi Calls from her kitchen: a pressure cooker whistles while the agent is talking.
fernandes-english en Speaks English only; asks the price for his Royal Enfield and how long it takes.
silent-line hi The line connects, then nobody speaks: a pocket dial, or a bad network.
...
Each caller is a small script that reacts to what it hears: it waits for a greeting, answers when Bolo stops talking, says "hello? aap sun rahe hain?" after six seconds of silence, interrupts when it is in a hurry, and hangs up when it is done. The line plays the caller in real time and records everything, so every chapter can measure Bolo the way a caller would.
The voices are the lab's toy speech: every word is three short tones, like a very fast R2-D2. That sounds silly, and it is on purpose. It has everything real speech has that a voice agent must handle: loudness and silence, words and pauses, timing, noise, a recogniser that can mishear. The speech-to-text and text-to-speech services in the lab behave like real streaming ones (the same messages, the same kinds of delay), and the code you write here works unchanged against real ones.
What arrives: frames
phone.ts is given in this chapter. dial(caller) takes a call, and call.next() gives you what the caller's phone sends: 20 ms of sound at a time, a frame. Here is a call from Mrs Iyer, after Bolo's greeting:
const call = await dial('kitchen-whistle');
await call.play(await speak(GREETING));
const first = await call.next();
console.log(first.length, Array.from(first.slice(0, 10)).join(' '));
320 110 34 -33 -41 -86 97 -108 114 -118 51
Sound is air pressure going up and down. A microphone measures it 16,000 times a second, and each measurement, a sample, is a whole number between -32,768 and 32,767. 20 ms is 320 samples. The ten above are the quiet of her kitchen: small numbers around zero. When she speaks, the numbers grow to thousands (the loudest in her first line is 9,961).
isSpeech(frame) asks one question: is this frame loud enough to be someone talking? Here are her 300 frames after the greeting, # for speech and . for quiet (trimmed to the first 150):
.........##########..........##########..##########..##########..##########..##########..##########..##########..##########..##########...............
(The longer quiet after the first word is the comma: "namaste, jupiter ki ...".)
Ten words, a short gap between each, then quiet. Chapter 2 opens up the samples and frames; chapter 6 makes isSpeech much smarter.
Three services and a loop
A voice agent turns a caller's sound into Bolo's sound, through three services in a row:
- Speech-to-text (STT): sound in, words out.
transcribe(pcm)here. - The model: the conversation in, a reply out.
lab-chat, the lab's model, with the same API as OpenAI's. - Text-to-speech (TTS): words in, sound out.
speak(text)here.
This chain is called a cascade. Some newer models take sound in and give sound out directly (speech-to-speech). Most voice agents on Indian phone lines today are cascades, because you can choose the best recogniser for Hindi, the best model, and the best voice separately, and you can see the text in between: what the caller said, and what the agent decided to say.
(lab-chat is a predictable stand-in for a real model: it reads the conversation and answers the same way every time, which is what lets the lab measure Bolo. Its habits are the ones real models have on a phone line, made exact. OPENAI_BASE_URL points the official openai package at it; point it and OPENAI_API_KEY at a real provider with your own key, on a machine with internet, and the same loop drives a real model.)
Version 0: a walkie-talkie
Imran's first Bolo works like a walkie-talkie. It greets the caller, then:
- Listens until the caller stops. It cannot know when someone has finished a sentence, so it waits for one full second of silence: 50 quiet frames in a row.
- Understands all of it: the whole turn goes to
transcribe. - Thinks: the whole conversation goes to the model, and it waits for all of the reply.
- Speaks all of it: the whole reply goes to
speak, andcall.playplays it. While it plays, it does not listen.
And round again, until the caller hangs up. The conversation is the same list of messages any chat app sends: a system prompt, Bolo's greeting as an assistant message, and each turn's words as a user message.
Task 2 writes this loop. It works: Mr Pandey gets his price, Shabnam gets her answer about Sunday. It is also slow, and it is deaf while it talks.
The call, as the caller lived it
The line keeps a record of every call: what the caller said and when, what Bolo said and when, and how long the caller waited. callRecord(id) reads it, and lab call prints the last one. Here is version 0 taking a call from Mr Gupta, who thinks while he talks:
0.54- 11.70 Bolo namaste kickstart mein aapka swagat hai main bolo bol rahi hoon bataiye main aapki kya ...
6.26- 9.99 caller haan ji, meri pulsar ki general service kitne ki hai? Bolo cut in at 9.94 s
17.70- 19.23 caller (nobody answered) hello? aap sun rahe hain?
21.78- 29.82 Bolo ji main sun rahi hoon bataiye main aapki kya madad kar sakti hoon maaf kijiye main sama... (2550 ms after the caller)
25.72- 28.39 caller aur pickup ka time kya hota hai? Bolo cut in at 28.36 s
...
outcome: caller_hung_up, not a success
✖ heard the bike price (ek hazaar ek sau ninyaanve)
✖ heard pickup hours (nau ... chhah)
cut-ins: 2, "hello?" from the caller: 1
(Trimmed: the rest of the call is two more lines.)
Mr Gupta said "haan ji, meri pulsar ki", then paused for a second to think. One second of silence is exactly what version 0 waits for, so it took his half-sentence as a whole turn, heard "haan ji" as a yes, and answered "Kya main aur kuch madad kar sakti hoon?" right over the rest of his question, which a walkie-talkie never hears. He waited, said "hello?", and never got his price. This is a cut-in, and chapter 7 is about it.
For every line the caller said, the record has latencyMs: from the end of the caller's last word to the start of Bolo's first. That is the silence the caller sat through, and it is the number every voice team watches.
Where the seconds go
In a conversation between people, the gap between one person stopping and the other starting is usually a fraction of a second. On a phone line, two seconds of silence feels like a dropped call. So how does version 0 spend almost 2.9 seconds on Mrs Iyer's first question? Time each stage on its own:
wait for silence: 1000 ms
speech to text: 371 ms
model: 886 ms
text to speech: 604 ms
total: 2861 ms
On her real call, the line measured 2,885 ms for that turn: the stages account for almost all of it. The biggest stage is not the model. It is the second of silence Bolo waits through before it does anything. Every stage after it waits for the one before to finish completely: all the audio before any words, all the reply before any speech.
That is the latency budget, and most of this lab spends it better: deciding faster that a turn is over (chapters 6 and 7), getting words while the caller is still talking (chapter 5), speaking the first sentence while the model writes the second (chapters 8 and 9).
The numbers
On Kickstart's thirteen callers:
| Version 0 | Bolo, chapter 20 | |
|---|---|---|
| calls that went well | 6 of 13 | 13 of 13 |
| caller's wait, typical (p50) | 2,540 ms | 1,105 ms |
| caller's wait, worst 1 in 20 (p95) | 3,519 ms | 1,980 ms |
| cut-ins | 2 | 0 |
| silent line | open for 90 s | ended at 34 s |
The interview task takes one turn apart, stage by stage, and checks your numbers against what the line measured.
Your turn
Take a call and look at its frames in task 1. Write Bolo's first loop in task 2. Print a call's timeline from the line's record in task 3. In the interview question, find where the seconds go.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
first-call.ts: listen to a call
Take a call and look at what arrives. dial('pandey-price') (from ./phone.ts), greet the caller with call.play(await speak(GREETING)), then read 300 frames with call.next() and hang up with call.hangup(). Print, in this order:
call ID: 300 frames, 6 seconds (ID is call.id)
speech in N frames (S s) (N: the frames isSpeech says are speech; S is N x 0.02 with 2 decimals)
caller: TEXT (the 300 frames joined into one Int16Array, then transcribed)
- Task 2
first-agent.ts: Bolo's first loop
Write and export listenTurn(call) and runCall(caller). listenTurn reads frames from the first one isSpeech says yes to until the 50th quiet frame in a row, keeping every frame in between (short pauses too), and returns them as one Int16Array, or null when call.next() gives null. runCall dials the caller, starts the messages with { role: 'system', content: SYSTEM } and { role: 'assistant', content: GREETING }, plays speak(GREETING), then until listenTurn gives null: transcribes the turn, adds it as a user message, asks 'lab-chat' with all the messages, adds the reply as an assistant message and plays speak(reply). Return { callId, turns, messages }, with turns a list of { heard, said }.
Under if (import.meta.main), run runCall('shabnam-sunday') and print each turn as caller: HEARD then Bolo: SAID, then call ID: N turns.
- Task 3
timeline.ts: the call, as the line heard it
Run runCall('meena-first-time') (from ./first-agent.ts), then read the line's record with callRecord(run.callId) (from ../../shared/kickstart.ts). Print every entry of its lines and replies sorted by start, as START WHO TEXT (START with 2 decimals, WHO caller or Bolo, two spaces apart). Then, for each line with a latencyMs, in order, print turn N: LATENCY ms, and last slowest: LATENCY ms.
- Task 4 · interview question
Interview: budget.ts, where do the seconds go?
Time one turn stage by stage with performance.now(): transcribe(...) on say('namaste ji , mere paas activa hai . uski general service ka kitna lagega ?'), one 'lab-chat' request with SYSTEM, GREETING (as the assistant) and the transcript (as the user), and speak(...) of the reply. Print wait for silence: 1000 ms, speech to text: N ms, model: N ms, text to speech: N ms, then total: N ms (the four added up). Then run runCall('pandey-price') and print the line measured: N ms (latency.turns[0] of its record), and last biggest: STAGE (the name of the slowest stage).
What you will learn
- Kickstart's phone line in the lab, its callers, and the lab's toy speech.
- What a call is made of: frames of 20 ms, each a list of numbers.
- The three services of a voice agent: speech-to-text, the model, text-to-speech.
- Bolo's first loop: listen, understand, think, speak, and listen again.
- A call's record: the timeline as the caller lived it, and the latency of every turn.
- The latency budget: where the seconds go, stage by stage.
More in Sound and the pipeline
Build "Meet Kickstart, and Bolo's first call" yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.