Build a voice agent (TypeScript) · Chapter 2 of 21
Sound as numbers
By the end of this chapter you can say what a sample, a sample rate, a bit depth and a channel are, read and write a WAV file byte by byte, cut audio into 20 ms frames, measure how loud each one is in RMS and dBFS, and work out how many bytes a call sends every second. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Sound and the pipeline
- 4 tasks
- TypeScript
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
In chapter 1, phone.ts hid the sound from you. call.next() gave you a frame, isSpeech(frame) said yes or no, and transcribe and speak took care of the rest. That was enough for a walkie-talkie. It is not enough for anything Bolo has to do next.
Imran puts it simply: "Every fix in this lab is a decision about numbers. When is the caller talking? How loud is the kitchen behind her? Is this the sound of a word or of a pressure cooker? Did the phone network send us 8,000 samples a second or 16,000? You cannot answer any of that until you can look at the numbers yourself."
Tejas has a more everyday reason. Kickstart keeps call recordings for complaints and for training, and when Bolo gets a call wrong, the bug report is a WAV file. Last week a script that read those files gave garbage on some of them and crashed on others. It assumed every WAV file starts its sound at byte 44. Some do not.
So in this chapter you open up the sound: the samples, the files they live in, the 20 ms frames a call is made of, and how loud each frame is.
The lesson
A sample is a number
Sound is air pressure going up and down very fast. A microphone turns pressure into a voltage, and a converter measures that voltage at a fixed pace. Each measurement is a sample: one number.
Three settings decide what the numbers look like:
- Sample rate: how many samples a second. Kickstart's line uses 16,000 (16 kHz). Phone networks use 8,000; music uses 44,100 or 48,000; many text-to-speech voices come out at 24,000.
- Bit depth: how many bits each sample has. Calls in this lab are 16-bit: whole numbers from -32,768 to 32,767. Zero is silence, the ends are as loud as the format can go.
- Channels: how many streams of samples. A call is mono (one). A recording of a call is often stereo: the caller on the left, the agent on the right (chapter 19 makes one).
Why 16,000? A sample rate can only capture sounds up to half of it (this is the Nyquist limit: you need at least two samples per up-and-down of a wave to see it). 16,000 a second catches everything up to 8,000 Hz, which holds all the sounds that make speech clear. Phones, at 8,000 a second, keep only up to 4,000 Hz, and that is chapter 3.
So one second of a call is 16,000 samples of 2 bytes each: 32,000 bytes. Remember that number.
In the lab, every voice is the lab's toy speech: a word is three short tones between 500 and 3,250 Hz, like a very fast R2-D2. Each tone is a real wave in real samples, so everything in this chapter is exactly what you would do with a real voice.
Inside a WAV file
A WAV file is a short header that says how to read the numbers, then the numbers. Here are the first 44 bytes of Mrs Iyer's recording, /workspace/calls/kitchen-whistle-1.wav, in hex:
00000000: 5249 4646 84df 0100 5741 5645 666d 7420 RIFF....WAVEfmt
00000010: 1000 0000 0100 0100 803e 0000 007d 0000 .........>...}..
00000020: 0200 1000 6461 7461 60df 0100 7600 1300 ....data`...v...
It is a list of chunks. The file starts with RIFF, the size of the rest, and WAVE. Then every chunk is a 4-letter id, its size in 4 bytes, and that many bytes of body:
fmtwith size10 00 00 00(16): format code01 00(1 means plain PCM samples), channels01 00(1), rate80 3e 00 00, and so on.datawith size60 df 01 00: the samples, two bytes each.
Numbers are little-endian: the least significant byte comes first, so 80 3e 00 00 is 0x3e80, which is 16,000. The first sample, 76 00, is 0x0076 = 118: the quiet of her kitchen.
Read those fields, and you have:
{ rate: 16000, channels: 1, bits: 16, samples: Int16Array(61360), seconds: 3.835 }
Here is the trap Tejas's script fell into. On a file like this one the samples start at byte 44, and plenty of code just jumps there. But chunks can be different sizes. The mu-law recording in the same folder has an 18-byte fmt chunk (two extra bytes at its end), so its data starts at byte 46, and byte 44 is in the middle of the word data. Other files carry a LIST chunk of song titles or a fact chunk before the sound. The rule is simple: walk the chunks by their sizes, skip the ones you do not know, and read data wherever it is. One more detail from the WAV format: a chunk with an odd size is followed by one pad byte.
When the format is not plain 16-bit PCM, say so instead of guessing:
Error: not 16-bit PCM (format 7, 8 bits)
Format 7 is mu-law, 8 bits a sample, what phone networks carry. That is chapter 3.
Reading samples safely
Each sample is two bytes, little-endian, signed. buf.readInt16LE(i) reads one. It is tempting to wrap the bytes in an Int16Array directly instead, and on most files that works. But a Node Buffer is often a slice of a bigger block of memory, and it can start at an odd byte of it. Then:
RangeError: start offset of Int16Array should be a multiple of 2
readInt16LE reads from any position, so use it and this never happens.
Frames
A voice agent never waits for a whole recording. It works on frames: small, equal pieces of audio, one after another. Kickstart's line sends one every 20 ms, which is 320 samples at 16 kHz. Twilio, Exotel and Plivo stream calls in 20 ms frames too, and speech recognisers and voice activity detectors are built around frames of 10 to 30 ms.
Why 20 ms? Short enough that Bolo can react within a few frames of something happening, long enough to hold several up-and-downs of even a low voice. Mrs Iyer's 3.835 seconds are 191 whole frames; the last 120 samples do not fill a frame and are dropped.
Here are the first twelve samples of frame 30, in the middle of her first word:
8957 7030 -3884 -9753 -3881 6849 9200 4 -9047 -6846 3859 9754
Big numbers, swinging from positive to negative and back: a tone.
How loud is a frame?
The biggest sample in frame 30 is 9,936. But the biggest sample is a bad measure of loudness: a single click in a quiet frame can reach 30,000. What you want is the typical size of all the samples. That is the RMS (root mean square):
- take each sample as a fraction of full scale (divide by 32,768),
- square it (so negative and positive both count),
- average the squares,
- take the square root.
The answer runs from 0 (silence) to about 1 (as loud as 16 bits can go). That lone click of 30,000 has an RMS of only 0.0512; frame 30 has 0.2122.
Engineers usually say loudness in decibels relative to full scale, dBFS: 20 × log10(rms). Full scale is 0 dBFS, and everything quieter is negative:
| RMS | dBFS | What it is |
|---|---|---|
| 0.5 | -6.02 | half of full scale |
| 0.25 | -12.04 | a quarter: every halving is about 6 dB |
| 0.2122 | -13.5 | Mrs Iyer's voice, frame 30 |
| 0.02 | -34.0 | isSpeech's threshold in chapter 1 |
| 0.0023 | -52.8 | her line's quiet, frame 3 |
A frame of all zeros has an RMS of 0, and log10(0) is minus infinity, which breaks every comparison and average you put it in. Give it -120 dBFS, far below anything 16 bits can hold (the smallest non-zero sample is about -90 dBFS).
That last row matters. A phone line is never silent: there is always a little hiss, the noise floor. Here are Mrs Iyer's first 120 frames, # above -20 dBFS, + between -34 and -20, . below -34:
.........................#########+..........#########+..#########+..##########..##########..##########..##########..###
Half a second of floor, then words: nine loud frames each (three tones of about 60 ms), a short gap between words, a longer one at her comma. The gap between the floor (-53) and her voice (-13.5) is about 40 dB. Chapter 6 builds a voice activity detector on exactly this.
Step through her first frames:
Writing a WAV
Writing is reading backwards: a 44-byte header with the numbers worked out from the samples, then the samples, each with writeInt16LE. The fields you compute are the byte rate (rate × channels × 2) and the block align (channels × 2: one sample of every channel).
One trap again comes from shared memory. Int16Array.subarray(a, b) gives a window onto a bigger array, not a copy. Take a tenth of a second of Mrs Iyer's call, w.samples.subarray(16000, 17600): that is 1,600 samples, 3,200 bytes. Buffer.from(part.buffer) is 122,720 bytes, the whole recording the window looks into. Write the samples one by one and the file is right: 3,244 bytes, header included.
The STT reads the rate from your header, so a WAV with a wrong header is not just ugly: it is heard wrong. Task 3 checks yours by sending it to the STT.
A call on the wire
On a real phone line the frames arrive over a WebSocket, as JSON text. Here is the line's start message for a call, and the first frame after it (trimmed):
{"event":"start","call_id":"CAmvx73p2xga0om1","caller":{"from":"+91 98810 66xxx"},"media_format":{"encoding":"audio/l16","sample_rate":16000,"channels":1},"frame_ms":20}
{"event":"media","seq":1,"t":20,"payload":"DADo/xcAVQB1AEkARQBcAPj/x/9xAJv/KwCDAOH/XABAACgA6/8VAB0ARwA9AFwAx/+M/2cAo/90A...
The start message tells you how to read everything after it: audio/l16 (linear 16-bit) at 16,000 Hz, one channel. Each payload is the frame's 640 bytes written as base64, because JSON can only carry text. Base64 spends 4 characters for every 3 bytes, so the payload is longer than the audio it carries. Twilio and Exotel send media in this same shape, with mu-law at 8,000 Hz inside (chapter 3).
The interview question is how many bytes that is per second and per hour, and you will measure it on a real call.
The numbers
| One frame (20 ms) | One second | |
|---|---|---|
| samples at 16 kHz | 320 | 16,000 |
| bytes at 16 bits | 640 | 32,000 |
| frames | 1 | 50 |
A call is a steady stream: fifty frames a second, every second, whether anyone is talking or not.
Your turn
Read a WAV file chunk by chunk in task 1. Cut a call into frames and find its loudest and quietest moments in task 2. Write a trimmed WAV that the STT hears in task 3. In the interview question, count what a live call costs on the wire.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
read-wav.ts: read a WAV file by hand
Write and export readWav(buf) giving { rate, channels, bits, samples, seconds }. Walk the chunks from byte 12 by their sizes (a chunk of odd size has one pad byte); read rate, channels and bits from "fmt ", and the samples from "data" with readInt16LE; skip any other chunk. Throw an Error when the format is not 1 or the bits are not 16. seconds is samples / channels / rate.
Under if (import.meta.main), read ../../calls/farhan-status-1.wav and print farhan-status-1.wav: RATE Hz, 1 channel, BITS bits, then N samples, S s (2 decimals), first 5: and the first five samples, and loudest: L (the largest sample, ignoring its sign).
- Task 2
frames.ts: frames and loudness
Write and export frames(pcm, size = 320) (whole frames of size samples, one after another; a short last piece is dropped), rms(frame) (each sample over 32768, squared, averaged, square-rooted) and dbfs(frame) (20 x log10(rms), and -120 for a frame of zeros).
Under if (import.meta.main), cut ../../calls/anil-reschedule-1.wav into frames and print anil-reschedule-1.wav: N frames of 20 ms, then loudest: and quietest: as frame I at T s, rms R, D dBFS (T 2 decimals, R 4, D 1; the first frame if two tie), then speech (above -34 dBFS): N frames.
- Task 3
write-wav.ts: write a WAV file by hand
Write and export writeWav(pcm, rate = 16000, channels = 1) giving a Buffer: the 44-byte header (RIFF, 36 + data bytes, WAVE, a 16-byte fmt chunk with format 1, channels, rate, byte rate rate x channels x 2, block align channels x 2, 16 bits, then data and its size) and every sample with writeInt16LE. pcm may be a part of a bigger array.
Under if (import.meta.main), read ../../calls/shabnam-sunday-1.wav, keep the frames from the first one above -34 dBFS to the last one (both included), write them to shabnam-trimmed.wav, POST the file to the STT (VOICE + /v1/listen), and print shabnam-sunday-1.wav: S s, kept frames FIRST to LAST: S s, wrote shabnam-trimmed.wav: N bytes, and the STT heard: TEXT.
- Task 4 · interview question
Interview: bytes.ts, how many bytes is a call?
Open a WebSocket to WS + /v1/line?caller=pandey-price and read its JSON messages: the start message has the call_id, each media message a base64 payload. Keep the first 150 payloads, then send {"event":"hangup"}. Print call ID: 150 media messages, 3.00 s of audio; one message: C characters of base64, B bytes, N samples; a second of audio: B bytes, C characters on the wire; base64 adds P% (whole number); an hour of one call: M MB (million bytes, 1 decimal).
What you will learn
- Samples, sample rate, bit depth and channels: what the numbers mean.
- A WAV file from the inside: chunks, little-endian numbers, and why the header is not always 44 bytes.
- Frames: 20 ms at a time, the unit every voice system works in.
- Loudness: RMS and decibels (dBFS), and the line's own quiet.
- Writing a WAV that players and the STT read back.
- What a call costs on the wire: bytes a second, and base64 inside JSON.
More in Sound and the pipeline
Build "Sound as numbers" yourself
Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.