Build a voice agent · Chapter 2 of 21

Sound as numbers

By the end of this chapter you can say what a sample, a sample rate, a bit depth and a channel are, read and write a WAV file byte by byte, cut audio into 20 ms frames, measure how loud each one is in RMS and dBFS, and work out how many bytes a call sends every second. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Sound and the pipeline
  • 4 tasks
  • Python
  • Free chapter

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.

Why you need this

In chapter 1, phone.py hid the sound from you. call.next() gave you a frame, is_speech(frame) said yes or no, and transcribe and speak took care of the rest. That was enough for a walkie-talkie. It is not enough for anything Bolo has to do next.

Imran puts it simply: "Every fix in this lab is a decision about numbers. When is the caller talking? How loud is the kitchen behind her? Is this the sound of a word or of a pressure cooker? Did the phone network send us 8,000 samples a second or 16,000? You cannot answer any of that until you can look at the numbers yourself."

Tejas has a more everyday reason. Kickstart keeps call recordings for complaints and for training, and when Bolo gets a call wrong, the bug report is a WAV file. Last week a script that read those files gave garbage on some of them and crashed on others. It assumed every WAV file starts its sound at byte 44. Some do not.

So in this chapter you open up the sound: the samples, the files they live in, the 20 ms frames a call is made of, and how loud each frame is.

The lesson

A sample is a number

Sound is air pressure going up and down very fast. A microphone turns pressure into a voltage, and a converter measures that voltage at a fixed pace. Each measurement is a sample: one number.

Three settings decide what the numbers look like:

  • Sample rate: how many samples a second. Kickstart's line uses 16,000 (16 kHz). Phone networks use 8,000; music uses 44,100 or 48,000; many text-to-speech voices come out at 24,000.
  • Bit depth: how many bits each sample has. Calls in this lab are 16-bit: whole numbers from -32,768 to 32,767. Zero is silence, the ends are as loud as the format can go. In numpy that is the dtype int16.
  • Channels: how many streams of samples. A call is mono (one). A recording of a call is often stereo: the caller on the left, the agent on the right (chapter 19 makes one).

Why 16,000? A sample rate can only capture sounds up to half of it (this is the Nyquist limit: you need at least two samples per up-and-down of a wave to see it). 16,000 a second catches everything up to 8,000 Hz, which holds all the sounds that make speech clear. Phones, at 8,000 a second, keep only up to 4,000 Hz, and that is chapter 3.

So one second of a call is 16,000 samples of 2 bytes each: 32,000 bytes. Remember that number.

In the lab, every voice is the lab's toy speech: a word is three short tones between 500 and 3,250 Hz, like a very fast R2-D2. Each tone is a real wave in real samples, so everything in this chapter is exactly what you would do with a real voice.

Inside a WAV file

A WAV file is a short header that says how to read the numbers, then the numbers. Here are the first 48 bytes of Mrs Iyer's recording, /workspace/calls/kitchen-whistle-1.wav, in hex (xxd -l 48 in the terminal):

00000000: 5249 4646 84df 0100 5741 5645 666d 7420  RIFF....WAVEfmt 
00000010: 1000 0000 0100 0100 803e 0000 007d 0000  .........>...}..
00000020: 0200 1000 6461 7461 60df 0100 7600 1300  ....data`...v...

It is a list of chunks. The file starts with RIFF, the size of the rest, and WAVE. Then every chunk is a 4-letter id, its size in 4 bytes, and that many bytes of body:

  • fmt with size 10 00 00 00 (16): format code 01 00 (1 means plain PCM samples), channels 01 00 (1), rate 80 3e 00 00, and so on.
  • data with size 60 df 01 00: the samples, two bytes each.

Numbers are little-endian: the least significant byte comes first, so 80 3e 00 00 is 0x3e80, which is 16,000. The first sample, 76 00, is 0x0076 = 118: the quiet of her kitchen.

Python's struct module reads them. struct.unpack_from("<I", data, 4) reads a 4-byte unsigned number at byte 4; the < means little-endian, I is 4 bytes, H is 2. One call can read several fields in a row:

code, channels, rate = struct.unpack_from("<HHI", body, 0)

Read those fields, and the samples, and you have:

Wav(rate=16000, channels=1, bits=16, samples=array([118,  19, -51, ..., -60,   2, 127], shape=(61360,), dtype=int16), seconds=3.835)

Here is the trap Tejas's script fell into. On a file like this one the samples start at byte 44, and plenty of code just jumps there. But chunks can be different sizes. The mu-law recording in the same folder has an 18-byte fmt chunk (two extra bytes at its end), so its data starts at byte 46, and byte 44 is in the middle of the word data. Other files carry a LIST chunk of song titles or a fact chunk before the sound. The rule is simple: walk the chunks by their sizes, skip the ones you do not know, and read data wherever it is. One more detail from the WAV format: a chunk with an odd size is followed by one pad byte.

When the format is not plain 16-bit PCM, say so instead of guessing:

ValueError: not 16-bit PCM (format 7, 8 bits)

Format 7 is mu-law, 8 bits a sample, what phone networks carry. That is chapter 3.

From bytes to samples

np.frombuffer(body, dtype="<i2") turns the data chunk's bytes into samples in one step: <i2 is a signed, 2-byte, little-endian number. It is fast because it does not copy anything: the array looks straight at the file's bytes. And Python's bytes cannot change, so neither can the array:

ValueError: assignment destination is read-only

That is what you get the first time anyone tries to change a sample, a bleep over a card number in chapter 19, say. Add .astype(np.int16) and the samples are a copy of their own, which you can change. (One more surprise from frombuffer: a byte count that is not a multiple of 2 raises ValueError: buffer size must be a multiple of element size.)

Frames

A voice agent never waits for a whole recording. It works on frames: small, equal pieces of audio, one after another. Kickstart's line sends one every 20 ms, which is 320 samples at 16 kHz. Twilio, Exotel and Plivo stream calls in 20 ms frames too, and speech recognisers and voice activity detectors are built around frames of 10 to 30 ms.

Why 20 ms? Short enough that Bolo can react within a few frames of something happening, long enough to hold several up-and-downs of even a low voice. Mrs Iyer's 3.835 seconds are 191 whole frames; the last 240 samples do not fill a frame and are dropped. In numpy a frame is a slice, pcm[at:at + 320].

Here are the first twelve samples of frame 30, in the middle of her first word:

8957 7030 -3884 -9753 -3881 6849 9200 4 -9047 -6846 3859 9754

Big numbers, swinging from positive to negative and back: a tone.

How loud is a frame?

The biggest sample in frame 30 is 9,936. But the biggest sample is a bad measure of loudness: a single click in a quiet frame can reach 30,000. What you want is the typical size of all the samples. That is the RMS (root mean square):

  1. take each sample as a fraction of full scale (divide by 32,768),
  2. square it (so negative and positive both count),
  3. average the squares,
  4. take the square root.

The answer runs from 0 (silence) to about 1 (as loud as 16 bits can go). That lone click of 30,000 has an RMS of only 0.0512; frame 30 has 0.2122.

Do the arithmetic in floats. numpy keeps an int16 array's arithmetic in int16, and the square of 9,000 does not fit in 16 bits: it wraps around without a word. Square frame 30's samples as they are, and its RMS comes out as 0.0011, quieter than the line's hiss. Dividing by 32,768 first makes the numbers floats (float64), and then nothing overflows.

Engineers usually say loudness in decibels relative to full scale, dBFS: 20 × log10(rms). Full scale is 0 dBFS, and everything quieter is negative:

RMSdBFSWhat it is
0.5-6.02half of full scale
0.25-12.04a quarter: every halving is about 6 dB
0.2122-13.5Mrs Iyer's voice, frame 30
0.02-34.0is_speech's threshold in chapter 1
0.0023-52.8her line's quiet, frame 3

A frame of all zeros has an RMS of 0, and math.log10(0) raises ValueError: math domain error (numpy's gives minus infinity, which breaks every comparison and average you put it in). Give it -120 dBFS, far below anything 16 bits can hold (the smallest non-zero sample is about -90 dBFS).

That last row matters. A phone line is never silent: there is always a little hiss, the noise floor. Here are Mrs Iyer's first 120 frames, # above -20 dBFS, + between -34 and -20, . below -34:

.........................#########+..........#########+..#########+..##########..##########..##########..##########..###

Half a second of floor, then words: nine loud frames each (three tones of about 60 ms), a short gap between words, a longer one at her comma. The gap between the floor (-53) and her voice (-13.5) is about 40 dB. Chapter 6 builds a voice activity detector on exactly this.

Step through her first frames:

Writing a WAV

Writing is reading backwards: a 44-byte header with the numbers worked out from the samples, then the samples. struct.pack("<I", n) writes a 4-byte number; the fields you compute are the byte rate (rate × channels × 2) and the block align (channels × 2: one sample of every channel).

The samples need care too. arr.tobytes() writes each number in the array's own size, and not every array is int16. Build samples from a list of Python ints, np.array([118, 19, -51, ...]), and numpy picks int64: 8 bytes a number. A tenth of a second of Mrs Iyer's call (1,600 samples, w.samples[16000:17600]) is 3,200 bytes as int16 and 12,800 as int64. A file of the second kind claims 16-bit samples and carries four times as many bytes, so every player and the STT read it wrong. np.asarray(pcm).astype("<i2").tobytes() gives 2 bytes a sample, whatever came in.

The STT reads the rate from your header, so a WAV with a wrong header is not just ugly: it is heard wrong. Task 3 checks yours by sending it to the STT.

A call on the wire

On a real phone line the frames arrive over a WebSocket, as JSON text. Here is the line's start message for a call, and the first frame after it (trimmed):

{"event":"start","call_id":"CAmvxabhpgy0jiir","caller":{"from":"+91 98810 66xxx"},"media_format":{"encoding":"audio/l16","sample_rate":16000,"channels":1},"frame_ms":20}
{"event":"media","seq":0,"t":0,"payload":"+v/W/0IAIQBDAHkAov9RAEYAf/95AKD/n/+BAIf/9//9/3AA4v+y/5n/h/8dAIL/UACA/3kAjf9//2YAUADx/67/...

The start message tells you how to read everything after it: audio/l16 (linear 16-bit) at 16,000 Hz, one channel. Each payload is the frame's 640 bytes written as base64, because JSON can only carry text. Base64 spends 4 characters for every 3 bytes, so the payload is longer than the audio it carries. Twilio and Exotel send media in this same shape, with mu-law at 8,000 Hz inside (chapter 3).

In Python, the websockets package opens the connection: async with connect(url) as ws: and then async for raw in ws: gives one message at a time, as it arrives. Sending is awaited too: await ws.send(text).

The interview question is how many bytes that is per second and per hour, and you will measure it on a real call.

The numbers

One frame (20 ms)One second
samples at 16 kHz32016,000
bytes at 16 bits64032,000
frames150

A call is a steady stream: fifty frames a second, every second, whether anyone is talking or not.

Your turn

Read a WAV file chunk by chunk in task 1. Cut a call into frames and find its loudest and quietest moments in task 2. Write a trimmed WAV that the STT hears in task 3. In the interview question, count what a live call costs on the wire.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    read_wav.py: read a WAV file by hand

    Write read_wav(data) giving a Wav (given: rate, channels, bits, samples, seconds). Walk the chunks from byte 12 by their sizes (a chunk of odd size has one pad byte); read rate, channels and bits from "fmt " with struct.unpack_from, and the samples from "data" with np.frombuffer(body, dtype="<i2").astype(np.int16) (a copy you can change); skip any other chunk. Raise ValueError when the format is not 1 or the bits are not 16. seconds is samples / channels / rate.

    Under if __name__ == "__main__":, read CALLS / "farhan-status-1.wav" and print farhan-status-1.wav: RATE Hz, 1 channel, BITS bits, then N samples, S s (2 decimals), first 5: and the first five samples, and loudest: L (the largest sample, ignoring its sign).

  2. Task 2

    frames.py: frames and loudness

    Write frames(pcm, size=320) (whole frames of size samples, one after another; a short last piece is dropped), rms(frame) (each sample over 32768 as a float, squared, averaged, square-rooted) and dbfs(frame) (20 x log10(rms), and -120.0 for a frame of zeros).

    Under if __name__ == "__main__":, cut CALLS / "anil-reschedule-1.wav" into frames and print anil-reschedule-1.wav: N frames of 20 ms, then loudest: and quietest: as frame I at T s, rms R, D dBFS (T 2 decimals, R 4, D 1; the first frame if two tie), then speech (above -34 dBFS): N frames.

  3. Task 3

    write_wav.py: write a WAV file by hand

    Write write_wav(pcm, rate=16000, channels=1) giving bytes: the 44-byte header (RIFF, 36 + data bytes, WAVE, a 16-byte fmt chunk with format 1, channels, rate, byte rate rate x channels x 2, block align channels x 2, 16 bits, then data and its size, all with struct.pack little-endian) and every sample as 2 bytes, whatever the array's dtype.

    Under if __name__ == "__main__":, read CALLS / "shabnam-sunday-1.wav", keep the frames from the first one above -34 dBFS to the last one (both included), write them to shabnam-trimmed.wav, POST the file to the STT (VOICE + /v1/listen), and print shabnam-sunday-1.wav: S s, kept frames FIRST to LAST: S s, wrote shabnam-trimmed.wav: N bytes, and the STT heard: TEXT.

  4. Task 4 · interview question

    Interview: call_bytes.py, how many bytes is a call?

    In an async def main(), connect with websockets to WS + /v1/line?caller=pandey-price and read its JSON messages (async for raw in ws): the start message has the call_id, each media message a base64 payload. Keep the first 150 payloads, then await ws.send of {"event": "hangup"} and stop reading. Print call ID: 150 media messages, 3.00 s of audio; one message: C characters of base64, B bytes, N samples; a second of audio: B bytes, C characters on the wire; base64 adds P% (whole number); an hour of one call: M MB (million bytes, 1 decimal). The file ends with asyncio.run(main()).

What you will learn

  • Samples, sample rate, bit depth and channels: what the numbers mean.
  • A WAV file from the inside: chunks, little-endian numbers, and why the header is not always 44 bytes.
  • Frames: 20 ms at a time, the unit every voice system works in.
  • Loudness: RMS and decibels (dBFS), and the line's own quiet.
  • Writing a WAV that players and the STT read back.
  • What a call costs on the wire: bytes a second, and base64 inside JSON.

More in Sound and the pipeline

Build "Sound as numbers" yourself

Kickstart services scooters at the door in Nagpur, and its phone line gets 600 calls a day in Hindi, English and Hinglish. You build Bolo, its voice agent, and measure every call: who it understood, how long callers waited and who it cut off. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.