How Node really works · Chapter 13 of 34
Readable and writable streams
By the end of this chapter you can read data a piece at a time, write it to something slow without flooding memory, and build streams of your own. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Events and streams
- 5 tasks
- Node.js
- In the all labs plan
The first 2 chapters of How Node really works are free. This one comes with the all labs plan.
Why you need this
Ravi's stock export now holds a year of deliveries, and the nightly script that reads it pauses the order server for a second. The kitchen printer has the opposite problem: it is slow, and at Saturday's 4 pm rush the server queued thousands of tickets in memory and fell over.
readFile from chapter 9 loads the whole file before you see a byte, so a bigger file means more memory and a longer pause. And a loop that hands tickets to a slow printer has no way to hear "wait, I am full". Streams fix both: data moves in small pieces, and the slow end can say when to stop.
What you build: 5 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
countBytes: count the chunks and bytes in Ravi's stock file
Write countBytes(stream). It reads the readable stream to the end with for await and returns a promise of { chunks, bytes }: how many chunks arrived, and how many bytes they held in all.
Chunks may be Buffers or strings. Count bytes, not characters: a string chunk '₹40' is 5 bytes, because ₹ takes three.
- Task 2
paidOrders: stream the paid orders as objects
Write paidOrders(orders). It returns a Readable stream in object mode whose chunks are one { id, total } object for every order with paid: true, in the same order, where total is qty times price. Unpaid orders are left out.
Return the stream itself (not a promise). An empty list gives a stream that ends with no chunks.
- Task 3
collectLines: rebuild whole lines from chunks that split them
Write collectLines(stream). It reads a stream of UTF-8 text (Buffer or string chunks) and returns a promise of its lines, without the '\n'. A chunk can end in the middle of a line, and even in the middle of a character such as ₹, and the lines must still come out whole and correct.
A last line with no '\n' after it still counts; a '\n' at the very end does not add an empty line.
- Task 4
createPrinter: the kitchen printer as a writable stream
Write createPrinter(). It returns a Writable in object mode with a tickets array on it (printer.tickets, empty at first). Each order written to it adds the ticket '#<id> <qty> x <item>' to printer.tickets, so { id: 7, qty: 2, item: 'masala chai' } gives '#7 2 x masala chai'.
An order with no item fails the stream with the error 'Order <id> has no item', so that finished on the printer rejects with it, and no ticket is added for it. The failure must come through the stream, not as an exception thrown out of write.
- Task 5 · interview question
Interview: respect backpressure when feeding a slow printer
Write writeAll(writable, chunks). It writes every item of chunks to the writable, in order, then ends it, and returns a promise that resolves once the writable has finished.
Whenever write returns false, stop and wait for the 'drain' event before writing the next item. The check uses a slow printer with a highWaterMark of 3 and 30 tickets: at no moment may it hold more than 3, and it must have emitted drain.
What you will learn
- Readable streams: data in chunks, read with for await.
- Object mode: streams of JavaScript values instead of bytes.
- Writable streams: your own write function, and how it fails.
- Backpressure: write returning false, and the 'drain' event.
More in Events and streams
Build "Readable and writable streams" yourself
Tapri’s order server must never stall at the 4 pm rush. You learn why it would. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of How Node really works are free. This one comes with the all labs plan.