How Node really works · Chapter 15 of 34

Compression with zlib

By the end of this chapter you can shrink data with gzip and brotli, choose the format a client can read, and compress or unpack a file of any size as a stream. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Events and streams
  • 5 tasks
  • Node.js
  • In the all labs plan

The first 2 chapters of How Node really works are free. This one comes with the all labs plan.

Why you need this

Anjali has numbers from the app: customers on the metro open the menu on a weak 4G signal, the menu response is 180 KB of JSON, and half of them give up before it loads. Ravi has a different complaint: a year of order archives is eating the server's disk.

Everything you have sent or saved so far went out byte for byte, as large as it was made. Text like orders repeats the same keys and item names thousands of times, and nothing you know yet takes advantage of that. node:zlib does, and it speaks Buffers and streams, which you already know.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    packOrders and unpackOrders: gzip the order list and get it back

    Write packOrders(orders). It turns the array of orders into JSON and gzips it, and returns a promise of the compressed Buffer (a real Buffer that starts with the gzip bytes 1f 8b, not a string). The check expects 200 orders to shrink to less than a quarter of their JSON.

    Write unpackOrders(packed), the reverse: it gunzips the Buffer and returns a promise of the parsed array of orders. It must work on any gzipped JSON, not only on what packOrders made.

  2. Task 2

    brotliPack and brotliUnpack: squeeze the menu text with brotli

    Write brotliPack(text). It compresses the UTF-8 text with brotli (not gzip) and returns a promise of the compressed Buffer, which must be smaller than the text.

    Write brotliUnpack(packed), the reverse: it decompresses a brotli Buffer and returns a promise of the original string, with characters such as ₹ intact.

  3. Task 3

    compressFor: pick the right compression for each customer's app

    Write compressFor(acceptEncoding, body). acceptEncoding is the client's Accept-Encoding header: a string such as 'gzip, deflate, br', or '' or undefined. body is a string or a Buffer. It returns a promise of { encoding, body }, where body is always a Buffer.

    A body under 1024 bytes is never compressed: encoding 'identity' and the body as it is. Otherwise use 'br' (brotli) if the client accepts it, else 'gzip' if it accepts that, else 'identity' with the body unchanged. Names are separated by commas, with any spaces around them, and may carry a weight after a semicolon, as in 'gzip;q=0.8' or 'gzip; q=0'. A weight of 0 means the client refuses that one. deflate alone gives 'identity'.

  4. Task 4

    archiveDay: stream a day of orders into a gzip archive

    Write archiveDay(orders, path). orders is any iterable of order objects (the check passes a generator). Write them to a gzip file at path as NDJSON: one JSON object per line, each line ending in '\n', in order. It returns a promise that resolves when the file is written.

    Stream it: lines go through a gzip stream into a file stream, rather than one big string compressed in one go.

  5. Task 5

    readArchive: read an archived day back, and refuse a corrupt one

    Write readArchive(path). It streams the gzipped NDJSON file at path back into an array of order objects, in order. Empty lines are skipped, and the last line may or may not end in '\n'.

    A file that is not valid gzip must make the promise reject, never resolve with what was read so far or with an empty array.

What you will learn

  • gzip and gunzip: compressing a Buffer or a string, and getting it back.
  • Brotli: smaller results for text on the web, at a CPU cost.
  • Content negotiation: the client says what it can read in Accept-Encoding, and small bodies are not worth compressing.
  • Compression streams: createGzip and createGunzip in a pipeline.

More in Events and streams

Build "Compression with zlib" yourself

Tapri’s order server must never stall at the 4 pm rush. You learn why it would. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of How Node really works are free. This one comes with the all labs plan.