How Node really works · Chapter 14 of 34

Transform and pipeline

By the end of this chapter you can build a stream that changes data as it passes, and chain streams so that a failure anywhere stops the whole chain and reaches your code. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Events and streams
  • 5 tasks
  • Node.js
  • In the all labs plan

The first 2 chapters of How Node really works are free. This one comes with the all labs plan.

Why you need this

Every night the order server writes the day's orders to orders.csv, one line per order, and Meera wants a report in the morning: how many orders, how many cups, how much money. On a monsoon Sunday the file is small. In Diwali week it was 300 MB, and the old script, which read the whole file and split it into lines, ran out of memory.

Chapter 13 gave you readable and writable streams, but the report needs several steps in a row: bytes to lines, lines to orders, orders to a total. Wired by hand, each pair of streams needs its own backpressure and its own error handling, and one missed error leaves a script that hangs.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    splitLines: split raw file chunks into lines

    Write splitLines(). It returns a Transform that takes UTF-8 text in chunks (Buffers or strings) and pushes one string per line, so its readable side is in object mode (and its writable side is not). Each line comes without its '\n', and without a '\r' just before it. An empty line is pushed as ''. A last line with no '\n' is pushed when the input ends; a '\n' at the very end adds no empty line.

    A chunk can end in the middle of a line, or in the middle of a character such as ₹, and every line must still come out whole and correct.

  2. Task 2

    parseOrder: turn each CSV line into an order object

    Write parseOrder(). It returns a Transform in object mode on both sides: a line such as '3,masala chai,2,40' in, the order { id: 3, item: 'masala chai', qty: 2, price: 40 } out, with id, qty and price as numbers.

    The header line 'id,item,qty,price' and empty lines are skipped. A line that does not have exactly four fields, or whose id, qty or price is not a number (an empty field is not one), fails the stream with an error whose message is 'Bad line: ' followed by the line.

  3. Task 3

    totals: add up the day's orders as they stream past

    Write totals(). It returns a Transform in object mode on both sides that takes orders and pushes nothing for each one. When the input ends, it pushes exactly one summary, { orders, cups, revenue }: how many orders there were, the sum of their qty, and the sum of qty times price.

    With no orders at all, it still pushes { orders: 0, cups: 0, revenue: 0 }.

  4. Task 4

    dailyReport: stream a big order file into the nightly report

    Write dailyReport(inputPath, outputPath). It streams the CSV file at inputPath through splitLines, parseOrder and totals, and writes the summary to outputPath as one line of JSON followed by '\n'. It returns a promise that resolves with the summary object, once the file has been written.

    Stream the file: the check uses 20,000 orders that arrive in many chunks, and the whole report must finish within two seconds.

  5. Task 5 · interview question

    Interview: make one bad row fail the whole pipeline

    dailyReport must reject, and quickly, when anything in the chain fails. A bad row in the middle of the file rejects it with the 'Bad line: ...' error from parseOrder, and an input file that does not exist rejects it with an error whose code is ENOENT. The check gives each case two seconds before it says the report never settled.

What you will learn

  • Transform streams: writable on one side, readable on the other, changing each chunk on the way.
  • flush: the last word, once the input has ended.
  • Object mode on one side: bytes in, values out.
  • pipeline: a chain of streams that cleans up and rejects when any part fails.

More in Events and streams

Build "Transform and pipeline" yourself

Tapri’s order server must never stall at the 4 pm rush. You learn why it would. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of How Node really works are free. This one comes with the all labs plan.