Python, properly · Chapter 30 of 42
Iterators and generators
By the end of this chapter you can say what a for loop really does, write generators with yield, and chain them into a pipeline that reads a file of any size in almost no memory. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Pythonic and typed
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Python, properly are free. This one comes with the all labs plan.
Why you need this
Bhavna has exported every order DabbaDrop's partner kitchens took this year, and the file is huge. Rafiq wrote the obvious script: read all the lines into a list, then count. On the small lab machine it crawled, and on next year's file it will not fit at all.
Everything you have written so far builds the whole list first and only then works on it. To handle a file bigger than memory, the program has to take one row, deal with it, and forget it before taking the next. That is what iterators and generators are for, and Python uses them everywhere: in every for loop, in open(), in range().
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
read_orders.py: read the orders one row at a time
Bhavna's order exports are too big for a list. Write read_orders(path), a generator: it opens the CSV file at path and yields one dict per order, with the header line as the keys and the total converted to an int (756, not "756").
The check reads a small file, then sums the totals of a 100,000-row file while measuring memory. Reading every row into a list before yielding still holds them all at once, and fails.
- Task 2
pipeline.py: a lazy pipeline
Write three functions that work on any iterable of order dicts (each has a status and an int total):
delivered(rows), a generator that yields only the rows whose status is delivered;
totals(rows), a generator that yields each row's total;
revenue(rows), which returns the sum of the totals of the delivered rows, using the two generators.
The check takes the first item from delivered and from totals on a source that fails if anything reads past the first row, and measures revenue's memory on 100,000 rows.
- Task 3
stats.py: count and add up in one pass
Rafiq wants two numbers from an order stream. Write order_stats(rows): it returns a tuple (count, total), the number of orders and the sum of their totals.
The check calls it with a list, with a generator of the same orders (3 orders totalling 1725 gives (3, 1725)), with no orders at all ((0, 0)), and on 100,000 generated orders while measuring memory.
Careful: a generator can be looped over only once, and has no len().
- Task 4 · interview question
Interview: batches.py, batches from any iterable
The kitchen system accepts orders in batches. Write batches(items, size), a generator that yields lists of size items, in order, and a last shorter list for whatever is left over (never an empty list).
batches(range(7), 3) gives [0, 1, 2], [3, 4, 5], [6]; batches of nothing gives nothing.
items can be any iterable, including a generator with no len() that cannot be sliced. The check takes the first two batches of a source that fails if anything reads past the items those two batches need.
What you will learn
- Iterators: iter(), next() and StopIteration, the machinery inside every for loop.
- Exhaustion: an iterator gives each item once, then it is empty.
- Generators: functions with yield that pause and resume.
- Generator expressions and pipelines: lazy steps chained together, measured on a million-row file.
More in Pythonic and typed
Build "Iterators and generators" yourself
DabbaDrop, a Mumbai tiffin startup, runs on Excel and WhatsApp. You are its first Python developer. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Python, properly are free. This one comes with the all labs plan.