NumPy · Chapter 4 of 18
Views and memory
By the end of this chapter you can say how an array lies in memory (one block, its strides, C order), tell a view from a copy and check which one you have (np.shares_memory, .base), make a copy when you need one, say which operations copy and which do not, transpose and flatten without surprises, and read one piece of a file too big for memory with np.load(..., mmap_mode="r"). Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Arrays from zero
- 4 tasks
- NumPy
- In the all labs plan
The first 2 chapters of NumPy are free. This one comes with the all labs plan.
Why you need this
Bhanwar keeps a map of dead panels. He takes the plant's grid at noon from the day, sets the dead panel to zero on his map, and hands the day to Imran for the evening report. The report's noon total has gone down, though the panels made what they made. He changed only his map. Or did he?
And Rukmini wants an archive of every day, one file that grows to hundreds of megabytes over a year. To look at one block of one day, np.load as you know it reads the whole file into memory first.
Both come down to one question chapter 3 left open: when you slice an array, what do you get back? With lists, a slice is always a new list. NumPy does something faster and, until you know it, more surprising.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
views.py: a view and a copy of one plant block
Write two functions in views.py. day has shape (96, 40, 60), and the plant's blocks are ten rows each: block k is rows 10*k to 10*k + 9 (k is 0 for A, 1 for B, 2 for C and 3 for D).
block_view(day, k) returns every time block of plant block k, floats of shape (96, 10, 60), as a view: it shares the day's memory, so a change to it shows in day.
block_copy(day, k) returns the same readings as a copy with its own memory, so a change to it leaves day as it is.
The Check tries k = 0, 2 and 3, on the day and on another made-up day, and asks NumPy whether each result shares memory with the day it was given.
- Task 2
strides.py: how far to the next panel
Write two functions in strides.py, each taking a 2-D array a and returning a whole number of bytes.
bytes_down(a) returns how many bytes apart in memory two neighbours down a column are (a[0, 0] and a[1, 0]). For the noon grid, (40, 60) float64, it is 480.
bytes_right(a) returns how many bytes apart two neighbours along a row are (a[0, 0] and a[0, 1]). For the noon grid, 8.
The Check tries the noon grid, a float32 grid of shape (40, 60), the noon grid transposed, every other row of it, every third column of it, and an int16 grid of shape (3, 5). For the transposed and stepped arrays, working it out from the shape and the itemsize gives the wrong answer.
- Task 3
read_block.py: one block from a file, without reading it all
Write read_block(path, b) in read_block.py. path is a .npy file of a day-like array (blocks first, then rows and columns) and b is a block number. Open the file with np.load and mmap_mode, so only block b is read from the disk, and return block b only (shape (40, 60) for a day) with the file's dtype, as an array in memory: a copy, not tied to the file.
The Check looks for mmap_mode in your code, tries blocks 0, 48 and 95 of a day's file and block 3 of a small int16 file of shape (5, 3, 4), and then changes the file after your function has returned: your array must not change with it.
- Task 4 · interview question
Interview: marked.py, the map that changed the day
Bhanwar's marked(day, b, r, c) returns block b's grid of the plant, shape (40, 60), with panel (r, c) set to 0.0, for his map of dead panels. The map is right, but after he called it with b = 48, r = 21, c = 12, day[48, 21, 12] was 0.0 as well, and every total made from day afterwards was different. Fix it so it returns the marked grid, floats of shape (40, 60), and leaves day exactly as it was given. The Check tries block 48 with panels (21, 12) and (7, 41), block 40 with (0, 0) and block 60 with (39, 59), on the day and on another made-up day, and checks that the day it gave is unchanged.
Imran will ask: Bhanwar set one panel of his grid to zero and the day changed. Why? And what does day[b] give back?
What you will learn
- Memory and strides: one block of bytes, and how many bytes to step along each axis.
- C order and F order: rows next to each other, or columns; the flags that say which.
- Views: a slice shares the array's memory, with its own shape and strides; writing into it writes into the original.
- Checking and copying: np.shares_memory, np.may_share_memory, .base, and .copy().
- What copies and what does not: slices and .T do not; arithmetic, astype and copy do.
- Transpose, ravel and flatten, and np.ascontiguousarray.
- mmap_mode: a view onto a file on disk.
More in Arrays from zero
Build "Views and memory" yourself
DhoopGrid runs a solar plant in the Thar desert with 2,400 panels. You turn its raw readings into answers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of NumPy are free. This one comes with the all labs plan.