NumPy · Chapter 1 of 18
Why NumPy, and your first arrays
By the end of this chapter you can say why DhoopGrid moves its numbers from Python lists to NumPy, measure the difference yourself, say what an ndarray is (one dtype, one block of memory, a shape), make arrays from lists and from nothing with np.array, np.zeros, np.ones, np.full, np.arange and np.linspace, read an array's shape, dtype, ndim and size, and do arithmetic on every element at once. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Arrays from zero
- 4 tasks
- NumPy
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
DhoopGrid runs a 1.3 MW solar plant in the Thar desert near Phalodi: 2,400 panels in 40 rows of 60, and every unit it makes is sold on the power exchange in 96 blocks of 15 minutes a day. Every panel reports its output every 15 minutes. That is 230,400 numbers a day from the panels alone, before the inverter logs, the weather, the prices and the drone's thermal images.
Rukmini, the founder, wants to know every evening which panels earned less than they should, because a panel that makes 40 % less in October loses money every sunny day until someone notices. Bhanwar, the field technician, walks the rows with a drone and a brush and needs a list of the panels to look at by morning. Meenal, who runs finance, turns each block's energy and the exchange's price into rupees and wants those sums right.
Today all of this is spreadsheets and a nightly Python script that loops over every reading, one at a time, in lists. It is slow, it gets slower every time the plant grows, and the code that answers "which panels in block C were below 80 % at noon?" is a page of nested loops that nobody wants to touch. Imran, DhoopGrid's staff data scientist, is moving it all to NumPy, the library that every data tool in Python is built on (pandas, Matplotlib and scikit-learn all hold their numbers in NumPy arrays). You do it with him. You know Python's lists, loops and functions; this lab starts from there.
The lesson
A million numbers, two ways
Each panel has an optimiser that reports its volts and amps, and power is volts times amps. Imran makes a million test readings and times both ways in IPython, the friendlier Python shell your lab has (type ipython in the terminal). Its %timeit runs a line many times and reports how long one run takes:
In [1]: import numpy as np
In [2]: volts = [36.0 + (i % 40) / 10 for i in range(1_000_000)]
In [3]: amps = [11.0 + (i % 25) / 10 for i in range(1_000_000)]
In [4]: v = np.array(volts)
In [5]: a = np.array(amps)
In [6]: %timeit [x * y for x, y in zip(volts, amps)]
37 ms ± 601 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [7]: %timeit v * a
322 μs ± 430 ns per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
The list comprehension is the fastest loop plain Python has, and it took 37 milliseconds. v * a did the same million multiplications in 0.32 milliseconds: about 115 times faster. (A μs is a millionth of a second; your numbers will differ a little from run to run.)
The memory tells the same story:
In [8]: import sys
In [9]: sys.getsizeof(volts) + sum(sys.getsizeof(x) for x in volts)
Out[9]: 32448728
In [10]: v.nbytes
Out[10]: 8000000
The list needs 32 MB for a million numbers; the array needs 8 MB, exactly 8 bytes each.
Why? A Python list does not hold numbers. It holds references to float objects, each one a separate object somewhere in memory with its own type and count of users. To multiply, Python's loop takes each reference, finds the object, checks its type, multiplies, and makes a new float object for the answer, a million times over. A NumPy array holds the numbers themselves, side by side in one block of memory, all of the same type. v * a hands the two blocks to a loop written in C that multiplies them straight through, with no type checks and no objects on the way.
What an ndarray is
NumPy's array type is the ndarray, an n-dimensional array. The day's panel readings are in /workspace/data/power_day.npy, a file NumPy wrote (chapter 2 covers files; np.load reads one):
import numpy as np
day = np.load("/workspace/data/power_day.npy")
print(type(day))
print(day.shape)
print(day.ndim, day.size)
print(day.dtype, day.itemsize, day.nbytes)
<class 'numpy.ndarray'>
(96, 40, 60)
3 230400
float64 8 1843200
Every ndarray is three things:
- A shape: the length of each dimension.
(96, 40, 60)is 96 blocks of the day, each a grid of 40 rows by 60 columns of panels. The dimensions are called axes. - One dtype: the type of every element, here
float64, a decimal number in 8 bytes (itemsize). - One block of memory:
nbytesis 230,400 times 8 bytes, 1,843,200, and nothing else.
ndim is the number of axes (3) and size the number of elements (96 × 40 × 60 = 230,400). These are attributes, values the array carries, so they have no brackets: day.shape, not day.shape().
One dtype for the whole array
Because an array is one block of same-sized elements, every element has the same dtype. NumPy picks one that can hold everything you give it:
print(np.array([450, 452, 0]).dtype)
print(np.array([450, 452.5, 0]))
print(np.array([450, "dead", 0]))
int64
[450. 452.5 0. ]
['450' 'dead' '0']
Whole numbers make an int64 array. One decimal makes every element a float (450. is how NumPy prints 450.0). One string makes them all strings (dtype <U21), and you can no longer do sums with them: a "dead" panel belongs in its own array or as 0, never as text among numbers.
Making arrays
From a list, or a list of lists for two dimensions:
row = np.array([441.2, 438.9, 0.0, 445.1])
print(row)
grid = np.array([[441, 438, 0], [447, 452, 449]])
print(grid)
print(grid.shape, grid.ndim, grid.size, grid.dtype)
[441.2 438.9 0. 445.1]
[[441 438 0]
[447 452 449]]
(2, 3) 2 6 int64
A shape is a tuple: (2, 3) is 2 rows of 3, and a 1-D array of 4 has the shape (4,), a tuple of one. The rows must be the same length, since an array is a rectangle:
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.
From nothing, when you know the shape and fill it later:
print(np.zeros(4))
print(np.zeros((2, 3)))
print(np.ones((2, 3), dtype=int))
print(np.full((2, 4), 540))
[0. 0. 0. 0.]
[[0. 0. 0.]
[0. 0. 0.]]
[[1 1 1]
[1 1 1]]
[[540 540 540 540]
[540 540 540 540]]
np.zeros and np.ones make floats unless you pass dtype=; np.full takes its dtype from the value. The shape goes in as one value, a tuple. Pass two numbers and the second is taken as the dtype:
TypeError: Cannot interpret '3' as a data type
Ranges come two ways:
print(np.arange(5))
print(np.arange(10, 20, 4))
print(np.linspace(0, 1, 5))
print(np.linspace(6, 7, 3))
[0 1 2 3 4]
[10 14 18]
[0. 0.25 0.5 0.75 1. ]
[6. 6.5 7. ]
np.arange(start, stop, step)works like Python'srange: it stops beforestop.np.linspace(start, stop, num)givesnumevenly spaced values and includesstop.
Use arange for whole numbers. With a decimal step it can surprise you, because 0.1 cannot be stored exactly and the last step lands just under the stop:
print(np.arange(1, 1.3, 0.1))
[1. 1.1 1.2 1.3]
1.3 was supposed to be left out. When you know how many values you want between two ends, use linspace.
Arithmetic on every element
Arithmetic between two arrays of the same shape works element by element: the first with the first, the second with the second. A single number is used with every element:
morning = np.array([212.4, 208.9, 0.0, 215.0])
noon = np.array([451.0, 447.3, 0.0, 455.8])
print(noon - morning)
print(noon / 1000)
print(noon * 2)
[238.6 238.4 0. 240.8]
[0.451 0.4473 0. 0.4558]
[902. 894.6 0. 911.6]
Every operation makes a new array and leaves the others as they were. / always gives floats, even from two int arrays. It works the same for any number of axes: two (40, 60) grids subtract cell by cell. Arrays of different shapes cannot be paired off one to one:
ValueError: operands could not be broadcast together with shapes (3,) (2,)
("Broadcast" is NumPy's rule for when shapes may differ, chapter 9.)
Lists are not arrays
The same operators mean something else on lists:
readings = [451.0, 447.3]
print(readings * 2)
print(readings + [455.8])
print(readings / 1000)
[451.0, 447.3, 451.0, 447.3]
[451.0, 447.3, 455.8]
Traceback (most recent call last):
File "/workspace/chapters/01-why-numpy/lists.py", line 4, in <module>
print(readings / 1000)
~~~~~~~~~^~~~~~
TypeError: unsupported operand type(s) for /: 'list' and 'int'
On a list, * 2 repeats it and + joins two lists; there is no / at all. A list is a container of anything; an array is numbers, and its operators do maths. Data read from a file or a spreadsheet often arrives as lists, so turn it into an array with np.array first.
Your turn
Four tasks, one file each. Task 1 makes an array from Bhanwar's noon readings and prints it with its shape, dtype, ndim and size; the Check runs it with other readings too, so print what the array says. Task 2 makes arrays from nothing: mind which range includes its end. Task 3 works out a block's energy in one line of arithmetic, with no loop. Task 4 is an interview question: Meenal's revenue function multiplies two lists, and Imran will ask why * works on arrays and not on lists. Run each file with python3 file.py, or try lines in ipython first.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
noon_row.py: one row at noon
Bhanwar read eight panels of row 12 at noon, in watts, and the starter keeps them in a list in the line watts = [470.4, 453.9, 463.1, 452.6, 455.3, 463.4, 454.0, 467.9]. Keep that line as it is. Make a NumPy array from the list, then print the array, and under it four lines: shape: and the array's shape, dtype: and its dtype, ndim: and its number of dimensions, size: and its number of elements. For this list the five lines are exactly:
[470.4 453.9 463.1 452.6 455.3 463.4 454. 467.9]
shape: (8,)
dtype: float64
ndim: 1
size: 8
The Check also runs the file with the watts line changed to whole numbers, watts = [440, 451, 0, 447], and to two rows, watts = [[440, 451, 0], [447, 452, 449]]. Then the array prints differently (two rows print as two lines) and its shape, dtype, ndim and size change, so print what the array says, never typed-in text.
- Task 2
layout.py: arrays from nothing
Write three functions in layout.py, each returning a new NumPy array:
blank_block(rows, cols) returns an array of zeros, as floats, of shape (rows, cols). The Check tries (10, 60) and (4, 3).
panel_numbers(first, last) returns the whole numbers from first to last with both ends included, as an int array: panel_numbers(1201, 1205) is [1201 1202 1203 1204 1205]. The Check tries 1201 to 1260, 7 to 9, and 5 to 5 (one number).
check_times(start, stop, n) returns n floats evenly spaced from start to stop with both ends included: check_times(6.0, 18.0, 5) prints as [ 6. 9. 12. 15. 18.]. The Check tries 49 times from 6.0 to 18.0 (every 15 minutes) and 3 from 6.5 to 7.5.
- Task 3
energy.py: a block's energy, all at once
Write energy_kwh(volts, amps, minutes) in energy.py. volts and amps are NumPy arrays of the same shape, one number per panel; minutes is one number. Return each panel's energy in kWh over that many minutes, as a float array of the same shape: the power in watts is volts times amps, and the energy in kWh is the watts times the minutes, divided by 60 and by 1000.
The Check calls it for a block of shape (10, 60) with 15 and with 5 minutes, and for one row of 3 panels with 60 minutes. Use no Python loop: a for, a while or a comprehension in energy_kwh fails the task.
- Task 4 · interview question
Interview: revenue.py, lists are not arrays
Meenal reads the exchange's file into two Python lists, each block's energy in kWh and each block's price in rupees a MWh, and her block_revenue(kwh, price_rs_mwh) stops with TypeError: can't multiply sequence by non-int of type 'list'. Fix it so it returns each block's revenue in rupees as a NumPy array of floats: the kWh times the price, divided by 1000 (a MWh is 1000 kWh). block_revenue([250.0, 275.5], [3100.0, 2950.0]) is an array of 2 floats.
The Check calls it with two lists of 4 blocks, with the same numbers as two NumPy arrays, and with two lists of whole numbers; every time it must return a float array of the revenues.
Imran will ask: why does * work on two arrays but not on two lists, and what does a list times 2 give?
What you will learn
- Why NumPy: the same work on a million numbers, in a loop and in one array operation, measured.
- The ndarray: one dtype, one block of memory, a shape.
- shape, dtype, ndim and size: what an array tells you about itself.
- Making arrays: np.array from lists, np.zeros, np.ones and np.full from nothing, np.arange and np.linspace for ranges.
- Elementwise arithmetic: +, -, * and / on every element at once, with no loop.
More in Arrays from zero
Build "Why NumPy, and your first arrays" yourself
DhoopGrid runs a solar plant in the Thar desert with 2,400 panels. You turn its raw readings into answers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.