NumPy · Chapter 2 of 18
Dtypes and files
By the end of this chapter you can choose a dtype for an array and say what it costs and what it can hold (np.iinfo, np.finfo), spot a silent overflow and a float that is only close, predict the dtype NumPy gives when two dtypes meet, convert on purpose with astype, make arrays shaped like others (np.empty, np.zeros_like, np.full_like, np.eye), and save and load arrays with np.save, np.load, np.savez_compressed, np.savetxt and np.loadtxt. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Arrays from zero
- 4 tasks
- NumPy
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
Rukmini wants every day of the plant kept, so that next October can be compared with this one. One day of panel readings is 1,843,200 bytes of float64; a year of them is 672,768,000 bytes, about 673 MB, and the readings are only good to 0.1 W anyway. Bhanwar brightened the drone's photo to see the dust better and the brightest panels went dark, with no error. Meenal's 30 days of blocks and prices arrive as a text file, blocks_30d.csv, and so far the only file you can read is a .npy.
All three are about the same two things. Every array has a dtype, and the dtype decides how many bytes each number takes, which numbers fit, and what happens when they do not. And arrays have to get in and out of files without losing their dtype and shape. With what you know from chapter 1 you would open the CSV, split every line, call float() on every piece in a loop, and still not know how to keep the result for tomorrow.
The lesson
A dtype is a size and a range
An array is one block of memory with every element the same size, so the dtype tells NumPy how to read the bytes. The name says it: int for whole numbers that may be negative, uint (unsigned) for whole numbers from 0 up, float for decimals, and the number is the bits each one takes (8 bits are one byte).
| dtype | bytes | holds |
|---|---|---|
| int8, uint8 | 1 | -128 to 127; 0 to 255 |
| int16, uint16 | 2 | -32,768 to 32,767; 0 to 65,535 |
| int32, uint32 | 4 | about ±2.1 billion; 0 to about 4.3 billion |
| int64, uint64 | 8 | about ±9.2 × 10¹⁸; 0 to about 1.8 × 10¹⁹ |
| float16, float32, float64 | 2, 4, 8 | about 3, 7 and 16 significant digits |
| bool | 1 | True or False |
You do not have to remember the table: NumPy tells you. np.iinfo gives a whole-number dtype's range and np.finfo a float dtype's:
In [1]: import numpy as np
In [2]: np.iinfo(np.int8)
Out[2]: iinfo(min=-128, max=127, dtype=int8)
In [3]: np.iinfo(np.uint8)
Out[3]: iinfo(min=0, max=255, dtype=uint8)
In [4]: np.iinfo(np.int64).max
Out[4]: 9223372036854775807
In [5]: np.finfo(np.float32)
Out[5]: finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)
In [6]: np.finfo(np.float64)
Out[6]: finfo(resolution=1e-15, min=-1.7976931348623157e+308, max=1.7976931348623157e+308, dtype=float64)
In [7]: np.array([440, 451]).dtype
Out[7]: dtype('int64')
A float's resolution is how finely it can tell two numbers apart: float32 keeps about 7 significant digits, float64 about 16. A reading of 453.9 W needs four, so float32 holds the plant's readings easily, in half the bytes. The last line shows the default int: whole numbers with no dtype given become int64, on the lab's Linux and on any 64-bit NumPy 2.
Overflow is silent in arrays
Suppose the thermal camera stored its temperatures as int8, one byte each. 127 is the largest int8, so watch what adding 10 does:
In [1]: import numpy as np
In [2]: temps = np.array([120, 85, 41], dtype=np.int8)
In [3]: temps + 10
Out[3]: array([-126, 95, 51], dtype=int8)
In [4]: temps + 200
---------------------------------------------------------------------------
OverflowError Traceback (most recent call last)
Cell In[4], line 1
----> 1 temps + 200
OverflowError: Python integer 200 out of bounds for int8
120 + 10 is 130, which does not fit in one byte, so the count runs past 127 and wraps round to -128 and on up: -126. NumPy gives no error and no warning, because checking every element of a big array would slow every sum down. The result keeps the array's dtype, and the wrong number looks like any other.
The second line is different: 200 itself does not fit in an int8, and NumPy refuses to turn it into one. So a number that is too big for the dtype raises OverflowError, but a sum that grows too big wraps quietly.
The cure is a dtype big enough for the answer, before the sum:
In [5]: temps.astype(np.int16) + 10
Out[5]: array([130, 95, 51], dtype=int16)
Floats are close, not exact
A float is stored in binary, and most decimals, like 0.1, have no exact binary form, the way 1/3 has no exact decimal one:
In [1]: import numpy as np
In [2]: x = 0.1 + 0.2
In [3]: x
Out[3]: 0.30000000000000004
In [4]: x == 0.3
Out[4]: False
In [5]: np.isclose(x, 0.3)
Out[5]: np.True_
So never compare floats with ==. np.isclose(a, b) asks whether they are within a small tolerance, element by element; np.allclose(a, b) asks whether every element is. Both take atol=, the largest difference you accept, in the array's own units:
In [6]: w = np.array([453.9, 467.9])
In [7]: w32 = w.astype(np.float32)
In [8]: w32
Out[8]: array([453.9, 467.9], dtype=float32)
In [9]: w32.astype(np.float64)
Out[9]: array([453.8999939, 467.8999939])
In [10]: w32 == w
Out[10]: array([False, False])
In [11]: np.allclose(w32, w, atol=0.05)
Out[11]: True
float32 prints 453.9, but held as float64 again it shows what it really stores, 453.8999939: off by six millionths of a watt. Not equal, and well within 0.05 W.
When two dtypes meet
Arithmetic between arrays of two dtypes gives one dtype that can hold both. This is called promotion:
In [1]: import numpy as np
In [2]: ints = np.array([440, 451])
In [3]: floats = np.array([0.5, 0.25])
In [4]: (ints + floats).dtype
Out[4]: dtype('float64')
In [5]: small = np.array([3, 4], dtype=np.int8)
In [6]: medium = np.array([300, 400], dtype=np.int16)
In [7]: (small + medium).dtype
Out[7]: dtype('int16')
In [8]: w32 = np.array([453.9, 467.9], dtype=np.float32)
In [9]: (w32 + 0.2).dtype
Out[9]: dtype('float32')
In [10]: (w32 + np.array([0.2, 0.2])).dtype
Out[10]: dtype('float64')
- Whole numbers with decimals give floats.
- A small int with a bigger int gives the bigger int.
- A plain Python number takes the array's dtype (NumPy 2's rule): float32 plus 0.2 stays float32, and int8 plus 10 stays int8, which is why the temperatures wrapped. A float64 array promotes it.
Converting on purpose
astype makes a new array with the dtype you ask for and leaves the original as it was. The drone's thermal image is float64; as float32 it takes half the bytes and prints the same:
thermal = np.load("/workspace/data/thermal_c.npy")
small = thermal.astype(np.float32)
print(thermal.dtype, thermal.nbytes)
print(small.dtype, small.nbytes)
print(thermal[41, 40:44])
print(small[41, 40:44])
float64 153600
float32 76800
[41. 53.3 54.4 59. ]
[41. 53.3 54.4 59. ]
(thermal[41, 40:44] is four pixels of row 41; chapter 3 teaches it.) Going from floats to whole numbers cuts off the decimals, toward zero; it does not round. And any number becomes a bool by "is it not zero":
kw = np.array([3.9, -3.9, 0.6, 451.97])
print(kw.astype(int))
flags = np.array([0, 2, 1, 0])
print(flags.astype(bool))
print(np.zeros(3, dtype=np.uint8))
print(np.array([1, 2, 3], dtype=np.float32))
[ 3 -3 0 451]
[False True True False]
[0 0 0]
[1. 2. 3.]
When you know the dtype from the start, pass dtype= when you make the array, as the last two lines do, instead of making it and converting.
Arrays shaped like others
Often you need a new array of the same shape as one you have: a fault map for the grid, a result to fill in. The _like functions copy the shape and dtype of an array, not its values:
row = np.array([441.2, 438.9, 0.0, 445.1])
print(np.zeros_like(row))
print(np.ones_like(row, dtype=int))
print(np.full_like(row, 540))
print(np.empty(3))
print(np.eye(3))
print(np.identity(2, dtype=int))
[0. 0. 0. 0.]
[1 1 1 1]
[540. 540. 540. 540.]
[1.39066133e-309 9.27121489e-310 9.80845537e-307]
[[1. 0. 0.]
[0. 1. 0.]
[0. 0. 1.]]
[[1 0]
[0 1]]
np.zeros_like,np.ones_likeandnp.full_liketake the shape and dtype from the array (full_likegave floats, from the row), unless you passdtype=.np.emptyonly reserves the memory and does not clear it, so it holds whatever bytes were there: fast, and safe only when you fill every element yourself before reading it.np.eye(n)andnp.identity(n)are a square grid with ones on the diagonal, which linear algebra (chapter 16) needs.
One array in a file: .npy
np.save writes one array to a .npy file and np.load reads it back, with its shape and dtype kept. The file is the array's bytes plus a short header:
import os
thermal = np.load("/workspace/data/thermal_c.npy")
np.save("thermal_copy.npy", thermal)
back = np.load("thermal_copy.npy")
print(back.shape, back.dtype)
print(os.path.getsize("thermal_copy.npy"), thermal.nbytes)
(120, 160) float64
153728 153600
153,600 bytes of numbers and 128 of header. (os.path.getsize from Python's standard library gives a file's size in bytes.)
Several arrays in one file: .npz
np.savez(path, name=array, ...) keeps several arrays in one .npz file, each under the name you give it; np.savez_compressed does the same and compresses them, like a zip file. np.load of an .npz gives an NpzFile, and you read each array by its name, with square brackets as from a dict:
day = np.load("/workspace/data/power_day.npy")
thermal = np.load("/workspace/data/thermal_c.npy")
np.savez("survey.npz", day=day, thermal=thermal)
np.savez_compressed("survey_small.npz", day=day, thermal=thermal)
print(os.path.getsize("survey.npz"))
print(os.path.getsize("survey_small.npz"))
survey = np.load("survey_small.npz")
print(survey)
print(survey.files)
print(survey["thermal"].shape, survey["thermal"].dtype)
1997306
222839
NpzFile 'survey_small.npz' with keys: day, thermal
['day', 'thermal']
(120, 160) float64
Compressed, the bundle is a ninth of the size: the day has hours of zeros at night, and repeated bytes compress well. .files lists the names. Give every array a name: passed without one, they are saved as arr_0, arr_1, and nobody remembers which is which.
Text files: loadtxt and savetxt
np.loadtxt reads a text file of numbers into an array. Meenal's CSV has commas between columns and a header line of names, so tell it both:
path = "/workspace/data/blocks_30d.csv"
noon = np.loadtxt(path, delimiter=",", skiprows=49, max_rows=3,
usecols=(1, 2, 4))
print(noon)
print(noon.shape, noon.dtype)
[[ 49. 968. 1069.4]
[ 50. 978.5 1076.6]
[ 51. 998.5 1094.2]]
(3, 3) float64
- delimiter=",": what separates the columns.
- skiprows=: how many lines to skip at the top. The header is one line; here 48 more rows of the first day are skipped too, so reading starts at block 49.
- max_rows=: how many rows to read after that.
- usecols=: which columns, counting from 0; one number gives a 1-D array, a tuple gives one column for each.
Everything becomes float64 unless you pass dtype=. Leave out what is not a number, or loadtxt stops at the first piece of text:
ValueError: could not convert string 'date' to float64 at row 0, column 1.
(That is the last line; the file has a date column and a header.) The price column, price_rs_mwh, has five blanks, which loadtxt cannot read either: chapter 13 handles missing numbers.
np.savetxt(path, array, fmt=..., delimiter=",") writes a 1-D or 2-D array as text; fmt="%.1f" writes one decimal, and header= adds a line of names (with comments="", or it starts with #):
noon_kw = np.array([[451.2, 0.0], [447.35, 455.8]])
np.savetxt("noon_kw.csv", noon_kw, fmt="%.1f", delimiter=",",
header="west,east", comments="")
print(open("noon_kw.csv").read())
west,east
451.2,0.0
447.4,455.8
Text or binary
Text is for people and spreadsheets; .npy is for NumPy. The whole day as text (made 2-D first with reshape, which chapter 7 teaches, because savetxt writes rows and columns):
day = np.load("/workspace/data/power_day.npy")
np.savetxt("day.csv", day.reshape(96, 2400), fmt="%.1f", delimiter=",")
print(os.path.getsize("day.csv"))
print(os.path.getsize("/workspace/data/power_day.npy"))
back = np.loadtxt("day.csv", delimiter=",")
print(back.shape, back.dtype)
1120484
1843328
(96, 2400) float64
The text is smaller here, because the night's zeros are three characters each, but it lost the shape: it comes back as (96, 2400), not (96, 40, 60), and with one decimal only. Reading it is much slower, because every number has to be parsed from characters:
In [1]: import numpy as np
In [2]: %timeit np.load("/workspace/data/power_day.npy")
125 μs ± 3.03 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
In [3]: %timeit np.loadtxt("day.csv", delimiter=",")
8.3 ms ± 1.03 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
0.125 ms against 8.3 ms, about 66 times. Keep your own arrays in .npy or .npz; read and write text when a person or another program needs it.
Your turn
Four tasks, one file each. Task 1 halves the day's bytes with float32 and counts the bytes saved; mind that float32 is not smaller than every dtype. Task 2 saves the day's power and the peak ratings in one compressed bundle and reads one back by name. Task 3 reads one column of one day out of a CSV with loadtxt: work out how many lines to skip. Task 4 is an interview question: Bhanwar's photo wrapped from 250 to 4, and Imran will ask why there was no error and what a bigger dtype changes. Try each function in ipython first (from shrink import shrink), then press Check.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
shrink.py: half the bytes for the day
Write two functions in shrink.py:
shrink(day) returns the same readings as an array of dtype float32, of the same shape: for the day, shape (96, 40, 60). No reading may move by more than 0.05 W. The Check tries the day and a float64 array of shape (2, 3).
saved_bytes(a) returns, as a whole number, how many bytes the array a takes minus how many it would take as float32. For the day it is 921600. The Check also tries a float64 array of shape (10, 60) and an int16 array of 5 numbers; for the int16 array the answer is negative, because float32 takes more bytes than int16.
- Task 2
bundle.py: a bundle of arrays in one file
Write two functions in bundle.py:
save_bundle(path, power, peak_w) saves one compressed .npz file at exactly path (the Check gives a path ending in .npz), holding power under the name power and peak_w under the name peak_w, each with its own shape and dtype, and returns nothing. The Check reads the file back, compares both arrays and their dtypes, and checks the file is smaller than the same arrays saved without compression. It tries the day (96, 40, 60) and the noon grid (40, 60) as power, with a (40, 60) int array of peak ratings.
read_peak(path) returns the array stored under the name peak_w in the .npz file at path. The Check's files may hold other arrays too, and may be compressed or not.
- Task 3
one_day.py: one column of one day, from a CSV file
Write one_day(path, col, d) in one_day.py. path is a CSV file like blocks_30d.csv: values separated by commas, one header line, then 96 rows for each day. col is a column's number, counting from 0 (in blocks_30d.csv, 2 is irradiance_wm2, 3 ambient_c, 4 plant_kw, 5 schedule_kw). d is a day's number, counting from 0 (0 is the first day, 29 the last of blocks_30d.csv).
Return the 96 values of column col for day d, as floats, shape (96,). one_day(path, 4, 29) is the plant's kW in each block of 14 October.
The Check tries columns 2 to 5 of blocks_30d.csv on several days (0, 7, 16 and 29), and a small file of its own with a header and two days of three columns.
- Task 4 · interview question
Interview: brighten.py, 250 plus 10 is 4
The drone's photo is uint8: whole numbers from 0 to 255 for each pixel's colour. Bhanwar's brighten(pixels, amount) returns pixels + amount, and with pixels = np.array([250, 251, 120], dtype=np.uint8) and amount 10 it gives [4 5 130], with no error. Fix it so it returns each pixel plus amount as whole numbers with no wrapping (250 plus 10 must be 260), in an array of the same shape as pixels. amount is a whole number from 0 to 255; the Check tries 10, 60 and 255, on the photo (180, 320, 3) and on small arrays. Floats fail: the task asks for whole numbers.
Imran will ask: why did 250 plus 10 give 4 with no error, and what does a bigger dtype change?
What you will learn
- The dtypes: int8 to int64, uint8 to uint64, float16 to float64 and bool, with their sizes and ranges (np.iinfo, np.finfo).
- Overflow and floats: whole numbers wrap with no error; decimals are close, not exact, so compare them with np.isclose.
- Promotion: the dtype NumPy picks when two dtypes meet.
- astype and dtype=: convert on purpose, or make the right dtype from the start; np.empty, the _like functions and np.eye.
- Files: np.save and np.load for one array, np.savez_compressed for several, np.savetxt and np.loadtxt for text.
More in Arrays from zero
Build "Dtypes and files" yourself
DhoopGrid runs a solar plant in the Thar desert with 2,400 panels. You turn its raw readings into answers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.