NumPy · Chapter 13 of 18
Missing data and nan
By the end of this chapter you can say what nan is and where it comes from, read a messy file with np.genfromtxt, find missing values with np.isnan (never with == np.nan), work around them with np.nansum, np.nanmean, np.nanmax and np.nanargmax, fill gaps with a fixed value, the previous value or np.interp, and tell nan from inf with np.isinf, np.isfinite and np.errstate. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Analysis
- 4 tasks
- NumPy
- In the all labs plan
The first 2 chapters of NumPy are free. This one comes with the all labs plan.
Why you need this
Until now every array in this lab was complete: every panel, every block, a number. Real logs are not. INV07's logger lost its link from 11:05 to 11:50 on the 14th, so inverter_log.csv has ten blank cells in its column, and INV11 wrote ERR twice in the afternoon. Five blocks of the 30-day file have no price at all, because the exchange's feed dropped them.
Rukmini still wants every inverter's energy for the day, and to know how much of the log is missing. Meenal needs a reading for every 5 minutes in her report, so the gap must be filled with something believable. Neither can be done by pretending the blanks are 0: a 0 says "INV07 made nothing for 50 minutes", which is false, and every total and average built on it is quietly wrong. What you know so far cannot even read the file: np.loadtxt stops at the first blank cell. This chapter gives the gap its own value, nan, and shows how NumPy works with it.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
load_log.py: read the inverter log, gaps and all
/workspace/data/inverter_log.csv has a header line, a time column, then one column of kW for each inverter, a row every 5 minutes. Some cells are blank (INV07 lost its link for 50 minutes) and two say ERR (INV11). Write load_log(path) in load_log.py: read the file at path and return the inverters' readings as a float array, one row a reading and one column an inverter, without the time column, with nan wherever a cell is blank or not a number. load_log("/workspace/data/inverter_log.csv") has shape (156, 12).
np.loadtxt stops at the first blank cell. The Check also reads a small log of 3 rows with 3 inverters, so keep every column after the time instead of counting them by hand. A blank or an ERR must become nan, never 0.
- Task 2
gaps.py: energy, and the minutes that are missing
Write two functions in gaps.py. log is the array from task 1: one row every 5 minutes, one column an inverter, in kW, with nan where a reading is missing.
inverter_kwh(log) returns each inverter's energy in kWh from the readings it has, skipping the missing ones: every reading in kW times 5 minutes (5 / 60 of an hour), added up. For a (156, 12) log it is a float array of shape (12,).
minutes_missing(log) returns, for each inverter, how many minutes of its log are missing, 5 for every nan, as an int array: shape (12,) for a (156, 12) log.
The Check uses the whole log and a log of 4 readings by 3 inverters whose last inverter has no readings at all (its energy is then 0.0, not nan).
- Task 3
fill_gap.py: fill INV07's gap
Write fill_gaps(minutes, kw) in fill_gap.py. minutes is the time of each reading in minutes since midnight (06:00 is 360), increasing; kw is one inverter's readings at those times, with nan where one is missing; both have shape (n,). Return a new float array of shape (n,) with no nan: the readings that were there, unchanged, and each nan replaced by the value on the straight line from the reading before it to the reading after it, at its own time. A nan before the first reading takes the first reading, and a nan after the last takes the last.
Go by the times, not by the positions: the Check tries INV07 and INV11 of the log, readings with no gap, and 5 readings at 600, 610, 615, 640 and 660 minutes with gaps at both ends. Do not change the kw array you are given.
- Task 4 · interview question
Interview: nan_count.py, readings == np.nan
Imran's count_missing(readings) in nan_count.py should return how many values in an array of readings, of any shape, are nan, as one whole number: count_missing(log) for the (156, 12) log is the count of nan in it. It says 0, but INV07 alone has 10 missing readings. Fix it.
The Check tries the whole log, one inverter's column, readings with no nan (the count is 0), and a grid of 2 by 3.
Imran will ask: why did readings == np.nan find nothing, and what finds a nan?
What you will learn
- nan: "not a number", a float that marks a value as missing.
- Where it comes from: a blank in a file, 0 / 0.
- np.genfromtxt: reads a messy text file, with nan for what it cannot read.
- nan spreads and nan is not equal to nan: why np.isnan and never == np.nan.
- np.nansum, np.nanmean, np.nanmax, np.nanargmax: answers that skip the gaps.
- Filling gaps: a fixed value, the previous value, or a straight line with np.interp.
- inf and -inf: division by zero, with np.isinf, np.isfinite and np.errstate.
More in Analysis
Build "Missing data and nan" yourself
DhoopGrid runs a solar plant in the Thar desert with 2,400 panels. You turn its raw readings into answers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of NumPy are free. This one comes with the all labs plan.