NumPy · Chapter 18 of 18

Capstone: the nightly report and the dust

By the end of this chapter you have rebuilt DhoopGrid's nightly report end to end with NumPy alone, from the pieces of the whole lab: the day's 230,400 readings and the inverter log loaded and their gaps filled, totals by block with a reshape and by inverter with a bincount, every panel's status against its row's median, the day's revenue and penalty against the schedule, all printed as eleven lines, and fast because you measured first. Then you solve Imran's puzzle: dust spreading across the plant by the rules of the Game of Life, with no loop over the panels. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Real work
  • 5 tasks
  • NumPy
  • In the all labs plan

The first 2 chapters of NumPy are free. This one comes with the all labs plan.

Why you need this

Every evening at seven Rukmini reads one page: how much the plant made, where it peaked, how each block and inverter did, which panels Bhanwar must look at tomorrow, and what the day earned. Today that page comes from the old script you met in chapter 1, which loops over every reading. It takes long enough that nobody runs it twice, and when the inverter log has a gap it quietly reports less energy than the plant made.

Meenal needs the money on the same page. The exchange pays for every kWh at its block's price, and DhoopGrid pays a penalty for every block in which it made under 90 % of the schedule it gave the grid: the missing energy at that block's price. A cloud from 14:00 to 15:15 on the 14th means that this page has penalties on it.

Imran has the last word: "There is nothing new to learn for the report. It is chapters 1 to 17, all at once." And then a puzzle for the end, because dust does not stay where Bhanwar found it.

What you build: 5 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    log.py: the inverter log, every gap filled

    Write clean_log(path) in log.py. path is a CSV file like /workspace/data/inverter_log.csv: a header line, then one line every 5 minutes with the time and 12 inverters' kW (INV01 to INV12); some cells are blank and some say ERR. Return a float array of shape (lines, 12), the kW with no nan left: each gap filled along a straight line between the readings before and after it in the same inverter's column, using the line numbers 0, 1, 2 ... as the times. A gap at the start or the end of a column takes that column's nearest reading. For the real log the shape is (156, 12).

    The Check tries the real log, and a log of 8 lines with gaps at the start, in the middle and at the end of columns, and two ERR.

  2. Task 2

    totals.py: totals by block and by inverter

    Write two functions in totals.py. day has the shape (blocks, 40, 60) in watts; a panel's energy over those blocks in kWh is the sum of its readings times 0.25 / 1000.

    block_kwh(day): the plant's blocks A, B, C and D are rows 0 to 9, 10 to 19, 20 to 29 and 30 to 39. Return each block's energy in kWh, a float array of shape (4,), A first.

    inverter_kwh(day, inverter_of): inverter_of is an int array of shape (40, 60), each panel's inverter from 0 (INV01) to 11 (INV12). Return each inverter's energy in kWh, a float array of shape (12,), INV01 first, always 12 values: an inverter with no panels gets 0.

    The Check tries the whole day and 20 of its blocks for block_kwh, the plant's real inverters, and 8 blocks with made-up labels in which INV12 has no panels.

  3. Task 3 · interview question

    Interview: faults.py, measure first, then make it fast

    faults.py holds the old nightly script's panel_status(day): right, but the slowest part of the report. Make it fast: the same result with no Python loop. day has the shape (blocks, 40, 60) in watts; a panel's day energy is the sum of its readings times 0.25 / 1000 (kWh). Each panel gets a status by its energy against the median day energy of its own row: 0 dead (energy exactly 0), 1 weak (under 75 % of its row's median, and not dead), 2 dusty (75 % or more but under 90 %), 3 ok (90 % or more). Return an int array of shape (40, 60); for the real day 3 panels are 0, 3 are 1 and 60 are 2.

    The Check compares the result on the real day, on a changed day and on a day where block A made 20 % less (each row has its own median); then a for, a while or a comprehension in panel_status fails the task; then it times it, and it must take under 0.01 s (the old one takes about 0.08 s on the lab's half CPU, a whole-array one well under 0.001 s).

    Imran will ask: where did the old function spend its time, and how do you know before you change anything?

  4. Task 4

    report.py: Rukmini's nightly report

    report.py uses your log.py, totals.py and faults.py: tasks 1 to 3 must pass first. Keep its three file lines as they are: DAY = "/workspace/data/power_day.npy", LOG and BLOCKS. The Check runs the report again with the DAY line changed to other days.

    First write money(plant_kw, schedule_kw, price_rs_mwh): three arrays of one day's 96 blocks, the plant's kW, the schedule's kW and the price in rupees a MWh. A block's energy is its kW times 0.25 (kWh). A block is short when the plant made under 90 % of the schedule; its penalty is the missing energy (schedule minus plant, in kWh) times the price / 1000. Return three values, (revenue, penalty, short): the day's revenue in rupees (every block's kWh times its price / 1000, summed), the penalties summed, and the number of short blocks as an int. The Check tries 14 October and 4 made-up blocks, and accepts rupees within half a paisa.

    Then, under if __name__ == "__main__":, print the report for the day in DAY. The schedule and the price are the last 96 lines of BLOCKS (its last two columns). The plant's kW in a block is the sum of all its panels / 1000, and its energy for the day the sum of those times 0.25. Panel (r, c) is on inverter (r // 10) × 3 + c // 20 (0 is INV01). The inverter log's energy is the sum of clean_log(LOG) times 5 / 60. For the real day it prints exactly these eleven lines:

    DhoopGrid nightly report

    Energy: 7708.5 kWh

    Peak: block 48 at 12:00, 1102.7 kW

    Blocks: A 1913.7, B 1913.8, C 1914.0, D 1966.9 kWh

    Inverters, best first: INV11 INV10 INV12 INV07 INV09 INV06 INV05 INV02 INV01 INV03 INV04 INV08

    Inverter log: 7555.9 kWh, 98.0 % of the panels

    Dead: P0462 P1084 P2033

    Weak: P1217 P1273 P1336

    Dusty: 60 panels

    Revenue: Rs 27276.55

    Short blocks: 4, penalty Rs 633.24

    Energies, kW and the percent have one decimal, rupees two. The peak is the block with the most kW, and its time is that block's start. Dead, weak and dusty are task 3's statuses 0, 1 and 2; the ids are in panel order (P0001 first), separated by spaces, and the word none when there are none. Inverters are ranked by task 2's inverter_kwh, the most first. The percent is the log's energy over the day's energy, times 100.

    The Check runs it with the real day, with another day (a panel fixed, new faults, dust and a second cloud), and with a day with no faults at all (Dead: none, Weak: none, Dusty: 0 panels), so work out every line from the data.

  5. Task 5 · interview question

    Interview: life.py, dust that jumped the edge

    Imran models dust spreading on the (40, 60) grid by the rules of Conway's Game of Life. dusty is a bool array, True where a panel is dusty; each panel has 8 neighbours, and in one step all panels change at once: a dusty panel stays dusty when 2 or 3 of its neighbours are dusty and is cleaned otherwise; a clean panel turns dusty when exactly 3 of its neighbours are dusty. Nothing is outside the plant: a panel on the edge has fewer neighbours, and those outside count as clean.

    Bhanwar ran the starter's dust_step with dust near the east edge (column 59), and a few steps later dust appeared on the west edge, column 0. On a (3, 3) grid that is all dusty, every panel comes out clean, though the four corners have only 3 neighbours and should stay dusty.

    Fix dust_step(dusty) in life.py so it counts only real neighbours and returns the next step as a bool array of the same shape, with no Python loop in it (a for, a while or a comprehension in dust_step fails the task). dust_after(dusty, steps) is done: it calls dust_step that many times. The Check tries the all-dusty (3, 3) grid, dust moving off the east and the south edges, a row of three in the middle for 1 and 2 steps, and a random (40, 60) grid for 5 steps.

    Imran will ask: why did the dust jump from column 59 to column 0, and how do you count neighbours without that?

What you will learn

  • Loading and gaps (chapters 2 and 13): np.load, np.genfromtxt, nan, np.interp.
  • Totals (7, 8 and 14): reshape into blocks, sums along axes, np.bincount as a group-by.
  • Statuses (5, 8 and 14): a median along an axis with keepdims, np.select.
  • Money (5 and 9): masks over the 96 blocks, sums.
  • Ranking and ids (6 and 12): argsort, np.nonzero, f-strings.
  • Speed (10): measure first, then remove the loops.
  • New for the puzzle: np.roll, and the rules of the Game of Life.

More in Real work

Build "Capstone: the nightly report and the dust" yourself

DhoopGrid runs a solar plant in the Thar desert with 2,400 panels. You turn its raw readings into answers. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of NumPy are free. This one comes with the all labs plan.