pandas · Chapter 20 of 22

Shift, diff and change over time

By the end of this chapter you can compare each row with the one before it (or seven before it) with shift, diff and pct_change, do it within each customer with a groupby, keep running totals and records with cumsum and cummax, rank values, and spot the two mistakes that make a comparison over time quietly wrong. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Time
  • 4 tasks
  • pandas
  • In the all labs plan

The first 2 chapters of pandas are free. This one comes with the all labs plan.

Why you need this

Chapter 19 gave Shabnam her daily and monthly revenue. Her next questions are all about change: did November beat October, and by how much? Is this Sunday better than last Sunday? When did a day first beat every day before it? Harpreet asks how far into the month's target the running total is, and how long a customer waits before ordering again.

Every one of these compares a row with an earlier row. What you know so far does not do that. Take the store's first five days of takings (chapter 19's takings) and subtract "the same Series without its first day" from "the same Series without its last", hoping for each day minus the day before:

t = takings.loc["2025-10-01":"2025-10-05"]
print(t.iloc[1:] - t.iloc[:-1])
time
2025-10-01    NaN
2025-10-02    0.0
2025-10-03    0.0
2025-10-04    0.0
2025-10-05    NaN
Freq: D, Name: line_rs, dtype: float64

Zeros. Chapter 2's alignment paired every day with itself, because arithmetic pairs values by label, so each day minus the same day is 0, and the two ends, on one side only, are NaN. Dropping the labels with to_numpy() gives the right four numbers, but as a bare array with no dates on it, one shorter than the Series, ready to be pasted beside the wrong rows. What is missing is a way to move the values down a row while the labels stay where they are.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    growth.py: month-on-month growth

    Write month_change(monthly) in growth.py. monthly is a Series of the website's revenue for each calendar month, labelled by the month's first day (2025-10-01 for October 2025), in date order, as resample with "MS" gives it.

    Return a DataFrame labelled by monthly's labels, in the same order, with exactly three columns in this order: revenue (monthly's values), change_rs (this month's revenue minus the month before's) and growth_pct (the change as a percent of the month before's revenue: 10.0 for a month 10 % up, -5.0 for 5 % down). All three are decimals. The first month has no month before it: its change_rs and growth_pct are missing (NaN).

    The Check tries the real year and March to June 2026, with a month that does not change and one that falls.

  2. Task 2

    week_ago.py: each day against the same weekday a week before

    Write against_last_week(daily) in week_ago.py. daily is a Series of the website's revenue on each day that had orders, labelled by the day (dates at midnight, in date order), as a groupby on the date gives it: a day with no orders is not in it at all, so 14 and 15 June 2026 are missing.

    Return a DataFrame labelled by every calendar day from daily's first day to its last, in date order, with exactly three columns in this order: revenue (that day's revenue, 0 for a day missing from daily), week_ago (the revenue on the date seven days before, the same weekday; missing (NaN) for the first seven days, which have no week before them) and change_rs (revenue minus week_ago). All three are decimals.

    The Check tries the real daily revenue and 1 to 16 February 2026 with 5 and 10 February missing.

  3. Task 3

    so_far.py: the month so far, and the best day so far

    Write month_so_far(daily) in so_far.py. daily is a Series of revenue for every calendar day, labelled by the day (dates at midnight, in date order, no day missing).

    Return a DataFrame labelled by daily's days, in the same order, with exactly three columns in this order: revenue (daily's values), month_so_far (the running total within the day's calendar month: the 1st of each month starts again from that day's revenue) and best_so_far (the highest daily revenue from daily's first day up to and including this day, across months). All three are decimals.

    The Check tries the real year and 29 January to 3 February 2026.

  4. Task 4 · interview question

    Interview: repeat.py, a previous order in the future

    Harpreet wants, for every website order, how many days it came after the same customer's previous order. Her days_since_last(orders) in repeat.py gives negative numbers: for many orders, the customer's "previous" order is a later one.

    orders has one row per order with the columns order_id, customer_id and placed_at (dates), labelled 0, 1, 2 and so on, and the Check gives them newest first, the way her export sorts them, and also in no order at all.

    Fix days_since_last so it returns a Series labelled like the orders' rows (the input's own labels, in any order): for each order, the whole days since the same customer's previous order (placed_at minus the previous order's placed_at, as the days of the duration), and a missing value (NaN) for a customer's first order. Whole days are decimals here, because of the missing values.

    The Check tries the real orders newest first, and 6 orders of 3 customers in no order.

    Devika will ask: why was the customer's previous order in the future, and what must be true of the rows before you shift within a group?

What you will learn

  • shift: move the values down (or up) by n rows; the labels stay.
  • diff and pct_change: a value minus the one n rows before, and the change as a fraction.
  • Like with like: shift(7) on a full daily calendar is the same weekday a week before.
  • groupby then shift or diff: the previous row within each customer, in time order.
  • cumsum, cummax and rank: running totals, records so far, and positions.
  • gt and lt: comparisons as methods.

More in Time

Build "Shift, diff and change over time" yourself

Kargha, a Varanasi handloom brand, sells through a website, a marketplace and a store. You make one true picture of its sales. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of pandas are free. This one comes with the all labs plan.