pandas · Chapter 21 of 22
Categories and method chaining
By the end of this chapter you can store a column of repeated text as a category and measure what it saves, put values like sizes in their own order so they sort and compare the way people mean, work with the .cat accessor, avoid the two traps of categories, and write a cleaning script as one readable chain of methods. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Real work
- 4 tasks
- pandas
- In the all labs plan
The first 2 chapters of pandas are free. This one comes with the all labs plan.
Why you need this
The capstone, next, holds a year of all three channels in one table. Many of its columns repeat a handful of values thousands of times: the website's payment column has 14,128 values and only 4 different ones, and as text it takes 165,910 bytes. The whole website table takes 2,618,712.
Some of those values also have an order that is not the alphabet's. Shabnam sorts products into collection tiers, everyday, festive and bridal, and wants them in that order. As text, pandas sorts them like words, and comparing them is worse:
tiers = pd.DataFrame({
"sku": ["KG-SAR-002", "KG-DUP-011", "KG-STL-004", "KG-SAR-019", "KG-KUR-007"],
"tier": ["bridal", "festive", "everyday", "bridal", "everyday"],
})
print(tiers.sort_values("tier"))
print((tiers["tier"] >= "festive").tolist())
sku tier
0 KG-SAR-002 bridal
3 KG-SAR-019 bridal
2 KG-STL-004 everyday
4 KG-KUR-007 everyday
1 KG-DUP-011 festive
[False, True, False, False, False]
Bridal first, and "festive or better" finds only the festive dupatta: "bridal" comes before "festive" in the alphabet, so as text it is "smaller". Gopal's kurta sizes have the same problem.
And Harpreet's cleaning scripts have grown into six or seven lines that each reassign the same name, lines = lines.... Run them out of order in IPython, or twice, and the table is quietly wrong: the second run drops nothing, adds line_rs again, and nobody can tell from the code which version lines is now.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
categories.py: city and status as categories, and the memory saved
Write two functions in categories.py, each given lines: the website's order lines from orders_site.csv without the duplicates and the test orders, with the cities already cleaned to 16 names (city and status are text, str). They must not change lines.
with_categories(lines) returns a DataFrame with the same columns, rows, labels and values as lines, where only city and status have the category dtype; every other column keeps its own dtype.
memory_saved(lines) returns how many bytes the table saves with city and status as categories: the table's total memory with memory_usage(deep=True), summed over every column and the index, as text minus with the two categories. One whole number (an int).
The Check tries the real lines and their first 40 lines (where the saving is much smaller).
- Task 2
sizes.py: kurta sales by size, in size order
Write size_sales(kurtas) in sizes.py. kurtas is the website's kurta lines (the lines whose size is not blank) with the columns order_id, placed_at, sku, size, qty and price_rs; size is text, one of S, M, L, XL and XXL, and qty is decimals with a few blanks.
Return a Series of the units sold (the sum of qty) of each size, labelled by the five sizes in size order, S, M, L, XL, XXL, not in alphabetical order. Every one of the five sizes is in the result even if no line has it: its units are 0. The labels may be text or an ordered category.
The Check tries the real kurta lines and 6 kurta lines with no XXL among them.
- Task 3
chain.py: Harpreet's six steps as one chain
chain.py has Harpreet's clean_lines(raw), written as six steps that each reassign a variable, and her own function fix_cities(table), which you keep as it is. raw is the website's file as read_csv reads orders_site.csv. Her steps, in order: drop the exact duplicate lines; drop the test lines (customer_id TEST); clean the cities with fix_cities; add line_rs, price_rs times qty, as the last column; make status a category; sort by placed_at with kind="stable" (lines with the same time keep their order).
Rewrite clean_lines as one chain: the function's body is a single return statement whose value is one expression of methods, one after another (put it in brackets, one method a line), with fix_cities as one step of the chain through pipe (query and assign fit the other steps). It must return exactly the same table as her six steps: the same columns in the same order, the same rows in the same order with their own labels, and the same dtypes (status a category).
The Check runs it on the real file and on 7 messy lines (a duplicate, a test line, cities like " bombay" and "new delhi", a blank qty), then reads the function: more than one statement in it, or no pipe with fix_cities, fails the task.
- Task 4 · interview question
Interview: big_sizes.py, L before M
Gopal restocks the bigger kurtas first, so he asked for large_and_up(kurtas) in big_sizes.py: the kurta lines of size L, XL or XXL, from L up. Its list has M and S kurtas in it, and its L lines come before M ones, though the data is right. kurtas is as in task 2: the website's kurta lines with the columns order_id, placed_at, sku, size, qty and price_rs, size as text.
Fix it so it returns a DataFrame of the lines whose size is L or bigger (L, XL or XXL), with all six columns in the same order and the lines' own labels, sorted by size from L to XXL (within one size, in any order). The size column may come back as text or as an ordered category.
The Check tries the real kurta lines and 6 kurta lines.
Devika will ask: why did sort_values("size") put L before M, and how do you make pandas sort and compare sizes in size order?
What you will learn
- The category dtype: codes and categories, astype("category"), and the memory it saves.
- Ordered categories: pd.CategoricalDtype(..., ordered=True), for sorting and comparing in your order.
- The .cat accessor: categories, codes, rename_categories, add_categories, remove_unused_categories, set_categories.
- Two traps: a value that is not a category, and the empty groups of observed=False.
- Method chaining: one method a line in brackets, with assign, query and pipe; debugging a chain, and when not to chain.
More in Real work
Build "Categories and method chaining" yourself
Kargha, a Varanasi handloom brand, sells through a website, a marketplace and a store. You make one true picture of its sales. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of pandas are free. This one comes with the all labs plan.