pandas · Chapter 3 of 22
Dtypes: what each column holds
By the end of this chapter you can say what each of pandas' dtypes stores, why a column of whole numbers with one blank comes back as float64, keep it whole with a nullable dtype such as Int64, change a column's dtype with astype, turn text into numbers with pd.to_numeric and count what would not convert, and measure what a table costs in memory with memory_usage and info. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Tables from zero
- 4 tasks
- pandas
- In the all labs plan
The first 2 chapters of pandas are free. This one comes with the all labs plan.
Why you need this
Harpreet pays the weavers from the website's order lines, and her payout sheet wants the pieces on each line as a whole number. Here is what read_csv gives her from the website's file:
import numpy as np
import pandas as pd
lines = pd.read_csv("/workspace/data/orders_site.csv")
print(lines[["order_id", "sku", "qty", "pincode"]].head(3))
print(lines["qty"].dtype, lines["pincode"].dtype)
order_id sku qty pincode
0 KS-100001 KG-DUP-025 1.0 700022.0
1 KS-100002 KG-SAR-006 1.0 500072.0
2 KS-100002 KG-DUP-025 1.0 500072.0
float64 float64
Every quantity is 1 or 2, and every pincode has six digits, yet both print with a .0. The payout sheet refuses "1.0 pieces", and a courier label that says 700022.0 goes back to the warehouse. The cause is 14 blank qty cells out of 14,128 lines, and 12 blank pincodes.
Gopal has the opposite problem. Sunil typed his recount into a form that saves every value as text, so on_hand holds "14", not 14, and adding up three counts glues them together. With chapter 1 you can see a column's dtype; you cannot yet change it. This chapter does.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
recount.py: Sunil's recount, typed as text
Write recount_types(sheet) in recount.py. sheet is Sunil's recount from a form that saves every value as text: four columns, every one of dtype str, sku, on_hand (a count, like "14"), cost_rs (like "8980.0") and counted_by.
Return a DataFrame with the same four columns in the same order and the same row labels, where on_hand has dtype int64 and cost_rs has dtype float64, while sku and counted_by stay str. The Check compares the dtypes exactly, so Int64 or float32 are not right here.
The Check tries a recount of 120 skus and a recount of 2 skus labelled 7 and 3.
- Task 2
to_numbers.py: hand-typed counts with n/a and -
Write two functions in to_numbers.py. Each is given typed, a Series of text with no blanks, labelled by sku: Gopal's hand-typed counts, most of them numbers as text like "14", some of them "n/a", "-" or a word.
as_numbers(typed) returns a Series of decimal numbers with the same labels in the same order: each text that is a number as that number, and every other value as a missing value (NaN).
count_not_numbers(typed) returns how many values of typed are not numbers, as one whole number (an int): 2 for "14", "n/a", "-" and "3", and 0 when every value is a number.
The Check tries 8 counts with "n/a", "-" and the word "lost", 3 counts that are all numbers, and 2 counts with no numbers at all.
- Task 3
memory.py: what a table costs in memory
Write two functions in memory.py. Each is given a DataFrame, table.
memory_by_column(table) returns a Series of how many bytes each column takes, counting the text itself in full, labelled by the column names in the table's order, with no entry for the index.
memory_saved(table, column, dtype) returns how many bytes the column named column takes now, minus how many it would take converted to dtype (a dtype's name, like "int32"), as one whole number (an int). It is negative when the new dtype takes more.
The Check tries the catalogue (products.csv), the weavers and a small table of 3 rows, with mrp_rs as "int32", cost_rs as "float32", looms as "int8" and a small table's qty as "Int64".
- Task 4 · interview question
Interview: whole_numbers.py, qty came back as float64
Harpreet's payout sheet needs qty and pincode as whole numbers, but read_csv gives float64 for both (qty prints 1.0, pincode 700022.0). Her whole_numbers(lines) stops with IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer. Fix it so it returns the lines with qty and pincode as whole numbers that can be missing, dtype Int64, a blank kept as a missing value, and every other column exactly as it was: the same columns, order, row labels and dtypes. lines is the website's order lines as read_csv reads orders_site.csv, so the result has 14128 rows.
The Check compares the dtypes exactly. It also tries a small table of 2 lines with no blanks, and a small table whose qty is all blank.
Devika will ask: why did qty come back as float64 when every quantity is a whole number?
What you will learn
- pandas' dtypes: int64, float64, bool, str, datetime64 and object, and what each stores.
- Why one blank makes whole numbers float, and the nullable dtypes Int64, Float64, boolean and string, with pd.NA.
- astype: a column (or several) in another dtype, and the errors when a value cannot convert.
- pd.to_numeric with errors="coerce": text to numbers, NaN where it is not a number.
- memory_usage(deep=True) and info(): what a table costs, and a first check of its types and blanks.
More in Tables from zero
Build "Dtypes: what each column holds" yourself
Kargha, a Varanasi handloom brand, sells through a website, a marketplace and a store. You make one true picture of its sales. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of pandas are free. This one comes with the all labs plan.