Machine Learning · Chapter 18 of 26
Class imbalance
By the end of this chapter you can say why a model trained on rare defaults hardly ever predicts one, weight the classes with class_weight="balanced" and work its weights out by hand, measure what that buys and what it costs, compare it with moving the threshold, resample the training rows without leaking, and let Farida's rupees choose. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Classifiers
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.
Why you need this
339 of Dobara's 2,400 loans defaulted: one in seven. Train the logistic regression of chapter 14 on them and ask it to predict, and it approves almost everyone. On this chapter's 600 test loans it calls 29 a default, and it catches 18 of the 85 that really defaulted. Each default it misses costs Farida ₹30,000.
Raju found the usual fix online: "balance the classes". He copied the defaults until there were about as many as repaid loans, trained a tree, and its recall on the test rows was 1.0. Rachna wants to know which of the fixes are real, and what each one costs in rupees.
Chapter 4 already showed one answer, moving the threshold. What you cannot yet do is change what the model learns: make a missed default count for more inside fit, or change the rows fit sees. Both are common, both change recall and precision, and one of them, done in the wrong order, gives a number that is simply false.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
weights.py: the balanced weights, by hand
Write balanced_weights(y) in weights.py. y is a pandas Series or a NumPy array of class labels: 0 and 1, or text, or more than two classes. Return a dict with one key per class present in y (the label itself) and, as its value, the weight class_weight="balanced" gives it: n_samples / (n_classes × the class's count), where n_samples is how many labels y has and n_classes how many different labels. Five labels [0, 0, 0, 0, 1] give {0: 0.625, 1: 2.5}.
The Check compares your weights with scikit-learn's compute_class_weight on the last 500 loans' defaulted, 300 labels written as "repaid" and "defaulted", the occupations of the first 600 loans (five classes) and 90 made-up labels of three classes.
- Task 2
weighted.py: what class_weight buys, and what it costs
Write with_and_without(X, y) in weighted.py. X is a DataFrame of number columns (blanks allowed) and y the label defaulted. Split with train_test_split, test_size=0.25, random_state=7 and stratify=y. Build two pipelines with make_pipeline, each of SimpleImputer(strategy="median", add_indicator=True), StandardScaler() and LogisticRegression(max_iter=1000): "plain" with no class_weight, and "balanced" with class_weight="balanced" on the LogisticRegression. Fit each on the training rows. On the test rows, compute recall_score and precision_score of its predict (of the defaults, class 1) and roc_auc_score of the second column of its predict_proba.
Return a DataFrame with exactly two rows, labelled "plain" then "balanced", and exactly the columns recall, precision and auc, in that order (floats from 0 to 1). The Check tries the loans' ten number columns and 800 made-up loans with other columns.
- Task 3
oversample.py: oversampling the training rows
Write oversample(X_train, y_train, random_state) in oversample.py. X_train is a DataFrame of training rows and y_train their labels (1 a default, 0 repaid), with the same row labels. Count how many more repaid loans there are than defaults; draw that many rows from the defaults in X_train with the DataFrame's sample method, replace=True and the random_state given; put the drawn rows after all the rows of X_train, and their labels from y_train after all of y_train.
Return two values separated by a comma: the new DataFrame and the new Series of labels, with every row's own row label kept (a drawn row repeats its label), so both classes have the same count. Every original row stays. The Check tries the loans' training rows with random_state 0 and 5, and 200 made-up loans.
- Task 4 · interview question
Interview: leak.py, a recall too good to be true
Raju oversampled all the loans with your oversample from Task 3 (so do Task 3 first), then split, then trained a tree with no depth limit, and its test recall was nearly 1.0. Fix honest_recall(X, y) in leak.py so that it splits X and y with train_test_split, test_size=0.25, random_state=7 and stratify=y before anything else, oversamples the training rows only (oversample with random_state=0), and fits DecisionTreeClassifier(random_state=0) on the oversampled training rows.
Return two values separated by a comma: the fitted tree and its recall of the defaults (class 1) on the test rows. The Check tries the loans' ten number columns and 600 made-up loans.
Tanvi will ask: why did Raju's recall jump, and why is the lower number the real one?
What you will learn
- Why imbalance happens: the model's loss treats every row alike, so the common class wins.
- class_weight="balanced": each class weighted n_samples / (n_classes × count), and compute_class_weight to check it.
- What weighting buys and costs: recall up, precision down, AUC about the same, probabilities no longer shares.
- Moving the threshold instead: a similar trade, with no refit.
- Oversampling and undersampling with pandas sample, on the training rows only.
- The resampling leak: copies of test rows in the training rows.
- The rupees decide: Farida's costs for each choice.
More in Classifiers
Build "Class imbalance" yourself
Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.