Machine Learning · Chapter 2 of 26
Estimators, and the split
By the end of this chapter you can read and change a model's settings, tell what it learned from what you chose, ask it how sure it is with predict_proba, swap one model for another in a line, and split the loans so the test score means what you think it means: stratified, with its luck measured, and by date when the future is what you want to know. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Learning from data
- 4 tasks
- Python
- Free chapter
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.
Why you need this
Chapter 1's tree scored 0.835 on 600 loans it had never seen. Farida read the number and asked two questions. First: "It says refuse or approve. How sure is it?" A buyer the tree barely calls risky and one it is certain about get the same 1, and a ₹30,000 loss sits behind every default approved. Second, from Rachna: "If you had held back another 600 loans, would it still say 0.835?"
What you know so far cannot answer either. predict gives only the class. And random_state=1 picked one shuffle out of thousands: with another seed the same tree on the same columns scores anywhere from 0.835 to 0.883, and the share of defaults in the test rows wanders from 11.7 % to 16.3 %, so part of any one score is luck. Worse, Dobara lends to next quarter's buyers, and defaults rose over the year: a random split mixes the easy months with the hard ones, and the score it gives is for a year that has already happened.
The lesson
Settings you choose: hyperparameters
A model's settings are its hyperparameters: you choose them before fit, and the data never changes them. Here is chapter 1's tree, in an IPython session (ipython in the terminal). The session starts with the imports and chapter 1's two columns:
import pandas as pd
from sklearn.base import clone
from sklearn.tree import DecisionTreeClassifier
loans = pd.read_csv("/workspace/data/loans.csv")
X = loans[["monthly_income_rs", "existing_emis"]]
y = loans["defaulted"]
In [7]: model = DecisionTreeClassifier(max_depth=2, random_state=0)
In [8]: model
Out[8]: DecisionTreeClassifier(max_depth=2, random_state=0)
In [9]: model.get_params()
Out[9]:
{'ccp_alpha': 0.0,
'class_weight': None,
'criterion': 'gini',
'max_depth': 2,
'max_features': None,
'max_leaf_nodes': None,
'min_impurity_decrease': 0.0,
'min_samples_leaf': 1,
'min_samples_split': 2,
'min_weight_fraction_leaf': 0.0,
'monotonic_cst': None,
'random_state': 0,
'splitter': 'best'}
In [10]: model.set_params(max_depth=4)
Out[10]: DecisionTreeClassifier(max_depth=4, random_state=0)
- Printing a model shows only the settings you changed from their defaults: the tree has thirteen, and you set two.
get_params()gives all of them as a dict, so a program can read a setting:model.get_params()["max_depth"].set_params(max_depth=4)changes a setting on the same model and gives the model back. Change a setting afterfitand nothing is relearned until you fit again.
Most of these thirteen are for later chapters (chapter 16 opens the tree). For now the point is the shape of things: settings go in when the model is made, by name.
What fit returns, and what it keeps
In [11]: hasattr(model, "classes_")
Out[11]: False
In [12]: fitted = model.fit(X, y)
In [13]: fitted is model
Out[13]: True
In [14]: [name for name in vars(model) if name.endswith("_")]
Out[14]:
['feature_names_in_',
'n_features_in_',
'n_outputs_',
'classes_',
'n_classes_',
'max_features_',
'tree_']
In [15]: model.set_params(max_depth=2).fit(X, y).predict(X.head(3))
Out[15]: array([0, 0, 0])
- Before
fitthe model has noclasses_: it has learned nothing yet. fitchanges the model in place and returns the same model (fitted is modelis True), so calls chain: make, fit and predict in one line.- Everything
fitlearned is stored in attributes ending in an underscore: the classes it saw, the columns, andtree_, the questions themselves.vars(model)lists everything on the model, and the ones ending in_are what it learned.
That underscore is the most useful habit scikit-learn has. No underscore: you chose it (max_depth). An underscore: it came from the data (classes_, tree_). When a number looks wrong, the underscore tells you whether to blame your settings or the rows the model saw.
clone: the same settings, nothing learned
In [16]: copy = clone(model)
In [17]: copy
Out[17]: DecisionTreeClassifier(max_depth=2, random_state=0)
In [18]: hasattr(copy, "classes_")
Out[18]: False
clone (from sklearn.base) makes a new, unfitted model with the same settings. It copies what you chose and none of what was learned. That is what you want when the same recipe must learn again on other rows: chapter 5 fits one clone per fold, and this lab's Checks fit a clone of your model on the training rows to see whether yours learned from those rows alone.
How sure is it? predict_proba
In [19]: applicants = pd.DataFrame({"monthly_income_rs": [12500, 31000, 15000], "existing_emis": [2, 0, 1]})
In [20]: model.classes_
Out[20]: array([0, 1])
In [21]: model.predict_proba(applicants)
Out[21]:
array([[0.49315068, 0.50684932],
[0.92419825, 0.07580175],
[0.78852459, 0.21147541]])
In [22]: model.predict(applicants)
Out[22]: array([1, 0, 0])
These are chapter 1's three applicants, and model is the depth 2 tree fitted on all the loans again (In [15]). predict_proba gives one row per applicant and one column per class, and the columns are in the order of classes_: here column 0 is "repaid" and column 1 is "defaulted". Each row adds up to 1.
Where do the numbers come from? A tree sends a row down its questions to a leaf, and its probability is the share of each class among the training loans in that leaf. apply gives the leaf each row lands in:
In [23]: leaf = model.apply(X)
In [24]: first_leaf = model.apply(applicants)[0]
In [25]: y[leaf == first_leaf].value_counts()
Out[25]:
defaulted
1 37
0 36
Name: count, dtype: int64
In [26]: 37 / 73
Out[26]: 0.5068493150684932
The first applicant lands in the leaf of 73 past loans of which 37 defaulted: 0.507. predict is simply the class with the larger share, so a leaf where half the loans defaulted and a leaf where all of them did both come out as 1. The probability is what Farida asked for. Chapter 4 moves the cut away from a half.
The columns follow classes_, and classes_ is sorted. With labels that are words, the order changes:
In [27]: words = y.map({0: "repaid", 1: "defaulted"})
In [28]: named = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X, words)
In [29]: named.classes_
Out[29]: array(['defaulted', 'repaid'], dtype=object)
In [30]: named.predict_proba(applicants)
Out[30]:
array([[0.50684932, 0.49315068],
[0.07580175, 0.92419825],
[0.21147541, 0.78852459]])
"defaulted" comes before "repaid" in the alphabet, so the chance of default is now column 0. Code that always takes column 1 silently reports the chance of repaying. Read classes_ first, every time.
The same API for other models
Every scikit-learn classifier is made, fitted, asked to predict and scored the same way, so trying another model is a one-line change. Two you will open up later, used here as black boxes with their default settings: LogisticRegression (a weighted sum of the columns turned into a probability; chapter 14) and KNeighborsClassifier (it looks up the past loans most like the new one and takes their vote; chapter 15).
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=1
)
models = [
DecisionTreeClassifier(max_depth=2, random_state=0),
LogisticRegression(max_iter=1000),
KNeighborsClassifier(),
]
for model in models:
model.fit(X_train, y_train)
print(model, model.score(X_test, y_test))
print(" ", model.predict_proba(X_test.head(1)))
DecisionTreeClassifier(max_depth=2, random_state=0) 0.835
[[0.93741497 0.06258503]]
LogisticRegression(max_iter=1000) 0.84
[[0.9025054 0.0974946]]
KNeighborsClassifier() 0.8283333333333334
[[1. 0.]]
The same loop works for all three, and each gives a probability its own way: the tree from a leaf's share, the logistic regression from its weighted sum, kNN from the vote of five neighbours (five said "repaid", so 1.0). max_iter=1000 gives logistic regression enough rounds to finish learning; chapter 14 says why it needs them. Not every estimator predicts: some transform a table into another table, such as a scaler that puts columns on one scale. They share fit and come in chapter 7.
The golden rule of the test rows
The test rows stand in for buyers nobody has seen. That works only if nothing about them leaks into your choices: not into fit, and not into you. If you try ten settings and keep the one with the best test score, the test rows have chosen your model, and the score is no longer about new buyers. So: choose with the training rows, and look at the test rows once, at the end. Chapter 5 shows how to compare settings without touching them.
stratify=y: the same share of defaults in both parts
Only 14.1 % of loans defaulted, and a random shuffle does not promise that a quarter of them land in the test rows:
print(round(y.mean(), 4))
for seed in range(5):
plain = train_test_split(X, y, test_size=0.25, random_state=seed)
strat = train_test_split(
X, y, test_size=0.25, random_state=seed, stratify=y
)
print(seed, round(plain[3].mean(), 4), round(strat[3].mean(), 4), strat[3].sum())
0.1412
0 0.1167 0.1417 85
1 0.16 0.1417 85
2 0.14 0.1417 85
3 0.1217 0.1417 85
4 0.1583 0.1417 85
plain[3] is y_test, the fourth part. Without stratifying, the test rows' share of defaults goes from 11.7 % to 16 % with the seed, and a test set with fewer defaults is easier to score well on. With stratify=y, the split shuffles the defaults and the repaid loans separately and cuts a quarter of each, so every test set has exactly 85 defaults in 600, 14.2 %, as close to the whole as 600 rows allow. For a rare class, stratify is the safe default.
How much luck is in one number?
Even stratified, which 600 loans you hold back changes the score. Measure it: the same depth 2 tree on the same two columns, 20 seeds, three test sizes:
import numpy as np
for test_size in (0.1, 0.25, 0.5):
scores = []
for seed in range(20):
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=test_size, random_state=seed
)
tree = DecisionTreeClassifier(max_depth=2, random_state=0)
tree.fit(X_train, y_train)
scores.append(tree.score(X_test, y_test))
scores = np.array(scores)
print(test_size, len(X_test), scores.min().round(3), scores.max().round(3))
0.1 240 0.812 0.883
0.25 600 0.835 0.883
0.5 1200 0.842 0.875
With a quarter held back, chapter 1's 0.835 was the lowest of 20 draws: the tree did not change, the test rows did. Hold back 240 loans and the score swings over seven points; hold back 1,200 and it is steadier, but the tree learns from only half the loans. A quarter is the usual compromise. With small data both halves are too small, and chapter 5's cross-validation uses every row for testing once instead.
When a random split is the wrong split
A shuffle assumes the loans are all alike in time. Dobara's are not:
applied = pd.to_datetime(loans["applied_on"])
past = applied < "2025-04-01"
print(past.sum(), (~past).sum())
print(loans.loc[past, "applied_on"].max(), loans.loc[~past, "applied_on"].min())
print(round(loans.loc[past, "defaulted"].mean(), 4), round(loans.loc[~past, "defaulted"].mean(), 4))
1797 603
2025-03-31 2025-04-01
0.1358 0.1575
The last quarter, from 1 April 2025, has 603 loans and defaulted at 15.8 %, against 13.6 % before. A model goes live on buyers who apply after the ones it learned from, so the honest test is the same: learn from the past, test on what came after. A random split lets the tree learn from June's loans and be tested on July's, which is the wrong way round, and the test rows hold only a quarter of the hard months. Its score answers "how good was it over last year?", not "how good will it be next quarter?". Splitting by date is plain pandas: a mask on the date, no shuffling.
The second trap is the same buyer twice. If one buyer had three loans, a random split can put two in training and one in the test rows, and the model is tested on someone it already knows. Dobara's loans have no buyer id, so it cannot happen here, but the fix is to split by groups: all of a buyer's loans on one side. Chapter 5 does that with GroupKFold, keeping each dealer's loans together.
Spot the bug
X_train, X_test, y_train, y_test = train_test_split(X, y, 0.25)
TypeError: Input should have at least 1 dimension i.e. satisfy `len(x.shape) > 0`, got scalar `array(0.25)` instead.
That is the last line of the traceback. train_test_split takes any number of tables first, and splits them all the same way, so 0.25 without a name is taken as a third table to split. Settings are always given by name: test_size=0.25.
Your turn
Four tasks, one file each. Task 1 makes a stratified split with the task's test_size and random_state; the Check compares every row of the four parts. Task 2 gives each application its chance of default: the trap is a model whose labels are words, where the default column is not column 1. Task 3 scores one tree on a split for each seed in a list and gives the range: only the split's seed changes. Task 4 is the interview question: the same tree scored on a random split and on a split by date, and Tanvi will ask why the first is higher and which one Rachna should believe. Try each in ipython first (from seeds import scores_over_seeds).
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
stratified.py: a stratified split
Write stratified_split(X, y) in stratified.py. X is a DataFrame of features and y the label (defaulted), one value for each row of X. Split the rows with train_test_split, test_size=0.25 and random_state=7, stratified on y, so that the training rows and the test rows each keep y's share of defaults.
Return the four parts, separated by commas, in the order train_test_split gives them: X_train, X_test, y_train, y_test (DataFrames for X, Series for y).
The Check compares every row of the four parts with its own split, so a split without stratify, or with another test_size or random_state, fails. It tries the loans' five features from chapter 1 (monthly_income_rs, emi_rs, existing_emis, down_payment_rs, age) and three columns of 300 made-up loans.
- Task 2
default_chance.py: the chance of a default
Write default_chance(model, applications, default_label) in default_chance.py. model is a fitted classifier (a decision tree or a logistic regression) with predict_proba; applications is a DataFrame of new rows with the columns the model was fitted on; default_label is the label that means "defaulted" in this model, 1 or a word such as "defaulted".
Return a NumPy array of floats from 0 to 1, one per application in order: the model's probability of default_label for that row (one column of predict_proba, not both, and not predict's 0s and 1s).
The Check gives models fitted with the labels 0 and 1, and models fitted with words ("repaid" and "defaulted", "paid" and "default"), where the default's column is not the second one. The columns of predict_proba follow the order of the model's classes_.
- Task 3
seeds.py: how much luck is in one score?
Write scores_over_seeds(X, y, seeds) in seeds.py. X is a DataFrame of features, y the label, and seeds a list of whole numbers. For each seed, in the list's order: split the rows with train_test_split, test_size=0.25 and random_state set to that seed (not stratified); fit a DecisionTreeClassifier with max_depth=3 and random_state=0 on that split's training rows only; and score it on that split's test rows.
Return two values separated by a comma: a NumPy array of the test scores (floats from 0 to 1), one per seed in the list's order, and their range, the largest score minus the smallest, as one float.
The tree's own random_state stays 0: only the split's random_state changes. The Check tries the loans' five features from chapter 1 with the seeds 0 to 9, and three columns of 400 made-up loans with the seeds [3, 1, 4].
- Task 4 · interview question
Interview: by_date.py, next quarter
Write random_and_dated(loans, features, cutoff) in by_date.py. loans is a DataFrame as pd.read_csv reads loans.csv (applied_on is a date as text, such as "2025-04-01"), features a list of column names, and cutoff a date as text. X is loans[features] and y is loans["defaulted"].
Fit a DecisionTreeClassifier with max_depth=3 and random_state=0 twice, and score each on its own test rows: first on a random split with train_test_split, test_size=0.25 and random_state=7 (not stratified); then on a split by date, training on the loans applied before the cutoff and testing on the loans applied on the cutoff day or later.
Return two floats from 0 to 1, separated by a comma: the random split's test score first, then the dated split's test score. The Check tries the loans' five features from chapter 1 with the cutoff "2025-04-01", and emi_rs, existing_emis and age with the cutoff "2025-01-01".
Tanvi will ask: why does the random split score higher, and which of the two numbers tells Rachna how the model will do on next quarter's buyers?
What you will learn
- The estimator in depth: settings (hyperparameters) with get_params and set_params, what fit returns, and the attributes ending in an underscore that hold what it learned.
- clone: a fresh, unfitted copy with the same settings.
- predict_proba: the model's probability for each class, in the order of classes_, and how predict comes from it.
- Other estimators, the same API: LogisticRegression and KNeighborsClassifier swapped in one line.
- stratify=y: a split that keeps the share of defaults the same in both parts.
- The luck in one number: the same tree over 20 seeds.
- When a random split is wrong: loans in date order, and the same buyer twice.
More in Learning from data
Build "Estimators, and the split" yourself
Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
Free with a Google or GitHub sign-in, like the first 2 chapters of every lab.