Machine Learning · Chapter 10 of 26
Pipelines and leakage
By the end of this chapter you can spot a column that leaks the answer, put every preprocessing step and the model into one Pipeline, cross-validate it so that each fold refits everything, and build the full preprocessing for Dobara's loans: blanks, scales and categories together. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Preparing the data
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.
Why you need this
Raju came back from the collections team with good news. He had added one more column to a small tree, and its test AUC jumped from 0.669 to 0.981. The column was collection_calls: how many times the collections team phoned the buyer. It is a wonderful predictor of default, and a useless one, because the calls happen after a buyer stops paying. On the day someone applies for a loan, nobody has called them yet. A model that needs this column cannot be used on a single real application.
That is a leak: information in the training data that would not be there when the model is used. Chapters 7 and 9 met a quieter kind, a scaler or an imputer that learned from the test rows. Doing every step by hand on the training rows, then again on the test rows, then again inside every fold of cross-validation, is how such leaks slip in: one forgotten line and the number is not honest. This chapter gives you the question that finds the loud leaks and the tool, the Pipeline, that stops the quiet ones.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
day_one.py: what Dobara knows on the day
Write day_one_columns(loans) in day_one.py. loans is a DataFrame with the columns of loans.csv as pd.read_csv reads it, or some of them (always defaulted); do not change it.
Return two values, X and y, separated by a comma. X is a DataFrame of every column of loans, in the table's order and with every row, except these: the label defaulted, the id loan_id, the date applied_on, and every column Dobara learns only after the loan was given (ask of each column: would Dobara know this on the day the buyer applies?). y is the column defaulted as a Series, with its name kept.
The Check tries all the past loans, the loans with their columns in another order, and 50 loans with only six of the columns, so drop the columns by name, and only those the table has.
- Task 2
full_model.py: the whole model, in one pipeline
Write make_model() in full_model.py. It takes nothing and returns a Pipeline that is not fitted yet (the Check fits it), with two steps in this order. Step 1 is a ColumnTransformer with two parts: the number columns, picked by dtype with make_column_selector, go through a Pipeline of SimpleImputer(strategy="median", add_indicator=True) then StandardScaler(); every other column (the text), also picked by dtype, goes through OneHotEncoder(handle_unknown="ignore", min_frequency=10). Step 2 is LogisticRegression(max_iter=1000).
Pipeline or make_pipeline are both fine, with any step names. The imputer comes before the scaler. The Check looks at the steps and their settings, then fits your pipeline on the loans' day-one columns (Task 1's X, split with test_size=0.25, random_state=7 and stratify=y) and on 300 made-up applicants with other columns, and compares its probabilities with those of the pipeline described here.
- Task 3
fit_and_score.py: cross-validate, fit, then test once
Write fit_and_score(model, X, y) in fit_and_score.py. model is a Pipeline that is not fitted yet, X its features and y the label. Split with train_test_split, test_size=0.25, random_state=7 and stratify=y. Cross-validate model on the training rows only with cross_val_score, cv=StratifiedKFold(5, shuffle=True, random_state=0) and scoring="roc_auc", and take the mean of the five scores. Then fit model on the training rows and find its ROC AUC on the test rows (roc_auc_score with predict_proba's column for class 1).
Return three values separated by commas: the fitted pipeline, the mean cross-validated AUC, and the test AUC, both floats from 0 to 1. The test rows are used once, at the end, and never in the cross-validation. The Check gives the pipeline of Task 2 with the loans' day-one columns, and a pipeline of a StandardScaler and a kNN with the loans' five features of chapter 1.
- Task 4 · interview question
Interview: cv_leak.py, a scaler outside the folds
Raju scaled all the loans with a StandardScaler and then cross-validated a kNN on the scaled rows. Fix honest_scores(X, y) in cv_leak.py so that the scaler is the first step of a pipeline, StandardScaler() then KNeighborsClassifier(), and cross_val_score gets that pipeline and the raw X, with the same cv=StratifiedKFold(5, shuffle=True, random_state=0) and scoring="roc_auc". X is a DataFrame of number columns with no blanks.
Return two values separated by a comma: the pipeline (cross_val_score fits copies of it, so it stays unfitted) and the five fold scores, the NumPy array cross_val_score gives. The Check tries the loans' five features of chapter 1, three other columns of the loans, and 400 made-up applicants.
Tanvi will ask: Raju's mean is only a little different, so why is it not honest, and what does a pipeline refit in each fold that Raju's code fitted once?
What you will learn
- Leakage: a column, a row or a fitted number that carries what the model could not know, and the question to ask of every column.
- Where leaks hide: columns recorded later, ids and dates, repeated rows, preprocessing fitted on all the rows, features built from y.
- Pipeline and make_pipeline: steps, named_steps, step names, and step__param in get_params and set_params.
- A pipeline is an estimator: fit, predict_proba, score, and cross_val_score that refits every step in every fold.
- The full preprocessing for loans: a ColumnTransformer with an imputer and a scaler for numbers, an encoder for text, then the model.
More in Preparing the data
Build "Pipelines and leakage" yourself
Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.