Machine Learning · Chapter 5 of 26
Cross-validation
By the end of this chapter you can test a model on every loan instead of one quarter of them, report its score with its spread, choose the folds that fit the data (stratified for a rare class, in date order for loans over time, by group for dealers), score several metrics at once, and give every loan a guess from a model that never saw it. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Learning from data
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.
Why you need this
Chapter 2 measured the luck in one split: the same depth 2 tree scored anywhere from 0.835 to 0.883 depending on which 600 loans were held back. Rachna cannot decide anything on a number that moves that much with a seed, and Tanvi cannot compare two settings when the difference between them is smaller than the luck.
With what you know, you would run twenty splits by hand and average them, as chapter 2 did. That works, but each split tests some loans several times and others never, and 1,800 training rows means a quarter of Dobara's history is never learned from in any one run. Chapter 4 made it worse: choosing a threshold took a third part, so the model learned from only 1,440 loans. Cross-validation rotates the test part so that every loan is tested exactly once, and every loan is learned from in the other rounds.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
fold_scores.py: five fold scores
Write fold_scores(X, y) in fold_scores.py. X is a DataFrame of features and y the label (defaulted). Score a DecisionTreeClassifier with max_depth=3 and random_state=0 with cross_val_score, cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=0) and scoring="roc_auc". Give cross_val_score the unfitted model and all of X and y.
Return three values separated by commas: the NumPy array of the five fold scores, in fold order; their mean; and their standard deviation (NumPy's std of the array).
The Check tries the loans' five features from chapter 1 (monthly_income_rs, emi_rs, existing_emis, down_payment_rs, age), emi_rs, existing_emis and age, and three columns of 400 made-up loans.
- Task 2
several_metrics.py: several metrics at once
Write cv_summary(X, y) in several_metrics.py. X is a DataFrame of features and y the label. Run cross_validate once on a DecisionTreeClassifier with max_depth=4 and random_state=0, with cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=0), scoring=["accuracy", "roc_auc", "recall"] and return_train_score=True.
Return a DataFrame with one row per metric, labelled "accuracy", "roc_auc" and "recall" in that order, and the columns test_mean (the mean of the five held-out fold scores), test_std (their std, NumPy's std of the array) and train_mean (the mean of the five training scores), in that order.
The Check tries the loans' five features from chapter 1 and three columns of 400 made-up loans.
- Task 3
out_of_fold.py: a guess for every loan
Write out_of_fold(X, y) in out_of_fold.py. X is a DataFrame of features and y the label. Use cross_val_predict on a DecisionTreeClassifier with max_depth=3 and random_state=0, with cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=0), so that each loan is guessed by the model fitted on the four folds without it.
Return two values separated by a comma: the NumPy array of guesses (0 or 1), one per row of X in its order, and their confusion matrix against y, the 2 by 2 NumPy array confusion_matrix gives with labels=[0, 1] (truth first).
The Check tries the loans' five features from chapter 1, emi_rs, existing_emis and age, and three columns of 400 made-up loans. A tree fitted on all the rows and asked about the same rows is not an out-of-fold guess.
- Task 4 · interview question
Interview: time_folds.py, folds in date order
Write shuffled_and_by_time(X, y) in time_folds.py. X is a DataFrame of features and y the label, both in the order the loans were applied for. Score the same DecisionTreeClassifier (max_depth=3, random_state=0) with cross_val_score and scoring="roc_auc" twice: with cv=KFold(n_splits=5, shuffle=True, random_state=0), and with cv=TimeSeriesSplit(n_splits=5). Do not shuffle or sort the rows yourself.
Return two floats separated by a comma: the mean of the shuffled folds' scores, then the mean of the time folds' scores.
The Check tries the loans' five features from chapter 1, emi_rs, existing_emis and age, and the first 1,200 loans' five features.
Tanvi will ask: why are shuffled folds optimistic for Dobara's loans, and which of the two numbers would you give Rachna?
What you will learn
- KFold: the rows cut into k folds; each fold is the test rows once.
- cross_val_score: fit and score once per fold, five scores back; report the mean and the spread.
- StratifiedKFold: folds that keep the share of defaults, what cv=5 means for a classifier, and scoring=.
- How many folds: the cost in fits and the size of each test fold.
- cross_validate for several metrics and the training scores, and cross_val_predict for a guess on every loan.
- TimeSeriesSplit and GroupKFold: when shuffled folds are wrong.
More in Learning from data
Build "Cross-validation" yourself
Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.