Machine Learning · Chapter 19 of 26
Search, and the honest estimate
By the end of this chapter you can let scikit-learn try every combination of settings with cross-validation, count what it costs, read its results table, search wide spaces at random, and give Rachna a score that the search did not flatter, from nested cross-validation. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Tuning
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.
Why you need this
Every chapter since the fifth has a dial: a tree's depth and leaf size, a forest's leaves, boosting's rounds, logistic regression's C. So far you turned them one at a time, with a loop over cross_val_score and an eye on the printout. Tanvi wants the search done properly: every combination, the same folds for each, the scores kept in one table, and the winner refitted at the end.
Rachna has a sharper question. Raju tuned a tree on random numbers, by mistake, and the search reported an AUC of 0.60 for columns that know nothing about defaults. If a search can "find" skill where there is none, how much of the number it reports for Dobara's loans is real?
A loop of cross_val_score can choose settings, but the score of the winner is the best of many noisy scores, picked because it was high. Reporting it is like announcing the highest of twenty dice rolls as the die's average. This chapter automates the search and then measures how much the choosing inflated the score.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
grid.py: a grid search
Write tree_search(X, y) in grid.py. X is a DataFrame of number columns (blanks allowed) and y the label defaulted; they are all building rows, so there is no split. Make a GridSearchCV of DecisionTreeClassifier(random_state=0) with exactly the grid {"max_depth": [2, 3, 4, 5], "min_samples_leaf": [10, 50, 100]}, cv=StratifiedKFold(3, shuffle=True, random_state=0) and scoring="roc_auc", and fit it on all of X and y.
Return three values separated by commas: the fitted search, its best settings (the dict best_params_ gives) and its best mean cross-validated AUC (a float). The Check runs the same search itself and compares every candidate's mean score, on the loans' ten number columns and 600 made-up loans.
- Task 2
fit_count.py: count the fits
Write count_fits(param_grid, n_folds) in fit_count.py. param_grid is a grid as GridSearchCV takes it: either a dict of setting names to lists of values, or a list of such dicts, each a grid of its own whose candidates are added to the others'. n_folds is the number of folds. A search fits every candidate once per fold and then fits the best one once more on all the rows.
Return the number of fits as a whole number. {"max_depth": [2, 3], "min_samples_leaf": [5, 20, 50]} with 5 folds is 2 × 3 = 6 candidates, so 6 × 5 + 1 = 31. The Check tries dicts of one to three settings, a setting with one value, and a list of two dicts.
- Task 3
nested.py: nested cross-validation
Write nested_scores(X, y) in nested.py. X is a DataFrame of number columns (blanks allowed) and y the label defaulted. The model is make_pipeline of SimpleImputer(strategy="median", add_indicator=True), StandardScaler() and LogisticRegression(max_iter=1000). The inner search is a GridSearchCV of that pipeline with the grid {"logisticregression__C": [0.01, 0.1, 1, 10]}, cv=StratifiedKFold(3, shuffle=True, random_state=0) and scoring="roc_auc". The outer loop is cross_val_score of the unfitted search itself on X and y, with cv=StratifiedKFold(3, shuffle=True, random_state=1) and scoring="roc_auc".
Return the three outer scores, as the NumPy array cross_val_score gives. The search's best_score_ is not it. The Check tries the loans' ten number columns and 500 made-up loans.
- Task 4 · interview question
Interview: optimism.py, the number for Rachna
Raju ran a RandomizedSearchCV of DecisionTreeClassifier(random_state=0) over {"max_depth": randint(1, 10), "min_samples_leaf": randint(1, 100)} with n_iter=10, cv=StratifiedKFold(3, shuffle=True, random_state=0), scoring="roc_auc" and random_state=0, and reported its best_score_ to Rachna as "what the model will do". His report(X, y) in optimism.py gives that number twice. Keep his search exactly as it is and fix report so that it returns two floats separated by a comma: the search's best_score_ after fitting it on X and y, then the honest estimate, the mean of cross_val_score of the same search with cv=StratifiedKFold(3, shuffle=True, random_state=1) and scoring="roc_auc".
The Check tries the first 1,200 loans' ten number columns and five columns of random numbers with a random label.
Tanvi will ask: on random numbers the search still finds an AUC above 0.5. Why is best_score_ optimistic, and which number do you give Rachna?
What you will learn
- GridSearchCV: every combination in param_grid, scored with cv= and scoring=, then the best refitted; best_params_, best_score_, best_estimator_.
- Counting the fits: candidates times folds, plus one refit.
- cv_results_: the whole search as a table.
- scoring decides the winner: accuracy and AUC pick different trees.
- RandomizedSearchCV: a fixed number of draws from distributions such as loguniform.
- Nested cross-validation: an outer loop around the search, for the honest number.
More in Tuning
Build "Search, and the honest estimate" yourself
Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.