Machine Learning · Chapter 17 of 26

Forests and boosting

By the end of this chapter you can build a forest of trees and say why it beats one tree, get a free validation score from the rows each tree left out, boost shallow trees one after another and stop them at the right round, and say honestly how these models compare with logistic regression on Dobara's loans. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Classifiers
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.

Why you need this

Chapter 16 ended on a tree's weakness: two bootstrap samples of the same loans grew two different trees. Farida cannot lend by a model that would change its mind if Dobara had happened to sign a few different buyers last year. And a single tree is not very good: with no limit it memorises (a test AUC of 0.598 on the loans), and with a depth limit it answers in a handful of steps of risk.

Rachna has heard that forests and boosting win most competitions on tables of data, and she wants to know whether Dobara should use one. Tanvi says the honest answer needs two things you do not have yet: how the many trees are combined, and a fair race between them and the logistic regression of chapter 14 on the same folds. What you know so far gives one tree and its instability. This chapter turns that instability into a strength, and then measures whether it was worth it.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    tree_vs_forest.py: one tree against a forest

    Write tree_and_forest(X, y) in tree_vs_forest.py. X is a DataFrame of number columns (blanks allowed) and y the label defaulted. Split with train_test_split, test_size=0.25, random_state=7 and stratify=y. Make a DecisionTreeClassifier(max_depth=4, random_state=0) and a RandomForestClassifier with n_estimators=50, min_samples_leaf=5 and random_state=0, and fit both on the training rows only. Score each on the test rows with roc_auc_score of its probability of default (the second column of predict_proba).

    Return four values separated by commas: the fitted tree, the fitted forest, the tree's test AUC and the forest's test AUC (floats from 0 to 1).

    The Check fits fresh copies of both on the same training rows and compares, so neither may see the test rows. It tries the loans' ten number columns and 600 made-up loans with other columns.

  2. Task 2

    out_of_bag.py: the free validation set

    Write oob_and_cv(X, y) in out_of_bag.py. X is a DataFrame of number columns and y the label defaulted; all of them are building rows, so there is no split. Make a RandomForestClassifier with n_estimators=40, min_samples_leaf=5, oob_score=True and random_state=0, and fit it on all of X and y. Its out-of-bag AUC is roc_auc_score of y and the second column of the forest's oob_decision_function_. Its cross-validated AUC is the mean of cross_val_score for a RandomForestClassifier with the same settings (n_estimators=40, min_samples_leaf=5, random_state=0; oob_score is not needed), cv=StratifiedKFold(3, shuffle=True, random_state=0) and scoring="roc_auc".

    Return three values separated by commas: the fitted forest, its out-of-bag AUC and the cross-validated AUC (one float, the mean of the three folds).

    oob_score_ is the out-of-bag accuracy, not the AUC. random_state=0 is fixed in both places so the Check's forests are the same as yours. The Check tries the loans' ten number columns and 400 made-up loans.

  3. Task 3

    boosting_rounds.py: the best boosting round

    Write best_round(X, y) in boosting_rounds.py. X is a DataFrame of number columns with no blanks and y the label defaulted. Split with train_test_split, test_size=0.25, random_state=7 and stratify=y into training rows and validation rows. Make a GradientBoostingClassifier with n_estimators=100, learning_rate=0.3, max_depth=3 and random_state=0, and fit it on the training rows only. For every round, compute the AUC on the validation rows of the model so far: staged_predict_proba of the validation rows gives the probabilities after round 1, round 2 and so on.

    Return three values separated by commas: the fitted model, the best round (a whole number, counting the first round as 1; on a tie, the earliest) and its validation AUC (a float).

    The best round is found on the validation rows, never on the training rows. The Check tries the loans' eight number columns with no blanks and 500 made-up loans.

  4. Task 4 · interview question

    Interview: deep_gap.py, deep trees alone and together

    Raju says a forest is just many overfitted trees, so it must overfit too. Write train_and_test_auc(X, y) in deep_gap.py to measure it. X is a DataFrame of number columns and y the label defaulted. Split with train_test_split, test_size=0.25, random_state=7 and stratify=y. Fit a DecisionTreeClassifier(random_state=0) and a RandomForestClassifier(n_estimators=50, random_state=0), both with no depth limit, on the training rows. For each, compute the AUC of its predict_proba (second column) on the training rows and on the test rows.

    Return a dict with exactly two keys, "tree" and "forest", each holding a tuple of two floats: the training AUC first, then the test AUC. The Check tries the loans' ten number columns and 600 made-up loans.

    Tanvi will ask: every tree in the forest memorises its rows, just like the single tree. Why does the forest do so much better on loans it never saw?

What you will learn

  • Bagging (BaggingClassifier): many trees, each on a bootstrap sample, their probabilities averaged.
  • Random forests (RandomForestClassifier): bagging plus a random handful of columns at every question (max_features).
  • Why averaging works: the trees' mistakes differ, so they cancel.
  • Out-of-bag rows (oob_score=True): each tree's left-out rows as a free validation set.
  • Boosting (GradientBoostingClassifier): shallow trees, each one fitted to the mistakes of those before, with learning_rate and n_estimators; staged_predict_proba to watch it learn.
  • HistGradientBoostingClassifier: fast boosting that accepts blanks and stops early by itself.
  • The scoreboard: all of them against logistic regression, and why the simple model can win.

More in Classifiers

Build "Forests and boosting" yourself

Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.