Machine Learning · Chapter 16 of 26
Decision trees, from inside
By the end of this chapter you can say how a decision tree chooses each question: work out a node's Gini impurity by hand, score a split by the impurity of its two halves, find the best threshold on a column by trying them all, read a fitted tree and its leaves, turn its dials, and say what trees are good and bad at. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Classifiers
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.
Why you need this
Dobara's first model, in chapter 1, was a decision tree, and it found "income of ₹15,750 or less" by itself. Since then you have used trees as a model that just works: fit, predict_proba, a score. Farida now wants more than a score before she lends by one. A tree is a list of questions, so she wants to know why it asks those questions, why 15,750 and not 16,000, and why the same tree trained on a slightly different year of loans asks different ones.
Rachna has a second worry. Of the 600 test loans in this chapter, a depth-3 tree calls 31 a default, yet 85 of them defaulted. If most of its answers are "repaid", what is it good for?
Until now you could only read a tree's printout. You could not say how it was built, so you could not say which of its numbers to trust. In this chapter you open it: the arithmetic fit does at every node is small enough to do by hand, and once you have done it, the printout, the leaves, predict_proba and the settings all make sense.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
gini.py: Gini impurity, by hand
Write two functions in gini.py.
gini(labels) is given the class labels of one pile of rows, as a pandas Series or a NumPy array (0 and 1, or text such as occupations, or more than two classes). It returns the pile's Gini impurity as one float: 1 minus the sum, over the classes present, of each class's share of the pile, squared. A pure pile gives 0, and [0, 0, 1, 1] gives 0.5.
split_gini(left, right) is given the labels of the two halves of a split (Series or arrays). It returns their Gini impurities averaged with weights: each half's Gini times its number of labels, added, divided by the number of labels in both. The plain mean of the two is not it.
The Check tries the last 300 loans' defaulted, small arrays, the first 500 loans' occupations, made-up labels of three classes, and splits of the loans by age and by EMI.
- Task 2
best_split.py: the best threshold on one column
Write best_threshold(x, y) in best_split.py. x is one column of numbers as a Series with no blanks, and y the labels (0 or 1), one per value of x. Find the cut a DecisionTreeClassifier with max_depth=1 chooses on that column alone: the candidates are the midpoints between neighbouring distinct values of x in sorted order (the values 3, 5 and 9 give the candidates 4 and 7); each candidate sends the rows with x less than or equal to it to the left half and the others to the right half; score each by the weighted Gini of its two halves, as in Task 1; keep the candidate with the lowest score.
Return two values separated by a comma: the best cut (a float) and its weighted Gini (a float). If two cuts tie, either is accepted. A value of the column itself is not a midpoint and is not accepted.
The Check tries the loans' age, the loans' emi_rs, the last 600 loans' down_payment_rs and 80 made-up loans' bike prices, and checks your cut against the tree's.
- Task 3
big_leaves.py: big leaves, and what they hold
Write leaf_table(X, y) in big_leaves.py. X is a DataFrame of number columns (blanks allowed, since trees accept NaN) and y the label defaulted, with the same row labels. Split with train_test_split, test_size=0.25, random_state=7 and stratify=y. Make a DecisionTreeClassifier with min_samples_leaf=100 and random_state=0 and no depth limit, and fit it on the training rows only. Then find the leaf each training row lands in, with the tree's apply method, and for each leaf count its training rows and the share of them with defaulted 1.
Return two values separated by a comma: the fitted tree, and a DataFrame with one row per leaf and exactly the columns leaf (the leaf's number as apply gives it), loans (how many training rows it holds) and default_share (the share of them that defaulted, from 0 to 1), sorted by default_share from highest to lowest (a tie by leaf, smallest first), with row labels 0, 1, 2 and so on.
Count the training rows only, never the test rows. The Check tries the loans' ten number columns, four of them, and 1,200 made-up loans with other columns.
- Task 4 · interview question
Interview: unstable.py, two trees from one set of loans
Farida retrained the tree on a slightly different year of loans and it asked different questions. Write two_trees(X, y) in unstable.py to show her how much a tree changes. X is a DataFrame of number columns and y the label defaulted, with the same row labels. For random_state 1 and then 2: draw a bootstrap sample of the rows with X's sample method, frac=1, replace=True and that random_state, take the labels of the same rows from y, and fit a DecisionTreeClassifier(max_depth=3, random_state=0) on that sample.
Return three values separated by commas: the tree of random_state=1's sample, the tree of random_state=2's sample, and the share of the rows of X (a float from 0 to 1) whose predicted probability of default (the second column of predict_proba) differs by more than 0.1 between the two trees. The Check tries the loans' ten number columns and 400 made-up loans.
Tanvi will ask: both trees learned from the same loans, so why do they disagree, and what would you do about it?
What you will learn
- Gini impurity: one number for how mixed a pile of loans is, 0 when pure.
- The weighted impurity of a split: how a tree scores a question.
- The best threshold: every cut between neighbouring values, tried one by one; the tree keeps the best.
- Reading a tree: export_text, the leaves, apply, and why predict_proba matters more than predict.
- The dials: max_depth, min_samples_leaf, max_leaf_nodes, ccp_alpha.
- feature_importances_: the impurity each column removed.
- What trees are good and bad at: no scaling, NaN accepted, readable; unstable, steps, no extrapolation.
More in Classifiers
Build "Decision trees, from inside" yourself
Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.