Machine Learning · Chapter 6 of 26

Over and underfitting

By the end of this chapter you can tell a model that is too simple from one that remembers its examples, find the setting between them by sweeping it with cross-validation, read the gap between the training and the cross-validated score, run the sweep in one call with validation_curve, and say with learning_curve whether more loans would help. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Learning from data
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.

Why you need this

Chapter 1 met both failures without naming them. A tree with no depth limit scored 1.0 on the loans it learned from and 0.763 on new ones: it had remembered its examples. And chapter 3 found a depth 2 tree that mostly says "repaid" for everyone: it had learned too little to tell buyers apart. Both are bad for Dobara, for opposite reasons, and the cure for one makes the other worse.

So far you chose max_depth by habit: 2 in the stories, 3 in the tasks. Rachna will ask why 3. Without a way to measure, the only answers are "it seemed fine" or "the test score was best", and chapter 2's golden rule rules out the second. Chapter 5 gave you the tool: cross-validated scores on the training rows. This chapter uses it to choose, and to say which way a model is wrong.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    depth_sweep.py: the depth sweep

    Write depth_sweep(X, y, depths) in depth_sweep.py. X is a DataFrame of features, y the label (defaulted) and depths a list of whole numbers, not always in order. For each depth, in the list's order, run cross_validate on a DecisionTreeClassifier with that max_depth and random_state=0, with cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=0), scoring="roc_auc" and return_train_score=True.

    Return a DataFrame with one row per depth, in the list's order, and the columns depth (the max_depth), train (the mean of the five training scores) and cv (the mean of the five held-out fold scores), in that order.

    The Check tries the loans' five features from chapter 1 (monthly_income_rs, emi_rs, existing_emis, down_payment_rs, age) with the depths [1, 3, 6, 4], and three columns of 400 made-up loans with the depths [5, 2].

  2. Task 2

    best_depth.py: the best depth, and its gap

    Write best_depth(sweep) in best_depth.py. sweep is a DataFrame like the one Task 1 returns: the columns depth, train and cv, one row per depth.

    Return two values separated by a comma: the depth of the row with the highest cv score, as a whole number, and that row's gap, its train score minus its cv score, as a float. Choose by the cv score, not the training score; when two rows tie for the highest cv score, take the first in the table's order.

    The Check tries a hand-made sweep of depths 1 to 6, a hand-made sweep with a tie, and a real sweep of the loans' five features.

  3. Task 3

    more_loans.py: would more loans help?

    Write learning_table(X, y, sizes) in more_loans.py. X is a DataFrame of features, y the label and sizes a list of training sizes, either shares of a training fold (such as 0.5) or numbers of rows (such as 200). Run learning_curve on a DecisionTreeClassifier with max_depth=4 and random_state=0, with train_sizes=sizes, cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=0) and scoring="roc_auc".

    Return a DataFrame with one row per size and the columns size (the number of training rows learning_curve used, which it gives back), train (the mean training score at that size) and cv (the mean held-out score at that size), in that order.

    The Check tries the loans' five features from chapter 1 with the sizes [0.2, 0.5, 1.0], and three columns of 400 made-up loans with the sizes [100, 200, 320].

  4. Task 4 · interview question

    Interview: diagnose.py, which one, and why?

    Write two functions in diagnose.py. train_and_test(X, y) splits X and y with train_test_split, test_size=0.25, random_state=7 and stratify=y, fits a DecisionTreeClassifier with max_depth=1 (the "stump") and one with no depth limit (the "deep" tree), both with random_state=0, on the training rows only, and returns a dict with the keys "stump" and "deep", each a tuple of two accuracies from the tree's own score: on the training rows first, then on the test rows.

    diagnose(scores) is given a dict like that, a name to a tuple of (training score, test score), with two or more names, and returns two names separated by a comma: the one that overfits, with the largest training score minus test score, then the one that underfits, with the lowest training score. A low test score alone does not mean underfit.

    The Check tries train_and_test on the loans' five features from chapter 1 and three columns of 400 made-up loans, and diagnose on three hand-made dicts.

    Tanvi will ask: which tree overfits, which underfits, and what would you change in each?

What you will learn

  • Underfitting and overfitting: too simple to see the pattern, or so flexible that it learns the noise.
  • The sweep: one setting from small to large, the training score against the cross-validated score.
  • The gap as the diagnosis: both low means underfit; high train and low cv means overfit.
  • Every model has a dial, and its name.
  • validation_curve: the sweep in one call, and its arrays' shapes.
  • learning_curve: the scores as the training rows grow, and whether more loans would help.

More in Learning from data

Build "Over and underfitting" yourself

Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.