Machine Learning · Chapter 23 of 26

Four classic datasets, four honest studies

By the end of this chapter you can load scikit-learn's bundled datasets, read what each one holds before you model it, and run a short honest study on each: the floor, a fair split, two or three models, the right metric for the class that matters, and the number reported with its spread and in its own units. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Real work
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.

Why you need this

Dobara's loans are one problem: two classes, one of them rare, money on each mistake. Rachna is hiring a second data scientist, and Tanvi wants every candidate, and you, to show the same routine on problems of other shapes before touching Dobara's data again: one where everything works, one where the label's order is a trap, one with ten classes, and a small regression that cannot be predicted well.

The problem is not the models: you know enough of them. It is the habits. On a new dataset it is easy to report the first number that prints: a score with no floor beside it, a recall of the wrong class, an R² from one lucky split, an accuracy that hides which classes it confuses. scikit-learn ships four small datasets that everyone knows, so a study on them can be checked against what is already known.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    iris_study.py: iris, and the flowers it gets wrong

    Write iris_study(X, y) in iris_study.py. X is a DataFrame of measurements and y the class of each row (whole numbers), as load_iris(as_frame=True) gives them in .data and .target. Split with train_test_split, test_size=0.3, random_state=9 and stratify=y. Fit a LogisticRegression(max_iter=1000) on the training rows, and predict the test rows.

    Return three values separated by a comma: the fitted model, its accuracy on the test rows (a float from 0 to 1), and a list of the row labels (X's own index labels, not positions) of the test rows whose prediction differs from their class, in the order the test rows come. The Check tries iris and iris's two sepal columns alone, where the model gets more wrong.

  2. Task 2

    malignant.py: the recall of malignant cases

    In load_breast_cancer, label 0 is malignant and 1 benign (target_names), so the class that matters is 0. Write malignant_recall(X, y) in malignant.py. X is a DataFrame of measurements and y the labels, 0 or 1, as load_breast_cancer(as_frame=True) gives them in .data and .target.

    Split with train_test_split, test_size=0.25, random_state=0 and stratify=y. Make a Pipeline of a StandardScaler then a LogisticRegression(max_iter=1000) and fit it on the training rows. Predict the test rows and measure the recall of label 0: of the test rows that are 0, the share the model predicts as 0.

    Return two values separated by a comma: the fitted pipeline, and that recall (a float from 0 to 1). The Check also tries the ten "mean" columns alone and a made-up table whose label 0 is the rarer class.

  3. Task 3

    digits_race.py: kNN against logistic regression on digits

    Write knn_against_logistic(X, y) in digits_race.py. X is a DataFrame of 64 pixel columns (an image of 8 by 8 pixels, flattened) and y the digit each row shows, 0 to 9, as load_digits(as_frame=True) gives them in .data and .target.

    Split with train_test_split, test_size=0.25, random_state=2 and stratify=y. On the training rows, fit a KNeighborsClassifier with its default settings, and a Pipeline of a StandardScaler then a LogisticRegression(max_iter=1000). Score both on the test rows (accuracy).

    Return a dict with exactly two keys, "knn" and "logistic", each the model's test accuracy as a float from 0 to 1. The Check tries all the digits and the first 900.

  4. Task 4 · interview question

    Interview: wrong_class.py, 0.97 for the wrong class

    Raju wants every malignant case caught, so he flags a test row as malignant when the model's probability of malignant is at least a cut. At a cut of 0.5 his malignant_caught says 0.97, but the confusion matrix shows more malignant cases missed than that.

    malignant_caught(model, X_test, y_test, cut) in wrong_class.py is given a fitted classifier on breast cancer (label 0 malignant, 1 benign), the test rows, their labels and a cut from 0 to 1. Fix it so it takes the probability of label 0 (the column of predict_proba that the model's classes_ gives for 0), flags the rows whose probability is at least cut, and returns the share of the malignant test rows (label 0) that are flagged, one float from 0 to 1. The Check tries cuts of 0.5, 0.2 and 0.05 on two models of its own.

    Tanvi will ask: which class was 0.97 the recall of, and what do you check first on any new dataset before you trust a recall?

What you will learn

  • sklearn.datasets: load_* gives a dataset bundled with scikit-learn, as_frame=True as pandas; fetch_* needs the internet.
  • iris: three classes almost apart, so every model scores well, and what that teaches.
  • breast cancer: label 0 is malignant, so a metric's default class is the wrong one.
  • digits: ten classes, a floor of 0.1, kNN at its best, a 10 by 10 confusion matrix.
  • diabetes: a regression with a ceiling, reported with its spread and in its own units.

More in Real work

Build "Four classic datasets, four honest studies" yourself

Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.