Machine Learning · Chapter 3 of 26

Accuracy, and the confusion matrix

By the end of this chapter you can say what a score is worth by comparing it with the simplest possible guess, count a model's four kinds of right and wrong in a confusion matrix, compute precision, recall and F1 by hand and with scikit-learn, read a classification report line by line, and turn a model's mistakes into Farida's rupees. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Learning from data
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.

Why you need this

Chapter 1 ended with a question: is 0.835 good? It sounds good. Here is the simplest model there is: say "repaid" for every loan and approve everyone. On the same 600 test loans it scores 0.84. Chapter 1's tree, which learned from 1,800 loans, is worse than approving everyone.

That cannot be the whole story, and it is not. Approving everyone never catches a single default: on those 600 loans it lets all 96 defaults through, and at Farida's ₹30,000 a default approved that is ₹28.8 lakh. Accuracy cannot see this, because it counts every right guess the same, and when 86 of every 100 loans are repaid, "repaid" is right most of the time whatever the buyer looks like. What you know so far, the share of right guesses, hides the one thing Dobara lends by: which guesses were wrong, and what each kind costs.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    baseline.py: compared to what?

    Write baseline_lead(X, y) in baseline.py. X is a DataFrame of features and y the label (defaulted). Split the rows with train_test_split, test_size=0.25, random_state=7 and stratify=y. Fit a DummyClassifier(strategy="most_frequent") and a DecisionTreeClassifier with max_depth=3 and random_state=0, both on the training rows only, and score both on the test rows.

    Return two floats separated by a comma: the baseline's test accuracy, then the tree's lead over it (the tree's test accuracy minus the baseline's, which can be small or below zero).

    The Check tries the loans' five features from chapter 1 (monthly_income_rs, emi_rs, existing_emis, down_payment_rs, age) and three columns of 400 made-up loans.

  2. Task 2

    counts.py: the four counts

    Write confusion_counts(y_true, y_pred) in counts.py. y_true is what really happened (a Series of 0 and 1) and y_pred a model's guesses (a NumPy array of 0 and 1), one per loan in the same order. A default (1) is the positive class.

    Return a dict of four whole numbers (ints) with the keys "tn" (repaid, guessed repaid), "fp" (repaid, guessed default), "fn" (default, guessed repaid) and "tp" (default, guessed default). Use confusion_matrix from sklearn.metrics with labels=[0, 1].

    The Check tries a tree's guesses on 600 test loans, Farida's rule on all the loans, 6 loans that all repaid and were all approved (no 1 anywhere, and the four counts must still be there), and 5 loans that all defaulted and were all refused.

  3. Task 3

    by_hand.py: precision, recall and F1 by hand

    Write scores_from_counts(counts) in by_hand.py. counts is a dict of four whole numbers with the keys "tn", "fp", "fn" and "tp", as Task 2 returns it. Compute with plain arithmetic, not with sklearn: precision (of the loans guessed 1, the share that really defaulted), recall (of the loans that really defaulted, the share guessed 1), and f1 (the harmonic mean of precision and recall).

    Return a dict with the keys "precision", "recall" and "f1", each a float from 0 to 1. When a division would be 0 by 0 (no loan guessed 1, or both precision and recall 0), that value is 0.0, as scikit-learn gives with zero_division=0.

    The Check gives the counts of two trees' guesses, of Farida's rule and of approving everyone, and compares with precision_score, recall_score and f1_score.

  4. Task 4 · interview question

    Interview: accuracy_lies.py, what Farida should look at

    Write report(y_true, predictions) in accuracy_lies.py. y_true is what really happened (a Series of 0 and 1); predictions is a dict from a name (such as "approve everyone") to a NumPy array of guesses, 0 or 1, one per loan of y_true. A default (1) is the positive class.

    Return a DataFrame with one row per name, labelled by the name, in the dict's order, and the columns accuracy, precision, recall and cost_rs, in that order. accuracy is the share of guesses that are right; precision the share of loans guessed 1 that defaulted (0.0 when none is guessed 1); recall the share of defaults guessed 1; cost_rs Farida's rupees: 30000 for each default approved (a false negative) plus 9000 for each good buyer refused (a false positive), as a whole number.

    The Check gives approving everyone, a tree and Farida's rule on 600 test loans, and refusing everyone on 8 hand-made loans.

    Tanvi will ask: approving everyone scores 0.86; why does accuracy lie here, and what should Farida look at instead?

What you will learn

  • DummyClassifier: the baseline, a model that ignores the features, and the score every real model must beat.
  • The confusion matrix: the four counts of right and wrong, with confusion_matrix and its labels= order.
  • Precision and recall: of the loans refused, how many would have defaulted; of the defaults, how many were caught.
  • F1: one number for both, and why it is not the plain mean.
  • Specificity and balanced accuracy, and classification_report.
  • Farida's costs: the rupees inside a confusion matrix.

More in Learning from data

Build "Accuracy, and the confusion matrix" yourself

Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.