Machine Learning · Chapter 4 of 26
Thresholds, ROC and PR
By the end of this chapter you can move a model's cut away from 0.5 and say what it does to precision and recall, choose the cut that costs Dobara the least on rows kept for that purpose, read a ROC curve and its AUC, read a precision-recall curve and its average precision, and explain why, with a rare class, the second tells more. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Learning from data
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.
Why you need this
Chapter 3 ended with Farida's rupees: a default approved costs ₹30,000 and a good buyer refused ₹9,000. A model that refuses a few more buyers and catches a few more defaults can save money even when its accuracy falls. But predict gives no choice: it refuses a loan only when the model thinks a default is more likely than not. For a logistic regression on Dobara's numeric columns that happens for 1 loan in 480, and that one is a good buyer: all 68 defaults among the 480 are approved, ₹20.4 lakh of Farida's money.
What you know so far gives one confusion matrix per model, at a cut you did not choose. Farida needs the whole range: at each possible cut, how many defaults caught and how many good buyers refused, and which cut costs least. Rachna needs one number to compare models that does not depend on a cut at all, and to know when that number flatters.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
at_threshold.py: guesses at a threshold
Write two functions in at_threshold.py. proba is a NumPy array of a model's probabilities of default (floats from 0 to 1), one per loan; threshold is a float; y_true is what really happened, a Series of 0 and 1, one per loan of proba.
guesses_at(proba, threshold) returns a NumPy array of whole numbers (0 and 1, not True and False), one per loan: 1 where the probability is the threshold or more, 0 where it is less. A probability exactly equal to the threshold gets 1.
precision_recall_at(y_true, proba, threshold) returns two floats from 0 to 1, separated by a comma: the precision and then the recall of those guesses, a default (1) being the positive class. When no loan is guessed 1 the precision is 0.0.
The Check tries a tree's and a logistic regression's probabilities on 600 test loans, and 8 hand-made probabilities with one exactly on the threshold 0.3 and a threshold of 0.95 that none reaches.
- Task 2
cheapest_threshold.py: the cheapest threshold
Write cheapest_threshold(y_true, proba, thresholds) in cheapest_threshold.py. y_true is what really happened on the validation rows (a Series of 0 and 1), proba a NumPy array of the model's probabilities of default on the same rows, and thresholds a list of floats, not always in order.
For each threshold, guess 1 where the probability is the threshold or more and 0 elsewhere, and price the guesses with Farida's costs: 30000 rupees for each default approved (a false negative) plus 9000 for each good buyer refused (a false positive).
Return two values separated by a comma: the threshold with the smallest cost (one of the list's values) and that cost in rupees, as a whole number. When two thresholds cost the same, return the first of them in the list's order.
The Check tries a tree's and a logistic regression's probabilities on 600 loans, and 8 hand-made probabilities where two thresholds tie.
- Task 3
auc_by_pairs.py: the AUC by counting pairs
Write auc_by_pairs(y_true, scores) in auc_by_pairs.py. y_true is what really happened (a Series of 0 and 1, both present) and scores a NumPy array of a model's scores, one per loan.
Pair every default with every repaid loan. A pair is a win when the default's score is higher, half a win when the two scores are equal, and nothing when it is lower. The AUC is the wins divided by the number of pairs.
Return two floats separated by a comma: your AUC from counting pairs, then roc_auc_score's for the same loans (the two should agree). The Check tries 5 and 8 hand-made scores with ties, and a tree's and a logistic regression's probabilities on 600 loans.
- Task 4 · interview question
Interview: rare_class.py, a fine-looking AUC
Write roc_and_pr(y_true, proba, min_recall) in rare_class.py. y_true is what really happened (a Series of 0 and 1), proba a NumPy array of the model's probabilities of default, and min_recall a float, the share of defaults Farida wants caught.
Return a dict of three floats from 0 to 1: "auc", the roc_auc_score of the loans; "average_precision", their average_precision_score; and "best_precision", the highest precision among the points of precision_recall_curve whose recall is min_recall or more.
The Check tries a tree's probabilities on 600 loans with a recall of 0.8, a logistic regression's with 0.5, and 8 hand-made probabilities with a recall of exactly 2 / 3, which is one of the curve's points.
Tanvi will ask: with defaults this rare, why does the ROC look good when the precision at a useful recall does not?
What you will learn
- The threshold: predict is predict_proba cut at 0.5; any other cut is a comparison you write.
- Choosing the cut from costs, on validation rows, never the test rows.
- roc_curve and roc_auc_score: every cut at once, and the chance a default is ranked above a repaid loan.
- precision_recall_curve and average_precision_score: the same cuts seen from the refused loans' side.
- Why PR tells more when the positive class is rare.
- FixedThresholdClassifier and TunedThresholdClassifierCV: the cut kept inside the model.
More in Learning from data
Build "Thresholds, ROC and PR" yourself
Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.