Machine Learning · Chapter 15 of 26
Decision boundaries, and k nearest neighbours
By the end of this chapter you can draw where a classifier changes its mind on two features, as a map of 0s and 1s made with a grid of points, say why logistic regression, a tree and kNN draw different shapes, explain what kNN does inside, and set its dial, k, honestly. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Classifiers
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.
Why you need this
Farida thinks about applicants in two numbers before any other: what the buyer earns and what the EMI will be. Somewhere on that page of incomes and EMIs is a line: on one side Dobara lends, on the other it refuses. Every model you have trained draws such a line, but you have only ever seen its scores. Rachna wants to see the line itself: is it a sensible shape? Does it refuse a buyer on ₹54,000 a month with a ₹13,000 EMI and approve one on ₹48,000 with the same EMI, for no reason anyone can give?
A score cannot show that. Printing the predictions for the 600 test loans does not either, because they sit where the past buyers happened to be. You need the model's answer everywhere on the page, and that is a grid. The map also explains the third model of chapter 2, kNN, which until now has been a black box: its line is the strangest, and the reason is how it decides.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
grid_map.py: a map of a model's answers
Write two functions in grid_map.py. model is a fitted classifier (a pipeline is fine) that learned from a DataFrame with exactly two columns, monthly_income_rs then emi_rs, and predicts 0 or 1; incomes and emis are 1-D NumPy arrays of values to try.
grid_map(model, incomes, emis) makes every pair of an income and an EMI with np.meshgrid(incomes, emis), predicts each pair, and returns a 2-D NumPy array of the predictions of shape (len(emis), len(incomes)): row i is emis[i], column j is incomes[j]. The model learned from a DataFrame with two named columns, so give predict a DataFrame of those columns with one row per point, not the grids themselves.
default_share(model, incomes, emis) returns the share of the grid's points the model calls a default (1), as a float from 0 to 1.
The Check tries a scaled logistic regression, a depth 3 tree and a scaled kNN with k=9, each on a grid of 13 incomes by 14 EMIs and on one of 3 by 2.
- Task 2
k_sweep.py: the k dial
Write auc_by_k(X, y, ks) in k_sweep.py. X is a DataFrame of numeric columns of the loans, y their defaulted, and ks a list of whole numbers. Split the rows once with train_test_split, test_size=0.25, stratify=y and random_state=3. For each k, in the list's order, fit a Pipeline of a StandardScaler then a KNeighborsClassifier with n_neighbors=k (its other settings at their defaults) on the training rows, and measure roc_auc_score on the training rows and on the test rows, from the probability of default (class 1), not from predict's 0s and 1s.
Return a DataFrame with one row per k, in the list's order, and exactly three columns: "k", "train_auc" and "test_auc" (floats, not rounded).
The Check tries income and EMI with ks [1, 15, 101], and five loan columns with ks [51, 5].
- Task 3
neighbours.py: the applicant's nearest past loans
Write nearest_loans(X_train, y_train, applicant, k) in neighbours.py. X_train and y_train are training rows of the loans (numeric columns, defaulted), applicant a DataFrame of one row with the same columns, and k a whole number. Fit a Pipeline of a StandardScaler then a KNeighborsClassifier with n_neighbors=k on X_train and y_train. Scale the applicant with the pipeline's own fitted scaler, and ask the KNeighborsClassifier step for its k nearest training rows with its kneighbors method.
Return two values separated by a comma: a list of the k nearest loans' row labels (X_train's index, nearest first), and the share of them that defaulted (a float from 0 to 1, equal to the pipeline's predict_proba for class 1). kneighbors gives positions (0, 1, 2 ... in X_train's order), not row labels.
The Check tries five loan columns with two test loans (k=5 and k=12), and income and EMI with an applicant on ₹16,000 and an EMI of ₹5,200 (k=9).
- Task 4 · interview question
Interview: k_one.py, a perfect neighbour
Raju tried a kNN with n_neighbors=1 and told Rachna it is right about 99 % of the time. His knn_scores(X, y) gives the same high number twice. X is a DataFrame of numeric columns of the loans and y their defaulted. Fix it so that it splits the rows with train_test_split, test_size=0.25, stratify=y and random_state=5, and fits a Pipeline of a StandardScaler then a KNeighborsClassifier(n_neighbors=1) on the training rows only.
Return three floats separated by commas: its accuracy on the training rows, its accuracy on the test rows, and its roc_auc_score on the test rows (from the probability of class 1). The Check tries income and EMI, and five loan columns.
Tanvi will ask: why does k=1 score almost 1.0 on its training rows, and why not exactly 1.0?
What you will learn
- A decision boundary: where the predicted class changes, found by predicting every point of a grid made with np.meshgrid.
- The shapes a model can draw: a straight line for logistic regression, steps for a tree, anything for kNN.
- How kNN works: store the training rows; to predict, find the k nearest by distance and take the vote. kneighbors shows them.
- Fitting is storing: fast fit, slow predict, measured with %timeit.
- The dial runs backwards: k=1 memorises, a large k is too smooth; weights="distance".
- What kNN needs: scaled columns, and not too many of them.
More in Classifiers
Build "Decision boundaries, and k nearest neighbours" yourself
Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.