Machine Learning · Chapter 14 of 26
Logistic regression
By the end of this chapter you can say what LogisticRegression computes inside, turn its weighted sum into a probability by hand, read its coefficients as odds ratios, explain why it is already regularised and needs scaled columns, and say where it stands among Dobara's models. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.
- Part: Classifiers
- 4 tasks
- Python
- In the all labs plan
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.
Why you need this
Since chapter 2 you have used LogisticRegression as a black box that gives each applicant a probability of default, and since chapter 10 it has been the last step of Dobara's pipeline. It scores well. But Farida approves the loans, and a number she cannot explain is a number she cannot defend to Rachna or to a buyer who was refused. She asks plain questions: what does one more EMI do to a buyer's risk? Why is this applicant 0.08 and that one 0.02?
Chapter 11's linear regression answered questions like these with coefficients in rupees. Why not use it for defaults, with y as 0 and 1? Because its answer is a straight line, and a straight line does not stop at 0 or at 1: for 41 of 600 test loans it gives a "probability" below zero. Logistic regression keeps the weighted sum and bends it into a probability. To read it you need to know how.
What you build: 4 tasks
Each task has tests. You write the code in the lab's editor and press Check.
- Task 1
by_hand.py: the probability, by hand
Write default_probability(pipe, X) in by_hand.py. pipe is a fitted Pipeline of a StandardScaler then a LogisticRegression, fitted on loans labelled defaulted (0 or 1); X is a DataFrame of loans with the columns it was fitted on. Compute the probability of default for every row of X yourself, from what the two steps learned: scale X with the pipeline's own fitted scaler (its first step); take the weighted sum z, each scaled column times its coefficient in coef_ (two-dimensional, one row for class 1), added up, plus intercept_; then the sigmoid, 1 / (1 + e to the power -z).
Return a NumPy array of floats from 0 to 1, one per row of X, equal to the second column of predict_proba. Do not call predict_proba, predict_log_proba, decision_function or predict: the task is to do their sum yourself.
The Check tries pipelines on three and on eight loan columns (one with C=0.05) and on 300 made-up loans.
- Task 2
odds.py: odds ratios
Write odds_ratios(pipe) in odds.py. pipe is a fitted Pipeline of a StandardScaler then a LogisticRegression, fitted on a DataFrame of loans' columns.
Return a pandas Series named "odds_ratio", one value per column the pipeline was fitted on, in that order, indexed by the column names: e (np.exp) to the power of each column's coefficient. The coefficients are in log-odds, so the odds ratio is their exponential, not the coefficient itself.
The Check tries a pipeline on eight loan columns and one on 300 made-up loans.
- Task 3
compare.py: logistic regression against a tree
Write logistic_vs_tree(X, y) in compare.py. X is a DataFrame of numeric columns of the loans and y their defaulted. Split the rows with train_test_split, test_size=0.25, stratify=y and random_state=11. Fit a Pipeline of a StandardScaler then a LogisticRegression (its default settings), and a DecisionTreeClassifier(max_depth=4, random_state=0), both on the training rows only, and score each one on the test rows with roc_auc_score, from its probability of default (class 1).
Return two floats separated by a comma: the logistic regression's test AUC, then the tree's. An AUC needs probabilities, not predict's 0s and 1s.
The Check tries eight numeric columns of the loans, three of them, and 300 made-up loans.
- Task 4 · interview question
Interview: chance.py, a chance below zero
Raju wrote default_chance(X_train, y_train, X_new) to give Farida each new applicant's chance of default. It fits a LinearRegression on the 0s and 1s of defaulted, and some applicants get a "chance" below 0. Fix it so that it fits a Pipeline of a StandardScaler then a LogisticRegression (its default settings) on X_train and y_train, the training rows of loans (numeric columns, defaulted).
Return a NumPy array of floats from 0 to 1, one per row of X_new: each applicant's probability of default, the class 1 column of predict_proba. The Check tries the loans' training rows on eight columns with their test rows as the new ones, and 200 made-up loans with 100 new ones.
Tanvi will ask: why is linear regression the wrong model for defaulted?
What you will learn
- Why not linear regression for a 0 or 1 label: predictions below 0.
- The sigmoid: 1 / (1 + e⁻ᶻ) turns the weighted sum z into a probability, and decision_function gives z.
- Log-odds and odds ratios: what a coefficient means, and np.exp(coef_) as "multiplies the odds by".
- Odds are not probability: the same odds ratio moves a probability by different amounts.
- C: the penalty LogisticRegression always has, so scale first.
- max_iter and ConvergenceWarning: a real one, and its fix.
More in Classifiers
Build "Logistic regression" yourself
Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.
The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.