Machine Learning · Chapter 9 of 26

Missing values

By the end of this chapter you can find out why a value is missing before you touch it, fill blanks with SimpleImputer from the training rows only, keep the fact that a value was blank as a column of its own, and say what dropping rows does to a number you report. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Preparing the data
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.

Why you need this

572 of Dobara's 2,400 loans have no bureau score, because the buyer had never borrowed before and no credit bureau knew them. 134 have no years at the address. In bikes.csv, 195 bikes have no engine size. Give the loans with their blanks to the logistic regression of chapter 2, and it refuses:

ValueError: Input X contains NaN.
LogisticRegression does not accept missing values encoded as NaN natively. ...

(the last lines of the traceback, trimmed). The easy answers you already know from pandas are both risky. dropna throws away a quarter of the loans, and they are not a random quarter: buyers with no credit history defaulted at 21.9 %, almost twice the 11.7 % of the rest. And fillna(0) tells the model that those buyers have a score of 0, lower than anyone who ever applied. This chapter asks first why each blank is there, then fills it in a way the model can learn from, without hiding that it was blank.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    blank_rates.py: does a blank say something?

    Write blank_rates(X, y, column) in blank_rates.py. X is a DataFrame (the training rows of a split, for example), y its labels (1 defaulted, 0 repaid, one per row of X, with the same row labels) and column the name of a column of X that has blanks (NaN).

    Return two floats from 0 to 1, separated by a comma: the default rate (the mean of y) of the rows where column is blank, then the default rate of the rows where it is not blank.

    The Check tries the training rows of a split of the loans (test_size=0.25, random_state=7, stratify) with bureau_score and with years_at_address, and 300 made-up loans with a column called credit_score.

  2. Task 2

    fill_train_test.py: fill the blanks from the training rows

    Write fill_train_test(X_train, X_test) in fill_train_test.py. Both are DataFrames of number columns with some blanks, the two parts of a split. Make a SimpleImputer with strategy="median", fit it on X_train only, and transform both X_train and X_test with that one imputer.

    Return three values separated by commas: the fitted SimpleImputer, the filled training rows and the filled test rows. Both tables are DataFrames that keep the columns' names and the rows' labels (set_output(transform="pandas") does that); a NumPy array is not accepted.

    The test rows' blanks are filled with the medians of the training rows, never their own. The Check tries six and five columns of the loans, three columns of the bikes (engine_cc among them), and 80 made-up loans, and it looks at what your imputer learned.

  3. Task 3

    indicator_auc.py: a column that says "was blank"

    Write indicator_auc(X, y) in indicator_auc.py. X is a DataFrame of number columns with blanks and y the label. Split with train_test_split, test_size=0.25, random_state=7 and stratify=y. Then do this twice, first with add_indicator=False and then with add_indicator=True: fit a SimpleImputer(strategy="median", add_indicator=...) on the training rows; fit a StandardScaler on the imputed training rows; fit a LogisticRegression(max_iter=1000) on the imputed, scaled training rows; and find its ROC AUC on the test rows, imputed and scaled with the same two fitted transformers (roc_auc_score with predict_proba's column for class 1).

    Return three values separated by commas: the AUC without the indicator, the AUC with it (floats from 0 to 1), and the list of the column names that the imputer with add_indicator=True gives (its get_feature_names_out, as a list of strings).

    The Check tries six columns of the loans, five other columns of the loans, and 400 made-up loans.

  4. Task 4 · interview question

    Interview: dropped_rows.py, fewer defaults on paper

    An analyst dropped every loan with a blank anywhere and told Rachna that Dobara's default rate is lower than she thought. Fix default_rate_report(loans) in dropped_rows.py. loans is a DataFrame with the columns of loans.csv, or some of them (always defaulted). The report is about every loan, and shows what dropping the rows with a blank would hide.

    Return a dict with exactly these four keys: "loans", the number of loans, all of them (an int); "default_rate", the default rate of all of them; "dropped", the number of loans that have a blank in any column, the rows dropna would remove (an int); and "dropped_rate", the default rate of those loans. Rates are floats from 0 to 1. The Check tries all the past loans, the last 600, and 6 made-up loans.

    Tanvi will ask: why did the default rate fall, which buyers did dropping the rows hide, and what would a model trained without them do with this week's 49 applicants who have no bureau score?

What you will learn

  • What a blank does to a model: the NaN error, and trees, which accept NaN in scikit-learn 1.9.
  • Why it is missing: three kinds of blank in Dobara's data.
  • dropna and what it hides.
  • SimpleImputer: mean, median, most_frequent and constant, with statistics_ learned from the training rows.
  • What filling destroys, and add_indicator=True, which keeps it.
  • KNNImputer and IterativeImputer, in one section.

More in Preparing the data

Build "Missing values" yourself

Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.