Machine Learning · Chapter 13 of 26

Polynomials, bias and variance

By the end of this chapter you can bend a linear model with squares and products of its columns, use interactions to let one column's effect depend on another, measure the cost of going too far, and say what bias and variance are, with a number for each. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Regression
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.

Why you need this

Chapter 11's price line was right on average and wrong in a pattern: it priced young and old bikes too low and middle-aged ones too high, and it missed new cruisers by ₹50,000 and more. The reason was plain: a bike loses a share of its value each year, so the rupees it loses are not the same every year, and not the same for a ₹2 lakh cruiser as for a ₹30,000 scooter. A straight line cannot say either thing.

Raju knows both from the showroom floor, and Rachna would like the model to know them too. You could write the curve by hand, a new column for age squared, another for age times "is a cruiser", and so on for every pair. With 12 columns there are 78 pairs and squares, and you would still be guessing which ones matter. scikit-learn makes all of them in one step. The catch is that a model with hundreds of columns can learn the training bikes too well, and this chapter measures where that starts.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    expand.py: squares and products

    Write expand(X, degree) in expand.py. X is a DataFrame of numeric columns and degree a whole number. Make a PolynomialFeatures with this degree and include_bias=False (no column of 1s), and use it to turn X into every power and product of its columns up to that degree.

    Return a DataFrame with one column per new feature, named as the transformer's get_feature_names_out names them ("age_years^2", "age_years owners", ...), in its order, with X's own row labels. The transformer gives back a NumPy array, which has no column names and no row labels.

    The Check tries age_years and owners of all the bikes at degree 2, three numeric columns of the last 7 bikes (row labels from 1593) at degree 3, and two made-up columns at degree 4.

  2. Task 2

    degrees.py: the cost of a higher degree

    Write degree_scores(X, y, degrees) in degrees.py. X is a DataFrame of numeric columns of the bikes, y their sold_price_rs, and degrees a list of whole numbers. Split the rows once with train_test_split, test_size=0.25 and random_state=6. For each degree, in the list's order, make a Pipeline of a StandardScaler, then a PolynomialFeatures with that degree and include_bias=False, then a LinearRegression (in that order, the scaler first), fit it on the training rows, and score it (R²) on the training rows and on the test rows.

    Return a DataFrame with one row per degree, in the list's order, and exactly three columns: "degree", "train_r2" and "test_r2" (floats, not rounded).

    The Check tries the bikes' age_years, km_driven, owners and segment one-hot encoded (seven columns) with degrees [1, 2, 3, 4] and [2], and 200 made-up bikes.

  3. Task 3

    bootstrap.py: how much one bike's price wobbles

    Write price_spread(X_train, y_train, bike, degree, n_fits, seed) in bootstrap.py. X_train and y_train are training rows of the bikes, bike a DataFrame of one row with the same columns, and degree, n_fits and seed whole numbers.

    Make one generator, np.random.default_rng(seed), before any fit. Then n_fits times: draw len(X_train) row positions with the generator's integers method, from 0 up to (not including) len(X_train), one call per fit; fit a fresh Pipeline of a StandardScaler, a PolynomialFeatures with this degree and include_bias=False, and a LinearRegression on those rows, taken by position (so a row can come twice); and predict the bike's price.

    Return two floats separated by a comma: the mean of the n_fits predicted prices, and their standard deviation (NumPy's std, as it is by default). The Check tries degrees 1, 2 and 3, on the bikes' seven columns of Task 2 and on made-up bikes.

  4. Task 4 · interview question

    Interview: pick_degree.py, Raju's best degree

    Raju tried degrees 1 to 4 on the bikes and picked the one with the highest R² on the rows it learned from. It is always the highest degree he tries, and on the test rows that model does worst. Fix pick_degree(X, y, degrees) in pick_degree.py so that it splits the rows with train_test_split, test_size=0.25 and random_state=6; for each degree makes the Pipeline of Task 2 (a StandardScaler, a PolynomialFeatures with that degree and include_bias=False, a LinearRegression) and scores it with cross_val_score on the training rows, cv=5, taking the mean of the five R² scores; picks the degree with the highest mean (the first in the list if two tie); fits its pipeline on all the training rows; and scores it on the test rows.

    Return two values separated by a comma: the degree it picked (a whole number) and that model's test R² (a float). The Check tries the bikes' seven columns of Task 2 with degrees [1, 2, 3, 4], and 200 made-up bikes.

    Tanvi will ask: why does the training R² keep rising with the degree while the test R² does not?

What you will learn

  • PolynomialFeatures: every square and product of the columns, up to a degree; include_bias, interaction_only and get_feature_names_out.
  • Interactions: a product column lets one feature's effect depend on another, such as age by segment.
  • The cost of a degree: the training R² always rises, the test R² does not.
  • Bias and variance: wrong the same way every time, or changing with the sample, measured on bootstrap refits.
  • Rescuing a high degree with Ridge, and judging it honestly with cross-validation.

More in Regression

Build "Polynomials, bias and variance" yourself

Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.