Machine Learning · Chapter 21 of 26

Clustering, groups without a label

By the end of this chapter you can group Dobara's buyers with k-means, say what it does at every step, scale the columns first and read the centres back in rupees, choose k with inertia and the silhouette, show that k-means finds "groups" even in random numbers, and check clusters against something they were not built from. Build it in your browser: a real editor and terminal, AI that explains, and tests that tell you when it works.

  • Part: Tuning
  • 4 tasks
  • Python
  • In the all labs plan

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.

Why you need this

Rachna wants to design offers: a lower down payment for one kind of buyer, a longer tenure for another, a bigger bike for a third. She asks Tanvi for segments: groups of buyers who look alike. Nobody has labelled the buyers with a segment, so there is no y to learn from.

Every model so far was supervised: fit took X and y and learned to predict y. Here there is only X: each buyer's income, age, the price of the bike and the share paid down. You could invent groups by hand ("under 25 and under ₹15,000"), but then the groups are your guess, not the data's. Clustering looks for the groups in the data itself. It always finds some, which is exactly the danger: Raju clustered the buyers last month, gave the four groups names, and nobody checked whether they were real.

What you build: 4 tasks

Each task has tests. You write the code in the lab's editor and press Check.

  1. Task 1

    segments.py: k-means on scaled buyers

    Write kmeans_segments(buyers, k) in segments.py. buyers is a DataFrame of number columns with no blanks, one row per buyer, and k a whole number. Fit a StandardScaler on all the buyers and scale them; fit KMeans(n_clusters=k, n_init=10, random_state=0) on the scaled table; turn its cluster_centers_ back into the buyers' own units with the scaler's inverse_transform.

    Return two values separated by a comma: the cluster of each buyer (the NumPy array labels_ gives, one per row), and a DataFrame of the centres with the buyers' column names, one row per cluster in the clusters' order (row labels 0 to k − 1). The Check tries all 2,400 buyers with k=3 (income, age, bike price and the share paid down), the first 1,200 with k=5, and 300 made-up buyers in other columns.

  2. Task 2

    silhouettes.py: the silhouette for each k

    Write silhouette_by_k(buyers, ks) in silhouettes.py. buyers is a DataFrame of number columns with no blanks, and ks a list of whole numbers. Scale the buyers with a StandardScaler fitted on them; then for each k in ks, take the labels of KMeans(n_clusters=k, n_init=10, random_state=0) fitted on the scaled table, and the silhouette_score of the scaled table and those labels.

    Return a pandas Series of the silhouettes (floats from −1 to 1), one per k, labelled by k, in the order of ks. Score on the same scaled table the clusters were found in. The Check tries the first 1,200 buyers with ks [2, 3, 4, 5] and 300 made-up buyers.

  3. Task 3

    cluster_rates.py: check the clusters against something else

    Write cluster_table(loans, labels) in cluster_rates.py. loans is a DataFrame with the columns of loans.csv (any row labels) and labels a NumPy array of one cluster number per row of loans, in the same order. Return a DataFrame with one row per cluster and exactly the columns cluster (the cluster number), buyers (how many loans are in it) and default_rate (the share of them with defaulted 1, from 0 to 1), sorted by cluster from the smallest, with row labels 0, 1, 2 and so on. Do not change loans.

    The Check tries all the loans with k-means labels, the first 500 loans with made-up labels 0 to 4, and the last 50 loans with the labels 7 and 2.

  4. Task 4 · interview question

    Interview: noise.py, clusters in random numbers

    Raju clustered the buyers and named the "segments". Write noise_and_buyers(buyers, k, seed) in noise.py to compare them with clusters in numbers that mean nothing. buyers is a DataFrame of number columns with no blanks. Make random numbers of the same shape as buyers with np.random.default_rng(seed) and its random method. Then, for the random numbers and for the buyers, each on its own: scale with a StandardScaler fitted on it, fit KMeans(n_clusters=k, n_init=10, random_state=0), and compute the silhouette_score of the scaled table and the labels.

    Return two floats separated by a comma: the random numbers' silhouette first, then the buyers'. The Check tries all 2,400 buyers with k=3 and seed 7, and the first 800 with k=4 and seed 11.

    Tanvi will ask: k-means found clusters in the random numbers too. So what does a cluster prove, and what would you check before Rachna builds offers on them?

What you will learn

  • Learning without a label: clustering finds groups from X alone.
  • KMeans: assign each buyer to the nearest centre, move each centre to its buyers' mean, repeat; labels_, cluster_centers_, inertia_; n_init and random_state.
  • Scale first, and read the centres back in rupees with inverse_transform.
  • Choosing k: inertia for each k, and silhouette_score.
  • It always returns clusters: random numbers get clusters too.
  • Validate against something you did not cluster on.

More in Tuning

Build "Clustering, groups without a label" yourself

Dobara refurbishes used bikes in Indore and lends to its buyers. You build the models that decide whom to lend to and at what price to sell. A real editor and terminal in your browser, AI that explains, and a Check that says when it works.

The first 2 chapters of Machine Learning are free. This one comes with the all labs plan.