Clustering Isn't Discovery - It's a Hypothesis You Still Have to Test

An analyst runs KMeans(n_clusters=4) on dataset of customer purchase history, it gets back four groups, and he/she writes up the report as "from the data, it revealed four customer segments!" Management simply loves this - after long periods struggling to grasp at their customer base, they now have data-backed insights on real-names associated with each segment (Loyal Regulars, Bargain Hunters, At-Risk, New & Exploring), and pretty soon all the strategy meetings will silently assume those four groups are real and nobody even dares to question.

Unfortunately, those groups are not real and clustering does not "discover" hidden nuggets of insights in the data. Instead clustering imposes a partitioning based on what the analyst chooses as a similarity rule. If the analyst had chosen another reasonable rule, clustering can also fit the same dataset in a different way. Let's be clear - the four groups are a hypothesis about how to group the customers. To determine if the grouping is accurate, as in we can use this grouping to predict customer behavior, now that is a separate question that the clustering method can never answer by itself.

Same data, different "discovery"

To illustrate, try running KMeans clustering on an RFM table (recency, frequency, monetary value per customer) and only change the random seed:

from sklearn.cluster import KMeans

km_a = KMeans(n_clusters=4, n_init=1, random_state=7).fit(X_scaled)
km_b = KMeans(n_clusters=4, n_init=1, random_state=42).fit(X_scaled)

print(km_a.inertia_, km_b.inertia_)
# 8421.3   8103.9 - different local optima, neither one "wrong"

inertia: measure of how tightly grouped the data points are within the clusters

Notice that you get different inertia values. KMeans does not find the globally best grouping - it cannot. Instead the algorithm finds a local optimum based on randomly chosen initial centroids and two runs on the same data can converge to genuinely different groupings. In fact scikit-learn's user guide states "Given enough time, K-means will always converge, however this may be to a local minimum. This is highly dependent on the initialization of the centroids". As mitigation, the default behavior is to run the algorithm several times with different centroid seeds (n_init) and retain the run that scored the lowest on inertia - it does not guarantee a single answer.

What if you change to use a different method - say hierarchical clustering with average linkage on the same RFM dataset? In hierarchical clustering, you do not need to specify the number of clusters and hence, the group counts that appear "natural" in your dataset also shifts. This is because hierarchical clustering with average linkage and KMeans are optimizing on different objectives ( nested merge vs compact spherical clusters )

This dissonance between run-to-run, method-to-method is well studied under clustering stability - a collection of research whose overview by von Luxburg (2010) shows how sensitive clustering results are to the seed, sample, algorithm and why stability by itself is not sufficient for meaningful clustering.

Aren't there any methods to decide?

One might say: "We are very data-driven and we use the elbow method to pick k."

There are various metrics or methods to measure how well your clustering algorithm groups the dataset:

• elbow (inertia vs k)
• average silhouette method (silhouette score vs k)
• gap statistic

Side-by-side comparisons of various methods on same dataset often leads to contradictory recommendations on cluster count. There is no one-metric and final decision requires human judgement based on your domain knowledge.

At times, you might have another situation:

from sklearn.metrics import silhouette_score

for k in range(2, 7):
    labels = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(X_scaled)
    print(k, silhouette_score(X_scaled, labels))

# 2  0.41   <- highest score
# 3  0.33
# 4  0.29   <- the k the business narrative was built around
# 5  0.24
# 6  0.22

Here, the silhouette score shows k=2 is highest but you get push back from Marketing - "We need 4 separate personas to target!". Nothing wrong with that request, 2 clusters will usually lead to "Loyal" vs "Non-Loyal" segments and it is meaningless for marketing planning. 4 clusters do help in the business narrative to target specific market segments. If this is the case, a disclaimer is in order that k=4 was due to business narrative (instead of "insights from dataset" as authority)

Cluster on demographics may not mean different customer behavior

Your clustering tells you have four groups of customers and they look different based on the features you cluster on - recency, frequency, spend. Note that their behaviors that businesses care about - who leaves for a competitor, who upgrades to higher tier, who responds to coupons - may look different across those same groups. This discrepancy is what marketing teams struggle with using demographic and behavioral personas - the behavioral patterns often cut across various demographic personas.

The fix is not to have a better clustering metric but to validate externally based on your business objectives and goals (churn in the next quarter, response rate to campaign, etc) and check if they differ between clusters more than within the clusters.

What is the correct approach to clustering?

The above does not mean clustering is a useless technique to avoid. Instead what it means is that clustering is only a starting hypothesis for further investigations.

Before presenting the clustering results:

• re-run with different seeds and subset of data
  • this is to verify the stability of the cluster
• check with multiple metrics and methods (elbow, silhouette, gap statistics)
  • highlight in report when they disagree
• validate externally
  • select a business outcome
  • confirm it varies across clusters more than within cluster
• treat the chosen number of clusters as modeling decision

A clustering that complies with all four does not mean it is "true" but it does earn the right to be acted upon.

References

• Kleinberg, J. (2002). An Impossibility Theorem for Clustering. NeurIPS.
• von Luxburg, U. (2010). Clustering Stability: An Overview. arXiv.
• scikit-learn. Clustering - K-means, User Guide - local-minimum convergence and multiple-initialization behavior.
• scikit-learn. sklearn.cluster.KMeans API reference - n_init and random_state parameter definitions.
• Datanovia. Optimal Number of Clusters: Elbow, Silhouette, Gap Statistic.
• Ahmed, M. et al. (2022). K-Means Clustering Approach for Intelligent Customer Segmentation Using Customer Purchase Behavior Data. Sustainability, 14(12).
• MarTech. Why behavioral segmentation beats personas for real personalization.

---

SophiArch's Unsupervised Learning & Clustering course covers k-means, hierarchical clustering, and validation together for this reason - an algorithm that returns groups is the easy half; judging whether those groups mean anything to the business is the half that determines whether the analysis survives contact with a skeptical stakeholder.