The Hardest Part Wasn't Machine Learning: Building Customer Segments from 541,000 Retail Transactions

Vanshika — Tue, 14 Jul 2026 06:10:57 +0000

Every business has customers. Very few truly understand them.

A customer who bought something yesterday should not be treated the same as someone who has not returned in two years. And sending the same marketing offer to a one-time buyer and a regular shopper usually does not make much sense.

That was the idea behind this project.

The question was simple:

Can transaction data alone reveal useful customer groups that help businesses make better decisions?

The answer was yes—but not because of machine learning alone. The hardest part was not building a clustering model. It was cleaning and shaping more than 541,000 retail transactions into something reliable enough to analyze. This project follows that process, from raw transaction data to customer segments built with Python, RFM Analysis, and K-Means Clustering.

Understanding the Problem

Businesses collect a huge amount of transaction data every day, but raw data by itself does not tell much of a story.

Without understanding customer behavior, companies often:

Send the same campaign to everyone.
Miss customers who are about to stop buying.
Waste effort on customers who are unlikely to return.
Overlook their most valuable buyers.

Instead of treating every customer the same, businesses need a way to group them based on how they actually behave.

That became the goal of this project.

Using historical retail transactions, the aim was to find customers with similar buying patterns and turn those patterns into insights that a marketing team could actually use.

The Dataset

The project uses the Online Retail II dataset from the UCI Machine Learning Repository. It contains transaction records from a UK-based online retailer.

The dataset includes over 541,000 transaction records with fields such as:

Invoice Number
Product Description
Quantity Purchased
Invoice Date
Unit Price
Customer ID
Country

At first, it looked like the data was ready to go.

It was not.

Real Projects Start with Cleaning, Not Modeling

One thing became clear very quickly:

Machine learning is only as good as the data behind it.

Before calculating a single metric or training a model, the dataset needed a lot of cleaning.

A few issues stood out right away:

Missing Customer IDs
Cancelled invoices marked by invoice numbers starting with "C"
Negative quantities from returned items
Invalid or zero prices
Duplicate records

If these were left in the dataset, the customer behavior analysis would have been misleading.

The rule was simple:

If a transaction does not represent a real customer purchase, it should not be used for segmentation.

Using Pandas, each issue was handled step by step before moving on to feature engineering.

import pandas as pd

retail = pd.read_excel("online_retail_II.xlsx")

retail = retail.dropna(subset=["Customer ID"])
retail = retail[~retail["Invoice"].astype(str).str.startswith("C")]
retail = retail[(retail["Quantity"] > 0) & (retail["Price"] > 0)]
retail = retail.drop_duplicates()

The code looks small, but it made a big difference. Once the bad records were removed, the remaining data was much more trustworthy and much better suited for analysis.

Turning Transactions into Customer Behavior

A single purchase does not say much about a customer.

Two people might each have one transaction, but one may have bought yesterday while the other bought two years ago. Those are very different customers.

To make the data more useful, the project used RFM Analysis, a common customer segmentation method.

RFM looks at customer behavior through three questions:

Recency

How recently did the customer make a purchase?

Customers who bought recently are usually more engaged and more likely to come back.

Frequency

How often does the customer buy?

Customers who purchase often usually have a stronger relationship with the business.

Monetary Value

How much has the customer spent?

Customers who spend more often contribute a larger share of revenue.

Together, these three values give a compact but useful picture of each customer’s buying behavior.

The next step was to calculate these values for every customer.

snapshot_date = retail["InvoiceDate"].max() + pd.Timedelta(days=1)

rfm = retail.groupby("Customer ID").agg({
"InvoiceDate": lambda x: (snapshot_date - x.max()).days,
"Invoice": "nunique",
"TotalPrice": "sum"
})

rfm.columns = ["Recency", "Frequency", "Monetary"]

Instead of looking at hundreds of thousands of transactions, the data was now organized at the customer level, with each row representing one customer’s behavior.

That made the dataset much more useful for clustering.

Distribution plots of Recency, Frequency, and Monetary values.

Why K-Means?

At this point, each customer was represented by three numerical features.

The question was no longer how to calculate the metrics. It was how to find natural groups in the data.

K-Means Clustering was a good fit because it is simple, fast, and works well when the goal is to group customers with similar characteristics.

But before training the model, one important step was needed.

The three RFM features were on very different scales.

For example, Monetary values could range from a few pounds to several thousand, while Frequency values were much smaller.

If the data was used as-is, Monetary would dominate the clustering process.

To avoid that, the features were scaled using StandardScaler before fitting the K-Means model.

Choosing the number of clusters also took some testing. The Elbow Method was used to find a reasonable balance between simplicity and useful segmentation instead of picking a number at random.

The clusters showed clear differences in customer behavior. Some customers bought often and spent a lot. Some had only purchased once. Others looked like they were becoming inactive.

Instead of dealing with thousands of individual transactions, the business could now work with a few clear customer groups, each needing a different approach.

For example, loyal customers could be rewarded with special offers, while customers who seemed to be drifting away could be targeted with retention campaigns before they stopped buying altogether.

That was the point where the project stopped being just a machine learning exercise and started becoming something useful for the business.

What I Learned

The hardest part of this project was not the model—it was cleaning and understanding the data. Once the transactions were prepared properly, the clustering made much more sense.

I also learned that machine learning only matters when it leads to useful business decisions. In this case, customer segments can help identify loyal, inactive, and high-value customers.

If I built this again, I would try more features, compare other clustering methods, and make the analysis interactive.

Conclusion

Customer segmentation is about turning raw transaction data into decisions. With data cleaning, RFM analysis, and K-Means clustering, it becomes possible to find patterns that are hard to see manually.

If this post makes you think, "I could do this too," then it has done its job.

Resources

UCI Machine Learning Repository – Online Retail II Dataset
Python (Pandas, NumPy, Scikit-learn)
Matplotlib & Seaborn for visualization
Power BI for interactive dashboard creation

Thank you for reading!

DEV Community: Vanshika

The Hardest Part Wasn't Machine Learning: Building Customer Segments from 541,000 Retail Transactions

What I Learned

Conclusion

Resources