DBSCAN’s edge over K-Means clustering

The moment I saw this prompt, I knew I needed a dataset that was large and noisy enough to truly show how the two unsupervised models differ. So, despite the constraint to use a tiny made-up dataset, I chose an Airbnb dataset for NYC.

First, I cleaned the dataset and narrowed our focus from all of New York City to just Queens, reducing the data from nearly 49,000 rows to 4,540. I also filtered out long-term rentals by capping the minimum stay at 30 nights.

To prepare for clustering, I removed unhelpful columns like names and ID numbers (preventing the KMeans algorithm from misinterpreting ID values as mathematical weight), along with the neighbourhood group column, since everything is now in Queens.

Because the K-Means algorithm only works only with numbers, I handled the text columns. Instead of dropping neighborhoods entirely, I converted all 48 sub-categories into numerical frequency scores. I then dropped remaining text columns like room type and last review, and stored our finalized table as queens_df.

First five rows of queens_df (4,540 rows, 9 features)

Before building any machine learning model, especially gradient-based and distance-based models, it’s important to standardise the data. Without standardising, features with a larger range of values tend to look more important and dominate the results. I used Scikit-learn’s StandardScaler on queens_df, standardised it using fit_transform, and called the result X_scaled.

For the K-Means model, I followed these steps after standardising the data:

1. Build the K-Means model and fit it on X_scaled , with n_clusters=11 and random_state=101 for starters.
2. Find the appropriate number of clusters in two steps: plot the elbow curve to get a rough idea of possible k values, then plot the silhouette score of the base model.

Elbow method (left) and silhouette scores (right); silhouette peaks at k = 4

Note: The silhouette plot shows us a measure of how close each point in one cluster is to points in the neighbouring clusters. It provides a way to assess parameters like number of clusters visually (Silhouette scores range range from -1 to +1).

1. Build the final K-Means model using the k value derived from the plots, and store the cluster labels as a column in queens_df.
2. Plot scatter plots to see the clusters using various features (for example, price vs number_of_reviews, coloured by cluster).

Price vs number of reviews, coloured by K-Means cluster

1. Compare the median values of each cluster to see how their patterns differ. Using median() makes the clusters stand out and helps draw inferences, rather than looking at each cluster’s raw data individually.

Median feature values per K-Means cluster

Next, I’ll show you how I built the DBSCAN model , followed by a comparison of the two models, so it all makes more sense.

For the DBSCAN model, it’s worth remembering the main difference from K-Means: DBSCAN doesn’t need us to specify the number of clusters, it’s intuitive, and it handles noisy datasets extremely well.

Note: the DBSCAN documentation mentions that scikit-learn’s implementation has a worst-case memory complexity that can occur when eps is large and min_samples is low, whereas the original DBSCAN uses only linear memory. So I tried to be mindful of these parameters.

To build the DBSCAN model, I used the same X_scaled data from the previous model and followed these steps:

1. Rule of thumb for min_samples: 2 x n_features. Since I have 9 features in X_scaled, min_samples is 18.
2. For the first model, use min_samples (i.e. 18) as n_neighbors in NearestNeighbors, and call the model neigh.
3. Fit the neigh model to X_scaled.
4. Use tuple unpacking to get the distances and indices of the k-neighbours.
5. The distances for each data point always start with 0 (the distance from itself), followed by the distances to its neighbours in ascending order. We need the distance to its 18th nearest neighbour, which is the last column of the dist matrix.
6. Next, choose an eps value (it is simply the maximum distance at which two points still count as neighbours).
7. To find eps, plot these distances using matplotlib to get an elbow-like chart, which tells us what the eps value should be for the final model. We’re looking for the point where the curve bends sharply upward : below it, points are packed closely together. For my model, this turned out to be just after 1.5, so I chose eps=2.

k-distance plot (18th-neighbour distance, zoomed to 0–5)

1. Build the DBSCAN model with basic params, eps=2 and min_samples=18 , and call it dbscan.
2. Fit this model on X_scaled
3. Save dbscan.labels_ into queens_df[‘dbscan_clusters’].

Now, let’s compare both models:

KMeans clusters Vs DBSCAN clusters

1. Shape of the data: K-Means had to split Queens into 4 clusters and assign every listing to one, because it has no notion of noise. DBSCAN found only 2 clusters plus noise: one dense cluster holding 97.6% of listings, a tiny cluster of 0.8% (35 listings), and 1.7% noise (75 listings).

2. A hidden niche: DBSCAN’s tiny cluster is very specific. All 35 listings are private rooms with a 30-night minimum, the lowest median price (~$39), the highest availability (239 days), almost no reviews (median 1), and hosts running ~103 listings each, which suggests one or a few large operators. K-Means found a related group (cluster 1, 219 listings, ~7 listings per host), but it was diluted with mixed room types and smaller hosts.

3. Popular listings are the outliers: the noise points have the most reviews (median 132 per listing, ~5.4 per month). They resemble K-Means cluster 3, but DBSCAN treats them as rare, scattered listings rather than a group of their own.

4. The typical Queens listing: DB0’s hosts usually run a single listing (median 1), with ~14 reviews and a mix of private rooms (2,575), entire homes (1,710) and shared rooms (145).

---

This was Day 5 of my 30 Days of Data Science writing challenge. It has been by far the most interesting build so far. I learnt something new about how these two clustering models behave by reading scikit-learn’s documentation.

You can see the complete notebook on my GitHub: https://github.com/Rupali-Gauravaram/dev-prompt-generator/blob/main/posts/post_05.ipynb