Clustering is one of the most time-consuming parts of working with keyword data. Building and cleaning up a list of queries is usually pretty easy, but sorting thousands of keywords into clusters with the same search intent is where things get tricky.
Tools like Semrush, Ahrefs, and LowFruits make this job easier, but they come with limits on volume, settings, and how good the output actually is. This becomes especially obvious in complex niches, where automatic algorithms might group queries with different intents together or split apart phrases that are nearly identical in meaning.
Neural networks let you tackle this problem in a different way. Instead of matching individual words and lexical overlaps, queries get turned into multidimensional vectors called embeddings. You can then compare them by semantic similarity and use that to build clusters automatically.
In this article, we'll walk through how to build your own scalable clustering pipeline on a local machine or server, from automated collection of keyword and SERP data using an anti-detect browser to vectorization, grouping, and post-processing the results.
Collecting Keyword Data
There are a few ways to collect keyword data, but the basic process usually follows the same pattern: first, you build an initial pool of queries using services like Google Ads Keyword Planner and other tools that provide search volume data for your topic.
Then you expand that initial set with related queries, search suggestions, and other keyword sources. In the past, this kind of work was often handled by a single tool, but these days you usually need to combine several tools or write custom scripts to gather everything you need.
Using automation tools to collect keyword data
If commercial solutions don't fit your budget, feature needs, or limitations, you can automate part of the keyword collection yourself. For example, to pull search suggestions and other data from web interfaces, you can use headless browsers built on Puppeteer or Playwright.
If you're collecting keyword data across a large number of parallel sessions, you need to manage browser profiles and their environments safely. This is where an anti-detect browser like Octo Browser comes in handy—it lets you isolate sessions, manage profile settings, and connect different proxies. That simplifies your scraping infrastructure and cuts down on the manual setup required for each browser instance.
You can try Octo Browser for free with promo code DEVTO (4 days of the Starter subscription).
The main challenge with collecting data at scale is the restrictions that search engines put in place. Automated requests can trigger rate limits, CAPTCHAs, or other protection mechanisms, so when you're designing a pipeline like this, you need to plan for session stability, request frequency, and temporary blocks.
When working with browser automation, it's also important to control environment parameters like the User-Agent, window size, locale, WebGL, and other browser session characteristics. You can handle this with your own Puppeteer or Playwright configs, or with specialized browser solutions that let you spin up isolated profiles with different environment settings.
Network infrastructure is another separate task. You can use different types of proxies for distributed data collection: datacenter, residential, or mobile. The choice depends on request volume and your requirements for stability, speed, and cost. Datacenter proxies are usually cheaper and faster, but in some scenarios they're more likely to get restricted. Residential and mobile addresses tend to be more resilient, but they cost more and offer lower throughput.
Once the main data collection is done, you can round out your final list of queries with data from external keyword databases. The goal is to end up with as complete a set of queries as possible, which you can then pass on to the cleaning, normalization, and clustering stages.
Cleaning data before vectorization
After collecting your keyword data, you'll end up with a file containing tens of thousands of search queries. At first glance, it might look ready for vectorization and clustering, but the quality of your results depends heavily on how well the data is prepared beforehand.
At this stage, Pandas and NumPy are handy for cleaning and normalizing your input dataset. A raw query list almost always contains duplicates, unnecessary characters, technical clutter, irrelevant phrases, and other artifacts that creep in during parsing and when merging multiple sources.
An embedding model will turn those strings into vectors anyway, but that creates unnecessary computational overhead and can mess up the structure of your resulting clusters. That's why it's a good idea to get the data into a consistent, predictable format before vectorization.
Here are the main preparation stages:
Removing technical clutter. HTML tags, extra spaces, invisible characters, emojis, and other elements that may have slipped in during scraping get stripped out.
Global deduplication. When you combine queries from multiple sources, overlaps are almost inevitable. There's no point in vectorizing the same string more than once, so full duplicates should be removed upfront.
Stop-word filtering. At this stage, you can exclude queries containing irrelevant place names, unwanted markers, or words that don't fit your project requirements. For example, commercial keyword data might exclude queries with words like "free," "torrent," and similar modifiers.
Reducing words to their dictionary form (lemmatization) is optional. Modern models understand different forms of the same word and equivalent phrases well. For instance, they can recognize that "buy iPhone" and "I want to buy an iPhone" are basically the same query. So there's no need to deliberately change words before processing. That said, simple preprocessing can help you spot similar queries and cut down the amount of data.
The end result should be a clean, filtered set of unique queries without obvious technical noise. From there, the data can move on to the next stage—turning text into vector representations.
Vectorizing search queries
Vector clustering is different from traditional string comparison. Instead of matching exact words, it works with their semantic representations. Each search query gets converted into a numerical vector, an embedding, that encodes its meaning.
This is done using specialized embedding models. The text is first split into tokens, and then the model generates a fixed-size vector. As a result, semantically similar queries end up closer to each other in vector space.
For example, the phrases "buy iPhone 15" and "iPhone 15 Pro price" will have more similar vectors than "buy iPhone 15" and "Apple phone repair." That property is what makes it possible to use clustering algorithms to group queries together.
Commercial solutions (OpenAI, Claude)
For vectorization, you can use both cloud APIs and local models. The choice depends on the amount of data, quality requirements, your available infrastructure, and the acceptable processing cost.
Commercial APIs are convenient because they don't require you to deploy a model locally and let you get started quickly. The provider handles the infrastructure, model updates, and computational scaling for you.
For small and medium-sized datasets, this is one of the simplest options. But once you're processing hundreds of thousands or millions of queries, you need to think about API costs, request limits, and throughput.
On top of that, working with APIs requires a fault-tolerant architecture: things like bypassing API limits through multi-accounting, key rotation, and asynchronous requests via aiohttp.
You can test how commercial neural networks handle your queries yourself using the following code:
import os
from openai import OpenAI
# Initialize the client
client_ai = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
# Our raw dataset
queries = ["buy iphone 15", "iphone 15 pro price", "apple phone repair"]
# 1. Vectorize queries through OpenAI
response = client_ai.embeddings.create(
input=queries,
model="text-embedding-3-small"
)
# The result is a set of ready-made multidimensional vectors
embeddings = [data.embedding for data in response.data]
Local models from the Hugging Face ecosystem
An alternative to commercial APIs is open embedding models that run locally. For example, for multilingual tasks, you can use jinaai/jina-embeddings-v3 or Alibaba-NLP/gte-multilingual-large, which do not incur per-generation costs.
from sentence_transformers import SentenceTransformer
print("⏳ 1. Loading stable BGE-m3 model...")
model = SentenceTransformer('BAAI/bge-m3')
queries = ["buy iphone 15", "iphone 15 pro price", "apple phone repair"]
print(f"⏳ 2. Vectorizing {len(queries)} queries...")
embeddings = model.encode(queries, normalize_embeddings=True)
print("\n✅ Done! Let's look at the result:")
print(f"📊 Data dimensions: {embeddings.shape}")
print("🔍 Vector for the phrase 'buy iphone 15':")
vector_preview = [round(float(num), 4) for num in embeddings[0][:5]]
print(f"🔢 {vector_preview} ... and {len(embeddings[0]) - 5} more numbers.")
The main advantage of local models is that the entire processing pipeline stays within your own infrastructure. You process search queries without sending them off to a third-party API, and once the model is loaded, you can vectorize them without paying for each individual request.
Working with local models is often more cost-effective: they let you handle vectorization without paying for every API call. At the same time, they take up a fair amount of disk space. Along with model weights and cache, they can eat up several gigabytes. If you no longer need a model, it's a good idea to delete its local files and clear the cache once you're done working with it.
Storing and searching vector representations
After vectorization, each search query is represented as an embedding vector. The next question is where to store this data and how to efficiently find semantically similar queries.
For small datasets, vectors can stay in memory, for example, in NumPy arrays or Pandas structures. If you're only dealing with a few thousand queries, that's usually enough for experiments and local processing.
As the dataset grows, things change. If every vector gets compared directly against all the others, the number of operations grows quadratically. For tens or hundreds of thousands of queries, this quickly becomes resource-intensive in terms of both computation time and memory usage.
Vector databases solve this problem. They're optimized specifically for these kinds of tasks and support approximate nearest-neighbor search algorithms (HNSW in particular). Built-in indexes let you avoid comparing every vector against every other vector directly, and they deliver near-instant searches for the most similar phrases even across millions of records.
There are several popular solutions for this: Pinecone, Qdrant, PostgreSQL with the pgvector extension, and others. In our example, we'll use ChromaDB. It's well suited for local experiments: it runs without a separate server, stores data on disk, and integrates easily with Python code.
Let's save the embeddings we got from the previous stage and check how semantic search works:
import chromadb
# 1. Initialize the local database (creates the semantic_db folder)
client = chromadb.PersistentClient(path="./semantic_db")
# 2. Create the collection
collection = client.get_or_create_collection(
name="search_queries",
metadata={"hnsw:space": "cosine"}
)
# 3. Load our vectors and query texts
collection.add(
embeddings=embeddings.tolist(), # vectors from our local BGE-m3 model
documents=queries,
ids=["id_1", "id_2", "id_3"]
)
print("✅ Data saved to the database!\n")
# ==========================================
# SEARCH MAGIC: let's check how the database understands meaning
# ==========================================
test_phrase = "how much does the new iphone cost"
print(f"Searching the database for the phrase: '{test_phrase}'")
# Convert the test phrase into a vector using the same model
test_embedding = model.encode([test_phrase], normalize_embeddings=True)
# Ask the database to find the two most similar options
results = collection.query(
query_embeddings=test_embedding.tolist(),
n_results=2
)
print(f"Closest query: {results['documents'][0][0]} (Distance: {results['distances'][0][0]:.4f})")
print(f"Second closest: {results['documents'][0][1]} (Distance: {results['distances'][0][1]:.4f})")
Choosing a clustering algorithm
Once you've got your embedding vectors, you can move on to the next stage—grouping search queries. At this point, it's important to pick an algorithm that fits the structure of your data and doesn't force overly rigid assumptions about the number of clusters you'll end up with.
Why K-Means is not always suitable for SEO clustering
K-Means is a classic clustering algorithm that splits data into a predefined number of groups. You set that number with the k parameter.
For example, if you have 10,000 search queries and set k=500, the algorithm will create 500 centroids and assign each query to the nearest one in vector space.
The main limitation here is that you have to decide on the number of clusters in advance. That's not always convenient when you're building a keyword map: before you start processing, it's hard to know how many independent intent groups your dataset actually contains—50, 500, or 1,200.
If k is chosen poorly, related queries might end up split across several groups. The opposite can happen too: queries that are close in topic but differ in intent may get merged simply because the algorithm has to produce the specified number of clusters.
Clustering with DBSCAN
If the number of groups isn't known in advance, you can turn to density-based clustering algorithms. One of the best-known options is DBSCAN (Density-Based Spatial Clustering of Applications with Noise).
Unlike K-Means, DBSCAN doesn't require you to specify the number of clusters upfront. Instead, the algorithm looks for regions in vector space where objects are packed closely enough together and forms groups from them.
DBSCAN's behavior is driven by two main parameters:
eps(epsilon/distance): the maximum distance between points for them to count as neighbors. In our case, that's the cosine-similarity threshold of the vectors.min_samples: the minimum number of neighbors needed to form a full cluster.
DBSCAN starts with the first random query. If there are min_samples other queries within an eps radius of it, a cluster core forms. The algorithm then expands outward in all directions, pulling in new neighbors until the density runs out.
You can roughly think of the eps parameter as how strict the clustering is:
A smaller
epsmeans stricter grouping. Only queries with very similar vector representations land in the same cluster, so you usually get more groups, and they're more compact.A larger
epsloosens the merging conditions. Clusters get bigger and may cover broader semantics, but the risk of mixing queries with different intents goes up too.
Another handy property of DBSCAN is its ability to flag noise. If a query isn't in a sufficiently dense region of vector space, the algorithm doesn't try to force it into an existing cluster. Instead, it marks the query as an Outlier. You can export these queries to a separate file for manual review rather than compromising otherwise clean landing pages.
That said, DBSCAN is sensitive to the choice of eps: a single threshold doesn't always work well when some groups in your data are very dense and others are much sparser.
In those cases, consider HDBSCAN, a hierarchical extension of the density-based approach. It can find clusters with different densities, automatically adapting the eps parameter where queries are more tightly packed or, conversely, more spread out.
The clustering script
Let’s extract our vectors from the local ChromaDB database and run them through DBSCAN using the scikit-learn library:
import chromadb
import numpy as np
from sklearn.cluster import DBSCAN
print("⏳ 1. Connecting to the vector database...")
client = chromadb.PersistentClient(path="./semantic_db")
# Extract the collection with the vectors (use your own name)
collection = client.get_collection(name="search_queries")
# Extract all query texts and their mathematical vectors
data = collection.get(include=["documents", "embeddings"])
documents = data["documents"]
embeddings = np.array(data["embeddings"])
print(f"✅ Queries extracted from the database: {len(documents)}")
# ==========================================
# CLUSTERING (DBSCAN)
# ==========================================
print("⏳ 2. Starting the DBSCAN algorithm...")
# SETTINGS:
# eps = 0.15 (Allowed cosine distance. The smaller it is, the stricter the clusters);
# min_samples = 2 (Minimum of 2 queries to create a group);
# metric="cosine" (We explicitly specify that we measure angles between vectors, not linear distance).
dbscan = DBSCAN(eps=0.15, min_samples=2, metric="cosine")
# Run the grouping
labels = dbscan.fit_predict(embeddings)
# ==========================================
# OUTPUT RESULTS
# ==========================================
# The algorithm assigned each query a group number (0, 1, 2...).
# If a query is recognized as noise (Outlier), it receives the label -1.
clusters = {}
outliers = []
for doc, label in zip(documents, labels):
if label == -1:
outliers.append(doc)
else:
if label not in clusters:
clusters[label] = []
clusters[label].append(doc)
print("=== GROUPING RESULTS ===")
for cluster_id, docs in clusters.items():
print(f"\n Cluster #{cluster_id} (Queries: {len(docs)})")
for d in docs:
print(f" - {d}")
if outliers:
print(f"\n Outliers/Noise (Queries: {len(outliers)})")
for out in outliers:
print(f" - {out}")
The example above uses a local model, but when working with commercial embedding models, the eps value may require additional tuning. In particular, a threshold of 0.15 that works for one model may, with another configuration, cause a significant portion of the queries to merge into one large cluster or be incorrectly classified as noise.
Therefore, whenever you switch models, you should tune eps separately based on the distribution of distances between vectors and the actual quality of the resulting groups.
Hybrid clustering for separating search intents
Clustering can be done with embeddings alone, but in practice that's often not enough. Semantic similarity doesn't always mean the search intent is the same.
Take the queries "buy an anti-detect browser" and "what is an anti-detect browser." They're very close in topic. The embedding model correctly recognizes that both phrases refer to the same object, so the distance between their vectors will be small. As a result, DBSCAN will most likely drop both into the same cluster.
From an SEO standpoint, that's not what you want, because the intent differs. The first implies a commercial landing page, while the second calls for informational content.
Let's demonstrate this with a small test dataset:
queries = [
# Informational
"what are antidetect browsers for",
"what is an antidetect browser",
"how antidetect browsers work",
"antidetect browsers comparison",
# Commercial
"buy an anti-detect browser",
"buy proxies for an anti-detect browser",
"anti-detect browser trial",
# Download
"download anti-detect browser octo browser",
"octo browser anti-detect browser download",
"octo anti-detect browser download",
# Queries on a different topic
"ford everest 2024 review",
"buy used ford everest",
# Noise
"weather in pattaya in May",
"tom yum soup recipe"
]
When clustering only by vector similarity, queries related to anti-detect browsers may end up in the same group despite differences in intent.
At this stage, the anti-detect browser becomes part of the pipeline once again—but not for collecting semantic data. This time, it's used to collect SERP data. For each query, you need to pull the search results, then use URL overlaps as an additional clustering signal.
When you're dealing with a large number of queries, it helps to distribute this collection across isolated browser profiles—for example, by pairing Octo Browser with Playwright or Puppeteer.
For each query, grab the top 10 URLs from the search results. Then, before running DBSCAN, compare the results for semantically similar phrases. If two queries share no URLs, or the overlap falls below a set threshold, the distance between their corresponding vectors gets artificially increased.
So embeddings are used to find semantically similar queries, while the SERP analysis acts as an extra constraint and helps keep phrases with different search intents from being merged.
Once you add the search-result data, the test dataset we discussed earlier gets distributed differently:
The number of clusters increases, and the groups themselves better match the intended search intent of the queries.
Post-processing results and updating data
After forming the clusters, there is one more practical task—assigning a clear name to each group. Numbers such as “Cluster #42” are convenient for an algorithm but tell an SEO specialist, editor, or content writer very little.
Automatic cluster naming with an LLM
When you're doing clustering manually, a specialist has to review the contents of each group, figure out the primary intent, and come up with a name for the future page or piece of content. If you've got several hundred clusters, this stage eats up a serious amount of time.
That said, this part of the process can be automated with an LLM. The model gets a list of queries from one cluster, determines the overall intent, and generates an appropriate heading. You can use either cloud-based models or local solutions like Ollama.
It's important to define a strict output format upfront. If you just ask the model to come up with a name, you may get extra explanations and comments alongside the heading. That's why it's better to spell out explicitly in the system prompt that the output should contain only the heading and nothing else:
from openai import OpenAI
# 1. INITIALIZATION AND KEY
# Insert your actual API key here
client_ai = OpenAI(api_key="sk-YOUR_OPENAI_KEY")
# 2. OUR DATA (Result of hybrid clustering)
clusters = {
0: [
"what are anti-detect browsers for",
"what is an anti-detect browser",
"how anti-detect browsers work",
"anti-detect browsers comparison",
],
1: [
"buy an anti-detect browser",
"buy proxies for an anti-detect browser",
"anti-detect browser trial"
],
2: [
"download anti-detect browser octo browser",
"octo browser anti-detect browser download",
"octo anti-detect browser download"
]
}
# System prompt (set rules for the model)
prompt = """You are an expert SEO specialist. Analyze the following cluster of search queries. Determine the primary user intent and generate one highly relevant H1 title for a future category page or article. Return ONLY the title, without any additional text, quotes or explanations."""
print("⏳ Sending clusters to GPT-4o-mini for automatic naming...\n")
# 3. Iterate through all clusters
for cluster_id, queries in clusters.items():
# Send the queries from the current cluster to the API
response = client_ai.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": prompt},
{"role": "user", "content": "\n".join(queries)} # Combine the queries into a single text
],
temperature=0.3
)
# Get the response
h1_title = response.choices[0].message.content
# Print the result to the console
print(f"Cluster #{cluster_id}")
print(f"Phrases: {', '.join(queries)}")
print(f"Generated H1: {h1_title}\n")
For the test dataset, the result might look like this:
Cluster #0
Phrases: what are anti-detect browsers for, what is an anti-detect browser, how anti-detect browsers work
Generated H1: What are anti-detect browsers used for
Cluster #1
Phrases: buy an anti-detect browser, buy proxies for an anti-detect browser, anti-detect browser trial
Generated H1: Best anti-detect browsers and proxies for safe browsing
Cluster #2
Phrases: download anti-detect browser octo browser, octo browser anti-detect browser download, octo anti-detect browser download
Generated H1: Download anti-detect browser Octo Browser
In a real project, the number of clusters will be much larger, so storing the source data directly in the code isn't practical. Usually, clusters get loaded from a file or database, and the generated names are written back to the table for further work.
At this stage, the LLM isn't involved in the clustering itself. It's only used to post-process groups that have already been formed. This automates the routine part of the work and gives you a clear keyword set without having to manually name every cluster.
Adding new queries
Another advantage of storing embeddings in ChromaDB is that you can work with new data without manually running a nearest-neighbor search across the entire dataset again.
Once you get a new batch of semantic data, the queries go through the same pipeline: cleaning, vectorization, and adding them to ChromaDB. For each new embedding, you can find the nearest existing queries and check the distance to them.
If the nearest neighbors belong to a stable existing cluster and meet your defined similarity threshold, the new query can be added to that group. If no suitable cluster turns up, the query stays a candidate for forming a new group or gets sent for additional processing.
This approach lets you use the accumulated vector database as an index for handling new queries, so you avoid running a full pairwise search across the entire keyword set every time it gets updated.
Conclusion
Building your own pipeline based on embedding models, vector storage, and clustering algorithms takes time. You have to set it up, test it, and tune it. But once that's done, you end up with a system you can adapt to a specific topic, data volume, and project requirements.
The main advantages of this approach are:
Reduced dependence on specialized services. You don't need a separate SEO service with fixed plans and volume limits for clustering. The main limitations shift to your own infrastructure: compute resources, memory, and disk space.
Control over clustering logic. You can pick the embedding model yourself, configure
eps, use SERP data, and change the rules for combining queries depending on the task.Control over the data. When you use local models and local storage, your keyword data stays within your own infrastructure and isn't sent to third-party APIs.
The main value of this approach isn't completely replacing ready-made SEO tools. Rather, it's the ability to build your own controllable pipeline. Use Octo Browser to automate semantic and SERP data scraping, and handle the rest of the processing locally with embeddings, ChromaDB, and clustering algorithms.
Once you build this process and carefully tune it on real data, it becomes more than a one-off script. It turns into a working tool you can reuse and scale as your keyword set grows.
Top comments (0)