You can ship semantic search that looks fine on a demo and still retrieve the wrong chunk on the questions your users actually ask. The embedding you store is one of the decisions that matter. A single vector per chunk is the common starting point. A late-interaction model keeps more of the match in the score. Filters on category, date, or customer change how fast that search stays once the catalog is large. My suggestion: test it with about 50 queries from your own logs before you argue about indexes. It is a first read, not a final answer.
What does one embedding vector actually buy you?
An embedding turns a piece of data into a list of numbers, a vector, that captures its meaning. Two sentences that mean similar things produce vectors that sit near each other, even when they share no words. Semantic search compares those vectors and returns the closest ones. In retrieval-augmented generation you store document embeddings, embed the question, and pull the nearest chunks. How good the embedding model is decides how often "near" means "relevant."
A vector database is the store built for that job: hold the embeddings and, given a query vector, return the closest matches. A regular database matches exact values. This one matches by similarity. That split matters once you also want the nearest vectors that satisfy a condition, such as a product category, a date range, or a customer. Similarity and the filter are one query, not two features you can tune in isolation.
I would not pick an embedding from a public leaderboard and stop there. The only distance that counts is distance on your chunks and your questions.
When is a single vector the wrong shape?
Maxime Labonne, Head of Post-Training at Liquid AI, splits embedding models into three families. The first is the classic BERT-style embedding: give the model an input, get a vector out, and calculate its similarity with other vectors.
One vector in, one similarity score out. You can precompute every document vector and compare the query vector to them. That is cheap at query time, and I would default to it when the question is one idea and the chunk is about that idea.
He describes the cross-encoder, or reranker, next: the query and the document go through the model together, and you get a similarity score. He calls that good and very expensive, because you cannot treat the document side as something you fully pay for ahead of time.
The third family is the one I would put on the same afternoon test as the single vector. Liquid released a ColBERT-style late-interaction retrieval model.
And finally, you have this late interaction family where what you do is that you take kind of the best of both world, meaning that you can precompute your vectors, and, also, you can do re ranking in the same model.
Maxime Labonne, Head of Post-Training at Liquid AI, on Chain of Thought ep 43
Late interaction still lets you precompute document vectors, and the same model produces the ranking score. Labonne calls this an exciting way to do retrieval, and he sees RAG as a good fit for smaller models because the task is not too complex. Your job is narrower: on your queries, does the single vector already put the right chunk in the top results, or does the late-interaction score move the right chunk up?
If the single vector wins on your set, keep it. Extra scoring you do not need is just latency.
How do filters behave once the catalog is large?
Bob van Luijt, CEO and co-founder of Weaviate, hit this in production e-commerce search. Customers had billions of products. Filtered vector search, queries that came back with few hits, became significantly slower. Weaviate had not optimized that filtered path. He is specific about the scale where it shows up.
If you have a million products, you're not going to notice that. But if you go up to a billion, you start to notice it.
Bob van Luijt, CEO and co-founder of Weaviate, on Chain of Thought ep 8
He says Weaviate then implemented a method called ACORN, from the research literature, as a way to deal with filtering in vector search. The lesson I take is that a filter which feels free at a million rows can slow down badly at a billion, at least on a path nobody has optimized, especially when the filter leaves few candidates. Van Luijt argues that small use cases hide this: you can do almost anything with the database and not see it in the bill or the timings. Production size is when the operational cost shows up.
So the embedding bake-off and the filter check are the same exercise. A late-interaction score that looks better on unfiltered search can still lose if every query also has to restrict by tenant, category, or date, and that restricted search gets slow. Measure both.
How do I compare them on 50 queries this afternoon?
Pull 50 questions from real traffic, support tickets, or a notebook you trust. For each one, write down the chunk id you would accept as the answer. If you cannot name that id, drop the question. A comparison without a known good chunk only tells you which ranking looks plausible.
Run two retrieval setups you can already call, or two you mock with saved vectors. Setup A is a single embedding vector per chunk and a similarity score. Setup B is a late-interaction score on precomputed token or patch vectors, using whatever interface that model gives you. Do not invent a vendor client for this. Save top-5 ids and milliseconds. Then run the same 50 again with the metadata filter you expect in production, even if your file is still small. You will not reproduce a billion-row slowdown on a laptop. You will see whether the filter changes which ids come back, and you will have a timing function ready when the index grows.
The sketch below only tallies hits and time. You fill rows from your own runs.
# Sketch only. Not a pipeline a guest or a vendor ships.
# Each row: approach ("single" or "late"), filtered (bool),
# hit (bool), ms (float). One row per query per setup.
def summarize(rows):
buckets = {}
for row in rows:
key = (row["approach"], row["filtered"])
hit, ms, n = buckets.get(key, (0, 0.0, 0))
buckets[key] = (hit + int(row["hit"]), ms + row["ms"], n + 1)
report = []
for (approach, filtered), (hit, ms, n) in buckets.items():
report.append({
"approach": approach,
"filtered": filtered,
"hits_at_5": hit,
"queries": n,
"mean_ms": round(ms / n, 1) if n else None,
})
return report
I'd keep the single-vector setup when it matches late interaction on hits at 5, and when the filtered run is not the slow path you care about. I'd switch to late interaction when it picks up several queries the single vector misses, and the extra milliseconds still fit the product. I'd treat a filter that already reorders or empties the top 5 as a retrieval problem to fix before any prompt change. More interviews on chunking, vectors, and what breaks as retrieval grows are in the RAG and retrieval collection.
Checklist
- Write 50 real queries and the chunk id that should rank in the top 5 for each.
- Score the same set with a single vector and with late interaction. Record hits and milliseconds.
- Repeat with the metadata filter you will use in production (category, date, customer, or tenant).
- Stay on the single vector if late interaction does not move hits and the filter stays cheap at your size.
- Re-time filtered search as the catalog grows; van Luijt's customers saw it slow sharply at billions of products before Weaviate optimized that path.
More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)