Hubness in Vector Search: Why One Chunk Tops Every RAG Query

Chunk 4127 of your corpus reads: "All features are subject to regional availability. Contact your account team for details." It is in the top-5 for "how do I rotate an API key." It is also in the top-5 for "what is the refund window," "does SSO support SCIM," and roughly 3% of every query your users have ever typed.

You blame the chunker. You add a boilerplate filter. Two weeks later a different chunk takes its place.

This is hubness in vector search — a geometric property of high-dimensional nearest-neighbor search, first characterized by Radovanović et al. in 2010. A small set of vectors appears in the k-NN lists of a wildly disproportionate number of queries, and a long tail of vectors appears in none. It is not a content problem, so content fixes don't hold.

Key takeaways

• In high-dimensional spaces, the distribution of k-occurrence (how often a vector shows up in others' top-k) becomes strongly right-skewed. A few "hubs" are retrieved constantly; many "antihubs" are never retrieved at all.
• The driver is anisotropy: embedding sets have a dominant mean direction, and a document's alignment with it adds a near-constant, query-independent bonus to its cosine score.
• Measure it before you fix it: k-occurrence skewness and orphan rate, computed offline from your own index. Both are ~20 lines of NumPy.
• Two fixes stack. Mean-centering (all-but-the-top) is a linear transform you can bake into the index. A CSLS-style hub penalty applied at rerank time requires over-retrieval but is stronger.
• Only the document-side penalty changes ranking. The query-side term matters only if you compare scores across queries — which is exactly what a fixed similarity > 0.75 cutoff does.

Why do the same chunks appear in every top-k?

Take L2-normalized embeddings, so cosine similarity is just the dot product. Let μ̂ be the unit-length mean direction of your document set. Decompose any document vector:

d = α_d · μ̂ + d⊥        where  ⟨d⊥, μ̂⟩ = 0

Then for any query q:

cos(q, d) = α_d · ⟨q, μ̂⟩  +  ⟨q, d⊥⟩
            └── query-independent ──┘   └─ the part that actually answers ─┘

Embedding spaces are anisotropic. Random pairs from most text encoders do not sit near cosine 0 — they sit at a substantial positive similarity, because nearly every vector has a large positive component along μ̂. That means ⟨q, μ̂⟩ is positive and roughly stable across queries. So the first term is a fixed bonus per document, scaled by how "central" that document is.

Now compare spreads. Across your corpus, α_d varies. Across candidates for a given query, the discriminative term ⟨q, d⊥⟩ also varies. When the spread of the constant bonus is comparable to the spread of the discriminative term, ranking is partly decided by a number that has nothing to do with the query. High-α_d documents win ties everywhere — and in a corpus with millions of near-ties, winning ties everywhere means being in every top-k.

Generic text lands near the centroid because it is semantically generic. That part is intuitive. What isn't intuitive is that the effect grows with intrinsic dimensionality regardless of content, and that adding more documents makes it worse: more k-NN lists exist for the same hubs to enter.

How do I measure hubness in vector search?

Two numbers, computed offline from vectors you already have. k-occurrence skewness S_Nk (third standardized moment of the retrieval-count distribution) and orphan rate (fraction of vectors never retrieved by anyone).

import numpy as np

def knn_lists(X, k=10, block=4096):
    """X: (n, d) L2-normalized float32. Returns (n, k) neighbor indices."""
    n = X.shape[0]
    idx = np.empty((n, k), dtype=np.int32)
    for s in range(0, n, block):
        e = min(s + block, n)
        sims = X[s:e] @ X.T                       # (b, n)
        np.fill_diagonal(sims[:, s:e], -np.inf)   # drop self-match
        idx[s:e] = np.argpartition(-sims, k, axis=1)[:, :k]
    return idx

def hubness_report(X, k=10):
    idx = knn_lists(X, k)
    counts = np.bincount(idx.ravel(), minlength=X.shape[0]).astype(np.float64)
    mu, sd = counts.mean(), counts.std()
    top = max(1, counts.size // 1000)
    return {
        "k_occurrence_skew": float(((counts - mu) ** 3).mean() / (sd ** 3 + 1e-12)),
        "orphan_rate":       float((counts == 0).mean()),
        "top_0.1pct_share":  float(np.sort(counts)[-top:].sum() / counts.sum()),
    }

Rules of thumb from production corpora, not from a paper: skew near 0 is healthy, above ~1 is visible in eval, above ~3 means a handful of chunks are eating your context window. Orphan rate is the one that actually costs you money — those are documents you are paying to embed, store, and index that no query can reach.

Run it against real query embeddings too, not just doc-vs-doc. Queries and passages come from different distributions (short interrogative vs. long declarative), and the asymmetry matters: it is the document-side hubness measured against the query distribution that hurts.

What actually fixes it?

Three levers, cheapest first.

1. Mean-centering (bakeable into the index)

Remove the dominant direction from both sides. This is the single-component version of Mu and Viswanath's all-but-the-top postprocessing:

mu = D_raw.mean(axis=0)
mu /= np.linalg.norm(mu)

def decenter(V, mu):
    V = V - (V @ mu)[:, None] * mu          # project out the top direction
    return V / np.linalg.norm(V, axis=1, keepdims=True)

D = decenter(D_raw, mu)                      # index these
# at query time: q = decenter(q_raw[None, :], mu)[0]

This is a fixed linear transform followed by renormalization, so your HNSW or IVF index works unchanged — you are just storing different vectors. Two rules: apply the identical transform to queries (skip this and you destroy recall), and persist mu next to the index, keyed by embedding model version. Re-embed with a new model, recompute mu.

The generalization removes the top D principal components (D ≈ dim/100 in the original work). Diminishing returns past the first few, and each one you strip is signal you can't get back.

2. CSLS hub penalty (rerank-time)

Stronger, and it targets hubs individually rather than assuming a single global direction. Borrowed from Conneau et al.'s cross-lingual work:

CSLS(q, d) = 2·cos(q, d) − r_Q(q) − r_D(d)

where r_D(d) is the mean similarity of d to its k nearest queries, and r_Q(q) the symmetric term. A document that is close to everything gets a large r_D(d) and pays for it.

def hub_radius(D, R, k=10, block=4096):
    """r(d) = mean sim of each row of D to its k nearest rows of R.
       R = sampled real query embeddings if you have logs, else D itself."""
    r = np.empty(D.shape[0], dtype=np.float32)
    self_ref = R is D
    for s in range(0, D.shape[0], block):
        e = min(s + block, D.shape[0])
        sims = D[s:e] @ R.T
        if self_ref:
            np.fill_diagonal(sims[:, s:e], -np.inf)
        r[s:e] = -np.partition(-sims, k, axis=1)[:, :k].mean(axis=1)
    return r

R = hub_radius(D, Q_sample, k=10)             # precompute once, store per doc

def search(q, index, D, R, k_final=8, k_ann=64, lam=1.0):
    cand = index.search(q, k_ann)              # plain cosine ANN, untouched
    scored = (D[cand] @ q) - lam * R[cand]     # query term dropped: see below
    return cand[np.argsort(-scored)[:k_final]]

Two details that matter more than the formula.

Drop the query term for ranking. r_Q(q) is constant within a single query, so subtracting it cannot reorder anything. Keep it only if you compare scores across queries.

You must over-retrieve. The penalized score is not the metric your ANN index was built on, so the true top-8 under CSLS may not be in the raw cosine top-8. Pull k_ann at 5–10× k_final and rerank. If you skip this, you get the penalty's cost with a fraction of its benefit.

lam is a tunable knob, and it should be tuned. lam=1 is the textbook CSLS. Some hubs are hubs because they are genuinely the right answer to many questions — a well-written FAQ entry on authentication really does belong in a lot of top-ks. The penalty cannot tell deserved centrality from geometric centrality. Sweep lam over your eval set and watch nDCG, not skew.

3. Rank-based fusion

If you already run hybrid retrieval, Reciprocal Rank Fusion gives you partial hubness resistance for free: it consumes ranks, not scores, so a uniform per-document score bonus only matters where it flips a rank. It won't rescue a hub that is genuinely rank-1 everywhere, but it caps the damage.

Where does this interact badly with my index?

Don't delete hubs from the graph. In HNSW and its ancestors, high-degree nodes act as highways — the greedy descent relies on them to cross the graph in few hops. Pruning hubs to fix retrieval quality degrades navigability, and you'll pay in recall at the same ef_search. Penalize at scoring time; leave the graph alone.

Fixed similarity thresholds are where the query-side term bites. A rule like "only pass chunks with cosine > 0.75 into the prompt" assumes scores are comparable across queries. They aren't — r_Q(q) shifts the entire score distribution per query, so one global cutoff is strict for some queries and permissive for others. If you need a cutoff, compute it relative to the retrieved candidate distribution (z-score against the top-64, or a gap threshold against rank-1) instead of absolutely.

Recompute R on drift. Hub radii are a property of the corpus and the query mix. Bulk ingests and seasonal query shifts both invalidate them. Recomputing is one matmul pass — cheap enough to schedule nightly, expensive enough that you don't want it in the request path.

A cross-encoder reranker masks this, it doesn't solve it. Reranking fixes the ordering of what you retrieved. Hubness determines what you retrieved. If a hub displaced the correct chunk out of the top-64, no reranker recovers it — and you paid full reranker cost to reorder garbage.

So why does one chunk top every RAG query?

Because in high-dimensional embedding spaces, cosine similarity contains a query-independent component: a document's alignment with the corpus mean direction adds roughly the same bonus to its score no matter what was asked. Documents with a large such component become hubs, entering the top-k of a hugely disproportionate share of queries, while a long tail of documents becomes unreachable. It is geometry, not chunking, which is why boilerplate filters only move the problem to the next-most-central chunk. Measure it with k-occurrence skewness and orphan rate over your own index, remove the dominant direction from both queries and documents as a bakeable linear transform, and subtract a per-document CSLS hub penalty during an over-retrieved rerank pass. Tune the penalty weight on your eval set, because some documents are central for good reasons.