Embedding Hubness: Why One Chunk Shows Up in Every RAG Query

There is a chunk in your index that gets retrieved for "how do I rotate API keys," for "what is the refund window," and for "does the SDK support streaming." It's usually a legal disclaimer, a nav footer, or a two-line "Overview" paragraph. You delete it, and a different chunk takes its place within a week.

That is not a bug in your chunker and it is not a bad embedding model. It's embedding hubness: a structural property of nearest-neighbor search in high-dimensional spaces where a small number of vectors appear in a wildly disproportionate share of everyone's k-NN lists. It has been known in the ML literature since Radovanović et al. (2010), and almost nobody building RAG measures it.

TL;DR

• Embedding hubness is the skew in k-occurrence: how often a document appears in some query's top-k. In high dimensions this distribution becomes heavily right-tailed — a few "hubs" appear everywhere, and a long tail of "anti-hubs" appear never.
• The cause is a per-document query-independent bias term. For a doc d, its expected score over the query distribution is ⟨μ_Q, d⟩. Docs aligned with the mean query direction get a constant boost on every query, unrelated to relevance.
• Diagnose it with one matrix multiply: compute k-occurrence counts over a query sample and take the skewness. Skew above ~1.5 and any doc landing in >5% of top-10 lists means you have hubs.
• Fix it by scoring (q·d − μ_d) / σ_d instead of q·d. This is a per-doc affine transform, so you can bake it into the stored vectors with one extra dimension and keep using an inner-product ANN index — no reranking latency, no exhaustive search.
• Anti-hubs are the expensive half. Hub chunks look plausible in eyeball tests, so the recall hole they create stays invisible in aggregate metrics.

What is embedding hubness, and why does one chunk dominate RAG retrieval?

Hubness is the observation that as intrinsic dimensionality grows, the number of times a point appears in other points' k-nearest-neighbor lists stops being roughly uniform and becomes power-law-ish. If you have 100k chunks, 10k queries, and k=10, every chunk should average 1 appearance. In a hubby index, the top chunk shows up 2,000 times and 60% of chunks show up zero times.

The naive intuition — "the retriever thinks that chunk is relevant to everything" — is wrong in an important way. The retriever doesn't think anything about relevance. It computes a dot product, and the dot product has a component that doesn't depend on the query at all.

Why does high dimensionality create hubs?

Decompose the score. Assume L2-normalized embeddings, so cosine equals inner product. For a fixed document d, treat the query q as a random draw from your query distribution with mean μ_Q and covariance Σ_Q:

E_q[ q·d ]   = μ_Q · d           # bias: same for every query
Var_q[ q·d ] = dᵀ Σ_Q d          # signal: what actually varies with the query

Ranking only cares about differences in score across documents for one query. But the bias term differs across documents and is constant across queries. So the ranking you get is signal + bias, and whenever ‖μ_Q‖ is large relative to the spread of Σ_Q, the bias dominates.

This is where high dimensionality bites. Distance concentration means the query-dependent part of the score has shrinking relative variance as dimension grows, while the bias term does not shrink. The ordering of documents converges toward the ordering by μ_Q · d — i.e., toward "how close is this chunk to the centroid of query-space," which is a property of the chunk alone.

Two things make this much worse in practice than in theory:

Anisotropy. Transformer sentence encoders do not produce isotropic embeddings. Mean-pooled encoders in the E5/BGE/GTE family concentrate mass in a narrow cone; random unrelated pairs often score 0.6–0.8 cosine rather than ~0. That giant shared direction is μ_Q, and it's huge. Newer models (OpenAI's text-embedding-3-*, Voyage, Cohere v3) are better behaved, but "better" is not "centered."

Chunk length asymmetry. Short, generic chunks sit closer to the centroid because mean pooling over few tokens averages out topic-specific directions. Your boilerplate is short and generic. It's the perfect hub.

How do I detect embedding hubness in my vector index?

One matrix multiply over a sample. You need a few thousand real queries — production logs are ideal; synthetic queries generated per-document are an acceptable substitute.

import numpy as np
from scipy.stats import skew

# D: (N, dim) doc embeddings, Q: (M, dim) query embeddings — L2-normalized
D = doc_emb / np.linalg.norm(doc_emb, axis=1, keepdims=True)
Q = qry_emb / np.linalg.norm(qry_emb, axis=1, keepdims=True)

k = 10
S = Q @ D.T                                   # (M, N) cosine scores
topk = np.argpartition(-S, k, axis=1)[:, :k]  # top-k doc ids per query

Nk = np.bincount(topk.ravel(), minlength=len(D))   # k-occurrence

print(f"expected per doc : {k * len(Q) / len(D):.2f}")
print(f"k-occurrence skew: {skew(Nk):.2f}")
print(f"anti-hubs (N_k=0): {100 * (Nk == 0).mean():.1f}% of corpus")

for i in np.argsort(-Nk)[:10]:
    print(f"  doc {i:>7}: in {Nk[i]:>5} / {len(Q)} top-{k} lists "
          f"({100 * Nk[i] / len(Q):>5.1f}%)  |  {docs[i][:70]!r}")

Read it like this: skewness near 0 means a healthy, near-uniform index. Skew above ~1.5–2 means real hubs. The line that actually convinces people is the last one — printing the hub text. It is always something you'd never want retrieved.

If your corpus is too big for a dense Q @ D.T, run the same thing against your ANN index; hubness measured through HNSW is if anything more skewed, because hubs also become highly-connected graph nodes that every search traverses.

Does centering the embeddings fix hubness?

Partially, and it's the right first move: subtract the corpus mean from document vectors and the query mean from query vectors, then renormalize. Removing the dominant shared direction shrinks ‖μ_Q‖ and immediately flattens the k-occurrence distribution.

Two caveats that trip people up.

Use separate means for queries and documents. With asymmetric encoders (query: / passage: prefixes in E5, or dedicated query encoders), the query manifold and document manifold sit in different regions. Subtracting one global mean from both leaves the cross-manifold offset intact and can make things worse.

Centering is a rank-1 fix for a higher-rank problem. "All-but-the-top" style whitening (removing the top few principal components) goes further, but it changes every score in your system: your cosine > 0.75 cutoff, your rerank-or-not threshold, and any calibration you did downstream are all invalidated. Budget for refitting them.

How do I correct hubness without giving up ANN search?

Score against the document's own score distribution instead of raw cosine:

score(q, d) = (q·d − μ_d) / σ_d      where μ_d = E_q[q·d], σ_d = std_q[q·d]

This asks the right question: not "is this doc's score high," but "is this doc's score high for this doc." A hub with a baseline of 0.72 that scores 0.74 on your query loses to a specialist with a baseline of 0.31 that scores 0.58.

The catch appears to be that this is a per-document normalization, which sounds like it forces exhaustive scoring. It doesn't — it's affine in q, so you can fold it into the stored vectors and one extra dimension:

import faiss

# S from the diagnostic above: (M, N) scores over a held-out query sample
mu = S.mean(axis=0)
sd = S.std(axis=0)

# shrink sigma toward the global value when the query sample is small
lam = min(1.0, len(Q) / 5000)
sd = np.sqrt(lam * sd**2 + (1 - lam) * sd.mean()**2) + 1e-6

# q_aug · d_aug == (q·d - mu_d) / sd_d
D_aug = np.hstack([D / sd[:, None], (-mu / sd)[:, None]]).astype("float32")
Q_aug = np.hstack([Q, np.ones((len(Q), 1))]).astype("float32")

index = faiss.IndexHNSWFlat(D_aug.shape[1], 32, faiss.METRIC_INNER_PRODUCT)
index.add(D_aug)                       # build the graph on corrected vectors
scores, ids = index.search(Q_aug, 10)

The augmented vectors are no longer unit norm, so this must be an inner-product index, not cosine or L2 — and the graph has to be rebuilt on the corrected vectors, not patched onto an existing one. Query cost is unchanged apart from one extra dimension.

If you'd rather not touch the index, do the same correction at rerank time: pull top-100 with raw cosine, rescore with (s − μ_d)/σ_d, keep top-10. You need to store μ_d, σ_d per chunk (8 bytes) and you're done. This catches most of the win because hubs are still in the raw top-100 — they're just no longer at the top of it. It cannot fix anti-hubs that never make the top-100 at all; only the baked-in version does that.

What breaks when you deploy this?

Your score thresholds. Post-correction scores are z-scores, roughly centered near 0 with a scale of 1, not cosines in [0.3, 0.9]. Any "if best_score < 0.7, say I don't know" guard fires on everything. Refit it on the new scale before shipping.

Your query sample leaks. If you estimate μ_d on the same queries you evaluate on, the correction absorbs real relevance signal for those exact pairs and your eval numbers improve for the wrong reason. Estimate on production traffic or synthetic queries, evaluate on held-out.

Cold-start chunks. Newly indexed documents have no μ_d. Initialize from the corpus-wide mean of μ for chunks of similar length, and re-estimate on a schedule. Don't let a fresh chunk default to μ_d = 0 — it will outrank everything.

It is not a substitute for chunk hygiene. If nav bars and cookie banners are in your index, delete them. Hubness correction makes a well-built index behave better; it makes a garbage index produce differently-shaped garbage.

One more thing worth internalizing: contrastive fine-tuning with in-batch negatives reduces hubness — that's part of why fine-tuned retrievers beat off-the-shelf ones by more than the eval delta suggests — but it does not eliminate it. Measure after fine-tuning too.

So why does one chunk show up in every RAG query?

Because cosine similarity between a query and a document contains a term that doesn't depend on the query. In high-dimensional, anisotropic embedding space, that per-document bias — essentially how close the chunk sits to the centroid of query space — can outweigh the genuine query-document signal, so short generic chunks near the centroid become hubs that surface in nearly every top-k, while a long tail of specific chunks become anti-hubs that are never retrieved at all. Measure it with k-occurrence skewness over a sample of real queries; fix it by scoring (q·d − μ_d)/σ_d, which you can bake into your stored vectors with one extra dimension and serve from an ordinary inner-product ANN index. Then go refit your score thresholds.