Embedding Anisotropy: Why Cosine Similarity Never Hits Zero

Take two sentences that have nothing to do with each other — "The mitochondria is the powerhouse of the cell" and "I need to renew my parking permit by Friday." Embed both with a standard model, compute cosine similarity, and you will not get a number near zero. You will get something like 0.6, sometimes higher. Encode two random strings and you still get a stubbornly positive number.

This is embedding anisotropy, and if you have ever picked a similarity threshold for RAG retrieval and watched it behave nonsensically across corpora, this is why. The vectors your model produces do not fill the space uniformly. They live inside a narrow cone, and cosine similarity — which measures the angle between vectors — is being computed inside a space where every vector already points roughly the same way.

Key takeaways

• Embedding anisotropy means learned embeddings occupy a narrow cone in vector space, so cosine similarity between unrelated text is systematically positive (often 0.5–0.9 for older models), not 0.
• The cause is training dynamics: a few high-magnitude "rogue" dimensions and a nonzero mean vector dominate every embedding, adding a constant angular offset.
• Absolute cosine thresholds (score > 0.8) are therefore not portable across models or corpora — the baseline similarity floor shifts.
• Fixes: mean-centering, all-but-the-top whitening, and z-scoring the dimensions. Contrastively trained modern models (OpenAI text-embedding-3, Cohere v3) are more isotropic but still not centered at zero.
• The robust production move is to rank, not threshold — or calibrate the threshold per corpus against a measured random-pair baseline.

What is embedding anisotropy?

Anisotropy is the property that embeddings are not distributed uniformly over directions. In an isotropic space, if you sample two random vectors, their expected cosine similarity is 0 — they are as likely to point one way as another. In an anisotropic space, random vectors share a dominant direction, so their expected cosine similarity is a positive constant.

You can measure it directly. Take a batch of unrelated sentences, embed them, and compute the mean pairwise cosine similarity. In an isotropic space that number sits near 0. For BERT-family and many older sentence encoders it lands well above 0.5. That gap is the anisotropy.

Ethayarajh's 2019 analysis of contextualized representations quantified this: in the upper layers of models like GPT-2 and BERT, two random words can have cosine similarity above 0.95. Gao et al. named the underlying cause the "representation degeneration problem." The embeddings degenerate into a narrow cone during training.

Why do unrelated sentences get high cosine similarity?

Because the embedding distribution has a large nonzero mean and a handful of dominant dimensions, and cosine similarity does not remove either.

Decompose any embedding v into its mean and its deviation: v = μ + (v − μ). The mean vector μ — averaged over your whole corpus — is not zero. It is a real direction with real magnitude. Every single embedding contains that μ component. When you take the cosine between two embeddings, the shared μ contributes a positive term to the dot product regardless of what the sentences actually mean. That is your similarity floor.

On top of the mean, transformer embeddings are notorious for rogue dimensions (also called outlier or "massive activation" dimensions). A small number of coordinates carry variance orders of magnitude larger than the rest. Kovaleva et al. and others traced these to LayerNorm and training dynamics. Because cosine similarity is a normalized dot product, these few high-magnitude dimensions dominate the numerator. Two vectors that both have a large positive value in dimension 588 will look similar even if every other coordinate disagrees.

So the geometry is: a strong common mean, plus a couple of axes that soak up most of the magnitude. The genuine semantic signal — the part you actually want to retrieve on — lives in the low-variance residual, and it is being drowned out.

Why does this break RAG retrieval thresholds?

It breaks thresholds because the baseline is not zero and it is not constant. Developers write retrieval guards like:

hits = [d for d, score in results if score > 0.8]

That 0.8 encodes an assumption: unrelated content scores low, relevant content scores high, and 0.8 is a clean gap in between. Under anisotropy, unrelated content scores maybe 0.6, and truly relevant content scores 0.82. Your gap is 0.06, not 0.8, and it drifts every time you change the corpus, the model, or even the average document length.

Swap text-embedding-ada-002 for a different provider and the same 0.8 cutoff either returns nothing or returns everything, because each model has a different anisotropy floor. The threshold was never measuring semantic relevance in absolute terms. It was measuring relevance plus a per-model constant you never subtracted.

This is also why raw cosine scores are misleading in dashboards. A "0.72 similarity" looks like strong agreement to a stakeholder. It might be barely above the random-pair floor.

How do you measure and fix anisotropy?

First measure your baseline. Embed a set of mutually unrelated texts and compute the mean off-diagonal cosine similarity. That number is your anisotropy floor — the score two random things get for free.

import numpy as np

def cosine_matrix(X):
    Xn = X / np.linalg.norm(X, axis=1, keepdims=True)
    return Xn @ Xn.T

def anisotropy_floor(embeddings):
    """Mean cosine similarity of unrelated pairs. ~0 is isotropic."""
    S = cosine_matrix(embeddings)
    n = S.shape[0]
    off_diag = S[~np.eye(n, dtype=bool)]
    return off_diag.mean()

# embeddings: (N, d) from unrelated sentences
print("floor:", anisotropy_floor(embeddings))  # e.g. 0.61

The cheapest correction is mean-centering: subtract the corpus mean before computing similarity. This kills the shared μ term directly.

mu = embeddings.mean(axis=0, keepdims=True)
centered = embeddings - mu
print("floor after centering:", anisotropy_floor(centered))  # drops toward ~0

The stronger correction is all-but-the-top (Mu & Viswanath), which removes the top few principal components — the rogue directions — after centering. You mean-center, run PCA, and project out the top k components (typically k = 2 or 3 for hundreds of dimensions).

def all_but_the_top(embeddings, k=2):
    mu = embeddings.mean(axis=0, keepdims=True)
    X = embeddings - mu
    # top-k principal directions
    U, S, Vt = np.linalg.svd(X, full_matrices=False)
    top = Vt[:k]                     # (k, d)
    proj = X @ top.T @ top           # component along dominant axes
    return X - proj                  # residual: the semantic signal

cleaned = all_but_the_top(embeddings, k=2)
print("floor after ABTT:", anisotropy_floor(cleaned))

A related technique is whitening: center, then apply the inverse square root of the covariance matrix so every dimension has unit variance and dimensions are decorrelated. Whitening equalizes the rogue dimensions instead of deleting them. It reliably improves semantic-similarity benchmarks on older encoders, though it can overfit if you estimate the covariance on too little data — fit it on a representative sample of your corpus, not on ten documents.

One caveat: PCA/whitening transforms must be fit once on a reference set and reused, not recomputed per query. If you re-fit on each batch, your vectors stop being comparable across batches and retrieval falls apart.

Do modern embedding models still have this problem?

Less than older ones, but yes. Models trained with a contrastive objective — pulling positives together and pushing negatives apart — explicitly penalize the "everything is similar" degenerate solution. That is a large part of why text-embedding-3-large, Cohere Embed v3, and modern sentence-transformers models behave far better than raw BERT [CLS] vectors. Contrastive training with hard negatives is, in effect, an isotropy regularizer.

But "better" is not "centered at zero." The random-pair floor is smaller, yet still positive, and it still varies by model. Do not assume a fresh model frees you from measuring the baseline. Embed some unrelated pairs and check the floor before you hardcode any threshold.

The practical stance: prefer ranking over absolute thresholds. Top-k retrieval only cares about the order of scores, and a constant mean offset barely perturbs order for a fixed query. Where you must threshold — dedup, "no good answer" gating, semantic cache hits — calibrate the cutoff against a measured random-pair baseline for that exact model and corpus, and re-measure when either changes. Add a cross-encoder reranker on the shortlist when precision matters; rerankers score query–document pairs jointly and are not subject to the same shared-cone geometry.

The bottom line

Embedding anisotropy is why cosine similarity between unrelated sentences never hits zero: learned embeddings collapse into a narrow cone dominated by a nonzero mean vector and a few high-magnitude rogue dimensions, so every pair inherits a constant positive similarity floor before any semantic signal is counted. That floor is why absolute cosine thresholds like score > 0.8 do not port across models or corpora. Measure your random-pair baseline first, mean-center or apply all-but-the-top whitening to remove the shared directions, and prefer ranking or per-corpus calibrated thresholds over hardcoded cutoffs. Modern contrastively trained models shrink the floor but do not eliminate it — so measure, don't assume.