Cosine Similarity Threshold 0.8: Embedding Anisotropy Broke My RAG

My RAG bot had exactly one safety rule: if the best chunk scores below 0.8 cosine similarity, answer "I don't know." That cosine similarity threshold ran for months without complaint. Then I swapped the embedding model, redeployed, and the bot said "I don't know" to everything. Including "how do I reset my password," which has its own dedicated doc page.

Nothing crashed. No errors. Retrieval still returned the right chunk at rank 1. My gate just threw it away, every single time.

The worst part: when I went digging, I found the old setup had been broken too. In the other direction.

TL;DR

• A cosine similarity score has no universal meaning. 0.8 can be "near-duplicate" for one embedding model and "random noise" for another.
• Embedding anisotropy is the cause: most models pack all vectors into a narrow cone, so even unrelated texts score high. How narrow depends on the model.
• Contrastive training only teaches ranking, not absolute scores. Adding a constant to every similarity doesn't change the loss, so nothing pins 0.8 to "relevant."
• Fix: measure the baseline similarity of unrelated pairs in your corpus, calibrate the threshold on a small labeled set, and recalibrate on every model change.
• Better still, gate on relative signals (score vs. baseline percentile, top-1 margin) or a reranker, not a raw cosine number.

What happened when I switched embedding models?

The rank order survived the migration, but the score scale collapsed. Every relevant match dropped well below my 0.8 threshold, so the gate rejected all of them.

Here's what my logs showed, roughly, on the same internal docs corpus:

	Old model (ada-002)	New model (text-embedding-3-small)
Top-1 score, questions with a real answer	~0.84 median	~0.55 median
Score between two random unrelated chunks	~0.72 median	~0.18 median

Look at the right column first. The new model was better at separating relevant from unrelated. The gap between "real answer" and "noise" got wider. My threshold just sat above the whole distribution.

Now look at the left column. With the old model, random junk scored 0.72 and real answers scored 0.84. My "safety" threshold of 0.8 was sitting in a 0.12-wide band between garbage and gold. I pulled an off-topic test question about pizza in Naples and it scored 0.77 against a Kubernetes runbook. My guardrail was holding by three hundredths of a point.

So the threshold was never a guardrail. It was a coincidence that happened to work for one model.

Why is cosine similarity between unrelated texts so high?

Because most embedding models are anisotropic: their vectors don't spread over the whole sphere, they crowd into a narrow cone. Any two vectors inside a narrow cone have a small angle between them, so cosine similarity is high no matter what the texts say.

This isn't a quirk of one vendor. It showed up in research on BERT and GPT-2 representations years ago (Ethayarajh, 2019, found contextual embeddings occupy a narrow cone), and "representation degeneration" has been studied in language models since.

You can see why with ten seconds of math. Model every embedding as a shared component plus a unique one:

v = m + r
|m|^2 = a^2   (direction every text shares)
|r|^2 = b^2   (what makes this text unique)

If the unique parts are roughly orthogonal to each other and to m, the cosine of two unrelated texts is:

cos(v1, v2) ≈ a^2 / (a^2 + b^2)

If the shared direction carries 70% of each vector's squared length, two texts with nothing in common score about 0.7. All the actual meaning has to fit in the remaining 0.3 of the range. That's exactly what my ada-002 numbers looked like.

Different models end up with different a^2 / (a^2 + b^2). That ratio is your noise floor, and it's not printed on the box.

Why does the score scale change between embedding models?

Because the training objective never asks for a specific score. Most embedding models are trained with a contrastive loss (InfoNCE or a close cousin) that only cares whether the right pair beats the wrong pairs.

The loss for one query looks like:

loss = -log( exp(s_pos / τ) / Σ_i exp(s_i / τ) )

Add a constant c to every similarity s_i and the softmax doesn't change. The exp(c / τ) factor cancels top and bottom. So the model is free to put "unrelated" at 0.1 or 0.7. Nothing in training cares.

The temperature τ and the training data decide how spread out the scores end up. Different model, different recipe, different scale. Your 0.8 doesn't survive the trip.

One more trap: if you shorten embeddings yourself (slicing a Matryoshka-style vector down to fewer dimensions), the vector is no longer unit length. Renormalize after slicing, or your "cosine" is really a dot product with drifting norms.

Is a cosine similarity threshold ever safe to use?

Yes, if it was measured on your corpus, with your exact model, and you re-measure it whenever either changes. A threshold copied from a blog post, a tutorial default, or a previous model is a guess.

The ranking from vector search is usually trustworthy. The absolute number next to it is not, until you calibrate it.

How do I calibrate a cosine similarity threshold?

Two steps: measure the noise floor, then fit the threshold on labeled examples.

Step 1: the noise floor. Sample random pairs of chunks from your own index and look at the distribution. This takes a minute and tells you where "unrelated" lives for this model:

import numpy as np

def noise_floor(E, n=5000, seed=0):
    """E: (N, d) array of unit-normalized chunk embeddings."""
    rng = np.random.default_rng(seed)
    i = rng.integers(0, len(E), n)
    j = rng.integers(0, len(E), n)
    keep = i != j
    sims = np.einsum("nd,nd->n", E[i[keep]], E[j[keep]])
    return np.percentile(sims, [5, 50, 95, 99])

Random pairs from the same corpus overestimate "unrelated" a bit (chunks share topic and jargon), which is fine. You want a conservative floor.

Step 2: a labeled set. Write 40-50 questions your docs answer and 40-50 they don't. Record the top-1 score for each. Then pick the threshold that hits the precision you need:

def pick_threshold(pos_scores, neg_scores, min_precision=0.95):
    candidates = sorted(set(pos_scores) | set(neg_scores))
    for t in candidates:
        tp = sum(s >= t for s in pos_scores)
        fp = sum(s >= t for s in neg_scores)
        if tp and tp / (tp + fp) >= min_precision:
            return t, tp / len(pos_scores)  # threshold, recall
    return None, 0.0

If no threshold gets you there with decent recall, the problem isn't the threshold. Your embeddings can't separate those questions, and you need a reranker or better chunks.

Store the model name and version next to the threshold in config. When someone bumps the model, the stale threshold should fail loudly, not silently.

Does mean-centering embeddings fix anisotropy?

It helps a lot for models with a strong shared direction. Subtract the corpus mean vector, renormalize, and the cone opens up. This is the idea behind "all-but-the-top" post-processing (Mu and Viswanath, 2018), which also removes a few dominant principal components.

mu = E.mean(axis=0)

def center(x):
    x = x - mu
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

E_c = center(E)
q_c = center(q)  # queries get the SAME mu, always

The catches are operational, not mathematical:

• The query must be centered with the exact same mu as the corpus.
• mu drifts as the corpus grows. Recompute it on a schedule and re-embed or re-center the index.
• Your vector index (pgvector, FAISS, whatever) has to store centered vectors, so this is a reindex, not a flag.

Centering makes scores easier to read. It doesn't make them calibrated. You still run step 2.

What should I gate on instead of a raw score?

Gate on signals that survive a model swap. Three that have worked for me:

1. Score vs. noise floor. Accept the top hit only if it beats the 99th percentile of random-pair similarity. When the model changes, recompute the floor and the gate moves with it.
2. Top-1 margin. When the query matches something real, the top hit usually stands out from the next few. When it matches nothing, scores come out flat. top1 - median(top2..top10) is a cheap, scale-relative signal.
3. A cross-encoder reranker for the final relevance call. It reads query and chunk together, so it judges relevance directly instead of comparing two independent vectors. Its scores need calibration too (many output raw logits), but it separates relevant from irrelevant much more cleanly.

The checklist I run now on every embedding change

• Re-embed a sample, plot the random-pair distribution. Write down p50 and p99.
• Re-score the labeled set. Re-pick the threshold. Commit it with the model name.
• Check vector norms. If they aren't ~1.0, normalize or switch to dot product on purpose.
• Diff top-1 results old vs. new on the labeled set. Rank changes are the real migration risk.
• Alert when "I don't know" rate jumps. That metric would have caught my outage in an hour instead of a day.

So, does a cosine similarity of 0.8 mean anything?

On its own, no. A cosine similarity threshold like 0.8 only means something relative to one specific embedding model on one specific corpus, because embedding anisotropy and contrastive training leave each model with its own score scale and its own noise floor. Unrelated texts can score 0.7 in one model and 0.15 in another. Trust the ranking that vector search gives you. Treat the number as uncalibrated until you've measured random-pair similarity in your own index and picked the threshold from labeled examples. Then redo both every time the model changes.

---

Written by the developer behind Preterview, an interview prep platform.