Upgrading Your Embedding Model Doesn't Break RAG Loudly — It Breaks It Quietly

The problem

A team I was helping upgraded their embedding model to cut cost — swapped an older general-purpose embedding model for a newer, cheaper one. No schema change, no downtime, no errors in any log. Over the next three weeks, support tickets crept up: "the assistant is confidently answering with the wrong doc." Nobody connected it to the embedding swap because nothing had crashed. Retrieval doesn't throw an exception when it's wrong. It just returns the nearest vectors — and "nearest" quietly stopped meaning anything.

Why it happens

Here's the part that trips people up: embedding spaces are not portable across models. Two different embedding models can both output 1536-dimensional vectors, both be excellent, and still be totally incompatible with each other — because "dimension 47" in model A's space and "dimension 47" in model B's space encode nothing in common. Each model learns its own geometry during training, shaped by its own objective and data. There's no shared coordinate system, no translation layer, no reason two models would ever agree on what "close" means.

So when you re-embed only new documents with the new model but leave old vectors sitting in the same index — which is what happened here, because a full reindex looked expensive and "we'll backfill later" — you end up with a vector store where some entries speak model A and some speak model B. A query embedded with model B gets compared against both. Against the model-B vectors, cosine similarity is meaningful. Against the model-A vectors, it's closer to noise — sometimes high, sometimes low, with no reliable relationship to actual semantic relevance.

I ran a quick sanity check to see how bad "noise" actually looks in practice:

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# same-model vectors for related concepts cluster tight and high
same_model_sim = 0.83   # typical for genuinely related text, same model

# cross-model comparison: query embedded with model B,
# candidate embedded (weeks ago) with model A
cross_model_sims = [0.71, 0.44, 0.79, 0.52, 0.68]  # no relationship to actual relevance

The cross-model numbers aren't uniformly bad — that's the trap. Some land high by coincidence, which is worse than all of them landing low, because a high score that means nothing still gets retrieved with confidence and handed straight to your LLM as "relevant context." The model doesn't hesitate on garbage context. It writes a fluent, confident answer built on a document that was never actually related to the question.

What to do about it

Treat an embedding model change like a schema migration, not a config tweak. A few things that actually hold up in production:

1. Full reindex, not incremental backfill. If the model changes, every vector in that index needs to be re-embedded with it. Partial migrations are the exact failure mode above — a two-model index that looks fine and silently isn't.
2. Version-tag every vector's metadata with the embedding model name and version. It costs one field and lets you query "how much of my index is stale" instead of guessing.
3. Shadow-evaluate before flipping. Stand up the new index in parallel, run a fixed eval set of real queries through both, and compare retrieval@k and answer quality before it's live. This is the step that gets skipped under time pressure, and it's the one that would've caught this in an afternoon instead of three weeks of tickets.
4. Never mix embedding models in one index, even "temporarily." Temporary is exactly when nobody's watching.
5. Batch the re-embedding cost down, don't skip it — queue it, rate-limit it, run it overnight. It's still cheaper than a support queue full of confidently wrong answers.

Key takeaways

• Different embedding models produce vector spaces that are not comparable, even at matching dimensions.
• Mixing vectors from two models in one index doesn't fail loudly — it silently corrupts a subset of your retrieval, sometimes convincingly.
• Treat embedding model upgrades as full-index migrations with a version tag and a shadow evaluation, not a drop-in model swap.