The provider ships v2, or somebody changes the tokenizer. The index is still there, still returning five results, still with plausible scores.
Change the width and the vector is the wrong shape, so a typed index refuses it outright — and even with that check removed, precision@5 only falls from 0.9417 to 0.7750.
Change the hash seed or the tokenizer and the width is identical, nothing anywhere objects, and precision@5 falls to 0.0333, 0.0917 and 0.1000 — against a chance baseline of 0.0833.
👉 Live, the whole grid computed in your browser: https://dev48v.infy.uk/ai/days/day79-embedding-drift.html
Nothing here calls an embedding API
The embedder is a hashed bag-of-tokens implemented in full — FNV-1a with a stated seed, the signed hashing trick, optional idf, L2 normalisation, cosine. A "version" is a concrete configuration of that embedder, so a version change here is one people actually make.
The corpus is synthetic and declared: 12 topics × 8 characteristic terms, 10 documents each. Because every document's topic is known by construction, retrieval is scored for correctness, not just for agreement with an earlier run.
The detectable failures are the mild ones
| version change | does a typed store catch it? | precision@5 after a full re-index | precision@5 if only the query moves |
|---|---|---|---|
| same version (control) | — | 0.9417 | 0.9417 |
| width 128 → 256 | yes — wrong width | 0.9583 | 0.7750 |
| width 128 → 64 | yes — wrong width | 0.8583 | 0.6333 |
| hash seed 1 → 2 | no — same width | 0.9250 | 0.0917 |
| 4-grams → 3-grams | no — same width | 0.9083 | 0.0333 |
| 4-grams → words | no — same width | 0.8583 | 0.1000 |
| idf weighting on | no — same width | 0.9333 | 0.9250 |
Sort by damage and you get almost exactly the reverse of sorting by detectability. The 3-gram row lands at 0.0333 — below the 0.0833 you would get by returning five documents at random.
The vectors carry no version tag. Cosine is happy to compare any two of the same length. There is no point in that path where a mismatch can be detected.
The half-migrated index is the shape this actually takes
The migration script runs on new ingests and nobody backfills. Half the index is in one space, half in another:
| half-migrated | single-version | difference | |
|---|---|---|---|
| new documents' share of top-5 | 21.67% | 52.50% | −30.83 points |
| precision@5 | 0.4667 | 0.9417 | −50.4% |
| mean top-1 cosine | 0.3992 | 0.5689 | −29.8% |
| queries returning five documents | 24 of 24 | 24 of 24 | none |
Those documents are not ranked badly. They are unreachable — and the system returns five results every time, drawn from the half that still lines up. No error, never an empty result set.
The monitor you would reach for is the wrong shape too
Without labels, the one thing you can watch is the similarity score. On a total mismatch it does move: mean top-1 cosine falls from 0.5689 to about 0.23, a ~60% drop. A threshold catches that.
On the half-migrated index it falls 29.8% while precision falls 50.4%. Every query still has a well-matched document somewhere in the half that lines up, so the top score stays respectable while half the corpus has quietly left the building.
A threshold tuned on the loud failure will not fire on the quiet one, and the quiet one is the one that happens.
Even the correct migration moves your answers
Re-embedding everything is the right fix and it works — precision comes back to 0.858–0.958, in one case slightly above the original. But top-5 overlap with the old index runs from 0.933 down to 0.725: between 6.7% and 27.5% of retrieved documents are different afterwards.
Nothing is broken. It is just that any prompt, cache, eval set or human sign-off pinned to the old results is now pinned to results that no longer come back — on the day you did everything right.
The fix, and the honest scope
Write the embedder version into the index and refuse to serve a query whose version does not match. Every consequence above follows from the vectors not carrying one.
Scope, because it changes how you read the table: the embedder is a hashed bag-of-tokens, not a neural model. A hash-seed change produces two unrelated spaces — that is the extreme case and an upper bound, not a prediction for what v1→v2 of a trained model does. The tokenizer rows are the better analogue, and they are nearly as bad.
Nothing here models semantic drift: there is no meaning in this corpus beyond term overlap, so this says nothing about a v2 that is simply better at understanding a question. 120 documents, 24 queries, top-5, plain cosine, no chunking, no reranker, no hybrid keyword leg.
28 in-page checks, 95 verifier assertions, 0 failures.
Top comments (0)