A better embedding model is free to adopt and expensive to install. You have to re-embed the whole corpus, and for the days or weeks that takes, half your index is in one vector space and half in another. A cosine across that seam is not a degraded score, it is noise - the two models' axes have nothing to do with each other. Ranking 400 documents by it finds the answer 2.7% of the time, against 2.5% for picking ten at random.
The whole frontier, priced in embedding calls: https://dev48.infy.uk/ai/days/day67-embedding-migration.html
Cost is embedding calls, so the curves do not line up
planCost(f, docs, queries, encoders) = f*docs + encoders*queries
breakEvenQueries(f, docs) = docs * (1 - f) // f*D + 2Q == D + Q
A second query-side encoder costs one extra call on every request for as long as the migration runs, so the two-encoder curves start half a corpus to the right and only vertical comparisons are legal. At an identical 150k calls you buy either the finished re-index at recall 0.973 or a half-migrated dual-encoder one at 0.917.
Order beats the arithmetic, at an identical budget
Every row re-embeds the same 30% of documents. The only difference is which ones.
| plan | traffic covered | recall@10 |
|---|---|---|
| new encoder only, hot first | 81.0% | 0.803 |
| both encoders z-scored, random | 26.7% | 0.673 |
| both encoders raw, random | 26.7% | 0.593 |
| new encoder only, random | 26.7% | 0.310 |
Recall tracks the fraction of traffic whose answers are converted, not the fraction of documents, and those are the same number only if you convert in an order uncorrelated with demand. Traffic is Zipf: the hottest 30% of documents covers 81% of queries, a random 30% covers 27%.
Ordering does not make the hole shallower. It is 51 points deep either way, 0.613 down to 0.107, the instant the query encoder swaps and nothing has been converted. What hot-first changes is how long you stay down there: even again at 10% re-embedded against 60%, and 10% of the migration under water against 57%.
The recommendation the measurement took away
Keeping both encoders alive removes the seam, and merging their raw cosines is unfair: the better and more anisotropic model scores everything higher, so at 30% converted the migrated documents take 61.6% of the top-10 slots instead of 30%. Z-scoring each space's candidates restores the fair share to 31.5%. That was the page I meant to write.
Then under hot-first ordering the fair merge scores worse - 0.873 against 0.897 - because the documents the bias favoured are precisely the popular ones the queries want. Under a random order, where the favoured documents are a random sample, the correction is worth a clear 8 points. So the null result is in the test file, where the inconvenient half of a finding cannot be quietly deleted later:
assert(recall(dual, hot, 0.3) > recall(dualz, hot, 0.3)); // normalising HURTS here
Normalisation is neither right nor wrong. It is right exactly when your migration order is uncorrelated with your traffic.
On a real system
Never swap the query encoder first: it is the one-line change and the one that digs the valley. Sort the backfill by query count, which is free. Write to both indexes on ingest so the backlog stops growing. And version the vectors - a row that does not record which model produced it makes every one of these decisions unavailable to you.
Part of a from-scratch series - one AI concept a day, measured in-browser: https://dev48.infy.uk/aifromzero.php
Top comments (0)