- Book: AI That Reads
- The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
A better embedding model ships. Or you change your chunking. Or you add the
heading path to the embedded text. Any of those means every vector you have is
now inconsistent with every vector you are about to create.
The obvious approach is a script that walks the table and updates each row.
That approach has a window — minutes to hours, depending on corpus size —
during which your index contains vectors from two different models, and
distances between them are arithmetic nonsense.
No error is raised. Retrieval quality just degrades unpredictably, then
recovers, and users describe it as "the AI was weird this afternoon."
Why mixing spaces is worse than it sounds
Two embedding models do not produce comparable coordinates. A cosine distance
of 0.2 between a v1 vector and a v2 vector means nothing at all — it is not
"slightly related", it is a comparison between incommensurable things.
During a partial migration, ranking is a mix of meaningful and meaningless
distances. The results look plausible, which is the problem. There is no
signal that anything is wrong, and the effect varies with how far the
migration has progressed.
So: never let two spaces coexist in one queryable set.
Version the space, not the row
The schema change comes first, and it is what makes everything else possible.
ALTER TABLE chunks ADD COLUMN embed_model text NOT NULL DEFAULT 'v1';
ALTER TABLE chunks ADD COLUMN embed_dims int NOT NULL DEFAULT 1536;
CREATE UNIQUE INDEX chunks_unique
ON chunks (tenant_id, content_hash, embed_model);
Now the same chunk can exist twice — once per model — without collision, and a
query is always scoped to one space:
const ACTIVE = process.env.EMBED_MODEL ?? "v1";
export async function search(q: string, scope: Scope, k = 10) {
return db.$queryRaw`
SELECT id, text FROM chunks
WHERE tenant_id = ${scope.tenantId}
AND embed_model = ${ACTIVE}
AND deleted_at IS NULL
ORDER BY embedding <=> ${await embed(q, ACTIVE)}::vector
LIMIT ${k}`;
}
embed(q, ACTIVE) takes the model as an argument. A query embedded with v2
and compared against v1 rows is the same bug from the other direction, and
passing the model explicitly at both ends makes it hard to get wrong.
If your vector column is dimensioned (vector(1536)) and the new model has a
different size, you need a second column or a second table. Check that before
planning the rest.
Backfill in the background, without touching live rows
export async function backfill(target: string, batch = 500) {
for (;;) {
const todo = await db.$queryRaw<Chunk[]>`
SELECT c.id, c.text, c.content_hash, c.tenant_id, c.doc_id
FROM chunks c
WHERE c.embed_model = ${ACTIVE}
AND c.deleted_at IS NULL
AND NOT EXISTS (
SELECT 1 FROM chunks n
WHERE n.content_hash = c.content_hash
AND n.tenant_id = c.tenant_id
AND n.embed_model = ${target})
LIMIT ${batch}`;
if (!todo.length) return;
const vecs = await embedBatch(todo.map((t) => t.text), target);
await db.$transaction(
todo.map((t, i) => db.chunk.upsert({
where: { tenant_content_model: {
tenantId: t.tenantId, contentHash: t.contentHash, embedModel: target } },
create: { ...t, embedModel: target, embedding: vecs[i] },
update: { embedding: vecs[i] },
})),
);
metrics.increment("reindex.chunks", todo.length);
}
}
NOT EXISTS makes it resumable: kill it, restart it, and it continues from
wherever it stopped. No cursor to persist, no partial state to reason about.
Live traffic is untouched throughout, because ACTIVE still points at v1.
Dual-write while both spaces exist
The one thing that breaks a long backfill: documents change during it. A chunk
ingested after the backfill passed its position exists only in v1, and you cut
over to a v2 index that is silently missing this week's content.
So during the migration, ingest writes both:
const WRITE_MODELS = (process.env.EMBED_WRITE ?? "v1").split(",");
export async function indexChunk(c: NewChunk) {
for (const model of WRITE_MODELS) {
const vec = await embed(c.text, model);
await db.chunk.upsert({ /* ... keyed by (tenant, hash, model) */ });
}
}
EMBED_WRITE=v1,v2 for the duration, back to a single value afterwards. It
doubles ingest cost temporarily, which is the price of not needing a freeze.
Verify before you switch, not after
Cutting over on "the backfill finished" is how you discover a gap in
production. Two checks, both cheap.
Completeness — every live chunk exists in the target space:
SELECT count(*) FROM chunks c
WHERE c.embed_model = 'v1' AND c.deleted_at IS NULL
AND NOT EXISTS (SELECT 1 FROM chunks n
WHERE n.content_hash = c.content_hash
AND n.tenant_id = c.tenant_id
AND n.embed_model = 'v2');
Must be zero. Not "small" — zero.
Quality — the new space is not worse on your labelled questions:
const before = await recallAt("v1", GOLDEN, 10);
const after = await recallAt("v2", GOLDEN, 10);
console.log({ before, after });
if (after < before - 0.05) throw new Error("regression; do not cut over");
This is the step that justifies keeping twenty labelled questions around. A
"better" embedding model is better on someone's benchmark, not necessarily on
your corpus, and finding that out after the cutover is expensive.
Cut over with a flag, not a deploy
export function activeModel(scope: Scope): string {
return flags.enabled("embed-v2", scope.tenantId) ? "v2" : "v1";
}
Percentage rollout, per-tenant, instantly reversible. A deploy-based switch
means the rollback is another deploy, at the worst possible moment.
Keep v1 rows for a couple of weeks after full rollout. Deleting them the same
day removes your ability to revert, and the storage is cheap by comparison.
DELETE FROM chunks WHERE embed_model = 'v1'; -- after the soak, not before
The cache needs the model in its key
If you cache embeddings by content hash alone, the backfill will return v1
vectors for v2 requests, and silently produce exactly the mixed-space
corruption the whole plan was designed to avoid.
const key = `emb:${model}:${dims}:${sha256(normalise(text))}`;
Model and dimension in the key. This is the single most likely way to get this
migration wrong while believing it went fine.
The shape, in five lines
Add embed_model to the schema and to every query. Backfill into the new
space with NOT EXISTS so it resumes. Dual-write during the window. Verify
completeness is zero and recall has not regressed. Cut over behind a flag,
soak, then delete the old space.
The version that skips all of this — a script that updates rows in place — is
faster to write and produces an afternoon nobody can explain.
If this was useful
AI That Reads covers the operational
side of a RAG corpus — model migrations, dual-write, reconciliation, and the
labelled set that tells you whether a change was an improvement.
The full series is at
xgabriel.com/ai-in-typescript.
Top comments (1)
We hit this in our pipeline when we switched embedding models mid-flight. Entity nodes encoded with ada-002 and nodes with text-embedding-3-small don't share a meaningful cosine distance space, so nearest-neighbor was returning junk for about 12% of lookups over a 3-week window before we caught it. The cache key must include model_name plus dims; we'd missed that for months and the phantom cache hits in Redis were masking the contamination. Not sure if your versioned schema caught this before queries ran or if you saw similar silent degradation first.