A safe embedding-model migration is a data migration, not an in-place update: build a second index with an explicit embedding version, keep the old index serving, shadow queries against both, and move reads only after the new index meets a retrieval-quality gate and a latency SLO. Never mix vectors produced by different models in one searchable population. For an edtech system that watches course pages and alerts instructors about meaningful changes, that rule preserves alert coverage while chunks are re-embedded in the background.
TL;DR: dual-write newly fetched page revisions, backfill an isolated target index from immutable source text, compare ranked results on a fixed query set, then switch a small routing pointer. Retain the old index until the rollback window closes. The decisive trade-off is retrieval quality versus query and indexing latency; the migration itself should not decide that trade-off under pressure.
How can you reindex after changing an embedding model without downtime?
The obvious failure is downtime, but the quieter failure is worse: the service remains available while recall changes underneath it. A watcher may still return ten chunks quickly, yet miss the paragraph where a prerequisite changed from Algebra I to Algebra II. HTTP availability says nothing about whether the right source reached the alerting stage.
Treat one page revision as the durable unit of work. Store its normalized text, content hash, fetch time, document identity, and chunking version before requesting an embedding. A vector is derived data. If the only surviving representation is the old vector, there is no defensible way to reproduce the target index or explain why an alert changed.
The invariant is compact: a read request targets exactly one complete embedding generation. Do not overwrite old rows chunk by chunk and do not query across dimensions or model families. Cosine similarity compares coordinates inside a shared representation space; changing the function that creates those coordinates invalidates the assumption that old and new vectors are comparable.
A migration also exposes stale-content bugs. Suppose a course page loses a tuition paragraph while the backfill is running. The crawler writes revision 42, the backfill reads revision 41, and an unconstrained worker can resurrect a deleted chunk in the target index. The target write therefore needs both the document ID and expected source revision. If the revision no longer matches, discard the result and enqueue current content.
The two-generation data path
Use a control-plane record to identify the active generation, while each vector row carries enough lineage to reject ambiguity. A practical identity is (document_id, revision, chunk_id, generation). The generation binds the embedding model and its configuration to a chunking version; changing either creates another generation. During backfill, normal ingestion performs dual writes. The old generation continues to satisfy user traffic. The new generation receives fresh revisions plus historical work from a bounded queue, so completion is defined by current source revisions rather than by a misleading count of jobs attempted. Deletes and access-control changes must be applied to both generations too. An index that contains every insertion but misses tombstones is not complete. The route switch should be one transactional update, cached briefly and observed as configuration rather than compiled into application deployments. Each request reads one generation and reports it in traces and metrics. Don't perform a per-query fallback that silently combines old and new rankings; it hides regressions, doubles tail latency, and makes the result impossible to attribute. If the new generation fails its serving SLO, route the next request back to the old generation.
Here is the preventative write path in Go. The interfaces are deliberately generic because queue, vector store, and embedding-provider choices do not change the concurrency contract.
type SourceChunk struct {
DocumentID string
Revision int64
ChunkID string
Text string
}
type VectorRow struct {
DocumentID string
Revision int64
ChunkID string
Generation string
Values []float32
}
type SourceStore interface {
CurrentRevision(ctx context.Context, documentID string) (int64, error)
}
type Embedder interface {
Embed(ctx context.Context, generation string, text string) ([]float32, error)
}
type VectorStore interface {
Upsert(ctx context.Context, row VectorRow) error
}
func ReembedCurrent(
ctx context.Context,
source SourceStore,
embedder Embedder,
vectors VectorStore,
chunk SourceChunk,
generation string,
) error {
values, err := embedder.Embed(ctx, generation, chunk.Text)
if err != nil {
return fmt.Errorf("embed %s/%s: %w", chunk.DocumentID, chunk.ChunkID, err)
}
revision, err := source.CurrentRevision(ctx, chunk.DocumentID)
if err != nil {
return fmt.Errorf("read current revision: %w", err)
}
if revision != chunk.Revision {
return ErrStaleRevision
}
return vectors.Upsert(ctx, VectorRow{
DocumentID: chunk.DocumentID,
Revision: chunk.Revision,
ChunkID: chunk.ChunkID,
Generation: generation,
Values: values,
})
}
The database must enforce uniqueness on that identity, making retries idempotent. The worker should classify stale revisions as expected supersession, transient dependency failures as retryable, and malformed or oversized input as terminal work for inspection. Blind retries turn one bad page into queue pressure.
Quality gates before traffic moves
Backfill completion is necessary, never sufficient. Assemble a fixed evaluation set from the actual retrieval jobs: course-title changes, deadline edits, prerequisite substitutions, removed policies, reordered navigation, and boilerplate-only edits that should not alert. For each query, record relevant document and chunk judgments without using the candidate system's output as the answer key.
Compare generations with retrieval measures suited to ranked output, such as recall at a fixed cutoff and normalized discounted cumulative gain. Those metrics should be segmented by page type and change class; a single average can conceal a total miss on sparse catalog pages. The original RAG paper describes combining retrieved passages with generation, but this migration gate belongs before generation: if retrieval omits the changed paragraph, fluent downstream text cannot restore its provenance.
Latency needs the same discipline. Measure p50, p95, and p99 query latency for each generation under representative concurrency, including filter evaluation and reranking if those are part of the serving path. Set the acceptance threshold from the alert pipeline's remaining error budget, not from whichever result makes the migration look successful. A target generation that improves offline relevance but exhausts the fetch-to-alert latency budget is an engineering trade, not an automatic upgrade.
Keep the gate legible:
- coverage equals every live document revision represented in the target, with tombstones reconciled;
- retrieval quality is no worse than the agreed bounds on each critical change class;
- serving latency and error rate stay inside their SLOs during shadow load;
- generation, source revision, and chunk identity appear in every retrieval trace;
- rollback has been exercised before any broad traffic shift.
Then canary the route by a stable tenant or document partition, not a random coin flip on every request. Stable assignment lets repeated watches see consistent rankings and makes alert-rate comparison meaningful. Move in steps, pause for at least one normal crawl interval at each step, and stop automatically when an SLO or quality guardrail fires. The appropriate interval comes from the application's fetch schedule; inventing a universal number would be false precision.
Capacity is part of correctness
Backfills compete with live crawling, parsing, and alert delivery. Start with arithmetic: source chunks multiplied by average embedding input size gives the work volume; measured worker throughput gives an initial duration; retry rate, dual-write load, and headroom for live traffic make that duration credible. Rate-limit the migration below the capacity reserved for foreground work, and let queue age rather than CPU utilization alone control concurrency.
Slow is acceptable. Invisible contention is not.
Track target coverage by current revision, stale-job discard rate, embedding request latency, queue age, upsert latency, and query SLOs split by generation. Cardinality matters here: generation is a bounded label; document IDs belong in logs or traces, not metric labels. Alert when foreground latency consumes its budget, and have automation reduce backfill concurrency before an operator has to diagnose the shared bottleneck.
Storage planning must include both vector generations, source text, index-build amplification, and the rollback window. Do not delete the old generation immediately after full cutover. First stop its writes, retain the immutable snapshot, verify that no readers use it, and expire it under the same retention and access-control rules as the source corpus.
Buy, build, or combine?
The choice is about operational ownership, not a feature checklist. Managed infrastructure can remove index maintenance from the on-call queue, while self-hosting can provide tighter control over data placement and migration mechanics; neither choice removes the need for versioned data, evaluation, or rollback.
| Approach | Team owns | Useful when | Principal risk |
|---|---|---|---|
| Managed vector service | Source revision contract, evaluation, routing, provider limits | The team values lower storage on-call load over implementation control | Lock-in around indexing behavior and an external capacity boundary |
| Self-hosted vector index | Capacity, upgrades, backups, compaction, and query performance | Data placement or low-level tuning justifies sustained operator time | Backfill and foreground incidents share the same team |
| Postgres with a vector extension | Database operations plus vector index tuning | Corpus scale and query load fit an already operated relational system | Migration load can contend with transactional work |
| Split control and data planes | Routing metadata plus either storage model | Multiple generations and reversible cutovers are routine | More states to observe and test |
My default decision rule would be to estimate monthly operator hours and failure ownership before comparing service prices. Price changes; pager load and recovery complexity remain architectural facts. If a team cannot rehearse restoration, inspect generation lineage, and reserve backfill capacity in its self-hosted design, nominal control is not useful control. Conversely, an external service with opaque index readiness or hard-to-test export paths can make rollback an assumption.
Where dual indexing is the wrong answer
For a tiny, rebuildable corpus with a maintenance window that users have accepted, an offline rebuild and one cutover may be simpler. Dual indexing is also unnecessary when the representation function has not changed and the store supports a proven online index rebuild over identical vectors. Do not pay permanent complexity for a migration that has no availability requirement.
The pattern also does not make incompatible query behavior safe by itself. If the migration changes filters, chunk boundaries, reranking, or authorization semantics, isolate and evaluate those changes explicitly; otherwise a clean embedding comparison becomes confounded. One variable per migration is slower on a roadmap and much faster during an incident.
For the course-page watcher, the final decision remains concrete: promote the target only when current revisions are complete, meaningful changes remain retrievable, nuisance changes remain suppressed by the downstream policy, and end-to-end alert latency stays inside its SLO. Keep the old generation available for rollback, record the routing change, and remove it only after the rollback window and retention policy permit. Availability without retrieval integrity is a false success.
Top comments (0)