A retrieval index update is a production change, even when the model and prompt stay fixed. A new chunker, embedding model, document set, or access filter can change what the assistant sees. In a client workflow, that may change answers, citations, and who can read a source. This drill makes an index cutover a testable release rather than a silent re-embed.
Define the two index versions
Keep the current index readable while building the candidate. Give each index an immutable identifier and record the source snapshot, ingestion time, parser, chunking settings, embedding model, metadata schema, and access-control mapping. A label like “latest” is not enough for a rollback.
A small release manifest might contain:
release: client-policy-2026-11-17
active_index: policy-v17
candidate_index: policy-v18
source_snapshot: approved-docs-2026-11-16
chunker: semantic-v3
embedding_model: embed-v2
acl_mapping: tenant-and-role-v5
Build the candidate without changing live traffic. Check document counts, failed ingestions, duplicate document IDs, missing metadata, and records removed since the previous snapshot. If source material contains customer data, use approved test fixtures and preserve the same access rules in the test environment.
Run a paired canary set
Choose cases from the actual workflow: a common question, a rare exception, an amended policy, a revoked document, a tenant-specific document, a request with no approved answer, and a question whose answer spans two documents. Run each case against both index versions with the model, prompt, tools, and evaluation settings pinned. Save the retrieved document IDs and ranks as well as the final answer.
| Case | Candidate should show | Stop condition |
|---|---|---|
| Current policy | New approved source | Old superseded source is cited |
| Revoked document | No revoked chunk | Revoked text remains retrievable |
| Tenant boundary | Only authorized chunks | Cross-tenant result appears |
| Multi-document answer | All needed sources | A necessary condition disappears |
| Unanswerable request | Abstention or clarification | Unsupported answer appears |
| Rare exception | Exception clause retained | Generic rule overrides it |
Compare the business outcome and source support, not only text similarity. A fluent answer can still be wrong if it cites the wrong version. Investigate any regression before routing live requests to the candidate.
Cut over with an exit path
Start with an internal canary or a small authorized traffic slice. Tag every request with the index version so traces and user reports can be separated. Watch unsupported-answer reports, retrieval misses, citation quality, authorization failures, latency, and the number of cases sent to a human. Set the thresholds and decision owner before the canary starts.
If a stop condition occurs, restore the prior index alias and verify the whole journey: retrieval, answer, citation, and tool routing. Keep the candidate index for diagnosis, but prevent it from serving user traffic. Reconcile any cases processed during the canary; changing the alias does not undo a bad answer already delivered.
Retain the release evidence
Store the manifest, source approvals, paired case results, canary window, observed metrics, decision, and rollback test. This lets a later engineer explain why the index changed and reproduce the result. It also makes the next cutover faster: the same canary set can be rerun with a new source snapshot and explicit changed expectations.
For engineers preparing to own this kind of client deployment work, Forward Deployed Engineer Certification provides role context alongside practical delivery skills.
Top comments (0)