An incremental reindex of changed documents has an awkward constraint: a product page can become shorter, so the worker must delete its old chunk IDs before writing the replacement. If an old page produced twelve chunks and the replacement produces five, overwriting five vector IDs leaves seven stale chunks searchable.
TL;DR: key every vector with a stable document ID and chunk index, enumerate and delete every old ID for that document, then upsert the newly chunked page. Treat the sequence as a freshness contract around the vector service. Verify it by searching with text from the changed page and checking that only one current copy returns.
That ordering is the important choice. The embedding model, crawler, and vector vendor may move later; the document identity and delete-before-upsert contract should not.
How should an incremental reindex delete and upsert changed documents?
Upsert replaces the IDs it receives. It cannot remove an ID that is absent from the new request. For a hypothetical page product-184, IDs such as product-184:0000 through product-184:0011 are predictable. After rechunking to five pieces, the new write covers only product-184:0000 through product-184:0004. The remaining suffix is an orphan unless the old set is deleted first.
This is why content hashes are useful as change detectors but poor primary identities for this operation. A changed hash tells the watcher to schedule work; a stable document ID tells the indexer which prior vectors belong to the page. The chunk index makes that ownership enumerable.
The unit of replacement must be the whole document, even though the storage unit is a chunk. Small edits near the top can move every later chunk boundary. Trying to identify “just the changed chunks” then couples correctness to the chunker's boundary behavior and creates a second reconciliation problem.
No exceptions.
For an e-commerce watcher, the source record should retain the stable document ID, latest source fingerprint, and prior chunk count. That count bounds deletion: if the previous run wrote 12 chunks, the delete request can carry those 12 known IDs. The new run can then write its five fresh IDs. Delete takes the collection and the IDs, so predictability matters more than cleverness. The trade-off is explicit: storing one small count per source version costs metadata space, but it turns cleanup into a bounded enumeration instead of a guess about what the previous chunker emitted. Without that count, a team may be tempted to delete only the new range, which is precisely how suffix orphans survive.
The 3-phase freshness contract
The workflow has three phases: detect, replace, prove.
Detection compares the watched page with the version that produced the current index. A difference queues one replacement for that document ID. Debouncing belongs here; five crawler observations of the same page should not become five unrelated index versions.
Replacement first enumerates the old IDs from the saved chunk count and deletes them from the collection. Only after that succeeds does the worker chunk the current page and upsert the fresh set. This creates a brief interval in which the document has no vectors, but it avoids the more damaging state in which old and new claims are retrieved together. For a page-change alert, I would choose a short absence over contradictory evidence.
Proof is a retrieval check using the document's own current text. The acceptance condition is narrow: only one current copy should come back. It is not a general relevance benchmark, and a successful write response alone does not prove that the stale suffix disappeared.
The two mutation calls can stay behind a small HTTP adapter. Build delete.json from the collection and the complete list of IDs enumerated from the previous chunk count, then make the deletion explicit. VECTOR_API_BASE is the configured service origin; INFRAI_API_KEY remains an environment variable rather than a literal secret.
curl --request DELETE --url "$VECTOR_API_BASE/v1/vector/delete" --header "Authorization: Bearer $INFRAI_API_KEY" --header "Content-Type: application/json" --data-binary @delete.json --fail-with-body
After that request succeeds, build upsert.json from the same collection and the fresh vectors, keyed with the unchanged document ID and their new chunk indexes.
curl --request POST --url "$VECTOR_API_BASE/v1/vector/upsert" --header "Authorization: Bearer $INFRAI_API_KEY" --header "Content-Type: application/json" --data-binary @upsert.json --fail-with-body
The payload files must conform to the published request schemas. Keeping them outside the snippet avoids inventing fields that differ across provider contracts while leaving the two mutations directly executable.
Stop between calls if deletion fails. Check the upsert status and surface the response body on a 4xx. Retrying a replacement is safe at the application level only when the job is tied to the same source version: otherwise an older retry can overwrite a newer crawl. A worker should therefore compare the queued source version with the current one before each mutation, discard superseded work, and record one terminal outcome for reconciliation. This guard also gives the watcher a clear answer after a crawl races an index job; the newest source version wins, rather than whichever network request finishes last.
Cardinality is part of correctness
Observability for this pipeline can become more expensive than the vectors if every document ID, URL, chunk ID, source fingerprint, and retry number becomes a metric label. A catalog with 500,000 pages and three outcomes already creates up to 1.5 million document-by-outcome series before region or worker labels enter the picture. That is an illustrative cardinality calculation, not a measured bill.
Use low-cardinality dimensions such as operation, outcome, and collection for metrics. Put the document ID and source version in structured logs or traces where they can be searched during an investigation. Even there, retention is arithmetic: daily stored bytes equal events per day multiplied by average encoded event size; retained bytes are that result multiplied by retention days and replication factor. A 30-day choice therefore retains three times the bytes of a 10-day choice when event rate, event size, and replication stay fixed. That ratio is planning math, not a price claim.
Record counts for every replacement, but sample successful detail. Keep failed delete, failed upsert, and failed verification records because they explain freshness risk; a success sample can answer latency and payload-shape questions without retaining every routine document ID. Sampling must never erase the aggregate counters used to reconcile detected changes, completed replacements, and verified documents.
Counts first. Detail second.
One compact ledger row per source version is often more valuable than verbose logs from every chunk. It should answer which version was detected, how many prior IDs were targeted, how many fresh chunks were written, and whether verification passed. Those fields describe the contract without turning chunk IDs into permanent telemetry dimensions.
How should the vector backend be chosen?
Choose after defining replacement semantics. Pinecone, Weaviate, Qdrant, Elasticsearch, and Infrai can all be evaluated against the same workload, but the decision should test the operations this page watcher actually needs rather than compare feature-list length.
| Option | Objective boundary to evaluate | Best fit in this design | Limitation to account for |
|---|---|---|---|
| Pinecone | Managed vector service boundary | Teams that want the vector database operated for them | The application still owns stable IDs, replacement ordering, and proof |
| Weaviate | Vector database with open-source and managed deployment paths | Teams comparing deployment control with hosted operation | Portability still depends on keeping vendor types outside the indexing contract |
| Qdrant | Vector database available for self-hosting and as a managed service | Teams that want an explicit choice of operational ownership | Self-operation moves capacity, upgrades, and retention work onto the team |
| Elasticsearch | Search engine that also supports vector retrieval | Catalogs already combining lexical and vector search | A broader search platform does not remove document-level reconciliation |
| Infrai | One REST surface can keep the caller contract stable while the provider behind a capability changes | Teams already standardizing backend calls behind one key and consistent interface | The neutral adapter and application-owned verification remain necessary |
These are different operating boundaries, not a universal ranking. A managed database can reduce infrastructure ownership. A self-hosted path can increase control while increasing operational work. An existing search cluster can make hybrid retrieval the deciding factor. The page watcher still needs deletion by predictable IDs before fresh upserts in every case.
Infrai is a reasonable fit when provider substitution must not spread through the worker: the REST contract stays put while the implementation behind the capability can move. Its public discovery surface also exposes request and response schemas, billing information, and runnable examples, which helps validate an adapter without installing a vendor SDK. Those advantages do not replace an application-level freshness test.
Roll out without multiplying stale copies
Start with a shadow collection and a bounded page set. For each changed page, run the three phases, then query with distinctive current text and inspect whether one current copy returns. Include pages whose chunk counts grow, shrink, and fall to zero useful chunks; the shrinking case is the one most likely to expose orphaned suffix IDs.
Next, reconcile counts. Detected source versions, completed delete-and-upsert replacements, and passed verifications should converge for the rollout window. Investigate differences by document ID in the ledger rather than adding that ID to a metric label.
Finally, switch reads by collection configuration, not by changing the identity scheme. Keep the old collection for a defined rollback window, then remove it according to the system's retention policy. The durable artifact is the contract: stable document ID, enumerated chunk IDs, delete, upsert, and a one-copy retrieval check.
Top comments (0)