A customer-support article can be fetched twice, split twice, and indexed twice while every individual write reports success. The constraint is that retries are normal in bulk ingestion, but duplicate search hits are not. Short answer: give each chunk a deterministic ID within a document revision, write with replace-by-ID semantics, and publish a new revision only after accounting for the old revision's IDs. Then check retrieval separately: two hits with the same text do not prove two vector records exist.
How do I debug duplicate chunks in vector search results?
Start with the evidence. For one repeated support-answer passage, log the returned vector IDs, document IDs, source revisions, chunk ordinals, and the exact stored text hashes. Two different IDs with the same document revision and ordinal point toward non-idempotent writes. The same ID returned twice suggests query assembly or result merging instead. Two different source documents containing the same answer may be legitimate duplicates in the corpus; deleting one by text hash could erase an independently maintained article.
One duplicate-looking answer is not enough evidence.
Do not infer an ingestion failure from the rendered answer alone. A retrieval pipeline can query multiple collections, merge result lists, or include adjacent chunks, and a support document may repeat a warranty clause in both its summary and body. Compare the raw hits before reranking or context packing with a direct lookup by vector ID. If a query response hides IDs, add them to your own retrieval diagnostics; text similarity is not a substitute for record identity. In a bulk run, isolate one source key and compare four snapshots: the input revision, the expected ID set before writing, the stored ID set after the retry, and the unmerged search results. If the expected set has three IDs but storage has six, investigate how IDs were allocated. If storage has three and the merged response has six, investigate the read path. If both have three yet the article appears outdated, inspect which revision the query is allowed to read. These are different repairs and deserve different alerts.
The failure is easy to reproduce conceptually. A worker reads article returns/42, creates three chunks, writes them, then loses its acknowledgment. A retry that allocates fresh random IDs creates three more records. The worker has performed the same logical operation twice, but the index sees six distinct keys. No search threshold can repair that write contract.
Retries are inevitable.
How should chunk identity survive a retry?
Separate three identities: the stable source document key, the source revision, and the chunk key within that revision. Normalize the input used by the chunker, fix the chunking configuration, and make the output ordering deterministic. A change in text normalization, overlap, tokenizer, or boundary rules can change every ordinal even if the document key stays fixed. Record the chunking recipe alongside the source revision so that a reprocessing job can explain why its output changed.
For a Node.js bulk-ingestion worker, the following Python sketch specifies the ID contract independent of a vector client. Implement the same canonical byte encoding in the worker and test it against shared fixtures; a hash is deterministic only when its input bytes are.
import hashlib
import json
def digest(parts):
payload = json.dumps(parts, ensure_ascii=False, separators=(",", ":"))
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def chunk_records(document_key, revision, recipe, chunks):
for ordinal, text in enumerate(chunks):
yield {
"id": digest([document_key, revision, recipe, ordinal]),
"document_key": document_key,
"revision": revision,
"recipe": recipe,
"ordinal": ordinal,
"text_hash": digest([text]),
"text": text,
}
The document key must be scoped to its tenant or corpus if those domains share an index. The revision can be a source-controlled version token or a digest of normalized source content; neither should be a processing timestamp, because a retry would then look like a new revision. The recipe names the boundary algorithm and settings. Keep embedding-model identity in the index namespace or recipe as appropriate: re-embedding under a different model must not silently masquerade as the same stored representation. Hashing is for stable identity here, not authorization or a guarantee that two support answers mean the same thing.
Why can deterministic IDs still leave stale results?
An upsert only replaces the IDs it receives. If revision A yielded five chunks and revision B yields three, the two IDs that disappeared from the new output still need removal. It gets worse when ordinals are reused: a partial retry can leave a mixture of A and B unless reads are gated by a published revision. Treat each document update as a set transition, not a loop of independent writes.
That missing pair matters.
| Choice | Retry behavior | Freshness boundary | Failure to watch |
|---|---|---|---|
| Random ID on every attempt | Creates another record | No reliable document-level boundary | Duplicate chunks after an ambiguous write |
| Stable ID for document, revision, recipe, and ordinal | Replaces the same logical chunk | Old revisions still require retirement | Both revisions searchable during transition |
| Stage a full revision, then change the visible revision | Retry can target the staged keys | Readers use one published revision | Orphaned staged records need cleanup |
The third approach gives the clearest read contract where the application can filter by a published revision. Store a manifest of expected IDs for each document revision outside the vector index, write and verify the new set, switch the document's visible revision in a durable metadata store, and retire the old IDs. That sequence does not make two independent stores atomic. If the process stops between publication and deletion, reads should exclude the old revision; a repair job can finish deletion later. If it stops before publication, readers keep using the prior complete revision. Define how long staged and retired records may remain and monitor that age, rather than treating a successful batch request as proof of freshness.
There is a real trade-off: retaining two revisions briefly consumes storage and complicates filtering, but deleting the old set first creates a window with missing support answers. When the index cannot filter on revision reliably, use an isolated generation or collection and switch the read target after validation; the exact switching mechanism depends on the storage system's consistency guarantees. Do not promise atomic publication unless the actual metadata and query path provide it.
What should the ingestion test prove?
Use an article fixture with a paragraph repeated in two different source documents, plus one document whose second revision produces fewer chunks. Submit the same bulk job twice and interrupt one run after writing only part of its batch. For each document and published revision, assert that the set of stored IDs matches the manifest exactly; then query and check that returned records belong to the visible revision. Test the negative case too: two distinct source documents may retain identical text. A global text-hash deduplication test would incorrectly collapse them.
Count attempted writes, distinct IDs, staged records, retired records, and the time between observing a source revision and publishing it. Alert on a persistent difference between manifest cardinality and indexed cardinality, not just on worker exceptions. Keep a small sample of raw result IDs in diagnostics without logging entire customer-support articles. Access control belongs on the source key and retrieval filter as well as the ingestion path; a stable ID does not enforce tenant isolation.
Bulk throughput matters, but the first operational limit to establish is the maximum unfinished revision backlog the team can inspect and repair. Batch writes can be retried after ambiguous timeouts with the same IDs; verify membership before publication, and bound repair work so a failed update does not accumulate invisible records indefinitely. This also makes an index migration measurable: run the new chunking recipe into a separate generation, compare expected counts and representative retrieval results, then switch reads and retire the previous generation after the rollback window.
Roll out one document at a time
Add the identity fields to write metadata and retrieval logs, establish a manifest for a small support-document subset, and verify retry invariance before enabling revision publication for the full corpus. Keep the previous generation readable during the first switch. A clean rollout is not a promise of perfect similarity rankings; it is the narrower, testable guarantee that an ambiguous retry cannot multiply records and that an outdated chunk cannot appear in a query scoped to the published revision.
Top comments (0)