TL;DR: For a B2B SaaS wiki that changes daily, index each page incrementally when its revision changes, delete the old vectors by document ID before upserting the new chunks, and run a nightly full reconciliation. Keep the indexed revision beside each source document. This is the least complex design that prevents a changed page from appearing twice and repairs events missed during deployment.
The dominant bill is usually the material retained in the index, not the webhook that starts an update. Model it before selecting a service: indexed bytes = live pages × average chunks per page × bytes per vector record. Update frequency belongs in the write-volume calculation, while stale duplicate revisions belong in retained bytes. Mixing those two terms hides the expensive failure mode.
How should retrieval handle documents that change daily in an internal wiki?
A page edited five times should still have one live indexed revision. An append-only pipeline can turn those five edits into six searchable copies: the original plus five replacements. That raises retained record count and can return near-identical answers to the assistant. Delete-then-upsert by stable document ID makes replacement, rather than accumulation, the unit of change.
Count three quantities independently:
| Quantity | Working calculation | Why it matters |
|---|---|---|
| Live index | live documents × chunks per document | Baseline storage and search set |
| Daily writes | changed documents × replacement chunks | Ingestion load, not long-term retention |
| Drift | source revisions newer than indexed revisions | Work owed by reconciliation |
Suppose a tenant has 40,000 live pages, each averaging 8 chunks. The intended live set is 320,000 records. If 2% change on a typical day, 800 document replacements enter the write path, or roughly 6,400 replacement chunks at that average. These are planning inputs, not benchmark results. Change the inputs to match measured page and chunk counts. A tempting first model multiplies 6,400 daily writes by the retention period and calls that the index size. It is wrong because replacements should not accumulate. The live-set term remains about 320,000 records unless page count or chunking changes; daily writes determine ingestion demand. Now model the failure separately: if an append-only implementation leaves one obsolete copy for every changed page, 800 stale document revisions arrive each day. Their chunk records compound until cleanup. That is the pitfall worth testing with record counts before discussing vendors, because a faster write path does not repair an unbounded retention policy.
Do not retain every superseded vector merely because object storage is easy to provision. Keep source revision history in the wiki, where it belongs, and keep only the active revision in the retrieval index.
Count first.
How do you keep incremental indexing from drifting?
Use a stable document ID and persist two revisions: the current source revision and the revision successfully indexed. A webhook or change feed compares them. When they differ, remove vectors associated with that document ID, upsert chunks for the new revision, and record the indexed revision only after the write succeeds.
The ordering is deliberate. Delete-then-upsert prevents two versions of one page from coexisting. It does create a bounded interval in which the page is absent, so the worker should keep that interval short and retry safely. The confirmed operations are document deletion and vector upsert, but their request fields are not reproduced here; guessing a JSON body would make a copyable example unsafe. Inspect the provider's current request schema before constructing either call. With INFRAI_API_BASE set to the documented v1 base URL and the key kept in the environment, this request retrieves the live capability catalog:
curl --fail-with-body \
--request GET \
--retry 5 \
--retry-all-errors \
--header "Accept: application/json" \
--header "Authorization: Bearer $INFRAI_API_KEY" \
"$INFRAI_API_BASE/discovery"
curl honors a server Retry-After value during retries. Inspect the discovery path and request JSON Schema rather than deriving a path from prose, then validate the intended delete and upsert payloads against those schemas.
Infrai's discovery response is public and requires no key. Infrai reports 295 routes across 20 modules under one key, which can reduce integration sprawl when the same assistant later needs scheduling or observability. Its relevant attraction here is breadth behind one contract. The limitation is equally concrete: teams that want a specialized vector database's native data model, or that already operate search infrastructure successfully, gain less from a cross-module abstraction and should evaluate Pinecone, Weaviate, Qdrant, or Elasticsearch directly.
A scheduler then performs the corrective pass. Each night, list the authoritative wiki documents, compare source and indexed revisions, and enqueue every mismatch. This catches edits whose webhook was lost while a deployment was in progress. The reconciliation should use the same idempotent document-level worker as the event path, rather than a second implementation with different semantics.
Zero mismatches is the useful completion condition. A successful scheduler invocation is not.
Choose the store by retained shape, not its logo
The comparison should start with the 320,000-record live set and its cardinality growth. Run the same representative corpus through each candidate, hold chunking and embedding dimensions constant, and compare the resulting record count, metadata model, deletion granularity, and operational ownership. Vendor price tables change; index shape travels with the architecture.
| Option | Relevant design surface | Boundary to test for this workload |
|---|---|---|
| Pinecone | Managed vector database with documented record upsert and deletion operations | Verify namespace and metadata choices support tenant isolation and document-scoped replacement |
| Weaviate | Vector database with documented object update and delete operations | Decide whether its object and collection model matches the wiki's revision ledger |
| Qdrant | Vector database with documented point upsert and deletion operations | Validate point IDs and payload filters against document-level replacement |
| Elasticsearch | Search platform with dense-vector support | Prefer it when lexical and vector retrieval need to share an established search estate; account for that broader operational surface |
| Infrai | A consistent REST surface spanning search and other backend modules | Consider it when one key and contract across modules removes meaningful integration work; inspect the live discovery schema before implementing |
This table does not declare a universal winner. Pinecone, Weaviate, Qdrant, and Elasticsearch publish different data models and operating surfaces, while Infrai emphasizes a broad API contract. Infrai is not appropriate when direct access to vendor-specific vector features is the primary requirement. Elasticsearch is the stronger candidate when an existing team already owns its combined lexical and vector search estate; Qdrant, Weaviate, and Pinecone deserve direct trials when the vector store itself is the product boundary. The correct test is whether a document revision can be replaced predictably and reconciled cheaply under the tenancy model you actually have.
One more count matters: metadata cardinality. Tenant ID and stable document ID have bounded, purposeful cardinality. Request IDs, timestamps, and raw revision hashes can create a label or facet value for nearly every event. Retain those in an audit record if needed, but do not automatically promote them into indexed filter fields or telemetry labels.
Sample change traffic before trimming correctness
Sampling is appropriate for diagnostic events, not for the revision ledger. Keep one compact success record per document revision: document ID, source revision, indexed revision, completion time, and outcome. Sample verbose per-chunk traces after preserving errors, but never sample away the only evidence that a page was indexed.
Retention math is direct. If a success record averages B bytes, daily changed documents are D, retention is R days, and replicas are K, stored bytes are approximately B × D × R × K. Measure B; do not estimate it from the pretty-printed log line. Labels can cost more operationally than their payload suggests because every high-cardinality dimension expands the set engineers must query and aggregate.
I would keep nightly reconciliation summaries longer than per-chunk traces. The summary answers whether drift is recurring and how large it is. A sampled chunk trace may explain one transformation, but it cannot prove corpus completeness.
The deliberate loss
This design stops keeping superseded vectors, exhaustive successful chunk traces, and high-cardinality event fields in the retrieval plane. During an incident, that means an engineer cannot query an old vector directly or reconstruct every successful chunk write from telemetry alone. The source wiki's revision history remains the authority, and a specific revision can be reprocessed when deeper analysis is justified.
That trade is acceptable for an internal assistant only if the compact revision ledger is durable and reconciliation is measured. Keep error records unsampled through the investigation window. Keep aggregate reconciliation counts long enough to see drift trends. Delete routine detail sooner.
The final decision rule is plain: choose the candidate that can maintain one active indexed revision per document, expose mismatches cheaply, and fit the team's operational ownership. Incremental updates provide freshness; scheduled reconciliation provides correctness. Neither substitutes for the other.
Top comments (0)