Adding semantic search to a private personal knowledge manager is mainly a versioning and capacity problem. Short answer: index immutable document revisions, retrieve only the latest permitted revision by default, and keep old vectors addressable for audit and time-travel queries. That rule keeps deleted notes out of normal results, bounds re-embedding work, and gives an SRE something precise to measure.
I learned this while reviewing an incident pattern in a healthtech content system: a note was edited, the embedding job lagged, and search returned a paragraph that the author had already removed. The request was fast; the answer was wrong. A second worker then indexed the same revision, doubling vector storage for no retrieval gain. The alert said latency was normal, so the first instinct was to tune the ranker; that sent us in the wrong direction because ranking can only choose among candidates that the consistency layer admits. The useful question was “which revision was allowed to be visible at commit time?” Once that became an explicit field, the replay was deterministic, the stale result had a clear owner, and the capacity model could count changed chunks instead of vague requests. I do not have a universal number for the right lag budget—your mileage may vary—but the failure mode is predictable when a mutable row is treated as the identity of an immutable vector.
How should document versioning shape retrieval design for a privacy-focused knowledge manager?
Start with an append-only revision record. Give every revision a stable document_id, a monotonically increasing version, an content_hash, an embedding model identifier, and a visibility state. The vector key should include the revision, not just the document. A tombstone is data, too: it tells the retrieval filter that an older revision must not leak while asynchronous deletion catches up.
Chunking needs the same discipline. Store chunk_id as (document_id, version, ordinal) and retain the character offsets used to create it. If a user changes one paragraph, hash-based chunk matching can reuse unchanged chunks; changed chunks get new embeddings. This is cheaper than re-indexing an entire notebook and makes a cost forecast possible: monthly embedding work is roughly changed-chunk count multiplied by average tokens per chunk.
The query path should apply authorization and “latest visible version” filters before ranking. Filtering after nearest-neighbor search is unsafe because a deleted or private chunk can occupy the top-k slots and crowd out valid evidence. Hybrid retrieval (lexical plus vector) is useful for identifiers and medication names, but both branches must consume the same revision visibility predicate.
The incident lesson: stale vectors are a consistency bug, not a ranking bug
The preventative path is an outbox transaction. The write that creates a revision also records an indexing event; a worker claims the event idempotently, computes embeddings, and atomically marks that exact revision searchable. A compare-and-swap on (document_id, version) prevents an older worker from publishing after a newer edit.
package retrieval
import "context"
type Revision struct {
DocumentID string
Version int64
Hash string
Visible bool
}
type Indexer interface {
Embed(ctx context.Context, text string) ([]float32, error)
Publish(ctx context.Context, r Revision, chunk int, vector []float32) error
}
func IndexRevision(ctx context.Context, ix Indexer, r Revision, chunks []string) error {
for i, text := range chunks {
vector, err := ix.Embed(ctx, text)
if err != nil {
return err
}
if err := ix.Publish(ctx, r, i, vector); err != nil {
return err
}
}
return nil
}
Retries must be keyed by event id, not by a timer. Record embedding model and tokenizer versions beside each vector; a model upgrade is a planned migration with its own error budget. For a privacy-focused product, encrypt source text and vectors at rest, keep tenant keys separate, and make logs contain identifiers rather than note contents. Retrieval traces should show revision ids, scores, and filter decisions, never the unredacted chunk.
Nine practices that keep index cost and recall explainable
- Define “current” as the greatest visible version in the authorization scope.
- Use immutable revision and chunk keys; never overwrite a vector in place.
- Deduplicate unchanged chunks by content hash, while retaining revision references.
- Put visibility, deletion, and retention filters in the candidate query itself.
- Gate model migrations with a shadow index and recall checks on a fixed corpus.
- Batch embedding work, but cap batch size to preserve queue-latency SLOs.
- Meter tokens, chunks, retries, and vector bytes per revision, not only per request.
- Keep a tombstone ledger long enough to cover replication and backup windows.
- Test edit, delete, restore, and concurrent-update sequences as state machines.
The list is intentionally operational. A perfect similarity score cannot compensate for a stale authorization decision, and an inexpensive index that cannot explain why a revision appeared will become an on-call tax.
Ship less.
Buy or build: which boundary matches your team?
| Boundary | Strength | Cost or limitation | Suitable when |
|---|---|---|---|
| Self-hosted vector store | Direct control of keys, retention, and network paths | Capacity planning, upgrades, and incident response stay with your team | Data residency and custom filtering dominate |
| Managed vector service | Faster initial operations and elastic capacity | Contractual lock-in, egress planning, and provider-specific filtering semantics | A small team values lower on-call load |
| Relational database with vector extension | One transaction boundary for revisions, ACLs, and vectors | Very large indexes may need careful partitioning and vacuum planning | The corpus is moderate and correctness is the first concern |
The catch is that none of these choices removes the version graph. A managed service is not suitable when its retention or key-management model cannot meet your threat model; stick with a relational design when cross-row authorization must be atomic. Conversely, self-hosting is a poor fit if nobody owns capacity forecasts, shard repair, and a 03:00 page.
Track three separate objectives: freshness (time from committed revision to searchable revision), correctness (no result from a non-visible revision), and retrieval quality (recall of approved answers at a fixed k). Alert on freshness and correctness; sample quality with a reviewed query set. Capacity planning should include peak edit bursts, re-embedding during model changes, tombstone retention, and a replica safety margin. I would rather lower top-k briefly than let an unbounded queue hide a deletion.
Choose the simplest boundary that can enforce revision-aware authorization in one auditable path. Revisit it when measured chunk churn, vector bytes, or queue latency crosses the budget you set with your SLOs. Document versioning is the invariant; the storage engine is an interchangeable implementation detail.
Top comments (0)