The In-Process Vector DB Trap: How ZVec's `optimize()` Can Silently Erase Crash-Recovered Data

#architecture #database #python #temporal

Introduction

Embedded, serverless vector databases are having a moment. The pitch is simple: skip the sidecar container, skip the network hop, just call a Python API and get a vector index that lives inside your own process. ZVec (PyPI package zvec, v0.5.1, built by Alibaba on a C++ core with Python bindings) is one of the newer entrants making that pitch. Opening a collection is genuinely trivial:

import zvec

collection = zvec.create_and_open("~/spike-zvec/data/my_collection", schema)

No Docker, no connection string, no separate process to keep alive. For a team running retrieval-augmented generation inside an orchestration layer — Temporal Activities, in our case — that simplicity is worth taking seriously. One fewer moving part is one fewer thing that pages you at 3am.

But "runs inside your process" cuts both ways. If the vector store lives inside the Activity's process, it dies exactly when the Activity's process dies — SIGKILL, an OOM kill, a host eviction, whatever your orchestrator throws at a worker mid-task. Before trusting any in-process store with durable state, the only honest question is: what does it look like on the other side of a hard crash?

So we tested it the direct way. We inserted 551 documents into a ZVec collection and, mid-run, killed the process with os._exit(1) — the Python equivalent of kill -9, no cleanup handlers, no graceful shutdown. Then we reopened the collection from a fresh process and checked what came back.

The result was reassuring on the point that matters most: the document data itself survived intact. stats().doc_count reported exactly 551 on reopen — no loss, no duplication. ZVec's write-ahead log (WAL) replayed automatically on open(), with no explicit "recover" call required from application code. For raw durability of inserted documents, this held up under a real kill-9 test.

What did not look reassuring at first glance was this line from the same stats() call:

>>> collection.stats()
{'doc_count': 551, 'index_completeness': {'embedding': 0.0}}

index_completeness: 0.0, right after a hard crash, reads like exactly the kind of number that should make you nervous — a half-built index, a corruption signal, something to fix immediately. We treated it that way at first too. It turned out to be the wrong read.

We reopened three other collections from earlier, entirely uneventful test runs — collections that had never crashed, never been killed, shut down cleanly every time. All three reported the identical index_completeness: 0.0. The number has nothing to do with crash history. It is simply the default state of any ZVec collection — pristine or crash-recovered — that has never had optimize() called on it. And critically, a collection sitting at index_completeness: 0.0 is not degraded in any functional sense: query() against it returns correct, ranked results. It's an unbuilt index, not a broken one.

That distinction — "unbuilt" versus "broken" — turns out to matter a great deal, because the natural next move is to reach for optimize() to build the index and push that number to 1.0. That move is where the real danger in this system lives, and it's the subject of the rest of this piece.

Section 2: The Deep Pitfall — Silent Data Loss in `optimize()`

The instinct, once you've seen index_completeness: 0.0, is straightforward: call collection.optimize() and move on.

collection = zvec.open("~/spike-zvec/data/crashed_collection")
print(collection.stats())
# {'doc_count': 551, 'index_completeness': {'embedding': 0.0}}

collection.optimize()
print(collection.stats())
# {'doc_count': 551, 'index_completeness': {'embedding': 1.0}}

On the same process handle, this looks like a clean repair: doc_count is still 551, index_completeness is now 1.0. No exception was raised. Nothing in the return value or the logs suggests anything went wrong.

Reopen that same collection from a fresh process, though, and a different number comes back. We ran this against a collection that had been killed with os._exit(1) mid-insert at document #551, under two different flush cadences:

Flush cadence before crash	Docs inserted before crash	Last durable `flush()`	`doc_count` after `optimize()`, verified on fresh reopen
No `flush()` calls at all	551	none	0
`flush()` every 100 docs	551	doc #500	501 (IDs 501–550 confirmed missing via `fetch()`)

In the first case, every one of the 551 documents that had survived the crash and reappeared correctly after open() was gone after optimize() — a complete round-trip from "fully recovered" to "fully lost," with optimize() as the only operation in between. In the second case, the loss was smaller but followed the same pattern: everything after the last confirmed flush() disappeared. stats().doc_count reported 551 correctly right up until the moment optimize() was called; afterward, the number silently regressed to whatever had last been durably segmented to disk. There is no warning, no exception, no flag in the optimize() return value — the only way to see the loss is to reopen from a separate process and check, or to fetch() individual document IDs and find them missing.

It would be easy — and it was, in fact, our own first conclusion — to generalize this into a blanket rule: "optimize() discards unflushed data." That framing turned out to be wrong, and the correction matters more than the original finding, because a rule that's too broad leads to over-cautious workarounds (excessive flush() calls everywhere) that don't actually address the real risk.

To isolate the real cause, we built a collection with 20 documents inserted and flushed, then 10 more inserted with no flush, then shut the process down normally — no crash, no os._exit(), just an ordinary exit. Calling optimize() on that collection preserved all 30 documents, unflushed ones included, verified on a fresh reopen with every ID present. Unflushed data, on its own, is not what optimize() discards.

The actual mechanism is narrower and more specific: optimize()'s data loss applies only to documents recovered via WAL replay from a genuine crash. A collection that has crashed and been auto-recovered on open() carries that recovered data in a state that optimize()'s index-rebuild path silently excludes — regardless of whether it happens to be flushed or not, in the crashed case. A collection with unflushed data but no crash history is fine. The determining factor isn't flush state; it's whether the collection has ever been killed mid-write. Nothing in the public API — no field in stats(), no flag, no exception type — tells you a collection carries this history at read time. The only external evidence is a one-time "crash residue" warning ZVec's C++ layer logs to stderr the first time such a collection is reopened after the crash; there is no persistent, queryable marker after that.

For anyone evaluating ZVec as durable storage inside a workflow engine, this is the finding that should change how a recovery path gets written: index_completeness: 0.0 after a crash is not the danger signal it appears to be, and reaching for optimize() to clear it is precisely the operation that turns a fully-recovered collection into a partially or completely empty one. The next section covers the recovery procedure that was measured to avoid this — reinserting recovered documents into a fresh, crash-free collection before ever calling optimize() — along with the architectural gap that procedure exposes in ZVec's API surface.

Section 3: The Architectural Solution — Fetch, Reinsert, Then Optimize

If optimize() can't be trusted against a collection that carries crash history, the practical question is whether there's a path that gets both properties at once: zero data loss and a fully built index (index_completeness: 1.0). We tested one, and it held up.

The idea is to stop treating the crashed collection as something to be repaired in place, and instead treat it as a read-only source to migrate out of:

import zvec

# STEP 1 — open the crashed collection. This alone triggers WAL replay;
# doc_count comes back correct even though index_completeness is still 0.0.
crashed = zvec.open("~/spike-zvec/data/crashed_collection")
print(crashed.stats())
# {'doc_count': 551, 'index_completeness': {'embedding': 0.0}}

# STEP 2 — fetch every document by ID. In testing, all 551/551 were
# retrieved with zero missing.
all_ids = [str(i) for i in range(551)]
docs = crashed.fetch(all_ids, include_vector=True)
assert len(docs) == 551

# STEP 3 — create a brand-new collection. It has no crash history,
# so it is not subject to the exclusion behavior from Section 2.
recovered = zvec.create_and_open("~/spike-zvec/data/recovered_collection", schema)

# STEP 4 — reinsert through the normal insert() path, not a raw copy.
result = recovered.insert(docs)
recovered.flush()

# STEP 5 — only now, on the crash-free collection, call optimize().
recovered.optimize()

Verified on a fresh process reopen of recovered_collection: doc_count came back as 551 — no loss — and index_completeness came back as {'embedding': 1.0}. Sampled vectors (IDs at both ends of the range) matched the originals byte-for-byte via np.allclose, and query() against the collection returned correctly ranked results. This is the only procedure in the entire investigation that achieved full index completeness and zero data loss at the same time.

The mechanism is exactly what Section 2 predicted: recovered_collection was built entirely through ordinary insert() calls in a process that never crashed. It carries no WAL-replay history for optimize() to silently exclude. The crash-specific exclusion bug simply doesn't apply to a collection that was never itself a crash victim — moving the data sidesteps the defect rather than working around it in place.

The Temporal Integration & Ledger Catch

Step 2 above — crashed.fetch(all_ids, ...) — has a dependency that's easy to gloss over: it requires already knowing all_ids. In the test, that was possible only because the test harness used a predictable, sequential ID scheme ("0" through "550"). ZVec's API was checked for a general-purpose "list every document ID in this collection" or full-collection-scan primitive, and none was found. If a production collection's ID space isn't known in advance — UUIDs generated per-document, IDs assigned by an upstream system, IDs accumulated across many separate insert calls over time — step 2 has no way to execute. fetch() requires you to already know what you're fetching.

This is not a defect to route around cleverly inside ZVec; it's a boundary that another system in the stack needs to own. And for teams already running ZVec inside Temporal Activities, that system is usually already present: Temporal's own workflow state.

A Workflow that owns document ingestion into a ZVec collection is a natural place to also maintain the ID ledger the recovery path needs — not as an afterthought, but as a first-class piece of the workflow's durable state, updated in the same Activity call that performs the insert(). Concretely:

Every insert() Activity records the document IDs it just wrote into Workflow state (or an external store the Workflow tracks — a small side table, a durable queue offset, whatever fits the existing stack).
A recovery Workflow, triggered on detecting the "crash residue" warning at open() time, reads the ID ledger from Temporal state rather than guessing at ZVec's contents.
The recovery Workflow then runs Steps 2–5 above: fetch() using the ledger's ID list, reinsert into a fresh collection, optimize(), and — as the final step — atomically repoint whatever downstream code holds the collection path so it now reads from the recovered collection instead of the crashed one.

None of these steps require ZVec to expose an enumeration API it doesn't have. The ledger question is answered by the layer that was already responsible for orchestrating the writes in the first place, which is precisely the kind of durable bookkeeping problem Temporal workflow state exists to solve. ZVec's gap in ID enumeration isn't a reason to avoid it — it's a reason not to treat it as a self-sufficient system of record. Pair it with something that already remembers what it wrote, and the gap closes.

Conclusion

Taken as a whole, this investigation supports a specific, narrow, and actionable set of conclusions rather than a verdict on ZVec as a whole:

Raw document durability across a hard crash is solid. Every kill-9 test in this investigation recovered doc_count exactly, with zero loss and zero duplication, via automatic WAL replay on open() — no manual recovery call required for the document data itself.
index_completeness: 0.0 is a normal, harmless default, not a crash indicator. It appears identically on collections that have never crashed and collections that just finished recovering from one. query() works correctly against it.
optimize() is unsafe specifically — and only — on a collection with crash history. The exclusion of WAL-replayed data from the index rebuild is not a general property of unflushed data; a crash-free collection with unflushed documents survives optimize() intact. The determining variable is whether the collection was ever killed mid-write, not whether it was recently flushed.
The one procedure measured to avoid the loss entirely is migration, not in-place repair: fetch every document from the crashed collection, reinsert it into a freshly created collection via the normal insert() path, and only call optimize() on that new, crash-free collection.

The operational rule that follows is simple enough to put directly into a runbook: if a collection has ever shown the crash-residue warning on reopen, do not call optimize() on it directly. Reinsert its contents into a new collection first. And because that migration path depends on knowing the full ID space — something ZVec itself cannot enumerate — any production deployment inside an orchestrator like Temporal should treat the orchestrator's own durable state as the ID ledger of record, not an optional convenience. ZVec earns its keep as a fast, in-process vector store; it does not, on its own, earn the role of being the only source of truth for what it contains.

Top comments (1)

Mads Hansen • Jul 14

This is exactly the kind of failure mode that gets missed when “embedded” or “in-process” is treated as automatically simpler.

The dangerous part is not just data loss. It is that the system may look healthy after recovery while the semantic index has quietly drifted from the source of truth.

For vector-backed workflows, I’d want a few invariants to be explicit: