Postmortem: Silent Data Loss in an In-Process Vector Store, and the Case for Durable Execution

TL;DR

• Bug: Calling optimize() on a ZVec Collection that had recovered from an unclean process termination silently discarded the documents restored by WAL replay. The call raised no exception, and stats() on the same handle reported index_completeness: 1.0 — a false signal of success. The loss only became visible when the collection was reopened from a separate process.
• Impact: In a minimal repro, doc_count regressed from 150 to 0 after optimize() on a crash-recovered collection with no prior flush() calls.
• Fix latency: 54 hours 55 minutes from issue filing to merged fix (alibaba/zvec #594 → #600).
• Root cause: A WAL handle lifecycle bug — segment recovery replayed the WAL into a local handle and closed it without reattaching it to the segment, so optimize() treated the writing segment as having no WAL records and skipped persisting the recovered data.
• Core argument: Fixing this one bug and preventing this class of bug are two different problems. The patch closes a specific code path; it does not change the fact that the storage engine's durability guarantees are implicit and handle-dependent. The second half of this post looks at what changes when state ownership is pushed to a system whose recovery model has no "last flush boundary" at all — a durable execution / event-sourcing layer such as Temporal.

---

Background: Why an In-Process Vector Store Has to Earn Its Durability Claims

ZVec is an in-process vector database: no server process, no connection string, storage backed by write-ahead logging (WAL) on the local filesystem. That simplicity is attractive for embedding retrieval directly inside a workflow orchestrator, where the vector store is expected to behave like any other durable resource an orchestrator's Activities touch.

For that use case, "crash-durable" is not a nice-to-have — it is a precondition for adoption. If a storage engine can silently lose state on process failure, it cannot be trusted as the backing store for anything a workflow engine expects to survive a crash and resume correctly.

Baseline: WAL Replay Works

Before this bug surfaces, the base case is worth establishing, since it is where trust in the system starts. A hard-killed process (os._exit(1), equivalent to kill -9) with unflushed inserts fully recovers on zvec.open():

crashed = zvec.open("./zvec_crash_test")
print(crashed.stats)
# doc_count=150 index_completeness={'embedding': 0.0}

doc_count is correct immediately after a crash, purely from automatic WAL replay — no manual recovery call required. index_completeness: 0.0 at this point is not an anomaly; it is the default state of any collection, crashed or not, until optimize() has been called on it, and query() still returns correct results while it holds.

---

Anatomy of the Silent Loss

A Deceptive Success Signal on the Same Handle

Calling optimize() on a crash-recovered collection does not raise. Worse, stats() on the same handle immediately afterward reports the pre-existing doc_count unchanged, alongside index_completeness: 1.0 — every observable signal on that handle says the optimize succeeded cleanly:

[optimize] before: doc_count=150 index_completeness={'embedding': 0.0}
[optimize] same-handle after optimize(): doc_count=150 index_completeness={'embedding': 1.0}

Neither the return value nor an exception nor the immediately-following stats() call exposes the defect. This is the core hazard: a call that completes cleanly and reports a plausible, successful-looking state can still have discarded data.

The Loss Surfaces Only Across a Process Boundary

The regression is invisible on the handle that called optimize() because that handle still holds the in-memory view of the data. It becomes visible only when a separate process reopens the same collection from disk:

[after-optimize] doc_count=0 index_completeness={'embedding': 0.0}

doc_count regresses to whatever was last durably segmented to disk before the crash — in this case, 0, since no flush() was ever called. The gap between the in-memory state a handle believes is true and the state actually persisted to disk is exactly what optimize()'s bug exploited.

---

Isolating the Cause: A Control Experiment

A plausible but wrong hypothesis at this point is "optimize() loses unflushed data." A control experiment rules that out: a collection with unflushed data but no crash history — created, populated, and shut down normally — retains 100% of its documents after optimize(), unflushed data included.

The loss is therefore not a property of "unflushed data" in general. It is specific to documents that were recovered via WAL replay from a genuine unclean termination. optimize()'s index-rebuild path excluded that WAL-replayed data from the rebuilt segment rather than merging it in.

A second experiment narrowed the loss boundary precisely. With a periodic flush() cadence — every 100 documents, 551 total inserted — optimize() on the crash-recovered collection left exactly 501 documents; IDs 501–550, everything inserted after the last flush() at ID 500, were missing. The loss boundary is exactly the last durably-flushed segment, not "everything since the crash" and not "everything unflushed." A single experiment result is not treated as proof of a causal claim here — the crash-history-vs-no-crash-history comparison is what isolates the trigger.

---

The Fix: Root Cause and Patch Design (PR #600)

Root cause, in one sentence: segment recovery replayed WAL records into a local WAL handle, then closed it without restoring the member WAL handle on the segment — so optimize()'s subsequent flush treated the writing segment as having no WAL records and skipped persisting them.

The patch to src/db/index/segment/segment.cc:

@@ -4212,8 +4212,6 @@ Status SegmentImpl::recover() {
               wal_file_path.c_str());
     return Status::OK();
   }
-  AILEGO_DEFER([&]() { recover_wal_file->close(); });
-
   std::array<uint64_t, static_cast<size_t>(Operator::DELETE) + 1>
       recovered_doc_count{};
   uint64_t total_recovered_doc_count{0};
@@ -4297,7 +4295,17 @@ Status SegmentImpl::recover() {
       (size_t)recovered_doc_count[3]   // DELETE
   );

-  return Status::OK();
+  if (recover_wal_file->close() != 0) {
+    return Status::InternalError("Failed to close recovered wal file: ",
+                                 wal_file_path);
+  }
+  recover_wal_file.reset();
+
+  // Keep the recovered WAL attached to the segment. Operations such as
+  // optimize() flush the writing segment before sealing it; without an open
+  // member WAL, flush() treats the recovered memory components as empty and
+  // returns without persisting them.
+  return open_wal_file();
 }

Instead of discarding the recovery WAL handle once replay finishes, recover() now reopens the segment's member WAL handle (open_wal_file()) before returning. optimize()'s subsequent flush step then sees a live WAL attached to the segment and persists the recovered records instead of treating the segment as empty.

This is a handle-lifecycle bug, not a logic bug in the index-rebuild algorithm itself — which is also why the patch is narrow: three lines removed, roughly a dozen added, confined to the recovery path.

---

Structural Mitigation: Why "Be Careful With optimize()" Isn't a Strategy

A Patch Closes This Bug, Not This Class of Bug

PR #600 closes the specific intersection of crash recovery and optimize(). It does not change the underlying design: an in-process storage engine that owns its own, implicit durability contract, with no external signal exposed at the time of a call like whether a collection currently carries unreplayed crash-recovery state. The next unknown intersection between recovery and some other operation is a design property, not a one-off accident, and a single patch cannot retire that property.

Event-Sourced Replay vs. Segment/WAL Replay

ZVec's WAL-based recovery model has a boundary: the last segment durably flushed to disk. Everything after that boundary depends on the WAL being correctly replayed and correctly reattached on every code path that might act on it — which is exactly the class of invariant that PR #600 had to repair by hand.

Durable execution frameworks built on event sourcing, such as Temporal, do not have an equivalent boundary. A Workflow's state is not a periodically-flushed snapshot; it is the deterministic replay of its complete event history — every signal received, every activity result recorded — reconstructed from that history on every recovery. There is no concept of "the last flush point" for the replay to fall behind, because the event history is the durable state, not a cache of it.

Reference Implementation: Signal-Sourced State With Verified Crash Recovery

A companion Temporal-based agent workflow used elsewhere demonstrates the pattern concretely. A long-running Workflow exposes signal handlers that mutate its in-memory state:

@workflow.signal
def add_task(self, task: str) -> None:
    self._task_queue.append(task)

@workflow.signal
def update_task_priority(self, priority: int) -> None:
    self._priority = max(1, min(10, priority))

@workflow.signal
def inject_human_feedback(self, feedback: str) -> None:
    self._human_feedback = feedback

Every signal is recorded into Temporal's event history as it is received. This is not a proposal — it has an automated resilience test that actually exercises the failure mode: start the Workflow, send signals (inject feedback, change priority), forcibly kill the worker process, restart it, and query the Workflow's state to confirm the signal-driven changes survived the crash intact. The recovery path here is full deterministic replay of the event history against a fresh worker, not a WAL segment reattachment.

This does not make the underlying storage problem disappear entirely. ZVec still exposes no full-collection enumeration API, so any migration-based recovery workaround depends on the caller already knowing every document ID. That is precisely why, in an architecture where ZVec sits inside a Temporal-orchestrated pipeline, the orchestrator's own durable state — not the storage engine's internal bookkeeping — is the right place to keep the ID ledger of record.

---

Post-Fix Verification: Does the Patch Work as Expected?

Constraint: Not Yet on PyPI

As of this verification pass, pip index versions zvec still reports 0.5.1 as latest — the same version the bug was originally found on. The merge commit containing the fix (e89be98b) is not included in any published PyPI release. A v0.6.0 git tag exists in the repository, but comparing it against the fix commit shows it is behind the fix — it was cut before the patch landed. Re-running the original repro script against a released package is therefore not currently possible, and that limitation is stated here rather than worked around.

Code-Level Verification

The change described above was confirmed directly against the PR's diff, not inferred from the PR description: the root-cause explanation ("the recovered WAL was not reattached to the segment's member handle") matches the actual patch (recover() now calls open_wal_file() before returning, instead of leaving the segment without a live WAL handle).

Test-Level Verification

The PR adds a regression test that mirrors the originally reported reproduction almost exactly:

TEST_F(CrashRecoveryTest, OptimizeAfterCrashRecoveryPersistsReplayedData) {
  // ... create collection, crash during insertion ...

  auto result = Collection::Open(dir_path_, options_);
  auto collection = result.value();
  recovered_doc_count = collection->Stats().value().doc_count;
  ASSERT_GT(recovered_doc_count, 0)
      << "No documents were recovered from the WAL";

  auto status = collection->Optimize();
  ASSERT_TRUE(status.ok()) << status.message();
  ASSERT_EQ(collection->Stats().value().doc_count, recovered_doc_count);

  // Reopen from a fresh handle and assert nothing was lost.
  auto reopened = Collection::Open(dir_path_, options_);
  ASSERT_EQ(reopened.value()->Stats().value().doc_count, recovered_doc_count)
      << "Optimize discarded documents restored from the WAL";
}

Crash, reopen, optimize(), reopen again from a fresh handle, assert doc_count is unchanged — the same crash-then-verify-from-a-fresh-process structure used to originally isolate the bug.

CI Evidence: Where the PR Description and the Public Logs Diverge

The PR description states that "the full CI matrix passed on the same commit: Linux, macOS, Windows, Android, iOS, lint, and clang-tidy." Pulling the actual check-runs from the GitHub Actions API tells a more precise story.

On the PR's own head commit (aaa4aab4), the Build & Test jobs for every platform, along with lint and clang-tidy, show as skipped — consistent with a permission gate that withholds full CI from external-contributor pull requests until a maintainer authorizes it. "The same commit" did not, in fact, run the full matrix.

What actually ran, and passed, was the merge commit onto main (e89be98b). Its CI log shows:

66/152 Test  #67: optimize_recovery_test ............................   Passed  354.84 sec
109/152 Test  #68: write_recovery_test ...............................   Passed  1687.70 sec

write_recovery_test is the binary that contains the new OptimizeAfterCrashRecoveryPersistsReplayedData case; it passed on the merge commit, along with the C++ test suite, the Python test suite, lint, clang-tidy, and a linux-x64 build. That is solid first-party evidence that the fix behaves as intended.

It is also worth being precise about what did not run: the Windows, macOS (multiple variants), iOS, and Android Build & Test jobs remained skipped even after the merge to main. "Full CI matrix" is therefore accurate for the core Linux test suite and static analysis, but not for the full platform matrix implied by the PR description.

What Was Not Done

This verification did not include building ZVec from source and re-running the original Python repro script against the patched code — there is no PyPI release yet to install, and a from-source build was out of scope for this pass. The natural follow-up, once a release beyond 0.5.1 ships, is to re-run repro_zvec_optimize_dataloss.py against the released package and confirm doc_count no longer regresses to 0.

---

Engineering Checklist

1. Verify data persistence from a fresh process and a fresh handle, never from the same handle that performed the write — a same-handle read can report a plausible, successful-looking state while the disk-persisted state has already diverged.
2. If a storage API cannot distinguish "unflushed" from "crash-recovered" in its return values or metrics, that distinction has to be tracked explicitly by the caller.
3. Do not treat a storage engine's internal durability model as a self-sufficient system of record inside an orchestrated pipeline. Treat the orchestration layer's own durable state as the ledger of record instead.
4. Do not generalize a single experiment's result into a causal claim ("crash-specific," "a general bug," etc.) without a control experiment to isolate the actual variable.

Comments

[Mads Hansen]

Excellent isolation of the crash-history condition and the cross-process validation step. One architectural boundary is worth keeping explicit: durable execution can make workflow transitions replayable, but it cannot make an activity's storage engine correct. A replay may call the same buggy optimize() again—or duplicate a side effect—unless the activity has an idempotency contract and verifies persisted state from a fresh process.

For retrieval systems, I prefer treating the vector index as a rebuildable projection, not the sole system of record. Keep a canonical manifest/event stream with document ID, content hash, source version, deletion tombstone, embedding model/version, and projection checkpoint. An indexing activity should compare-and-set that checkpoint only after reopening the collection and validating counts/checksums plus sampled retrieval invariants. On recovery, reconcile the manifest against the projection and rebuild missing generations atomically, then switch an active-generation pointer. Temporal or another durable orchestrator is valuable for retries, checkpoints, and compensation; the event/manifest model is what makes silent projection loss detectable and repairable. Workflow durability and data durability reinforce each other, but they need separate invariants and failure tests.