DEV Community: t-obara

Postmortem: Silent Data Loss in an In-Process Vector Store, and the Case for Durable Execution

t-obara — Tue, 21 Jul 2026 01:47:00 +0000

TL;DR

Bug: Calling optimize() on a ZVec Collection that had recovered from an unclean process termination silently discarded the documents restored by WAL replay. The call raised no exception, and stats() on the same handle reported index_completeness: 1.0 — a false signal of success. The loss only became visible when the collection was reopened from a separate process.
Impact: In a minimal repro, doc_count regressed from 150 to 0 after optimize() on a crash-recovered collection with no prior flush() calls.
Fix latency: 54 hours 55 minutes from issue filing to merged fix (alibaba/zvec #594 → #600).
Root cause: A WAL handle lifecycle bug — segment recovery replayed the WAL into a local handle and closed it without reattaching it to the segment, so optimize() treated the writing segment as having no WAL records and skipped persisting the recovered data.
Core argument: Fixing this one bug and preventing this class of bug are two different problems. The patch closes a specific code path; it does not change the fact that the storage engine's durability guarantees are implicit and handle-dependent. The second half of this post looks at what changes when state ownership is pushed to a system whose recovery model has no "last flush boundary" at all — a durable execution / event-sourcing layer such as Temporal.

Background: Why an In-Process Vector Store Has to Earn Its Durability Claims

ZVec is an in-process vector database: no server process, no connection string, storage backed by write-ahead logging (WAL) on the local filesystem. That simplicity is attractive for embedding retrieval directly inside a workflow orchestrator, where the vector store is expected to behave like any other durable resource an orchestrator's Activities touch.

For that use case, "crash-durable" is not a nice-to-have — it is a precondition for adoption. If a storage engine can silently lose state on process failure, it cannot be trusted as the backing store for anything a workflow engine expects to survive a crash and resume correctly.

Baseline: WAL Replay Works

Before this bug surfaces, the base case is worth establishing, since it is where trust in the system starts. A hard-killed process (os._exit(1), equivalent to kill -9) with unflushed inserts fully recovers on zvec.open():

crashed = zvec.open("./zvec_crash_test")
print(crashed.stats)
# doc_count=150 index_completeness={'embedding': 0.0}

doc_count is correct immediately after a crash, purely from automatic WAL replay — no manual recovery call required. index_completeness: 0.0 at this point is not an anomaly; it is the default state of any collection, crashed or not, until optimize() has been called on it, and query() still returns correct results while it holds.

Anatomy of the Silent Loss

A Deceptive Success Signal on the Same Handle

Calling optimize() on a crash-recovered collection does not raise. Worse, stats() on the same handle immediately afterward reports the pre-existing doc_count unchanged, alongside index_completeness: 1.0 — every observable signal on that handle says the optimize succeeded cleanly:

[optimize] before: doc_count=150 index_completeness={'embedding': 0.0}
[optimize] same-handle after optimize(): doc_count=150 index_completeness={'embedding': 1.0}

Neither the return value nor an exception nor the immediately-following stats() call exposes the defect. This is the core hazard: a call that completes cleanly and reports a plausible, successful-looking state can still have discarded data.

The Loss Surfaces Only Across a Process Boundary

The regression is invisible on the handle that called optimize() because that handle still holds the in-memory view of the data. It becomes visible only when a separate process reopens the same collection from disk:

[after-optimize] doc_count=0 index_completeness={'embedding': 0.0}

doc_count regresses to whatever was last durably segmented to disk before the crash — in this case, 0, since no flush() was ever called. The gap between the in-memory state a handle believes is true and the state actually persisted to disk is exactly what optimize()'s bug exploited.

Isolating the Cause: A Control Experiment

A plausible but wrong hypothesis at this point is "optimize() loses unflushed data." A control experiment rules that out: a collection with unflushed data but no crash history — created, populated, and shut down normally — retains 100% of its documents after optimize(), unflushed data included.

The loss is therefore not a property of "unflushed data" in general. It is specific to documents that were recovered via WAL replay from a genuine unclean termination. optimize()'s index-rebuild path excluded that WAL-replayed data from the rebuilt segment rather than merging it in.

A second experiment narrowed the loss boundary precisely. With a periodic flush() cadence — every 100 documents, 551 total inserted — optimize() on the crash-recovered collection left exactly 501 documents; IDs 501–550, everything inserted after the last flush() at ID 500, were missing. The loss boundary is exactly the last durably-flushed segment, not "everything since the crash" and not "everything unflushed." A single experiment result is not treated as proof of a causal claim here — the crash-history-vs-no-crash-history comparison is what isolates the trigger.

The Fix: Root Cause and Patch Design (PR #600)

Root cause, in one sentence: segment recovery replayed WAL records into a local WAL handle, then closed it without restoring the member WAL handle on the segment — so optimize()'s subsequent flush treated the writing segment as having no WAL records and skipped persisting them.

The patch to src/db/index/segment/segment.cc:

@@ -4212,8 +4212,6 @@ Status SegmentImpl::recover() {
               wal_file_path.c_str());
     return Status::OK();
   }
-  AILEGO_DEFER([&]() { recover_wal_file->close(); });
-
   std::array<uint64_t, static_cast<size_t>(Operator::DELETE) + 1>
       recovered_doc_count{};
   uint64_t total_recovered_doc_count{0};
@@ -4297,7 +4295,17 @@ Status SegmentImpl::recover() {
       (size_t)recovered_doc_count[3]   // DELETE
   );

-  return Status::OK();
+  if (recover_wal_file->close() != 0) {
+    return Status::InternalError("Failed to close recovered wal file: ",
+                                 wal_file_path);
+  }
+  recover_wal_file.reset();
+
+  // Keep the recovered WAL attached to the segment. Operations such as
+  // optimize() flush the writing segment before sealing it; without an open
+  // member WAL, flush() treats the recovered memory components as empty and
+  // returns without persisting them.
+  return open_wal_file();
 }

Instead of discarding the recovery WAL handle once replay finishes, recover() now reopens the segment's member WAL handle (open_wal_file()) before returning. optimize()'s subsequent flush step then sees a live WAL attached to the segment and persists the recovered records instead of treating the segment as empty.

This is a handle-lifecycle bug, not a logic bug in the index-rebuild algorithm itself — which is also why the patch is narrow: three lines removed, roughly a dozen added, confined to the recovery path.

Structural Mitigation: Why "Be Careful With `optimize()`" Isn't a Strategy

A Patch Closes This Bug, Not This Class of Bug

PR #600 closes the specific intersection of crash recovery and optimize(). It does not change the underlying design: an in-process storage engine that owns its own, implicit durability contract, with no external signal exposed at the time of a call like whether a collection currently carries unreplayed crash-recovery state. The next unknown intersection between recovery and some other operation is a design property, not a one-off accident, and a single patch cannot retire that property.

Event-Sourced Replay vs. Segment/WAL Replay

ZVec's WAL-based recovery model has a boundary: the last segment durably flushed to disk. Everything after that boundary depends on the WAL being correctly replayed and correctly reattached on every code path that might act on it — which is exactly the class of invariant that PR #600 had to repair by hand.

Durable execution frameworks built on event sourcing, such as Temporal, do not have an equivalent boundary. A Workflow's state is not a periodically-flushed snapshot; it is the deterministic replay of its complete event history — every signal received, every activity result recorded — reconstructed from that history on every recovery. There is no concept of "the last flush point" for the replay to fall behind, because the event history is the durable state, not a cache of it.

Reference Implementation: Signal-Sourced State With Verified Crash Recovery

A companion Temporal-based agent workflow used elsewhere demonstrates the pattern concretely. A long-running Workflow exposes signal handlers that mutate its in-memory state:

@workflow.signal
def add_task(self, task: str) -> None:
    self._task_queue.append(task)

@workflow.signal
def update_task_priority(self, priority: int) -> None:
    self._priority = max(1, min(10, priority))

@workflow.signal
def inject_human_feedback(self, feedback: str) -> None:
    self._human_feedback = feedback

Every signal is recorded into Temporal's event history as it is received. This is not a proposal — it has an automated resilience test that actually exercises the failure mode: start the Workflow, send signals (inject feedback, change priority), forcibly kill the worker process, restart it, and query the Workflow's state to confirm the signal-driven changes survived the crash intact. The recovery path here is full deterministic replay of the event history against a fresh worker, not a WAL segment reattachment.

This does not make the underlying storage problem disappear entirely. ZVec still exposes no full-collection enumeration API, so any migration-based recovery workaround depends on the caller already knowing every document ID. That is precisely why, in an architecture where ZVec sits inside a Temporal-orchestrated pipeline, the orchestrator's own durable state — not the storage engine's internal bookkeeping — is the right place to keep the ID ledger of record.

Post-Fix Verification: Does the Patch Work as Expected?

Constraint: Not Yet on PyPI

As of this verification pass, pip index versions zvec still reports 0.5.1 as latest — the same version the bug was originally found on. The merge commit containing the fix (e89be98b) is not included in any published PyPI release. A v0.6.0 git tag exists in the repository, but comparing it against the fix commit shows it is behind the fix — it was cut before the patch landed. Re-running the original repro script against a released package is therefore not currently possible, and that limitation is stated here rather than worked around.

Code-Level Verification

The change described above was confirmed directly against the PR's diff, not inferred from the PR description: the root-cause explanation ("the recovered WAL was not reattached to the segment's member handle") matches the actual patch (recover() now calls open_wal_file() before returning, instead of leaving the segment without a live WAL handle).

Test-Level Verification

The PR adds a regression test that mirrors the originally reported reproduction almost exactly:

TEST_F(CrashRecoveryTest, OptimizeAfterCrashRecoveryPersistsReplayedData) {
  // ... create collection, crash during insertion ...

  auto result = Collection::Open(dir_path_, options_);
  auto collection = result.value();
  recovered_doc_count = collection->Stats().value().doc_count;
  ASSERT_GT(recovered_doc_count, 0)
      << "No documents were recovered from the WAL";

  auto status = collection->Optimize();
  ASSERT_TRUE(status.ok()) << status.message();
  ASSERT_EQ(collection->Stats().value().doc_count, recovered_doc_count);

  // Reopen from a fresh handle and assert nothing was lost.
  auto reopened = Collection::Open(dir_path_, options_);
  ASSERT_EQ(reopened.value()->Stats().value().doc_count, recovered_doc_count)
      << "Optimize discarded documents restored from the WAL";
}

Crash, reopen, optimize(), reopen again from a fresh handle, assert doc_count is unchanged — the same crash-then-verify-from-a-fresh-process structure used to originally isolate the bug.

CI Evidence: Where the PR Description and the Public Logs Diverge

The PR description states that "the full CI matrix passed on the same commit: Linux, macOS, Windows, Android, iOS, lint, and clang-tidy." Pulling the actual check-runs from the GitHub Actions API tells a more precise story.

On the PR's own head commit (aaa4aab4), the Build & Test jobs for every platform, along with lint and clang-tidy, show as skipped — consistent with a permission gate that withholds full CI from external-contributor pull requests until a maintainer authorizes it. "The same commit" did not, in fact, run the full matrix.

What actually ran, and passed, was the merge commit onto main (e89be98b). Its CI log shows:

66/152 Test  #67: optimize_recovery_test ............................   Passed  354.84 sec
109/152 Test  #68: write_recovery_test ...............................   Passed  1687.70 sec

write_recovery_test is the binary that contains the new OptimizeAfterCrashRecoveryPersistsReplayedData case; it passed on the merge commit, along with the C++ test suite, the Python test suite, lint, clang-tidy, and a linux-x64 build. That is solid first-party evidence that the fix behaves as intended.

It is also worth being precise about what did not run: the Windows, macOS (multiple variants), iOS, and Android Build & Test jobs remained skipped even after the merge to main. "Full CI matrix" is therefore accurate for the core Linux test suite and static analysis, but not for the full platform matrix implied by the PR description.

What Was Not Done

This verification did not include building ZVec from source and re-running the original Python repro script against the patched code — there is no PyPI release yet to install, and a from-source build was out of scope for this pass. The natural follow-up, once a release beyond 0.5.1 ships, is to re-run repro_zvec_optimize_dataloss.py against the released package and confirm doc_count no longer regresses to 0.

Engineering Checklist

Verify data persistence from a fresh process and a fresh handle, never from the same handle that performed the write — a same-handle read can report a plausible, successful-looking state while the disk-persisted state has already diverged.
If a storage API cannot distinguish "unflushed" from "crash-recovered" in its return values or metrics, that distinction has to be tracked explicitly by the caller.
Do not treat a storage engine's internal durability model as a self-sufficient system of record inside an orchestrated pipeline. Treat the orchestration layer's own durable state as the ledger of record instead.
Do not generalize a single experiment's result into a causal claim ("crash-specific," "a general bug," etc.) without a control experiment to isolate the actual variable.

The In-Process Vector DB Trap: How ZVec's `optimize()` Can Silently Erase Crash-Recovered Data

t-obara — Tue, 14 Jul 2026 02:02:51 +0000

Update (July 2026): We have published a follow-up postmortem detailing the official fix (PR #600) for this bug and the structural case for durable execution (Temporal). Read the full analysis here: [https://dev.to/obataka/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for-durable-execution-16i2]

Introduction

Embedded, serverless vector databases are having a moment. The pitch is simple: skip the sidecar container, skip the network hop, just call a Python API and get a vector index that lives inside your own process. ZVec (PyPI package zvec, v0.5.1, built by Alibaba on a C++ core with Python bindings) is one of the newer entrants making that pitch. Opening a collection is genuinely trivial:

import zvec

collection = zvec.create_and_open("~/spike-zvec/data/my_collection", schema)

No Docker, no connection string, no separate process to keep alive. For a team running retrieval-augmented generation inside an orchestration layer — Temporal Activities, in our case — that simplicity is worth taking seriously. One fewer moving part is one fewer thing that pages you at 3am.

But "runs inside your process" cuts both ways. If the vector store lives inside the Activity's process, it dies exactly when the Activity's process dies — SIGKILL, an OOM kill, a host eviction, whatever your orchestrator throws at a worker mid-task. Before trusting any in-process store with durable state, the only honest question is: what does it look like on the other side of a hard crash?

So we tested it the direct way. We inserted 551 documents into a ZVec collection and, mid-run, killed the process with os._exit(1) — the Python equivalent of kill -9, no cleanup handlers, no graceful shutdown. Then we reopened the collection from a fresh process and checked what came back.

The result was reassuring on the point that matters most: the document data itself survived intact. stats().doc_count reported exactly 551 on reopen — no loss, no duplication. ZVec's write-ahead log (WAL) replayed automatically on open(), with no explicit "recover" call required from application code. For raw durability of inserted documents, this held up under a real kill-9 test.

What did not look reassuring at first glance was this line from the same stats() call:

>>> collection.stats()
{'doc_count': 551, 'index_completeness': {'embedding': 0.0}}

index_completeness: 0.0, right after a hard crash, reads like exactly the kind of number that should make you nervous — a half-built index, a corruption signal, something to fix immediately. We treated it that way at first too. It turned out to be the wrong read.

We reopened three other collections from earlier, entirely uneventful test runs — collections that had never crashed, never been killed, shut down cleanly every time. All three reported the identical index_completeness: 0.0. The number has nothing to do with crash history. It is simply the default state of any ZVec collection — pristine or crash-recovered — that has never had optimize() called on it. And critically, a collection sitting at index_completeness: 0.0 is not degraded in any functional sense: query() against it returns correct, ranked results. It's an unbuilt index, not a broken one.

That distinction — "unbuilt" versus "broken" — turns out to matter a great deal, because the natural next move is to reach for optimize() to build the index and push that number to 1.0. That move is where the real danger in this system lives, and it's the subject of the rest of this piece.

Section 2: The Deep Pitfall — Silent Data Loss in `optimize()`

The instinct, once you've seen index_completeness: 0.0, is straightforward: call collection.optimize() and move on.

collection = zvec.open("~/spike-zvec/data/crashed_collection")
print(collection.stats())
# {'doc_count': 551, 'index_completeness': {'embedding': 0.0}}

collection.optimize()
print(collection.stats())
# {'doc_count': 551, 'index_completeness': {'embedding': 1.0}}

On the same process handle, this looks like a clean repair: doc_count is still 551, index_completeness is now 1.0. No exception was raised. Nothing in the return value or the logs suggests anything went wrong.

Reopen that same collection from a fresh process, though, and a different number comes back. We ran this against a collection that had been killed with os._exit(1) mid-insert at document #551, under two different flush cadences:

Flush cadence before crash	Docs inserted before crash	Last durable `flush()`	`doc_count` after `optimize()`, verified on fresh reopen
No `flush()` calls at all	551	none	0
`flush()` every 100 docs	551	doc #500	501 (IDs 501–550 confirmed missing via `fetch()`)

In the first case, every one of the 551 documents that had survived the crash and reappeared correctly after open() was gone after optimize() — a complete round-trip from "fully recovered" to "fully lost," with optimize() as the only operation in between. In the second case, the loss was smaller but followed the same pattern: everything after the last confirmed flush() disappeared. stats().doc_count reported 551 correctly right up until the moment optimize() was called; afterward, the number silently regressed to whatever had last been durably segmented to disk. There is no warning, no exception, no flag in the optimize() return value — the only way to see the loss is to reopen from a separate process and check, or to fetch() individual document IDs and find them missing.

It would be easy — and it was, in fact, our own first conclusion — to generalize this into a blanket rule: "optimize() discards unflushed data." That framing turned out to be wrong, and the correction matters more than the original finding, because a rule that's too broad leads to over-cautious workarounds (excessive flush() calls everywhere) that don't actually address the real risk.

To isolate the real cause, we built a collection with 20 documents inserted and flushed, then 10 more inserted with no flush, then shut the process down normally — no crash, no os._exit(), just an ordinary exit. Calling optimize() on that collection preserved all 30 documents, unflushed ones included, verified on a fresh reopen with every ID present. Unflushed data, on its own, is not what optimize() discards.

The actual mechanism is narrower and more specific: optimize()'s data loss applies only to documents recovered via WAL replay from a genuine crash. A collection that has crashed and been auto-recovered on open() carries that recovered data in a state that optimize()'s index-rebuild path silently excludes — regardless of whether it happens to be flushed or not, in the crashed case. A collection with unflushed data but no crash history is fine. The determining factor isn't flush state; it's whether the collection has ever been killed mid-write. Nothing in the public API — no field in stats(), no flag, no exception type — tells you a collection carries this history at read time. The only external evidence is a one-time "crash residue" warning ZVec's C++ layer logs to stderr the first time such a collection is reopened after the crash; there is no persistent, queryable marker after that.

For anyone evaluating ZVec as durable storage inside a workflow engine, this is the finding that should change how a recovery path gets written: index_completeness: 0.0 after a crash is not the danger signal it appears to be, and reaching for optimize() to clear it is precisely the operation that turns a fully-recovered collection into a partially or completely empty one. The next section covers the recovery procedure that was measured to avoid this — reinserting recovered documents into a fresh, crash-free collection before ever calling optimize() — along with the architectural gap that procedure exposes in ZVec's API surface.

Section 3: The Architectural Solution — Fetch, Reinsert, Then Optimize

If optimize() can't be trusted against a collection that carries crash history, the practical question is whether there's a path that gets both properties at once: zero data loss and a fully built index (index_completeness: 1.0). We tested one, and it held up.

The idea is to stop treating the crashed collection as something to be repaired in place, and instead treat it as a read-only source to migrate out of:

import zvec

# STEP 1 — open the crashed collection. This alone triggers WAL replay;
# doc_count comes back correct even though index_completeness is still 0.0.
crashed = zvec.open("~/spike-zvec/data/crashed_collection")
print(crashed.stats())
# {'doc_count': 551, 'index_completeness': {'embedding': 0.0}}

# STEP 2 — fetch every document by ID. In testing, all 551/551 were
# retrieved with zero missing.
all_ids = [str(i) for i in range(551)]
docs = crashed.fetch(all_ids, include_vector=True)
assert len(docs) == 551

# STEP 3 — create a brand-new collection. It has no crash history,
# so it is not subject to the exclusion behavior from Section 2.
recovered = zvec.create_and_open("~/spike-zvec/data/recovered_collection", schema)

# STEP 4 — reinsert through the normal insert() path, not a raw copy.
result = recovered.insert(docs)
recovered.flush()

# STEP 5 — only now, on the crash-free collection, call optimize().
recovered.optimize()

Verified on a fresh process reopen of recovered_collection: doc_count came back as 551 — no loss — and index_completeness came back as {'embedding': 1.0}. Sampled vectors (IDs at both ends of the range) matched the originals byte-for-byte via np.allclose, and query() against the collection returned correctly ranked results. This is the only procedure in the entire investigation that achieved full index completeness and zero data loss at the same time.

The mechanism is exactly what Section 2 predicted: recovered_collection was built entirely through ordinary insert() calls in a process that never crashed. It carries no WAL-replay history for optimize() to silently exclude. The crash-specific exclusion bug simply doesn't apply to a collection that was never itself a crash victim — moving the data sidesteps the defect rather than working around it in place.

The Temporal Integration & Ledger Catch

Step 2 above — crashed.fetch(all_ids, ...) — has a dependency that's easy to gloss over: it requires already knowing all_ids. In the test, that was possible only because the test harness used a predictable, sequential ID scheme ("0" through "550"). ZVec's API was checked for a general-purpose "list every document ID in this collection" or full-collection-scan primitive, and none was found. If a production collection's ID space isn't known in advance — UUIDs generated per-document, IDs assigned by an upstream system, IDs accumulated across many separate insert calls over time — step 2 has no way to execute. fetch() requires you to already know what you're fetching.

This is not a defect to route around cleverly inside ZVec; it's a boundary that another system in the stack needs to own. And for teams already running ZVec inside Temporal Activities, that system is usually already present: Temporal's own workflow state.

A Workflow that owns document ingestion into a ZVec collection is a natural place to also maintain the ID ledger the recovery path needs — not as an afterthought, but as a first-class piece of the workflow's durable state, updated in the same Activity call that performs the insert(). Concretely:

Every insert() Activity records the document IDs it just wrote into Workflow state (or an external store the Workflow tracks — a small side table, a durable queue offset, whatever fits the existing stack).
A recovery Workflow, triggered on detecting the "crash residue" warning at open() time, reads the ID ledger from Temporal state rather than guessing at ZVec's contents.
The recovery Workflow then runs Steps 2–5 above: fetch() using the ledger's ID list, reinsert into a fresh collection, optimize(), and — as the final step — atomically repoint whatever downstream code holds the collection path so it now reads from the recovered collection instead of the crashed one.

None of these steps require ZVec to expose an enumeration API it doesn't have. The ledger question is answered by the layer that was already responsible for orchestrating the writes in the first place, which is precisely the kind of durable bookkeeping problem Temporal workflow state exists to solve. ZVec's gap in ID enumeration isn't a reason to avoid it — it's a reason not to treat it as a self-sufficient system of record. Pair it with something that already remembers what it wrote, and the gap closes.

Conclusion

Taken as a whole, this investigation supports a specific, narrow, and actionable set of conclusions rather than a verdict on ZVec as a whole:

Raw document durability across a hard crash is solid. Every kill-9 test in this investigation recovered doc_count exactly, with zero loss and zero duplication, via automatic WAL replay on open() — no manual recovery call required for the document data itself.
index_completeness: 0.0 is a normal, harmless default, not a crash indicator. It appears identically on collections that have never crashed and collections that just finished recovering from one. query() works correctly against it.
optimize() is unsafe specifically — and only — on a collection with crash history. The exclusion of WAL-replayed data from the index rebuild is not a general property of unflushed data; a crash-free collection with unflushed documents survives optimize() intact. The determining variable is whether the collection was ever killed mid-write, not whether it was recently flushed.
The one procedure measured to avoid the loss entirely is migration, not in-place repair: fetch every document from the crashed collection, reinsert it into a freshly created collection via the normal insert() path, and only call optimize() on that new, crash-free collection.

The operational rule that follows is simple enough to put directly into a runbook: if a collection has ever shown the crash-residue warning on reopen, do not call optimize() on it directly. Reinsert its contents into a new collection first. And because that migration path depends on knowing the full ID space — something ZVec itself cannot enumerate — any production deployment inside an orchestrator like Temporal should treat the orchestrator's own durable state as the ID ledger of record, not an optional convenience. ZVec earns its keep as a fast, in-process vector store; it does not, on its own, earn the role of being the only source of truth for what it contains.

Building Fault-Tolerant AI Agent Workflows with Temporal and CrewAI

t-obara — Wed, 08 Jul 2026 01:13:51 +0000

A reference pattern for running multi-agent LLM systems under strict human governance in production.

Reference Architecture & Demo Video: [https://project-sy5bk-qyr66bsfr-obataka123.vercel.app/lp.html]

1. The Challenge of Enterprise AI Agents

Most "AI agent" demos share three properties that make them unfit for enterprise use: they hold state in process memory, they treat human approval as an optional UI affordance rather than a hard gate, and they have no answer for what happens when a call to the LLM provider times out at 2am.

None of these are edge cases. In a real deployment:

State outlives the process. A document-generation or approval workflow can legitimately wait days for a human reviewer. A Python script with an input() call, or a LangGraph graph checkpointed to a database you manage yourself, does not survive a deploy, a container restart, or a crash during that wait — unless you build and maintain your own durability layer.
LLM calls fail in mundane, frequent ways. Rate limits, transient 5xx errors, network blips. These are not exceptional — they are the normal operating condition of any system that calls an external LLM API at volume. Retry logic bolted onto application code tends to be inconsistent: some call sites retry, some don't, backoff parameters drift between them.
Human-in-the-loop is not a webhook. If "wait for approval" is implemented as "poll a database row," you've built a distributed system without admitting it, and you inherit all its failure modes (missed writes, race conditions between the poller and the approver) without any of the tooling to reason about them.

The design question this article answers is narrow: how do you get a state machine that survives process death, retries failures according to an explicit policy, and blocks on human judgment for an unbounded period — without writing that infrastructure yourself?

The reference implementation here is an SOP (Standard Operating Procedure) auto-improvement pipeline: an LLM drafts a document in three phases, a human approves or sends back feedback at each phase, an autonomous validation/fix loop (CrewAI, two agents) cleans up the result, and a final human gate approves a GitHub PR. The domain is incidental. The pattern — orchestration engine as the source of truth for state, agent framework as a stateless reasoning unit invoked from inside an Activity — is the point.

2. Architecture Overview

Why Temporal, specifically

The alternative most teams reach for first is a graph-based agent framework (LangGraph and similar) running as a long-lived process, or a hand-rolled script with a job queue. Both push the durability problem back onto the application:

Requirement	Plain script / job queue	LangGraph (self-managed checkpointing)	Temporal
Survives process crash mid-wait	No — in-memory state is gone	Depends on your checkpoint store's consistency guarantees	Yes — event history is the state; a new worker replays it
Multi-day human approval wait	Requires a polling job + your own timeout handling	Requires your own checkpoint schema for "waiting" state	`workflow.wait_condition` on a `@workflow.signal`, no polling
Retry policy for LLM/API calls	Ad hoc, per call site	Ad hoc, per node	Declared once (`RetryPolicy`), enforced by the runtime
Deploying new code while workflows are in flight	Breaks in-flight state silently	Breaks in-flight state silently	`workflow.patched()` — explicit, versioned branching
Audit trail of what happened and when	You build it	You build it	Event history is authoritative by construction

The last two rows are the ones that actually matter at enterprise scale, and they're the ones demos never show.

Determinism and replay. A Temporal Workflow function is not "running" in the conventional sense — it is replayed from its event history every time the worker needs to reconstruct its state (after a crash, a deploy, a worker restart). This means the workflow function itself must be deterministic: no wall-clock time, no random numbers, no direct I/O. All non-deterministic work — LLM calls, GitHub API calls, validation — is pushed into Activities, which are allowed side effects and are retried independently by the runtime. sop_workflow.py enforces this at the import level:

with workflow.unsafe.imports_passed_through():
    from activities.sop_activity import generate_sop_phase_activity
    from activities.fix_sop_activity import fix_sop_with_crew_activity, ...
    from activities.github_activity import GitHubActivity

CrewAI itself imports os, makes network calls, and is not sandbox-safe — it cannot be imported inside the workflow's deterministic sandbox at all. It only ever runs inside an Activity.

Versioning in-flight workflows. This is the detail most articles skip. A workflow that's been running for two days is mid-way through its event history when you ship new code. If you change the code path a running instance already took, replay diverges from history and the workflow breaks. sop_workflow.py handles this with an explicit branch:

if workflow.patched("split-writer-reviewer"):
    return await self._call_fix_decomposed(sop_text, failures, human_feedback)
return await workflow.execute_activity(
    fix_sop_with_crew_activity,
    args=[sop_text, failures, human_feedback, self._fix_attempt],
    start_to_close_timeout=timedelta(minutes=7),
    retry_policy=LLM_RETRY_POLICY,
)

This is a real production concern: the fix-loop was refactored from one monolithic CrewAI kickoff() call into two separate Activities (Writer, then Reviewer). Workflow instances already in flight when that code shipped keep using the old single-Activity path (fix_sop_with_crew_activity); new instances take the split path. Without patched(), this refactor would non-deterministically break any workflow that was mid-execution at deploy time. There is no equivalent primitive in a hand-rolled state machine — you'd need to build and version your own replay-compatibility layer.

Mapping Temporal to CrewAI

CrewAI's contribution here is narrow and deliberate: it is a reasoning unit, not an orchestrator. It never owns state, never decides when to wait for a human, and never retries itself. Two agents — Writer and Reviewer — are defined per the standard CrewAI Agent/Task API:

writer = Agent(
    role="SOP 修正担当",
    goal=(
        "SOP のバリデーション失敗項目を全て解消し、"
        "最小限の変更で明確・再現性の高い改善版を Markdown 形式で出力する。"
    ),
    backstory="5 年以上のテクニカルライター経験を持つ専門家。...",
    llm=llm,
    verbose=True,
    allow_delegation=False,
)

Critically, in the current code path this pair is not invoked as a single CrewAI Crew.kickoff() from inside one Activity. It's decomposed into two independent Activities, each with its own retry policy:

async def _call_fix_decomposed(self, sop_text, failures, human_feedback=""):
    self._active_agent = "Writer"
    writer_result = await workflow.execute_activity(
        writer_task_activity,
        args=[sop_text, failures, human_feedback, self._fix_attempt],
        start_to_close_timeout=timedelta(minutes=7),
        retry_policy=LLM_RETRY_POLICY,
    )

    self._active_agent = "Reviewer"
    reviewer_result = await workflow.execute_activity(
        reviewer_task_activity,
        args=[writer_result.text],
        start_to_close_timeout=timedelta(minutes=7),
        retry_policy=LLM_RETRY_POLICY,
    )
    ...

The consequence: if the Reviewer's LLM call fails transiently, Temporal retries only the Reviewer Activity. The Writer's already-committed output is not re-run, re-billed, or re-generated (which would risk non-identical output on a second pass). This is the practical argument for keeping the agent framework's unit of work small and pushing sequencing up into the orchestrator — multi-agent frameworks that bundle several agents into one opaque kickoff() call give you all-or-nothing retry semantics, which is the wrong grain for cost- and latency-sensitive LLM pipelines.

Retry policy itself is centralized and explicit, not scattered:

LLM_RETRY_POLICY = RetryPolicy(
    maximum_attempts=3,
    initial_interval=timedelta(seconds=2),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(seconds=30),
)

Every LLM- or API-backed Activity in the workflow — generation, validation, fix, GitHub PR creation — takes this same policy. There is one place to change backoff behavior, and it is enforced by the Temporal server, not by application code that might forget to call it.

3. Implementing Strict Human-in-the-Loop with Temporal Signals

The product enforces four hard human judgment gates. None are advisory; the workflow physically cannot proceed past any of them without an external Signal.

Gate 1–3: per-phase approval (outline → draft → review). Each phase runs the same pattern: generate, then block on a Signal.

@workflow.signal
def approve_step(self, feedback: str) -> None:
    """
    "" (空文字): 現フェーズを承認して次フェーズへ進む。
    非空文字:   フィードバックとして同フェーズを再生成する。
    """
    self._step_feedback = feedback
    self._signal_received = True

for phase in PHASES:  # ["outline", "draft", "review"]
    self._current_phase = phase
    self._attempt_in_phase = 0
    last_feedback = None

    while True:
        self._signal_received = False   # reset BEFORE the Activity runs
        result = await self._call_llm(request)
        self._current_output = result.text
        self._status = "awaiting_approval"

        await workflow.wait_condition(lambda: self._signal_received)

        feedback = self._step_feedback
        self._step_feedback = ""
        self._signal_received = False

        if not feedback:
            self._approved[phase] = self._current_output
            break                        # move to next phase
        else:
            last_feedback = feedback
            self._attempt_in_phase += 1  # regenerate same phase

One detail matters more than it looks: self._signal_received = False is reset before await self._call_llm(...), not after. Signals delivered while the Activity is still executing (e.g., an eager reviewer clicking "approve" mid-generation) are not lost — wait_condition picks up the flag the instant control returns to the workflow. A polling-based approval mechanism would have a real race window here; Temporal's Signal delivery guarantees there isn't one.

An empty-string feedback approves and advances; any non-empty string is treated as a rejection with revision instructions, and the same phase regenerates with that feedback injected into the next LLM request (SOPRequest.feedback). This gives reviewers a single Signal type for two distinct actions, and gives the workflow author-level (not binary) revision control instead of just approve/reject.

Gate 4: PR approval, with a reject path. The final gate — before creating a GitHub PR — supports both approval and a full round-trip back into the autonomous fix loop:

@workflow.signal
def approve_pr(self) -> None:
    self._pr_approved = True

@workflow.signal
def reject_with_feedback(self, input_data: dict) -> None:
    self._human_feedback = (
        input_data.get("comment", "") if isinstance(input_data, dict) else str(input_data)
    )

if github_params.require_approval:
    self._status = "awaiting_pr_approval"
    self._pr_approved = False
    await workflow.wait_condition(
        lambda: self._pr_approved or bool(self._human_feedback)
    )
    if self._human_feedback:
        continue   # loop back to Phase 4 (autonomous_fix) with the feedback
self._status = "creating_pr"
pr_result = await self._call_github_pr(...)

wait_condition here blocks on a disjunction of two Signals, which lets a single wait point resolve two different human outcomes without a secondary polling loop. A rejection doesn't just pause — it re-enters the fix loop with the human's comment injected as a first-class input alongside the validator's own failure list:

if human_feedback:
    fix_result = await self._call_fix(final_sop, [], human_feedback)

Why not just a database flag checked by a cron job? Because that reintroduces every problem Temporal exists to solve: you'd need your own polling interval (latency vs. cost tradeoff), your own guarantee that a write isn't missed between poll cycles, and your own crash recovery for the poller itself. wait_condition on a Signal is push-based, has no polling interval, and is backed by the same event history that makes the rest of the workflow durable — the wait is the crash-recovery mechanism, not a separate one.

Observability without extra infrastructure. Every phase and fix attempt — approved or not — is appended to self._history, exposed via a @workflow.query:

@workflow.query
def get_history(self) -> list:
    """全フェーズ・全試行の比較ログ。人間介入の Evidence として使用する。"""
    return list(self._history)

This is the audit trail: every LLM output the workflow ever produced, whether a human approved or rejected it, and what feedback drove each regeneration — queryable at any point, including on a workflow that's still running, with no separate logging pipeline to maintain.

4. Conclusion & Reference Implementation Pattern

Stripped of the SOP-specific domain logic, the pattern is:

Workflow owns state and sequencing; Activities own side effects. LLM calls, validation, and GitHub operations are Activities with an explicit, shared RetryPolicy. The workflow function itself does nothing non-deterministic.
Human approval is a blocking Signal wait, not a poll. workflow.wait_condition(lambda: flag) combined with @workflow.signal handlers gives push-based, race-free blocking that survives process restarts for however long a human takes — minutes or days, no difference to the implementation.
Multi-agent reasoning is decomposed to Activity granularity. CrewAI agents run as separate Activities (Writer, then Reviewer) rather than one opaque multi-agent kickoff(), so retries are scoped to the agent that actually failed.
Code changes to long-running workflows are versioned explicitly, via workflow.patched(), rather than silently breaking workflow instances that are mid-flight at deploy time.

None of this is specific to SOP generation. The same shape applies to any pipeline where an LLM produces something a human must sign off on before it has real-world effect — code review automation, contract drafting, financial report generation — where "the process might be paused for two days waiting on a person" is not a corner case to be handled, but the normal operating condition the architecture has to be built around from the start.

About the Reference Implementation

The code patterns discussed in this article were extracted from a production-ready SOP (Standard Operating Procedure) improvement platform built to demonstrate high-durability AI agents.

Architecture & Video Demo: [https://project-sy5bk-qyr66bsfr-obataka123.vercel.app/lp.html]
Connect on LinkedIn: [https://www.linkedin.com/in/takashi-obara-1a5305150/]
Contact / Feedback: If you are an enterprise engineering leader tackling similar state-management or human-governance challenges with LLMs, I would welcome a high-context technical discussion. You can reach out to me via LinkedIn or through the contact form on the architecture page above.

Note: I am a hands-on Japanese engineer. While the technical architecture and code in this article are 100% mine, I use AI to assist with English translation for global communication.

🛠️ Reference Implementation & Code

The complete, production-ready codebase with Docker Compose setup is fully documented and open-sourced here:
👉 GitHub: obataka/temporal-demo