When an AI agent moves from answering questions to modifying real business systems — posting replenishment drafts, reconciling invoices, or updating warehouse tables — prompt engineering stops being the hardest problem. Crash recovery and duplicate side effects take over.
Many engineering teams start with a graph framework like LangGraph and later consider adding Temporal plus its visibility stack once worker crashes start causing partial writes. While Temporal is battle-tested for large microservice fleets, running multiple extra stateful clusters is often heavy for mid-sized engineering teams that already run reliable PostgreSQL servers.
At Dalian Fidwell Technology, we built ZeroOS as a self-hosted Linux + PostgreSQL control plane using three pragmatic database patterns instead of external broker clusters: a 12-state step lease machine, monotonic fencing tokens, and a three-tier tool safety ledger.
1. The Split-Brain Worker Problem
Consider an agent step that calls a legacy ERP HTTP endpoint:
- Worker A claims step
seq=4with a 30-second lease. - Worker A sends the POST request, then hits a 40-second network stall or host pause.
- The lease expires; Worker B claims
seq=4and retries. - Both requests reach the ERP — creating duplicate purchase orders.
In-memory graph checkpoints cannot solve this because the failure occurs across process boundaries and external I/O.
2. Atomic Lease Claiming with Monotonic Fencing Tokens
In PostgreSQL, we track every step inside a lease table and increment a monotonic fencing_token on every claim or takeover:
UPDATE agent_step_leases
SET state = 'RUNNING',
lease_owner = $3,
fencing_token = fencing_token + 1,
lease_expires_at = NOW() + INTERVAL '30 seconds',
updated_at = NOW()
WHERE task_id = $1
AND step_seq = $2
AND (
state = 'READY'
OR (state = 'RUNNING' AND lease_expires_at < NOW())
)
RETURNING fencing_token, idempotency_key;
When the worker finishes and commits its output digest, the UPDATE requires both the exact lease_owner and the exact fencing_token:
UPDATE agent_step_leases
SET state = 'COMMITTED',
result_digest = $4,
updated_at = NOW()
WHERE task_id = $1
AND step_seq = $2
AND state = 'RUNNING'
AND lease_owner = $3
AND fencing_token = $5
AND lease_expires_at > NOW()
RETURNING task_id, step_seq;
If zero rows are updated, the stale worker knows it was preempted and aborts any downstream commit.
3. Classifying Tools Before They Touch Production
Blind retries are dangerous when legacy ERPs lack deduplication keys. Every tool registered in ZeroOS must declare one of three safety tiers:
-
READ_ONLY: Safe to retry with exponential backoff. -
IDEMPOTENT_WRITE: Downstream accepts an external key; ZeroOS generates a deterministicidempotency_keyfromtask_id + step_seq + payload_hashand logs it prior to dispatch. -
NON_IDEMPOTENT: No automatic retry on ambiguous timeouts. If the network drops after dispatch, the step transitions intoRECONCILE_REQUIREDand routes through a fail-closed approval or reconciliation check before proceeding.
4. Secret Hygiene & Internal Staging Observation
To avoid leaking credentials through /proc/*/environ or crash dumps, ZeroOS injects short-lived credentials using an FD-Unlink descriptor pattern rather than plaintext environment variables.
In an isolated internal staging test on self-hosted Linux hardware, killing an active worker process mid-step resulted in lease expiry detection and standby worker takeover in ~6.99s (6,994 ms) with zero duplicate step commits. Because failover timing depends directly on heartbeat TTL, CPU load, and PostgreSQL I/O settings, treat this strictly as an internal test observation rather than an external uptime guarantee.
Paired optionally with LongMemOS (~3.1 MB standalone local memory engine, embedded_models=false, 463/500 or 92.6% in our internal LongMemEval test with O13 source-line evidence replay), teams can keep both agent execution state and long-term corporate memory inside their own Linux + PostgreSQL perimeter.
Top comments (1)
The separation between fencing a stale worker’s local commit and preventing a duplicate ERP effect is crucial. A fencing token can stop A from committing its step record, but it cannot recall a POST already accepted by the ERP. For a
NON_IDEMPOTENTstep, does lease takeover enterRECONCILE_REQUIREDbefore B can dispatch anything? A kill-after-send fixture with an independent ERP-side delivery counter would make that boundary very clear.