Exactly-Once Effects for AI Agents: The Zombie Worker Problem

At-least-once execution. Idempotent side effects. And a worker that came back from the dead.

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

What I Built

Your AI Agent Just Executed Twice. What Could Go Wrong?

Imagine an AI agent processing a customer refund.

It receives a request, examines the circumstances, approves a $100 refund, and calls a payment tool.

The payment provider successfully processes the refund.

Then the worker crashes — before the workflow can save its checkpoint.

The recovery system discovers an unfinished step and executes it again.

The agent calls the payment tool a second time.

The customer receives another $100.

From the workflow engine's perspective, nothing particularly strange happened. The step wasn't recorded as completed, so it had to be retried.

But from the business's perspective, we just lost $100.

This is one of the most important problems to solve when moving AI agents from prototypes into production:

Retrying an AI agent is easy. Retrying it without duplicating its side effects is the hard part.

And it isn't limited to payments.

AI agents create orders, send emails, update CRM records, publish documents, modify infrastructure, and call external APIs. In all these cases, successfully completing a workflow is only half the problem.

We also need to know what happens when the same work is executed more than once.

Distributed systems engineers know this problem as the distinction between at-least-once execution and exactly-once effects.

For AI agents, the distinction becomes especially important because recovery may repeat model calls, tool invocations, or entire sequences of actions.

I wanted to explore this problem experimentally using AGBoss, an open-source TypeScript framework I'm developing on top of LangGraph.

The result is Distributed Agent Recovery, a reproducible showcase built for the Hacktoberfest Ghost in the Machine challenge.

We'll freeze a worker halfway through a task, let another worker take over, and then bring the original worker back to life.

The question is simple:

Can a zombie worker corrupt an operation that another worker has already completed?

At-Least-Once Execution Is Not Exactly-Once Effects

Consider a distributed workflow that executes steps and persists checkpoints.

If a process crashes between performing an external operation and committing the next checkpoint, the runtime may not know whether the operation succeeded.

From its perspective, the step is unfinished.

So recovery may execute it again.

This is a normal consequence of at-least-once execution.

The same execute() function might run twice. But that doesn't necessarily mean the underlying business operation must happen twice.

For example, if a payment provider accepts the same idempotency key on both requests and atomically deduplicates them, the second request can return the original result without processing another payment.

There are four important concepts here:

• At-least-once execution: A step may execute multiple times after failures.
• Exactly-once effect: A logical external operation changes external state once.
• Idempotency: Repeating the same operation doesn't produce an additional effect.
• Fencing: A worker that has lost ownership cannot commit stale writes.

The last two mechanisms solve different problems.

Idempotency protects against repeated operations.

Fencing protects against stale workers that continue running after their work has been reassigned.

In practice, we often need both.

Why an Idempotency Key Alone Isn't Enough

Suppose two workers try to perform the same operation concurrently.

Both check whether payment-123 has already been processed.

Both see that it hasn't.

Both send the payment.

If deduplication isn't enforced atomically with the operation, the presence of an idempotency key doesn't prevent anything.

Now consider a different race.

Worker A begins a long-running action. Its lease expires. Worker B takes ownership, completes the work, and commits a newer state.

Then Worker A wakes up and attempts to write its old result.

Even if we use stable idempotency keys, we still need to determine whether Worker A is authorized to write.

That's what fencing is for.

AGBoss provides deterministic step idempotency keys, execution leases, checkpoint recovery, and protection against stale checkpoint writes.

The external application must complement those mechanisms with atomic deduplication and fencing at the side-effect boundary.

The goal of this project is to demonstrate how these mechanisms work together under actual process failures — not merely describe them in documentation.

The Experiment: A Distributed AI Release Factory

For the showcase, I built a small AI-assisted software release pipeline.

It prepares a release containing three modules:

• auth-service
• payment-engine
• telemetry-collector

A coordinator decomposes the release into independent tasks and distributes them across multiple AGBoss instances.

Each child workflow prepares module metadata, stages an artifact, and records the operation in shared storage.

After the child workflows finish, the coordinator runs an integration smoke gate and, if verification passes, promotes the release.

If verification fails, an AI agent analyzes the diagnostic information and proposes a corrected release plan.

We deliberately inject two different failures.

Failure 1 — Partial commit: All three artifacts are staged, but the final integration check fails. We need to compensate the already completed operations.

Failure 2 — Zombie worker: A worker freezes halfway through a task. Another worker takes over and finishes it. The original worker later resumes and attempts a stale write.

The first scenario demonstrates Saga compensation.

The second demonstrates checkpoint recovery, idempotency, and lease fencing.

They solve different problems, and it's important not to confuse them.

Demo

The demo runs a distributed AI release coordinator and two independent worker processes.

It deliberately fails an integration gate, performs Saga compensation, invokes a real LLM recovery agent, and launches a corrected second attempt.

During the second attempt, one worker is frozen with a real operating-system SIGSTOP signal.

After its execution lease expires, another worker recovers the exact same child execution from its checkpoint.

Then the frozen worker receives SIGCONT.

The zombie wakes up and attempts to commit its stale result.

The application ledger rejects its external write. AGBoss rejects its stale checkpoint.

The replacement worker's result remains intact.

What Actually Happened

In the recorded live experiment:

• The coordinator started two independent worker processes.
• The initial release attempt staged three artifacts.
• The smoke gate failed because a required compatibility flag was missing.
• Saga compensation removed all three failed-attempt staging artifacts.
• The LLM recovery agent inspected the failure and produced corrected build flags.
• During Attempt 2, the payment-engine worker froze after completing Step 1.
• Another worker took over after the real 30-second lease expired.
• The replacement resumed the same child execution under leaseEpoch = 2.
• The original worker woke up holding stale leaseEpoch = 1.
• Both the external stale write and the stale checkpoint write were rejected.

The central recovery used this child execution ID:

f9907abf-19fd-498b-8397-49152e25345f

The ID remained unchanged across takeover.

That distinction matters: the replacement worker didn't start a separate task with the same input. It recovered the original execution from its committed checkpoint.

Verified Results

The showcase includes an independent invariant verifier that examines the filesystem, the application's SQLite ledger, and AGBoss's internal execution tables.

Verification	Result
Independent worker processes	2
Original lease epoch	1
Lease epoch after takeover	2
Child execution ID	Preserved
Checkpoints in recovered execution	6
Competing checkpoint branches	0
Orphaned child queue entries	0
Active artifacts from failed attempt	0
Successfully promoted modules	3
Verified invariants	8/8

The checkpoint result is particularly interesting.

All six checkpoints for the recovered child execution form a single linear chain through their parent_checkpoint_id relationships.

There is no competing checkpoint branch committed by the zombie.

Meanwhile, the execution lease table confirms that ownership advanced to epoch 2 and the replacement worker became the valid owner.

We're not merely verifying that the workflow eventually completed.

We're verifying that recovery didn't leave competing persisted states or duplicated committed artifacts.

Recorded evidence:

• Live real-clock execution report
• Runtime log with recovery events
• GitHub Actions verification run

What About a Real Container Crash?

We use SIGSTOP and SIGCONT to demonstrate the zombie-worker problem because the original worker needs to come back.

But we also tested a more conventional failure: a Docker worker container terminating abruptly in the middle of a step.

In this case, the original worker never returns.

The surviving worker must wait for lease expiration, reclaim the execution, and continue from the last checkpoint.

This scenario is covered by a separate GitHub Actions CI job.

The repository also runs the recovery application in three environments:

• Independent Node.js processes sharing a local SQLite database.
• Docker Compose containers with a shared volume, including hard worker-crash testing.
• A single-node Kubernetes kind cluster with two worker pods and a coordinator Job.

All three CI verification jobs passed in the recorded GitHub Actions run.

These are separate operating-system processes and containers performing real recovery — not predetermined log sequences.

Code

The entire application is open source and available on GitHub:

agbossjs/distributed-agent-recovery

The AGBoss framework itself is also open source:

agbossjs/agboss

Run the Demo Locally

You'll need Node.js 22.13+ and npm.

The default tests and demos use a deterministic offline model transport, so no LLM API key is required.

The commands below are documented in the showcase README:

git clone https://github.com/agbossjs/distributed-agent-recovery.git
cd distributed-agent-recovery

npm ci
npm run check
npm test
npm run demo:cluster
npm run verify

To reproduce the real-clock experiment with an actual OS process freeze:

npm run build
node dist/src/demo.js --cluster --real-clock

To run with live OpenRouter inference:

OPENROUTER_API_KEY="..." \
  node dist/src/demo.js --cluster --live --real-clock

To run the containerized version:

npm run demo:compose

The repository includes the workflow implementation, fault-injection scenarios, Docker and Kubernetes configurations, automated tests, and recorded execution evidence.

How I Built It

Open-Source AI Stack

The project combines open-source orchestration with a live LLM recovery agent.

Its main components are:

• AGBoss — the open-source TypeScript framework providing declarative agent workflows, dependency injection, static validation, leases, checkpoint recovery, and execution coordination.
• LangGraph — the underlying workflow execution and checkpointing foundation.
• OpenRouter — the inference gateway used for the live LLM experiment.
• openai/gpt-4o-mini — the model used by the recovery-planning agent.
• SQLite WAL — the shared local database used for execution coordination and the application's transactional artifact ledger.
• Docker Compose and Kubernetes kind — deployment environments used to verify recovery across isolated workers.

The AI model itself is accessed through an external API; this showcase does not claim to run an open-weight model locally.

The open-source contribution is the agent framework, its inspectable execution machinery, and the reproducible fault-injection application built around them.

1. Declarative Distributed Workflows

AGBoss uses a declarative, typed workflow DSL.

Rather than manually assigning tasks to particular workers, the coordinator declares a parallel child-workflow operation with ChildPlacement.AnyInstance.

Here is an excerpt from ReleaseCoordinatorWorkflow:

start(ReleaseCoordinatorWorkflowStart)
  .returns(ReleaseRequestDto)
  .toAction(DecomposeReleaseTasksAction)
  .returns(listOf(ModuleTaskDto))
  .toParallelWorkflows(ModuleWorkerWorkflow, {
    maxConcurrency: 4,
    maxItems: 8,
    placement: ChildPlacement.AnyInstance,
  })
  .join()
  .returns(
    listOf(
      oneOf(StagedModuleArtifactDto, ModuleFailedDto)
    )
  )
  .toAction(LockReleaseCandidateAction)

The coordinator doesn't need to know whether Worker A or Worker B will execute a particular module.

AGBoss coordinates child executions through shared repositories, leases, and checkpoints.

The DSL makes input/output contracts and concurrency constraints explicit. The agboss check command statically validates workflow composition, DTO transitions, dependency injection bindings, and other declared constraints before invoking an LLM.

Static verification doesn't replace integration testing or fault injection. But it catches composition errors independently of distributed runtime failures.

Now let's break the application.

2. Act I — The Release Fails After the Artifacts Are Already Staged

The first release attempt distributes three child workflows across two independent worker processes.

All three modules prepare their staging artifacts successfully.

Then the smoke gate runs.

For this experiment, we modeled an integration compatibility failure: payment-engine requires an ABI compatibility flag that wasn't included in the initial release configuration.

The initial flags are --opt-level=3 --deterministic.

In the showcase, these represent an optimized, reproducible build configuration.

The missing flag is --abi-compat=v2, representing a requirement that payment-engine must be compatible with version 2 of a defined application binary interface.

These are illustrative flags for our release-planning scenario, not commands sent to an actual compiler.

The smoke-gate failure is intentionally injected so the recovery path can be exercised deterministically.

The implementation is in VerifyReleaseSmokeGateAction.

The action records a diagnostic message and throws ReleaseGateFailedError.

The problem is that all three staging artifacts already exist.

If the workflow simply terminates, it leaves external state partially modified.

Saga Compensation

AGBoss lets actions declare compensating operations through compensateWith.

In our showcase, LockReleaseCandidateAction declares the following relationship (abbreviated):

@Action({
  name: "lock-release-candidate-action",
  // Other metadata omitted
  compensateWith: CompensateReleaseCandidateAction,
})

The coordinator also declares an explicit error-handling branch.

From ReleaseCoordinatorWorkflow:

.toAction(VerifyReleaseSmokeGateAction)
.onError(
  ReleaseGateFailedError,
  toAction(CompensateReleaseCandidateAction)
    .toAction(SummarizeCompensatedAttemptAction)
    .returns(CompensatedReleaseCandidateDto)
    .toAgent(ReleaseRecoveryPlannerAgent)
    .returns(RecoveryPlanDto)
    .toFinish(ReleaseRecoveryPlannedWorkflowFinish),
)

The compensation removes the staging files, marks the corresponding SQLite ledger records as COMPENSATED, and releases the candidate lock.

There's an important distinction here.

AGBoss orchestrates compensation, but it cannot automatically undo arbitrary external operations.

The application is responsible for physically deleting files, reversing supported business operations, and making those operations safe to retry.

Our verifier therefore checks the actual filesystem and ledger state, not merely whether a compensation handler was called.

After the failed attempt, all three staging artifacts are compensated and no active staging files from Attempt 1 remain.

The system can now prepare a corrected attempt.

3. Where Does the AI Come In?

After compensation, the coordinator invokes ReleaseRecoveryPlannerAgent.

The agent is implemented in release-recovery-planner.agent.ts.

It uses InspectFailedAttemptTool to read the actual diagnostic and compensation status from SQLite before proposing a new plan.

In our recorded live run, the agent used openai/gpt-4o-mini through OpenRouter.

It made two model calls, invoked the inspection tool, and produced a typed RecoveryPlanDto containing the corrected configuration:

--opt-level=3 --deterministic --abi-compat=v2

The recorded inference used 1,705 input tokens and 178 output tokens.

This illustrates an architectural boundary I consider essential.

The LLM is responsible for application-level reasoning. The workflow infrastructure is responsible for consistency.

Diagnosing an integration failure and proposing a correction is a reasonable task for an AI agent.

Managing lease ownership, deduplication, checkpoint consistency, and transactional writes is not something we should delegate to prompt instructions.

4. Act II — The Worker Dies. Long Live the Worker.

The second release attempt starts with the corrected configuration.

One worker claims the payment-engine child workflow.

It completes the first step, PrepareModuleCandidateAction, and saves a checkpoint.

Then it enters StageModuleArtifactAction.

At this point, the fault-injection scenario freezes the entire OS process using SIGSTOP.

The freeze logic is implemented in StageModuleArtifactAction.

This is a real operating-system signal, not a simulated JavaScript exception.

The process stops executing.

It doesn't finish its action, renew its lease, or commit another checkpoint.

But its memory remains intact.

Unlike SIGKILL, SIGSTOP lets us later resume the process using SIGCONT.

The original worker acquired its execution lease with epoch 1.

In this experiment, the lease has a 30-second TTL.

When the worker stops renewing its lease and the lease expires, another AGBoss instance can reclaim the orphaned execution.

But notice what happens next:

The replacement worker doesn't start a different task with the same input. It resumes the exact same child execution from its last committed checkpoint.

In the recorded experiment, the execution ID was:

f9907abf-19fd-498b-8397-49152e25345f

After takeover:

• The executionId remains unchanged.
• leaseEpoch advances from 1 to 2.
• The replacement loads the previous checkpoint.
• Step 1 isn't repeated.
• Step 2 completes on the replacement worker.
• The step's deterministic idempotency key is preserved.

The runtime emits the event execution.recovered.

The replacement worker successfully commits the staged artifact.

So far, the recovery worked.

But the original process is still frozen.

And now comes the interesting part.

5. Act III — The Zombie Worker Returns

The fault-injection scenario sends SIGCONT to the frozen process.

It wakes up.

From its perspective, it is still executing StageModuleArtifactAction.

It still has its original lease information in memory: leaseEpoch = 1.

Unfortunately for the zombie, another worker has already completed the task under leaseEpoch = 2.

What happens if the old worker attempts to commit?

This is the central test.

Defense 1: Stable Step Idempotency Keys

AGBoss provides a deterministic key for an individual workflow step.

In StageModuleArtifactAction, the action obtains the key from ExecutionContext.

The following is an excerpt:

const executionId = this.executionContext.executionId;
const idempotencyKey =
  this.executionContext.step.idempotencyKey;

For the recovered child execution, the resulting key was:

f9907abf-19fd-498b-8397-49152e25345f:stage-module-artifact-action:1

The key identifies the execution, step, and visit number.

Importantly, it remains the same after recovery.

This gives the external ledger a stable identity for deduplicating repeated attempts.

But idempotency doesn't answer whether an old worker is still authorized to write.

We need another defense.

Defense 2: Lease Fencing at the External Side-Effect Boundary

An AGBoss @Action can perform external I/O inside its execute() method.

That happens before the workflow runtime commits the step's checkpoint.

Therefore, checkpoint fencing alone cannot protect arbitrary databases, payment systems, or external APIs.

Our application captures the worker's current lease information when the action begins.

See StageModuleArtifactAction, which captures the lease and forwards the captured owner and epoch to the ledger.

Before committing an artifact, the ledger validates the fencing token against current ownership.

The following is a simplified excerpt from LeaseFencingService.checkAndAdvanceFence():

if (
  params.capturedLeaseEpoch < activeLeaseEpoch ||
  (
    activeHolder !== params.capturedInstanceId &&
    params.capturedLeaseEpoch <= activeLeaseEpoch
  )
) {
  // Record ZOMBIE_WRITE_REJECTED
  return new StaleLeaseFenceError(/* ... */);
}

The actual implementation performs the check as part of the application's SQLite transaction.

When the zombie resumes, its captured epoch is 1, but the current epoch is 2.

The ledger rejects its operation and records ZOMBIE_WRITE_REJECTED.

The stale worker cannot overwrite the committed artifact.

This is application-level protection of the external side effect.

Defense 3: Fencing the Workflow Checkpoint

There's one more thing to protect.

Even after the external write is rejected, the old worker might attempt to persist a workflow state transition.

AGBoss uses LeasedCheckpointSaver to validate execution ownership at the checkpoint boundary.

The zombie can no longer renew its old lease.

As a result, its stale checkpoint write is rejected with ExecutionLeaseLostError, carrying the code execution.lease-lost.

We now have two distinct layers of protection:

Application-level fencing prevents stale writes to external business state.

AGBoss runtime fencing prevents stale writes to the workflow's checkpoint state.

Both matter.

Without application-level protection, a stale worker might mutate an external service before the runtime detects that it has lost its lease.

Without runtime-level protection, a stale worker might corrupt the workflow's persisted execution history.

This is why reliable recovery requires more than simply retrying failed tasks.

6. Independent Verification Against Runtime State

It's easy to print ZOMBIE_WRITE_REJECTED and declare success.

But a log message alone doesn't prove that the system remained consistent.

That's why the showcase includes an independent invariant verifier.

It inspects the application's SQLite ledger, filesystem artifacts, and AGBoss's internal cluster tables.

The relevant implementation is ReleaseInvariantVerifierService, which uses ClusterSnapshotRepository.

We examine three runtime tables:

• execution_leases — who owns an execution and how its lease epoch changed.
• checkpoints — the persisted state history of the workflow.
• child_execution_queue — outstanding child executions.

The verifier confirms the transition to epoch 2, the preserved execution identity, a linear checkpoint chain, no competing zombie branch, and no orphaned child tasks.

Directly reading internal AGBoss tables is a diagnostic technique tied to the tested framework version, not a recommended public application integration API.

7. What AGBoss Guarantees — and What It Doesn't

In distributed systems, overclaiming guarantees can be just as dangerous as underengineering failure handling.

AGBoss provides several building blocks for resilient AI workflows:

• Declarative workflow composition and static verification.
• Distributed child execution and shared queue coordination.
• Execution leases and recovery after ownership loss.
• Checkpoint continuation and stale checkpoint fencing.
• Deterministic step idempotency keys.
• Saga compensation for supported reversible operations.

However, AGBoss cannot automatically guarantee exactly-once side effects for arbitrary external systems.

If an action sends an HTTP request to an API without idempotency support, a checkpoint saver cannot retroactively make that API operation transactional.

Likewise, protecting internal workflow state doesn't automatically fence external writes.

The external system must enforce its own atomicity and deduplication contract.

In our showcase, the application-level SQLite ledger provides those mechanisms through transactional checks, unique constraints, and monotonic fencing tokens.

The accurate guarantee is:

At-least-once execution with effectively exactly-once external effects, where the external system enforces atomic idempotency and fencing.

And one more distinction matters: this does not imply exactly-once LLM inference.

A model invocation may also be repeated after a crash if the runtime hasn't durably recorded its result.

Compensation Is Not Crash Recovery

Saga compensation and lease-based crash recovery solve different problems.

In Attempt 1, the workflow receives a managed ReleaseGateFailedError and executes compensating operations.

In Attempt 2, a worker loses its lease and another worker resumes the unfinished execution.

If a container is hard-killed, it cannot immediately run compensation inside the terminated process. Recovery must first occur elsewhere.

Compensation itself also requires operations with meaningful reversal semantics.

Deleting a staged, unpublished artifact is straightforward. Undoing a payment, retracting an email, or reversing an irreversible third-party operation may require a fundamentally different business process.

Fixed Workers vs. Autoscaling

AGBoss separates work orchestration from infrastructure scaling.

ChildPlacement.AnyInstance allows eligible worker instances connected to shared execution repositories to participate in distributed execution.

Our showcase uses a fixed worker pool because the experiment is about recovery, idempotency, and fencing — not autoscaler behavior.

Dynamic Kubernetes scaling based on queue depth, resource usage, or other metrics is a separate infrastructure concern.

Production Kubernetes support, including distributed storage and autoscaling integration, is tracked in the AGBoss Kubernetes roadmap epic.

Single-Host SQLite vs. Multi-Node High Availability

The showcase uses shared SQLite WAL storage across independent processes and containers on one host.

This provides a reproducible environment without requiring PostgreSQL or a separate database service.

But it is not a demonstration of multi-host high availability.

A production multi-node Kubernetes deployment needs a suitable network-backed transactional storage layer and corresponding repository implementations.

The execution model can support newly available workers, but the storage and deployment guarantees must be satisfied independently.

Toward a Public Fencing API

The showcase also highlights an API-design opportunity.

AGBoss already manages execution ownership and checkpoint fencing. Applications currently need additional integration code to carry lease ownership into external side-effect transactions.

A type-safe, storage-independent public fencing contract could make this easier.

That doesn't mean AGBoss should pretend to enforce transactions inside arbitrary third-party systems.

It means giving developers a clearer, safer way to integrate external systems with execution ownership.

This is a direction for the framework's future API, not a guarantee claimed by the current release.

Why Does Open Innovation Matter?

There's a tremendous amount of attention in the AI-agent ecosystem devoted to models, prompts, tool calling, and evaluation.

All of that matters.

But once agents start taking real actions, we encounter problems distributed systems engineers have dealt with for decades: retries, duplicate delivery, stale ownership, partial commits, idempotency, checkpoint consistency, and recovery after process failure.

These problems cannot be solved by better prompt engineering alone.

They require explicit execution contracts.

I'm building AGBoss to combine typed agent composition, dependency injection, static workflow verification, and resilient execution mechanisms in a single TypeScript framework built on LangGraph.

Open source is especially important here because reliability guarantees need to be inspectable.

A closed black-box service might promise durable agent execution, but understanding exactly what happens between an external side effect and a checkpoint can be difficult.

With an open runtime, we can inspect the execution contracts, build reproducible fault-injection tests, and independently verify the persisted state.

We can freeze a worker, inspect the lease table, reproduce a stale write, and verify whether the system truly rejects it.

We can also identify missing guarantees and improve the framework through public API discussions, tests, and contributions.

The use of a hosted LLM for recovery planning doesn't change that principle. The model provides application-level reasoning; the infrastructure responsible for consistency remains open and testable.

The open-source ecosystem also makes it possible to build on established LangGraph execution concepts rather than reinvent every component from scratch.

Open innovation matters because reliability shouldn't require trusting a marketing claim. It should be something developers can reproduce and verify.

The Spirit of It

The most unsettling moment wasn't when the worker stopped.

It was when it came back.

Another worker had already recovered the execution, completed the unfinished step, and committed the result.

The original process had no idea.

It simply resumed from the exact instruction where the operating system had frozen it.

For a moment, two workers had been executing the same logical job.

But only one still had the right to change the result.

The zombie tried to write.

The ledger rejected it.

Then the runtime rejected its stale checkpoint.

The workflow survived.

The zombie didn't get the last word.

An AI agent may execute twice. That doesn't mean it should change the world twice.

---

Source code and documentation

• AGBoss — GitHub
• AGBoss — Documentation
• Distributed Agent Recovery — Showcase
• Recorded Live Experiment
• Kubernetes & Autoscaling Roadmap