Last week, a free-tier server rebooted at 02:47. The agent running on it had been accumulating context in memory, and when the process came back, the context was gone. Worse, a model call that had already triggered an external side-effect — opening a GitHub issue — ran again, creating a duplicate. The in-memory flag that was supposed to prevent that had melted with the process.
That event taught me a simple invariant: if your AI agent runs on any disposable server, every model call that can produce a side-effect must remain idempotent under process death. Not under a normal crash, not under a network partition — under a total loss of local state.
MonkeyCode's free model access and free server option give you exactly that kind of environment: a remote model endpoint and a compute node, both without SLAs. (Disclosure: This article was prepared as part of MonkeyCode's product outreach.) That combination is where the crash-only executor pattern turns a demo into something that behaves like a service.
The data flow
Here's the architecture I used, assuming the server can vanish at any instant:
- An orchestrator (on reliable infrastructure) places a task into a durable queue.
- The free server consumes the task and tries to acquire a lease in external storage (Redis, DynamoDB, even Postgres). The lease includes the task ID and an expiry.
- Before calling the model, the executor writes a side-effect record with status
pendingto the same external store. - It calls the model endpoint. The call itself is safely retryable — it just produces text.
- If the model result requires an external side-effect (e.g., creating an issue, sending an email), the executor applies that effect after the model call, then flips the record to
done. - If the server dies at any point, the lease expires. The orchestrator sees the expired lease, re-enqueues the same task ID, and a fresh executor picks it up. It checks the side-effect record:
donemeans skip,pendingmeans finish the finalization,absentmeans start from step 3.
That last check is the entire trick. The side-effect record lives outside the disposable server, so a reboot cannot erase it.
Minimal code sketch
I'm simplifying the real client for clarity. The key is that the state lives in an external key-value store, not in the process.
def run_with_lease(task_id, storage, model_call, apply_effect):
if not storage.acquire_lease(task_id, ttl=30):
return # someone else owns the task
status = storage.get_effect_status(task_id)
if status == 'done':
return
if status == 'absent':
storage.set_effect_status(task_id, 'pending')
result = model_call(task_id)
apply_effect(task_id, result)
storage.set_effect_status(task_id, 'done')
That's the heart of the pattern. The lease prevents two executors from working the same task simultaneously. The side-effect record makes the operation idempotent across process death.
Failure analysis
Let's walk through three concrete failure modes and check whether the invariant holds.
| Failure point | What happens | Invariant preserved? |
|---|---|---|
| Server dies before acquiring lease | Task stays in queue; orchestrator picks it up later | Yes — no side-effect attempted |
Server dies after pending, before model call |
Lease expires; new executor sees pending, re-calls model, applies effect |
Yes — at-most-once (the effect is applied only once) |
Server dies after applying effect, before marking done
|
Lease expires; new executor sees pending (not done), re-calls model, applies effect again |
No — duplicate side-effect |
The third row is the dangerous one. If you write the done marker after the effect, a crash in between creates a duplicate. The fix is to make the effect itself idempotent (e.g., unique constraint on the GitHub issue title) or to move the done marker into the same atomic transaction as the effect.
Tradeoffs
This pattern has real costs, and you should know them before adopting it.
| Approach | Pros | Cons |
|---|---|---|
| In-memory flag | Zero latency, no external calls | Lost on reboot, breaks idempotency |
| External lease + side-effect log | Survives crashes, enables safe retries | Adds latency and infrastructure complexity |
| Idempotent side-effects only (no record) | Simplest, no extra state | Requires each effect to be naturally idempotent, which is rare |
For a free server that can be reclaimed at any moment, the second row is the right baseline. The first row is why I hit the duplicate issue in the opening story.
How to validate the invariant
Don't trust the pattern — test it. Here's a minimal failure-injection harness that simulates a crash at every possible point:
class SimStorage:
def __init__(self):
self.leases = {}
self.effects = {}
def acquire_lease(self, task, ttl):
import time; now = time.time()
if task in self.leases and self.leases[task] > now:
return False
self.leases[task] = now + ttl
return True
def get_effect(self, task):
return self.effects.get(task, 'absent')
def set_effect(self, task, status):
self.effects[task] = status
def run_executor(storage, task, crash_point):
if not storage.acquire_lease(task, 30):
return 'skipped'
status = storage.get_effect(task)
if status == 'done':
return 'already-done'
if crash_point == 'before-model':
raise RuntimeError('crash before model')
storage.set_effect(task, 'pending')
result = f'result-{task}'
if crash_point == 'before-effect':
raise RuntimeError('crash before effect')
storage.effects[task] = 'applied'
if crash_point == 'after-effect':
raise RuntimeError('crash after effect')
storage.effects[task] = 'done'
return 'completed'
for crash_point in [None, 'before-model', 'before-effect', 'after-effect']:
s = SimStorage()
try:
run_executor(s, 't1', crash_point)
except RuntimeError as e:
print(f'crash at {crash_point}: {e}')
s.leases['t1'] = 0
result = run_executor(s, 't1', None)
print(f' after retry with crash at {crash_point}: status={result}, effects={s.effects}')
If you run this, you'll see that after-effect produces two applied entries in the log. That's the duplicate you need to design around.
What I would change next
The honest answer: I would not run this pattern without making the side-effect itself idempotent. The lease gives you at-most-once for the executor, but exactly-once for the side-effect requires either an atomic finalizer or a unique constraint in the target system. And I would add a lease-renewal loop so long model calls don't drop their lease mid-request.
Here's the counterexample question for your design: if your server applies the side-effect and then crashes before marking done, and the lease expires, will your system reject the duplicate, replay the model call, or compensate with a new operation? Answer that question before you ship it.
Top comments (0)