Architecture Review: Map Four Failure Domains Before Agent Steps Land on a Free Server

Here is an event order that ends with a duplicate deployment. A worker claims step s1, applies the patch, and the substrate disappears before the acknowledgement is written. The scheduler sees an open intent, no heartbeat, and no result, so it re-queues s1 on a fresh worker. The fresh worker applies the same patch again, and both attempts report success to different observers.

Nothing in that story is exotic: no network partition required, no Byzantine node, no exotic clock. It only requires that the place where a step runs and the place where the record of that step lives are allowed to fail independently. The common implementation fails to preserve one invariant: every step has at most one observable effect, and every effect has a durable intent that precedes it.

This is an architecture review of the substrate that keeps tempting teams to skip that invariant. I keep seeing the same move: put agent execution on a free server option, wire it to free model access, and treat the whole thing as a development laptop that happens to be remote. That framing is the bug. So let me write down the constraints, the data flow, the failure domains, and what I would change next.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

1. What "free" actually buys you, and what it does not

The operator of the MonkeyCode project lists two availability claims that make this review worth writing: free model access, and a free server option, with a stated free allowance currently quoted at ten million tokens. I am treating both as a snapshot with an expiry date, not as an architectural property, and you should re-verify them before you depend on them. (MonkeyCode is described by its operator as an open-source project; I have not audited its internals for this review, so nothing below assumes anything about how it is built.)

What is an architectural property is this: a substrate you did not buy, did not size, and cannot page anyone about is a borrowed substrate. Borrowed substrates are fine. Borrowed substrates with local-state assumptions are not. So before you run a single step on one, declare your assumptions out loud, because every later table depends on them.

• A1 (revocation): the free server may stop, restart, or be reclaimed mid-step without notice.
• A2 (state): local disk and process memory on that server are caches, not records.
• A3 (clock): wall-clock time there is not comparable to your scheduler's clock; only sequence numbers you control are.

If A1 is false for your provider, you can delete half of this article. If A1 is true and you have not designed for it, you have already shipped the duplicate deploy above — you just have not observed it yet.

2. Four failure domains, one invariant

An agent step is not a function call; it is a small distributed transaction whose participants are a planner, a substrate, an external effect, and a ledger. Four domains fail independently, and each needs its own detection signal and its own recovery owner.

Failure domain	What actually fails	Detection signal	Recovery owner
Step identity	payload drifts between attempts (new prompt, new revision)	intent hash mismatch	scheduler — reject, do not retry
Effect	effect applied, acknowledgement lost	intent committed, no result row	ledger replay — dedupe by effect key
Clock	lease/TTL expires while the step is alive	monotonic gap vs. lease deadline	lease fencing by token, never by timestamp
Substrate	server vanishes mid-step	missing heartbeat and open intent	re-queue with the same step_id

The invariant that ties them together: intent before effect, effect keyed by step identity, replay keyed by the same identity. Note what it does not require: exactly-once delivery. Delivery stays at-least-once, which is the only thing a borrowed substrate can honestly promise. Exactly-once is achieved at the effect, not on the wire.

3. A minimal harness that injects all four

You cannot review what you cannot execute. The following is a ~60-line fixture I use to argue about this in design reviews, with the ledger deliberately placed outside the "server" so the two can fail apart. Standard library only; run it locally.

# harness.py — python harness.py
import hashlib, json, sqlite3


def key(payload):
    return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()


class Ledger:
    """Lives OUTSIDE the borrowed substrate. Never on the free server."""

    def __init__(self):
        self.db = sqlite3.connect(":memory:")
        self.db.executescript("""
            CREATE TABLE intent(step_id TEXT PRIMARY KEY, phash TEXT, state TEXT);
            CREATE TABLE effect(step_id TEXT PRIMARY KEY, ekey TEXT);
            CREATE TABLE side(skey TEXT PRIMARY KEY);   -- the external system, simulated
        """)

    def intend(self, step_id, payload):
        ph = key(payload)
        self.db.execute("INSERT OR IGNORE INTO intent VALUES (?,?,'intended')", (step_id, ph))
        stored = self.db.execute("SELECT phash FROM intent WHERE step_id=?", (step_id,)).fetchone()
        if stored[0] != ph:
            raise ValueError(f"payload drift on {step_id}: reject, do not retry")

    def commit(self, step_id, ekey):
        with self.db:
            first = self.db.execute("INSERT OR IGNORE INTO side VALUES (?)", (ekey,)).rowcount
            self.db.execute("INSERT OR IGNORE INTO effect VALUES (?,?)", (step_id, ekey))
            self.db.execute("UPDATE intent SET state='committed' WHERE step_id=?", (step_id,))
        return first   # 0 == this effect already existed

    def state(self, step_id):
        row = self.db.execute("SELECT state FROM intent WHERE step_id=?", (step_id,)).fetchone()
        return row[0] if row else None

    def effects(self):
        return self.db.execute("SELECT count(*) FROM side").fetchone()[0]


class Fault(Exception):
    pass


def execute(ledger, step_id, payload, fault=None):
    ledger.intend(step_id, payload)
    if ledger.state(step_id) == "committed":
        return "replayed"                                  # idempotent no-op
    if fault == "kill_before_commit":
        raise Fault(step_id)
    first = ledger.commit(step_id, key({"step": step_id, "payload": payload}))
    if fault == "kill_after_commit":
        raise Fault(step_id)                               # caller never learns it worked
    return "applied" if first else "deduped"


def trial(fault):
    ledger = Ledger()
    payload = {"cmd": "apply-patch", "rev": 7}
    try:
        execute(ledger, "s1", payload, fault)
    except Fault:
        pass                                               # substrate is gone
    replay = execute(ledger, "s1", payload)                # new worker, same ledger
    return replay, ledger.effects(), ledger.state("s1")


for fault in [None, "kill_before_commit", "kill_after_commit", "duplicate_delivery"]:
    replay, side, state = trial(fault)
    assert side == 1, f"{fault}: {side} side effects, invariant violated"
    print(f"{str(fault):20} replay={replay:9} effects={side} state={state}")
print("all trials: exactly one observable effect per step")

The four steps I would walk a reviewer through, in this order:

1. Pin identity before anything else. intend() refuses a retry whose payload hash moved. That is the payload-drift domain closed with one assertion, not with a convention.
2. Commit the intent transactionally with the effect key. commit() inserts into side and effect in one transaction, so "effect applied but unrecorded" cannot be observed from the ledger.
3. Inject the kill on both sides of the commit. kill_before_commit must yield zero effects until replay; kill_after_commit must yield exactly one, with the second attempt classified as replayed.
4. Inject duplicate delivery explicitly. Run the same step twice and assert deduped, because at-least-once delivery is what you actually get — and the harness should say so rather than hope.

Expected output, which is the whole argument in four lines:

None                 replay=replayed  effects=1 state=committed
kill_before_commit   replay=applied   effects=1 state=committed
kill_after_commit    replay=replayed  effects=1 state=committed
duplicate_delivery   replay=replayed  effects=1 state=committed
all trials: exactly one observable effect per step

4. Failure analysis: what each domain costs when you skip it

Skipped control	Observable symptom	Blast radius	Cheap fix
Intent hash	stale revision re-applied after a prompt edit	silent wrong state, no error	reject on hash mismatch
Effect key	duplicate PR comments, double charges, doubled file edits	user-visible, hard to unwind	effect key = (step_id, rev)
Lease fencing	two workers own the same step after a long GC pause	concurrent conflicting writes	fencing token, monotonic counter
Open-intent reaper	a step that committed but never acked is retried forever	unbounded retries, budget burn	replay returns replayed, not a new effect

The third row is the one that surprises people, because it looks like a clock problem and is not. Any timestamp-based lease on a substrate whose clock you do not control will eventually hand the same step to two workers. Fencing tokens cost you an integer and remove the domain entirely.

5. Tradeoffs I would sign off on

Property	Without a ledger	With intent + effect ledger
Latency per step	one write	two writes + one hash
Duplicate effects after revocation	unbounded	zero, by construction
Recovery complexity	manual reconciliation	replay with the same step_id
Cost of a borrowed substrate	hidden in incident time	visible as ledger writes
Failure to reason about	unknown	classified into four domains

The honest accounting: you pay roughly one extra durable write per step to make replay boring. On a borrowed substrate, boring is the feature. What you should not do is put the ledger on the same free server, because then A1 takes the record with the effect and no amount of idempotency saves you.

6. What I would change next

If this were my review, three things would move before anything else shipped.

1. Split the data flow explicitly. Planner and ledger on durable infrastructure; only the step executor and its model calls are allowed onto the free server. Draw that boundary in the diagram, not in a wiki sentence.
2. Add a chaos gate to CI. Run the harness with each fault class and fail the build on any trial where effects != 1. It takes seconds and it is the cheapest regression test you will ever own.
3. Fence leases with tokens, never timestamps. Then delete any code that compares a remote wall clock to a local deadline, because that comparison is the seed of every double-owner bug.

7. Limitations, and who should not use this

This design assumes you can place a durable ledger somewhere outside the borrowed substrate; if you cannot, stop and fix that first, because the rest is decoration. It also assumes effects are keyable — file writes and API calls with an idempotency key yes, an irreversible external action with no key no, and for those you want a human gate rather than a clever retry. And if your steps are short, synchronous, and run on infrastructure you own and size, the ledger is probably overhead you should skip.

I would also not use this to argue that a free server option is production infrastructure. It is a substrate for work whose record lives elsewhere, and the moment you forget that sentence you are back to the duplicate deploy in the first paragraph.

If you want a substrate to run the harness against rather than your own laptop, MonkeyCode's free model access plus its free server option are a reasonable place to start — just re-check the currently stated allowance and the server terms yourself before you write them into a design doc.

So, the question I would put to you in review: which event order breaks your invariant first — a revocation between commit and acknowledgement, a replayed step after a payload edit, or two workers fenced by the same expired timestamp? And for the one that breaks it, does your system reject the attempt, replay it, or compensate after the fact?

Comments

[Alex Shev]

Mapping failure domains before deployment is especially important on free infrastructure because availability, quotas, storage, and identity can fail independently. I’d turn the map into a recovery drill: simulate a provider outage, missing secret, exhausted quota, and corrupted state, then record the operator action and recovery time. Architecture becomes credible when the repair path is exercised.