Treat Tool Intent as a Deduped Outbox Before Planner Retries Double-Commit

Last Tuesday I sat with a sequence dump that looked successful until I counted the tickets twice. The planner had asked for one incident record, and the board showed two identical rows minutes apart. Have you ever trusted a tool-failed retry line without asking whether the side effect already landed? I had trusted that line, and that is exactly how this architecture review started for me.

The violating order was simple, and it did not need a clever adversary to appear in production. The worker posted the ticket, then the process vanished before the result reached the planner queue. The planner treated silence as failure, emitted the same step again, and a second worker committed. Does a missing ack mean the tool never ran, or only that nobody survived to say so?

The invariant the common retry loop drops

I want one planner step to produce at most one committed side effect, unless named compensation runs. That sentence is the invariant, and everything else in this review exists to protect or refute it. Most agent diagrams draw a happy arrow from model output to tool call to observation, then bolt on retries. Is a timeout a hard failure you may replay, or an unknown that still owns the side effect?

I am reviewing a two-queue tool bus, not narrating a named outage from a company I will not invent. The fixture below is a proposed discrete-event model, not a production service I operate. If a design cannot name the reject, replay, and compensate paths on paper, I will not trust its retry flag in a real agent.

Declared assumptions

I am assuming at-least-once queues, crash-stop workers, and a planner that re-emits a step after timeout. I am assuming external tools are not naturally idempotent, because ticket and mail APIs usually are not. I am assuming the planner may sit on a shared model endpoint, while the worker may die without a graceful drain. I am not assuming exactly-once brokers, sticky sessions, or a private cluster with reserved capacity.

Constraints I would not negotiate

I will not let the model transcript be the source of truth for external writes, because transcripts are not ledgers. I will not let a planner timeout delete an in-flight intent, because silence is not an abort. I will not accept a tool adapter that mints a new external request whenever the queue redelivers a payload. If those three constraints feel expensive, the double-ticket failure is the cheaper-looking alternative that I refuse.

Here is the constraint set I use when I review the bus:

1. Every planner step must mint a stable intent_id before any worker may touch the outside world.
2. The outbox, not the model message, is the only record that authorizes a side-effecting call.
3. A worker may execute an intent only after a compare-and-set from queued into executing.
4. A late result for a superseded generation must be rejected, not folded into the next observation.

Which of those four would you drop first under schedule pressure, and what event order would punish that drop?

Data flow of the tool bus

I want the planner, the outbox, the worker, and the tool API in separate failure domains. The model may hallucinate a repeated tool call, and the queue may redeliver, but the outbox still owns the write. The result queue carries observations back, and it must key them by intent_id plus generation, not by arrival time. If a result cannot name those keys, I treat it as noise and I do not advance the plan.

sequenceDiagram
    participant P as Planner
    participant O as Intent outbox
    participant W as Tool worker
    participant T as External API
    participant R as Result queue
    P->>O: put(intent_id, gen, tool, args)
    O->>W: deliver at-least-once
    W->>O: CAS queued to executing
    W->>T: call with Idempotency-Key
    Note over W,R: worker may die here
    W->>R: observe(intent_id, gen, result)
    R->>P: bind only if gen still current
    P->>O: timeout does not mint a new intent_id

Happy path, written as a protocol rather than a vibe:

1. The planner assigns intent_id and generation zero, then inserts the row in queued state.
2. A worker claims the row with a compare-and-set, then calls the tool with that same key on the wire.
3. The worker writes the observation to the result queue, then marks the outbox row committed or failed.
4. The planner consumes the observation only when the generation still matches the live step.

If step two fails the compare-and-set, the second worker must stop. Replay is a delivery fact, not a license to spend again. Compensation is a different protocol, and I will not pretend a second create is a compensate.

Failure domains I actually budget for

I split the design into four domains because they fail on different clocks. The planner domain fails by timeout, duplicate decode, or a model that repeats a tool call with new prose. The outbox domain fails by lost write, lost ack, or a crash between state transitions. The worker domain fails by preemption, slow tool I/O, or a redelivered message after a successful post. The tool domain fails by accepting a body twice when no idempotency header was sent.

The dangerous coupling is the planner treating worker silence as a negative observation. That coupling turns one crash into two commits, and it hides inside retry middleware that looks responsible. Do we reject the late result, replay the same intent, or enqueue a compensate? If the outbox still says executing, I replay nothing and I wait or I escalate.

I also budget for reordered results, because preempted workers come back at the worst time. A generation that the planner already abandoned must not bind the next step. That is not a logging problem. That is a binding rule, and the result consumer has to enforce it.

A minimal simulator you can run

I want a fixture that can print the double-commit without standing up Kafka. The model below is discrete-event Python, and it is intentionally small enough to argue about in review. It is not a performance test, and I am not attaching invented latency numbers to it. Run it locally, then change the event order until the invariant breaks or holds.

# proposed fixture: not a production bus, not a benchmark
from dataclasses import dataclass
from typing import Dict, List, Optional, Tuple

QUEUED, EXECUTING, COMMITTED, FAILED = "queued", "executing", "committed", "failed"

@dataclass
class Intent:
    intent_id: str
    gen: int
    tool: str
    args: dict
    state: str = QUEUED
    side_effects: int = 0

class Outbox:
    def __init__(self) -> None:
        self.rows: Dict[str, Intent] = {}

    def put(self, intent: Intent) -> None:
        if intent.intent_id in self.rows:
            return  # dedupe: same intent_id never mints a second row
        self.rows[intent.intent_id] = intent

    def claim(self, intent_id: str, gen: int) -> bool:
        row = self.rows.get(intent_id)
        if not row or row.gen != gen or row.state != QUEUED:
            return False
        row.state = EXECUTING
        return True

    def complete(self, intent_id: str, gen: int, ok: bool) -> None:
        row = self.rows[intent_id]
        if row.gen != gen:
            return  # stale generation: reject
        row.state = COMMITTED if ok else FAILED

class Tool:
    def __init__(self) -> None:
        self.seen: Dict[str, str] = {}
        self.commits = 0

    def call(self, idem_key: str, body: dict) -> str:
        if idem_key in self.seen:
            return self.seen[idem_key]
        self.commits += 1
        ticket = f"T-{self.commits}"
        self.seen[idem_key] = ticket
        return ticket

def run(events: List[Tuple[str, dict]]) -> Dict[str, int]:
    outbox, tool = Outbox(), Tool()
    planner_timeouts = 0
    rejected_stale = 0
    for name, payload in events:
        if name == "plan":
            outbox.put(Intent(**payload, state=QUEUED))
        elif name == "claim_and_call":
            iid, gen = payload["intent_id"], payload["gen"]
            if not outbox.claim(iid, gen):
                rejected_stale += 1
                continue
            ticket = tool.call(iid, outbox.rows[iid].args)
            outbox.rows[iid].side_effects += 1
            if payload.get("ack", True):
                outbox.complete(iid, gen, True)
                payload["ticket"] = ticket
        elif name == "planner_timeout_replay":
            planner_timeouts += 1
            row = outbox.rows[payload["intent_id"]]
            # illegal: minting a new intent_id on timeout
            if payload.get("mint_new_id"):
                new_id = row.intent_id + "-retry"
                outbox.put(Intent(new_id, 0, row.tool, row.args))
            # legal: redeliver the same intent_id
        elif name == "late_ack":
            iid, gen = payload["intent_id"], payload["gen"]
            if outbox.rows[iid].gen != gen:
                rejected_stale += 1
            else:
                outbox.complete(iid, gen, True)
    return {
        "external_commits": tool.commits,
        "planner_timeouts": planner_timeouts,
        "rejected_stale": rejected_stale,
        "outbox_rows": len(outbox.rows),
    }

BAD = [
    ("plan", {"intent_id": "step-7", "gen": 0, "tool": "create_ticket", "args": {"title": "db blip"}}),
    ("claim_and_call", {"intent_id": "step-7", "gen": 0, "ack": False}),
    ("planner_timeout_replay", {"intent_id": "step-7", "mint_new_id": True}),
    ("claim_and_call", {"intent_id": "step-7-retry", "gen": 0, "ack": True}),
    ("late_ack", {"intent_id": "step-7", "gen": 0}),
]

GOOD = [
    ("plan", {"intent_id": "step-7", "gen": 0, "tool": "create_ticket", "args": {"title": "db blip"}}),
    ("claim_and_call", {"intent_id": "step-7", "gen": 0, "ack": False}),
    ("planner_timeout_replay", {"intent_id": "step-7", "mint_new_id": False}),
    ("claim_and_call", {"intent_id": "step-7", "gen": 0, "ack": True}),  # claim should fail
    ("late_ack", {"intent_id": "step-7", "gen": 0}),
]

if __name__ == "__main__":
    print("bad ", run(BAD))
    print("good", run(GOOD))
    assert run(BAD)["external_commits"] == 2
    assert run(GOOD)["external_commits"] == 1

The acceptance rule is explicit, and I want it in the same file as the events. external_commits must stay one for a single planner step when the worker dies before ack. The bad trace mints a new identifier on timeout, and the tool cannot see that the two calls are the same intent. The good trace redelivers step-7, the second claim fails, and the late ack still binds generation zero.

Injected failures and how I would run them

I do not trust a green happy path when the interesting bugs live in reordering. The commands below assume the fixture is saved as tool_outbox_sim.py in the current directory. They do not measure tokens, dollars, or hosted throughput, because I will not invent those figures.

1. Run the paired traces and keep the assertion on external_commits as the gate, not a log skim.
2. Flip mint_new_id to true and confirm the fixture fails, because that is the common planner bug.
3. Drop the Idempotency-Key path in Tool.call and watch a legal redelivery become a second spend.
4. Insert a second late_ack with gen=0 after you have advanced gen to one, and demand a reject.

python tool_outbox_sim.py
python -c "from tool_outbox_sim import run, BAD, GOOD; print(run(BAD), run(GOOD))"

If you add a real HTTP tool, send the outbox intent_id as Idempotency-Key and treat any 2xx without that echo as a protocol miss. An empty success body is a separate discussion I will not reopen here. The property I care about is simpler: one live intent, one external commit, or a named compensate row.

Tradeoffs, with a denominator

I am comparing designs per planner step, not per day and not per model call, because those denominators hide duplicates. The table is a review aid. It is not a vendor scorecard.

Design	What it preserves	What it costs	When it still breaks
Retry on planner timeout, new tool payload	Almost nothing	Extra calls that look like diligence	Worker died after commit
At-least-once queue, no outbox row	Delivery, not intent	Duplicate spends under redelivery	Any crash between post and ack
Deduped outbox plus compare-and-set claim	At most one executor	An extra durable write before I/O	Tool ignores the idempotency key
Outbox plus compensating inverse	Liveness after a true stuck execute	You must write the inverse, and it can fail too	Compensate timeout with no idempotency

I would rather pay the extra durable write than explain two tickets to an on-call thread. Throughput can wait until the invariant holds under injected preemption. If your load model cannot name concurrent in-flight intents, you are not measuring the bus. You are measuring hope again.

What I would change next

I would split executing into executing and awaiting_ack, because those are different recovery actions. I would put a bounded wait on awaiting_ack, then move to a compensate intent with its own identifier derived from the original. I would fence result binding with the generation, and I would reject any observation that cannot quote both keys. I would not add more planner retries until those states exist in the outbox schema.

I would also stop letting the model invent tool arguments on replay. Replay must load the stored args from the outbox row, not from a freshly sampled completion. Otherwise the second call is not a retry. It is a different write that only rhymes with the first. Can your current agent runtime even show you the stored args without reading a chat log?

Limitations, and who should skip this

This review does not give you a hosted topology, a queue product, or an incident runbook. The simulator is single-threaded and discrete, so it will not catch disk-ordering bugs or clock skew across regions. Free shared processes are useful for forcing timeouts, and they are the wrong isolation boundary for customer-facing writes. If you need a hard latency SLO, dedicated workers, or regulated audit storage, do not use this fixture as a deployment plan.

Skip the outbox if your tools are truly pure reads and a duplicate call cannot spend money or create records. Skip it if you already have a transactional outbox in the same database as the agent state, and your workers already claim with fencing. Do not skip it if a model timeout currently mints a new tool payload. That path is the double-commit I opened with.

Which event order still breaks the invariant in your bus, and should the system reject, replay, or compensate? If the answer is only "retry the tool," I would send the design back before any planner scales up.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free models that can run this workflow. A free server option is enough to reproduce the setup.