Canary Agent Tool Calls With Duplicate Deliveries and an Idempotency Gate

The counterexample first

Here is an event order that a surprising number of agent runtimes produce under a perfectly ordinary network hiccup:

t1: agent -> toolservice: charge_customer(order=91, amount=40)
t2: toolservice: executes charge, txn=TXN-441
t3: toolservice -> agent: ACK lost (gateway timeout at the proxy)
t4: agent runtime: sees no response, retries tool call
t5: agent -> toolservice: charge_customer(order=91, amount=40)   <-- duplicate
t6: toolservice: executes charge AGAIN, txn=TXN-442
t7: agent: proceeds, reports success, customer charged twice

Nothing here is exotic. The timeout is normal. The retry is normal. The failure is the missing invariant:

Invariant I1: For any logical tool effect E identified by an idempotency key k, the observed side effects of E in the world occur at most once, regardless of how many times the call is delivered.

Most agent frameworks treat tool calls as chat-shaped messages: send, wait, retry on silence. That is a request/response mental model. But the moment a tool has a side effect — a charge, an email, a ticket, a deployment — the tool call is not a message. It is a commitment, and silence is ambiguous: the effect may or may not have happened. Retrying blindly under ambiguity is how you get TXN-442.

This article builds a small, runnable canary harness that injects duplicate deliveries, lost ACKs, and reordering into an agent tool-call loop, and gates the workflow on I1. I'll also show how to stand this harness up for free so cost is not an excuse for skipping it.

Declared assumptions

• A1: Tool calls have side effects that are not safely repeatable (payments, notifications, writes).
• A2: The network between agent runtime and tool service can drop, delay, duplicate, or reorder messages. This is the baseline condition, not an edge case.
• A3: The agent runtime (or its orchestrator) retries on timeout. We do not get to disable retries; we get to make them safe.
• A4: The tool service can persist a small amount of state (an idempotency ledger). If it cannot, the gate fails closed and the workflow is not eligible for automated tool execution.
• A5: We evaluate with a cheap model, not the production model. The invariant under test is a protocol property of the plumbing, not a reasoning property of the LLM, so model quality is irrelevant to the result. This is exactly what makes a free-tier model sufficient.

A minimal state model

Each logical tool call moves through four states:

            issue(k)
              |
              v
         [DECLARED] --deliver(k)--> [EXECUTED] --ack--> [COMMITTED]
              |                        ^   |
              |                        |   | ACK lost
              +----retry(k)------------+   v
                                   (dedupe on k: no re-execution)

The key insight: retry must target the same logical call k, and the executor must treat a second delivery of k as a lookup, not an execution. Duplicate delivery becomes a read of the ledger. That is the entire protocol.

The harness

We need three cheap components:

1. A model endpoint to drive the agent loop (decides which tool to call and produces arguments). Protocol behavior does not depend on model quality, so a free model is fine.
2. A server to host the tool service, the fault-injecting proxy, and the ledger. A small free instance is plenty; the workload is a few hundred requests.
3. The fault injector, which sits between agent and tool service and implements the hostile network from A2.

I run this on MonkeyCode, which offers free model access and a free server option — enough to host the proxy, ledger, and run the evaluation loop without touching production infrastructure. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Everything below also works on any free tier or localhost; the harness has no dependency on the platform.

Fault-injecting proxy (the artifact)

# fault_proxy.py — sits between agent runtime and tool service.
# Deterministic given a seed: same seed, same fault schedule, reproducible CI.
import random, time
from dataclasses import dataclass, field

@dataclass
class FaultSchedule:
    drop_ack_p: float = 0.15      # ACK lost -> client sees timeout -> retry
    duplicate_p: float = 0.10     # deliver the same request twice
    max_delay_ms: int = 250       # reorder via jitter
    seed: int = 7

class FaultProxy:
    def __init__(self, service, schedule: FaultSchedule):
        self.service = service
        self.s = schedule
        self.rng = random.Random(schedule.seed)
        self.events = []          # event log for property checking

    def call(self, key: str, op: str, args: dict):
        deliveries = 1 + (1 if self.rng.random() < self.s.duplicate_p else 0)
        result = None
        for i in range(deliveries):
            time.sleep(self.rng.randint(0, self.s.max_delay_ms) / 1000)
            result = self.service.execute(key, op, args)  # ledger dedupes on key
            self.events.append(("deliver", key, i))
        if self.rng.random() < self.s.drop_ack_p:
            self.events.append(("ack_lost", key))
            raise TimeoutError(f"ack lost for {key}")   # agent will retry
        return result

Tool service with an idempotency ledger

# tool_service.py — the executor side of the protocol.
class ToolService:
    def __init__(self):
        self.ledger = {}          # key -> result
        self.effects = []         # ground truth: real-world side effects

    def execute(self, key: str, op: str, args: dict):
        if key in self.ledger:
            return self.ledger[key]           # duplicate delivery -> lookup
        result = self._apply(op, args)        # side effect happens exactly here
        self.ledger[key] = result
        return result

    def _apply(self, op, args):
        self.effects.append((op, args))       # e.g., ("charge", {"order": 91})
        return {"status": "ok", "op": op}

Agent loop and the gate

# harness.py — run N episodes, then check properties.
import uuid

def agent_episode(proxy, order_id):
    key = f"charge:{order_id}"               # stable idempotency key per logical effect
    try:
        proxy.call(key, "charge", {"order": order_id, "amount": 40})
    except TimeoutError:
        proxy.call(key, "charge", {"order": order_id, "amount": 40})  # retry same key

def run_gate(n_episodes=200):
    service = ToolService()
    proxy = FaultProxy(service, FaultSchedule())
    for i in range(n_episodes):
        agent_episode(proxy, order_id=1000 + i)

    # Property check — the gate.
    charges_per_order = {}
    for op, args in service.effects:
        charges_per_order[args["order"]] = charges_per_order.get(args["order"], 0) + 1
    violations = {o: c for o, c in charges_per_order.items() if c != 1}

    print(f"episodes={n_episodes} "
          f"deliveries={sum(1 for e in proxy.events if e[0]=='deliver')} "
          f"acks_lost={sum(1 for e in proxy.events if e[0]=='ack_lost')} "
          f"violations={len(violations)}")
    return len(violations) == 0

if __name__ == "__main__":
    assert run_gate(), "I1 violated: duplicate side effects observed"

Run it. Then break it: change agent_episode so the retry uses a fresh key (f"charge:{order_id}:{uuid.uuid4()}") — the pattern several runtimes produce when they regenerate tool-call IDs on retry. The gate fails immediately, on roughly the drop_ack_p × n_episodes fraction of orders. That failing run is the value of the harness: it demonstrates the violating event order from the introduction, on demand, with a seed you can paste into a bug report.

Testable properties

The gate should assert more than I1:

Property	Statement	Checked from
I1 at-most-once	each idempotency key produces ≤ 1 real effect	service.effects vs ledger
I2 at-least-once	every intended effect eventually executes (retry completeness)	intended set vs service.effects
I3 determinism	same seed → same event log → same verdict	replay proxy.events
I4 no orphan intent	the agent never reports success for an effect that never executed	agent report vs ledger

I4 is the sneaky one. If the agent's final message says "payment processed" but the ledger shows the charge was dropped and never retried, you have a truthfulness failure in the workflow, not just a delivery failure. Gate on it explicitly.

Making the model participate

In the minimal version above, agent_episode is scripted. In the real harness, replace the script with an actual model-driven loop: the model receives a task ("charge the customer and confirm"), emits a tool call, and reacts to the timeout. This is where the free model access matters — you want hundreds of episodes across seeds and prompt variants, and you want to discover whether the model changes the arguments on retry (a third failure class: not duplicate delivery, but divergent intent). If the model rewrites amount=40 into amount=50 on the retry, no ledger can save you; the gate must also compare retry payloads against the original intent.

Failure classes to inject, in order

1. Lost ACKs (t3 in the counterexample) — forces the ambiguity window.
2. Duplicate deliveries — proxies and queues do this without being asked.
3. Reordering — delay jitter so the retry arrives before the original's effect is visible.
4. Crash between effect and ledger write — the two-phase window in execute; fix with an atomic write or a write-ahead record.
5. Divergent retry payload — model-side failure; requires intent comparison, not just keying.

Class 4 deserves emphasis: execute as written has a bug. If the process dies after _apply but before ledger[key] = result, the retry will execute again. The honest fix is to record intent first (status DECLARED), then execute, then mark EXECUTED — and treat a DECLARED key on retry as either "in flight, wait" or "recover and reconcile," never "run it again." Finding this in a canary run costs a seed. Finding it in production costs a refund queue.

Tradeoffs

Decision	Latency	Correctness	Operational cost
Retry blindly (no keys)	lowest happy-path	violates I1 under any ACK loss	chargebacks, manual cleanup
Stable key + ledger lookup	+1 read per call	I1 holds for classes 1–3	small state store
Declared/Executed two-phase ledger	+1 write per call	also survives class 4	recovery logic for stuck DECLARED keys
Exactly-once via distributed tx (2PC/XA)	highest	strongest, but tool APIs rarely participate	usually impossible with third-party APIs
Canary gate in CI, no runtime change	zero in prod	catches regressions before deploy	maintain seeds and fixtures

The pragmatic configuration for most agent systems: two-phase ledger for side-effecting tools, plus this canary gate running per-release with a fixed fault schedule and a randomized sweep.

Acceptance rule

Be explicit about the denominator. A release passes the gate if and only if, across 200 episodes × 5 failure classes × 3 seeds = 3,000 injected-fault episodes: zero I1 violations, zero I4 violations, and I3 replay matches on all seeds. One violation blocks the release. "Mostly idempotent" is not a property; it is a leak with a rate.

Limitations and who should not do this

• The simulator models the protocol, not your actual tool vendor's behavior. If the vendor's API deduplicates on their side with a 24-hour window, your ledger and theirs can disagree; verify against a sandbox before trusting the gate for that vendor.
• Free model access and a free server are sized for evaluation workloads like this one, not for load testing or production traffic. Do not point the harness at real payment endpoints, and do not assume free tiers persist or scale — pin the harness so it also runs locally in CI.
• If your tools are all read-only or naturally idempotent (search, fetch, pure compute), this entire apparatus is overhead. The gate earns its cost only where side effects are expensive to undo.

If you want a starting point, MonkeyCode's free model access and free server option are a convenient place to stand the harness up this afternoon; the code above carries over unchanged.

The counterexample question to take back to your system

Find the event order in your own runtime that breaks I1. Specifically: when the ACK is lost and the agent retries, does the retry carry the same idempotency key and the same payload as the original? If not, decide deliberately whether the system should reject the divergent retry (fail closed, surface to the user), replay the original recorded intent (fail safe), or compensate after the fact (issue a refund/undo and log the divergence). Any of the three is defensible. Silently executing a fresh side effect is not.