Canary Idempotent Webhooks With Duplicate, Timeout, and Reorder Gates

A payment webhook once ack=timeout -> retry -> reorder -> duplicate ack can look healthy in logs while applying the same intent twice. The common handler invariant — “return 200 fast, then process asynchronously” — is not sufficient unless a stronger invariant holds:

Invariant: for one intent_id, at most one externally visible effect may commit, no matter how many deliveries, retries, or reorderings occur.

This article gives a small canary workflow you can run before scaling webhook traffic. It uses a free model option and free server option from MonkeyCode only as a low-friction place to generate adversarial traces and execute a disposable harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming specific model names, quotas, hardware, duration, permanence, or benchmark results; treat free access as an operator-supplied availability option, not a dependency of the method.

Assumptions

• One producer emits business intents with stable intent_id; delivery to the consumer is at-least-once.
• The consumer can commit an externally visible effect: charge, email, provisioning call, ledger write, or ticket transition.
• Transport may duplicate, delay, reorder, and lose acknowledgements. It does not forge new intent_id values.
• The effect provider may be idempotent, but the handler must not assume that unless verified by a contract test.
• Clocks are advisory only; correctness must come from state transitions, not timestamps.

Sequence model and counterexample

producer          broker           consumer           effect_provider
   |  intent A       |                 |                    |
   |---------------->|  deliver A d1   |                    |
   |                 |---------------->| begin A            |
   |                 |                 |------------------->| apply A
   |                 |   ack d1 lost   |                    |
   |                 |<----------------| timeout            |
   |                 |  deliver A d2   |                    |
   |                 |---------------->| begin A?           |

A violating order is simple: begin(A) -> apply(A) -> ack_lost -> redeliver(A) -> begin(A) again -> apply(A) again. If begin is only “received HTTP 200”, d2 has no memory that A already reached a committed effect.

The design invariant is not “no duplicate delivery.” That is impossible under retries. The invariant is no duplicate committed effect.

Minimal simulator

This is intentionally small enough to run in CI or on an ephemeral free server. It models an effect commit as the dangerous boundary and injects duplicate, timeout, reorder, and crash-before-ack events.

from dataclasses import dataclass
from itertools import permutations

@dataclass(frozen=True)
class Event:
    kind: str
    intent: str
    delivery: int

EFFECTS_COMMITTED = []
DEDUP = {}

def handle(evt, state):
    key = (evt.intent,)
    record = state.setdefault(key, {"seen": set(), "committed": False})
    record["seen"].add(evt.delivery)

    if evt.kind == "deliver":
        # Unsafe handler: every delivery may commit.
        EFFECTS_COMMITTED.append(evt.intent)
        return "acked"

    if evt.kind == "deliver_idempotent":
        if not record["committed"]:
            EFFECTS_COMMITTED.append(evt.intent)
            record["committed"] = True
        return "acked"

    if evt.kind == "ack_lost":
        return "retry_scheduled"

    raise ValueError(evt.kind)

def effects_for(intent):
    return sum(1 for x in EFFECTS_COMMITTED if x == intent)

def check(trace, handler_kind):
    EFFECTS_COMMITTED.clear()
    state = {}
    for evt in trace:
        handle(evt, state)
    assert effects_for("A") <= 1, f"invariant violated: {trace}"

base = [
    Event("deliver_idempotent", "A", 1),
    Event("ack_lost", "A", 1),
    Event("deliver_idempotent", "A", 2),
]

# Exhaust small permutations of a bounded trace. Denominator: 6 orders.
for p in permutations(base):
    check(list(p), "deliver_idempotent")

print("ok: bounded orders preserve at-most-one committed effect")

The unsafe version is the same trace with deliver instead of deliver_idempotent; it fails as soon as two delivery events are both executed. That is the point: retries are normal, so the commit gate must be durable before the effect, not after the HTTP response.

Failure classes to canary

Run the same intent through four gates before enabling the handler for all traffic:

Gate	Injected class	Must hold	Common failure
Duplicate	same intent_id, same payload, twice	one committed effect	unique constraint only after side effect
Ack-loss retry	commit succeeds, response times out	retry sees committed state	handler re-executes because ack was lost
Reorder	cancel arrives before create for same aggregate	reject or buffer by state machine	last-write-wins by arrival time
Crash window	process dies between intent record and effect call	recovery replays or compensates exactly one path	“in-flight” state has no owner

A practical workflow is: generate 20 candidate traces with a coding assistant, keep only traces that name the state before and after each event, execute them against the simulator, then promote survivors into provider contract tests. MonkeyCode's free model access can help brainstorm trace candidates, and its free server option can host the disposable runner if you do not want to touch shared CI; the correctness criteria still come from your invariant, not from the tool.

Testable properties

Use property-style assertions rather than fixed expected outputs:

1. At-most-one effect: count(committed_effects(intent_id)) <= 1.
2. Progress: if a delivery is accepted and dependencies are reachable, the intent reaches committed, rejected, or compensated within N attempts.
3. No orphan in-flight: every intent row has a terminal state or a live lease owner after simulated crash.
4. Order independence for terminal states: permuting duplicate and ack-loss events cannot change the final committed set.
5. Explicit denominator: report violations over all bounded permutations, not “we retried once in staging.”

Example acceptance rule: for 3 delivery events plus one ack-loss and one crash event, exhaust all 120 bounded permutations; zero traces may commit two effects; at least 95% must terminate without manual repair, and the remaining 5% must terminate as explicit needs_compensation, never silent duplicate.

Tradeoffs

Design choice	Latency	Throughput	Correctness	Cost/ops risk
Synchronous commit before 200	Higher	Lower	Strongest simple gate	Provider outage blocks ack
Durable inbox + worker	Medium	High	Good if state machine is strict	More moving parts
Rely on provider idempotency	Low	High	Only as good as provider contract	Hard to verify across vendors
Deduplicate in memory	Lowest	Highest	Fails on restart and multi-instance	Unsafe beyond demo

Limitations and who should not use this

Do not use a tiny permutation simulator as proof for unbounded queues, multi-region active-active writes, or effects that cannot be made atomic or compensable. It also will not save a handler whose external API has no idempotency key, no queryable outcome, and no rollback path. If the business effect is irreversible and non-partitionable, use a human approval gate or a different protocol, not more retries.

If you want a starting point, fork the simulator, replace EFFECTS_COMMITTED.append with a staging sandbox call, and run the four gates before any real webhook rollout. The useful habit is tool-independent: canary the failure classes before you trust the happy path.

Final counterexample question: which event order breaks your invariant — ack_lost before commit, reorder after commit, or crash between reserve and apply — and should the system reject, replay, or compensate that order?