DEV Community: Robin

Make a MiniMax H3 Claim a Replay Invariant Before It Changes Your Model Queue

Robin — Fri, 14 Aug 2026 11:02:04 +0000

A platform team ran a model router. The router used a cheap primary model and an expensive fallback. A request that timed out would escalate. A request that returned malformed JSON would escalate too. The team wanted the cheapest reliable primary. That constraint created a fragile equilibrium.

One morning a model release started circulating. To keep the example concrete, call it MiniMax H3. The announcement claimed lower cost and stronger benchmarks. The team's chat filled with screenshots. An engineer opened a pull request. It swapped the primary model. The on-call engineer stopped the change. The release was a claim, not a migration order.

The team listed assumptions before touching the router.

Traffic is nonuniform.
Benchmark averages hide tail behavior.
A candidate can win on cost while losing on escalation rate.
A release announcement is not a benchmark run.

The team needed a cost-controlled baseline. They used MonkeyCode's free model access and a free server instance to host the comparator. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That setup kept the evaluation from spending production budget before a decision.

The core model was small. Each request traversed a path.

request -> router -> primary
router -> fallback on timeout or malformed output
comparator replays trace -> baseline and candidate

A comparator replayed the same request trace through a baseline and a candidate. The baseline used the team's current primary model, exposed through the free MonkeyCode tier. The candidate pointed at the external release. The comparator recorded four values per outcome: success, latency, token count, and error type.

import asyncio
from dataclasses import dataclass

@dataclass
class Request:
    id: str
    prompt: str
    max_tokens: int

@dataclass
class Response:
    text: str
    tokens: int
    latency_ms: int

@dataclass
class Outcome:
    model: str
    ok: bool
    latency_ms: int
    tokens: int
    error: str = ''

class ReplayComparator:
    def __init__(self, baseline, candidate):
        self.baseline = baseline
        self.candidate = candidate

    async def run(self, trace, inject):
        results = []
        for req in trace:
            for name, client in [('baseline', self.baseline), ('candidate', self.candidate)]:
                try:
                    if inject.get(name) == 'timeout':
                        await asyncio.sleep(0.05)
                        raise TimeoutError('injected timeout')
                    resp = await client.complete(req)
                    results.append(Outcome(name, True, resp.latency_ms, resp.tokens))
                except Exception as exc:
                    results.append(Outcome(name, False, 0, 0, type(exc).__name__))
        return results

The next step was property checking. The team did not compare benchmark scores. They compared invariants on the same trace. A candidate could pass only if it did not increase errors, did not blow the budget, and held tail latency under the router timeout.

async def test_properties(results, candidate_timeout_ms):
    base_errors = sum(1 for r in results if r.model == 'baseline' and not r.ok)
    cand_errors = sum(1 for r in results if r.model == 'candidate' and not r.ok)
    assert cand_errors <= base_errors, 'candidate errors exceed baseline'

    base_cost = sum(r.tokens for r in results if r.model == 'baseline')
    cand_cost = sum(r.tokens for r in results if r.model == 'candidate')
    assert cand_cost <= base_cost * 1.05, 'candidate cost exceeded 5% limit'

    cand_latencies = [r.latency_ms for r in results if r.model == 'candidate' and r.ok]
    tail_latency = sorted(cand_latencies)[int(len(cand_latencies) * 0.95)]
    assert tail_latency <= candidate_timeout_ms, 'candidate tail latency exceeds router timeout'

    return {'base_errors': base_errors, 'cand_errors': cand_errors,
            'base_cost': base_cost, 'cand_cost': cand_cost,
            'candidate_tail_latency_ms': tail_latency}

The team injected failure modes before accepting any result. Each failure class had a matching invariant.

Failure class	Injected behavior	Acceptance gate
Timeout	Sleep and raise TimeoutError	Candidate tail latency stays under router timeout
Duplicate delivery	Run the same request twice	Success result must come from one logical completion
Truncated output	Return fewer tokens	Valid JSON and minimum token count still hold
Rate limit	Raise an HTTP 429	Router must escalate without adding retries
Cost overrun	Force token growth	Candidate cost stays within 5% of baseline

The tradeoff table forced the team to state what they were optimizing.

Decision	Result	Risk
Replay on free baseline	No production spend, reproducible trace	Baseline may drift from live traffic
Replay on live samples	Real traffic distribution	Cost and privacy risk, harder to reproduce
Benchmark-score comparison	Fast, seductive	Hides tail latency and escalation behavior

The open part mattered. The team attached the comparator to version control. The free server meant another engineer could run the same trace without secret budget approvals. The external model remained an environment variable. That was the open-source spirit here. It was not a slogan. It was a testable artifact that separated a vendor claim from a routing decision.

The team did not migrate on the first day. They ran the trace, injected failures, and checked the invariant. They recorded the acceptance rule in the repository. If a future release changed the queue, the same comparator would judge it.

Which event order breaks the invariant? A request times out on the candidate. The router retries the primary. The fallback completes after the retry. The system now has two success stories for one logical request. It must either reject the duplicate, replay the topic, or compensate the fallback. If the new model only wins on that duplicated traffic, it has not won the invariant.

Make Escalation Rate an Invariant Before You Mix a Cheap Primary Model With an Expensive Fallback

Robin — Thu, 13 Aug 2026 03:07:35 +0000

Every time a cheap model drops — this week it's DeepSeek-V4-Pro-0813 in my feed, last month it was something else — the same architecture gets sketched on a whiteboard: route everything to the cheap one, and "occasionally" escalate to the expensive one (Grok 4.6, or whatever your premium endpoint is) when the cheap one struggles. Cheap and good, with a little premium on top. What could go wrong?

Here is an event order that did go wrong for a system I reviewed:

Cheap model degrades slightly (a silent provider-side change). Retryable, low-confidence responses rise from 2% to 6%.
The router's escalation rule is if confidence < threshold: escalate. Escalation traffic to the premium model triples.
Premium model rate-limits the account. Escalations start timing out.
The retry wrapper retries the escalation — which retries the premium call — which deepens the rate limit.
The monthly premium budget is consumed in 9 days. Finance asks why. Nobody can say which requests escalated or why.

The invariant the implementation failed to preserve: the escalation rate is a bounded, explainable system property, not an emergent side effect of a threshold. If you can't state your escalation budget and prove your router respects it under degradation, you don't have a routing strategy — you have a cost surprise on a timer.

Declared assumptions

"Cheap and good" is a claim to validate, not a fact. I have not benchmarked DeepSeek-V4-Pro-0813 or Grok 4.6; treat both names below as example endpoints. Your harness must verify quality before promotion.
Each request yields a primary response plus a scalar confidence/quality signal. Your signal may be a verifier, a critic model, or a heuristic — the protocol doesn't care which.
You have a premium budget expressed as a rate (e.g., ≤ 5% of requests may escalate sustained) and a cap (absolute count per window).
Escalations and retries share the same downstream rate limit — this is the coupling that creates the failure above.

The escalation protocol, as a state model

Treat each request as a state machine, not a function call:

PRIMARY_CALL ──ok, confident──▶ DONE
     │
     ├─ok, low confidence──▶ BUDGET_CHECK ──allowed──▶ ESCALATE ──▶ DONE(escalated)
     │                            │
     │                            └─denied──▶ DONE(degraded, logged, sampled for replay)
     │
     └─error──▶ RETRY_ONCE ──▶ (same states; never re-escalate a retried escalation)

Three rules make this convergent:

Budget check before escalation, not after. A token bucket gates escalations. Denied escalations degrade gracefully to the primary answer with a flag, so you can replay them later against the premium model offline.
Retries never re-enter escalation. A retried escalation that fails returns degraded, not another premium call. This breaks the retry ↔ rate-limit feedback loop.
Every escalation carries a reason code (low_confidence, verifier_reject, timeout_primary) written to an append-only log. If you can't explain your escalation distribution, you can't debug your bill.

Minimal simulator

This runs with the standard library only. It injects primary-model degradation and a premium rate limit, then checks whether your router preserves the escalation invariant:

import random
from dataclasses import dataclass, field

@dataclass
class Bucket:
    rate: float          # tokens per request-window
    cap: int
    tokens: float = 0.0
    def allow(self) -> bool:
        self.tokens = min(self.cap, self.tokens + self.rate)
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True
        return False

@dataclass
class Stats:
    n: int = 0
    escalated: int = 0
    degraded: int = 0
    premium_calls: int = 0
    reasons: dict = field(default_factory=dict)

def simulate(n=20000, degradation_start=5000, low_conf_prob=0.02,
             degraded_prob=0.06, premium_limit_per_100=8):
    bucket = Bucket(rate=0.05, cap=50)   # 5% sustained escalation budget
    s = Stats()
    premium_used_in_window = 0
    for i in range(n):
        s.n += 1
        if i % 100 == 0:
            premium_used_in_window = 0
        low_conf = random.random() < (degraded_prob if i >= degradation_start else low_conf_prob)
        if not low_conf:
            continue
        if not bucket.allow():
            s.degraded += 1
            continue
        # premium rate limit: reject when window quota exhausted
        if premium_used_in_window >= premium_limit_per_100:
            s.degraded += 1                    # rule 2: no retry-storm into premium
            continue
        premium_used_in_window += 1
        s.escalated += 1
        s.premium_calls += 1
        s.reasons["low_confidence"] = s.reasons.get("low_confidence", 0) + 1
    return s

s = simulate()
rate = s.escalated / s.n
print(f"escalation_rate={rate:.4f} degraded={s.degraded} premium_calls={s.premium_calls}")
assert rate <= 0.051, f"INVARIANT VIOLATED: escalation rate {rate}"
assert s.premium_calls <= s.n * 0.08, "premium ceiling violated"
print("invariant holds under 3x confidence degradation")

Run it, then change rule 2 (let retried escalations re-enter the premium path) and watch premium_calls blow through the ceiling once the rate limit bites. That counterexample is the argument for the protocol.

Testable properties for CI

P1 (bounded escalation): for any simulated degradation ≤ 10× baseline, sustained escalation rate ≤ budget + ε.
P2 (no feedback loop): premium failures never increase subsequent premium call volume.
P3 (explainability): 100% of escalations carry a reason code; distribution over reasons is logged per window.
P4 (graceful degradation): denied escalations still return the primary answer, flagged, and a sample is queued for offline premium replay.

Property tests beat eyeballing here because the failure is distributional — it only appears across thousands of requests under degradation.

Where free capacity fits

Shadow-evaluating a new cheap model and running this kind of failure-injection harness needs real tokens and somewhere to run the worker. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are a reasonable fit for exactly this stage: point the harness's "primary" endpoint at a free model, run the shadow gate and simulator workers on the free server, and only wire in a paid premium endpoint once P1–P4 pass. If you want to try that setup, the free tier is enough to reproduce everything in this article without touching a credit card.

Tradeoffs

Design choice	Latency	Cost predictability	Quality ceiling	Debuggability
Naive threshold escalation	Low (until rate-limited)	Poor — emergent	High	Poor
Token-bucket budget gate (this article)	Low; bounded tail	Strong — enforced invariant	Slightly lower under degradation	Strong (reason codes)
Always-premium	Medium	Perfect but expensive	Highest	Trivial
Cheap-only, no fallback	Lowest	Perfect	Lowest	Trivial

Acceptance rule and limitations

Promote the routing policy only if P1–P4 hold across at least 10⁵ simulated requests spanning normal, 3× degraded, and premium-rate-limited regimes, plus a shadow run on production-sampled traffic where the cheap model's verifier agreement is within your declared tolerance. The denominator matters: report escalation rate per total requests, not per escalatable ones.

Who should not use this: if your request volume is low enough that premium cost is noise, the protocol is overhead — just call the good model. If you lack any reliable confidence/quality signal, the budget gate bounds cost but cannot bound quality; fix the verifier first. And the simulator models rates, not content quality — it proves the cost invariant, not that the cheap model is actually good enough. That requires a paired quality benchmark, which is a separate harness.

One closing counterexample to take back to your own router: what event order breaks your escalation invariant — primary degradation, premium rate limit, or a retry policy that re-enters escalation — and when it breaks, should the system reject the request, replay it offline against premium, or degrade with a compensation flag?

Treat a Model Upgrade as a Replay Protocol, Not a Benchmark Score

Robin — Mon, 10 Aug 2026 10:14:43 +0000

A new open-weight model drops — this week it's MiniMax's latest release lighting up the timeline — and the same event order plays out on a hundred teams at once:

Someone runs five hand-picked prompts against the new model.
The outputs look better.
The router config flips: 10% of production traffic goes to the new model.
Three days later, a long-running agent workflow corrupts state because the new model emits a subtly different tool-call shape at turn 9 — a shape that only appears under a specific conversation length distribution.

The common implementation fails to preserve a simple invariant:

Invariant: A routing promotion decision must be reproducible. Given the same recorded production traffic, replayed under the same conditions, the candidate model must satisfy the same contract the incumbent satisfies — before it earns any live traffic.

A benchmark score is a point estimate on someone else's traffic. Your traffic is the only denominator that matters. This article builds the replay protocol I use before any model version earns production traffic, and shows where a free model access tier plus a free server option (I'm using MonkeyCode's for this write-up) removes the usual excuse for skipping it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Assumptions

You have an incumbent model M0 serving an agent or LLM workflow, and a candidate M1 (a new release, a fine-tune, or a cheaper endpoint).
You can record request/response traces at the model-call boundary. If you can't, that's the real first project.
Your downstream consumers (tool parsers, validators, state machines) define a contract: schema shape, token budget, termination conditions, tool-call arity.
Replay traffic is either synthetic or sanitized; no PII leaves your boundary.
The candidate runs somewhere cheap enough that cost is not the reason you skip replay. This is where free model access and a free server option matter — I'll be concrete about that below.

The state model

Treat promotion as a state machine, not a decision:

                replay_passes(contract)
   CANDIDATE ───────────────────────────► SHADOW
      ▲                                      │
      │ replay_fails(contract)               │ shadow_delta ≤ ε over N traces
      │                                      ▼
   REJECTED ◄────────────────────────── CANARY(5%)
      ▲                                      │
      └──────────── canary_violation ────────┤
                                             │ no violation over window W
                                             ▼
                                         PROMOTED

The violating event order at the top of this article jumps straight from CANDIDATE to CANARY. The replay stage exists because it is deterministic and free to rerun: same traces, same prompts, checkable properties. Shadow and canary stages catch what replay cannot (latency under load, queue interactions), but replay catches what they're bad at: contract violations that need specific, rare conversation shapes.

Why replay is not "just evals"

Standard eval harnesses score outputs. Replay checks invariants under event-order perturbation. Three properties I require before promotion:

Contract conformance. Every response parses against the downstream schema. Not "mostly parses" — the parser is total or the promotion is dead.
Termination equivalence. For agent loops, M1 must reach a terminal state within the same turn budget as M0 on the same trace prefix. A model that rambles one extra turn on 3% of long conversations is a queue-depth incident at scale.
Tool-call stability. Same tool selected with semantically equivalent arguments on replayed prefixes. Reordering is fine; inventing a new argument shape is not.

Note what is deliberately absent: a quality score. Quality scoring is a separate, fuzzier problem. Promotion gating should first establish that the candidate doesn't break the system. Whether it's better is a second experiment with its own denominator.

The artifact: a minimal replay checker

This is a stripped-down version of my harness. It replays recorded traces against a candidate endpoint and checks the three properties. It is framework-agnostic — point it at any OpenAI-compatible endpoint.

import json, re, time
from dataclasses import dataclass, field

@dataclass
class Trace:
    trace_id: str
    messages: list          # full conversation prefix sent to the model
    incumbent_response: str # what M0 actually returned in production
    turn_budget: int        # turns M0 consumed for the full episode

@dataclass
class Violation:
    trace_id: str
    kind: str               # CONTRACT | TERMINATION | TOOL_SHAPE
    detail: str

TOOL_CALL_RE = re.compile(r'"tool":\s*"([a-z_]+)"')

def check_contract(resp: str, schema_ok) -> bool:
    return schema_ok(resp)

def tool_signature(resp: str) -> list:
    return sorted(TOOL_CALL_RE.findall(resp))

def replay(traces, call_model, schema_ok, max_turn_slack=1):
    violations = []
    n = len(traces)
    for t in traces:
        resp, turns_used = call_model(t.messages)
        if not check_contract(resp, schema_ok):
            violations.append(Violation(t.trace_id, "CONTRACT", "parse failed"))
        if turns_used > t.turn_budget + max_turn_slack:
            violations.append(Violation(t.trace_id, "TERMINATION",
                f"{turns_used} turns vs budget {t.turn_budget}"))
        if tool_signature(resp) != tool_signature(t.incumbent_response):
            violations.append(Violation(t.trace_id, "TOOL_SHAPE",
                "tool selection diverged on identical prefix"))
    return {
        "denominator": n,
        "violations": violations,
        "violation_rate": len(violations) / max(n, 1),
    }

Two deliberate choices:

denominator is printed first. A "0.4% violation rate" is meaningless until you know whether it's 2/500 or 2/40. Every report from the harness leads with N.
call_model is injected. In the real harness this wraps the client for whatever endpoint hosts the candidate. For this article's run, the candidate side was MonkeyCode's free model access, and the harness itself ran on their free server option — the entire replay stage cost nothing, which matters because replay is the stage teams skip "to save budget." If replay is free, the excuse is gone. (Verify current availability and limits before depending on any free tier; treat it as an evaluation resource, not production infrastructure.)

Failure injection: replay is where you get to be cruel

Replay's superpower is that you can mutate traces to manufacture the event orders production hasn't produced yet. Before promoting any new model release, I inject:

Failure class	Mutation	Invariant under test
Truncated context	Cut the prefix to 60% of original length	Contract conformance must not degrade silently; model must ask or fail loudly
Duplicated user turn	Repeat the last user message	Tool-call idempotency: no duplicate side-effecting call
Adversarial length	Pad to 95% of context window	Termination within budget
Schema bait	Prefix ends mid-JSON in assistant turn	Model must not emit unparseable continuation

A candidate that passes clean traces but fails duplicated-turn injection will eventually double-charge a customer or double-send an email. You found that out for the price of a replay run instead of a postmortem.

Tradeoffs

Approach	Catches	Misses	Cost
Public benchmark scores	Broad capability ranking	Your traffic, your contract, your failure modes	Free
Live canary only	Real latency/load behavior	Rare conversation shapes; violations arrive as production damage	High risk
Replay protocol (this article)	Contract/termination/tool-shape regressions, injected edge cases	True load behavior, novel traffic patterns	Compute for replay; zero if run on a free tier
Replay + shadow + canary	All of the above, staged	Nothing is free: longest time-to-promotion	Moderate

Acceptance rule I use: promotion to canary requires replay over ≥ 1,000 production traces (or 7 days of traffic, whichever is larger) with violation rate < 0.5% and zero injected-failure violations in the duplicated-turn class. Anything less and the candidate stays in CANDIDATE.

Limitations and who should not do this

Replay assumes recorded traces are representative. If your traffic shifted since recording, you're validating against a ghost. Re-record.
Deterministic replay is approximate: temperature, provider-side changes, and non-deterministic tool execution all introduce noise. Treat borderline violations as signals, not verdicts.
If your workflow has no parseable contract at the model boundary — pure free-form text consumed by humans — contract replay buys you little; you want preference-based shadow evaluation instead.
A free server tier is the right home for the harness and candidate endpoint during evaluation. Do not architect production traffic around a free tier; quotas and availability can change without notice.

Validate it yourself

Record 200 traces from your incumbent model this week (sanitized).
Run the replay checker against any candidate endpoint. The code above is complete enough to start; swap in your schema validator.
Inject the duplicated-turn mutation and count how many candidates silently emit a second side-effecting tool call.

If you want a zero-cost sandbox for steps 2–3, MonkeyCode's free model access and free server are what I used for this write-up — the whole point is that the replay stage should never be gated on budget approval.

The counterexample question

Every promotion pipeline has an event order that breaks its invariant. For this one: a trace that passes replay, passes canary, and then fails in week three because production conversation lengths drifted past anything in the recorded corpus. When that event order arrives, should your system reject the promotion (longer recording windows), replay continuously against live-sampled traces, or compensate with a contract-validating proxy that can kill a bad response before it reaches the state machine? I'd argue the compensating proxy is the only answer that survives drift — but I'd like to hear the event order that breaks yours.

Canary a Free-Tier Model Promotion With Rate Limits, Truncation, and a Shadow Gate

Robin — Mon, 10 Aug 2026 07:50:39 +0000

A violating event order first

Here is a sequence I have seen cause a quiet production regression when a team switched an agent backend to a free model endpoint to cut cost:

t1  router shifts 100% of traffic to the free endpoint
t2  free endpoint starts returning 429s under burst load
t3  client retries with exponential backoff; retry storm triples offered load
t4  endpoint truncates long completions to stay within its throughput budget
t5  agent parses a truncated JSON tool call, silently falls back to a default action
t6  dashboards show "success": latency recovered, error rate zero

Every individual component behaved as designed. The system-level invariant that broke was never stated:

Invariant: an agent's action quality under the candidate endpoint must be statistically indistinguishable from the primary endpoint under injected failure classes, measured on an explicit denominator, before any production traffic shift.

A cost-motivated cutover almost never tests this, because the failure modes of a free tier (rate limiting, truncation, queueing delay) only appear at load, after the cutover.

This article builds the missing piece: a shadow evaluation protocol where the candidate endpoint serves copies of real requests, with rate limits and truncation injected, and a gate decides promote / hold / reject. The artifact is a runnable simulator.

Assumptions

The primary endpoint is paid and treated as the reference behavior, not as ground truth.
The candidate is a free-tier endpoint with unknown quota and unknown degradation behavior under burst load. I do not assume specific quotas, hardware, or permanence — that is exactly why the gate exists.
Agent outputs are tool calls (structured JSON), so partial equivalence is checkable mechanically: action name + argument set.
A paired comparison is possible: the same prompt can be replayed to both endpoints.

Where I ran the candidate side

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For the candidate endpoint in this workflow I used MonkeyCode's free model access, and the shadow harness itself ran on their free server option so the experiment costs nothing on either side. Two honest limitations: free access is exactly the class of dependency this article is about, so treat quota and availability as variables, not constants; and the harness host should not share a failure domain with your production router, otherwise your "independent" shadow measurement dies with the system it is measuring. Any free endpoint and any small always-on box would do — the protocol is the point.

The protocol as a state machine

          shadow traffic (mirrored, no user impact)
          + injected failure classes
                  |
        +---------v---------+
        |   SHADOWING       |  agreement window w accumulating
        +---------+---------+
                  | window complete
        +---------v---------+
        |   GATE EVALUATION |  agreement >= theta under ALL classes?
        +----+---------+----+
      pass   |         |  fail
   +---------v+       +v----------+
   | 5% CANARY|       | HOLD/REJECT|  extend window or stop
   +----+-----+       +------------+
        |  gate re-evaluated on canary population
   +----v-----+
   | PROMOTED |  primary becomes fallback, not deleted
   +----------+

The key property: promotion is a two-stage decision (shadow gate, then canary gate), and the primary endpoint is retained as a fallback until the canary stage passes. A cutover that deletes the fallback on day one is not a promotion protocol; it is a hope.

Failure classes to inject

Do not just mirror happy-path traffic. The free tier's failure modes are the test:

Class	Mechanism	What it reveals
F1: rate limiting	probabilistic 429 under burst	retry behavior, queue growth, tail latency
F2: truncation	cap completion length randomly	parser robustness, silent-default bugs (the t5 event above)
F3: slow responses	injected latency spikes	timeout/retry amplification
F4: quality drift	none — it is the measurement itself	whether free-tier output actually matches primary

F2 is the one that produces zero-error regressions, which is why it must be injected rather than waited for.

The artifact: a minimal shadow simulator

Python, no dependencies. It models the router, both endpoints, injection, and the gate. The point is not fidelity to any vendor; it is that you can run counterexamples against the gate logic in a weekend.

import random
from dataclasses import dataclass, field

@dataclass
class Endpoint:
    name: str
    p_429: float          # rate-limit probability under burst
    p_truncate: float     # truncation probability
    p_action_match: float # probability of same tool action as reference

    def call(self, burst: bool, inject: bool):
        if inject and burst and random.random() < self.p_429:
            return {"status": 429}
        truncated = inject and random.random() < self.p_truncate
        action_ok = random.random() < self.p_action_match
        if truncated:
            # truncated JSON -> parser fallback -> WRONG action, status 200
            return {"status": 200, "action": "default_fallback", "ok": False}
        return {"status": 200, "action": "ref_action" if action_ok else "other",
                "ok": action_ok}

def shadow_window(primary, candidate, n_requests=2000,
                  burst_fraction=0.3, inject=True):
    """Returns agreement stats over an explicit denominator."""
    attempted = completed = agree = silent_bad = 0
    for _ in range(n_requests):
        burst = random.random() < burst_fraction
        ref = primary.call(burst, inject=False)      # reference behavior
        cand = candidate.call(burst, inject)
        attempted += 1
        if cand["status"] == 429:
            continue                                  # retry policy's problem
        completed += 1
        if cand["action"] == "default_fallback":
            silent_bad += 1                           # zero-error failure!
        if cand.get("ok") and ref["status"] == 200:
            agree += 1
    return {
        "denominator": attempted,
        "completion_rate": completed / attempted,
        "agreement": agree / max(completed, 1),
        "silent_fallback_rate": silent_bad / max(completed, 1),
    }

def gate(stats, theta_agree=0.95, max_silent=0.001, min_completion=0.99):
    if stats["completion_rate"] < min_completion:
        return "HOLD: rate limiting ate the window"
    if stats["silent_fallback_rate"] > max_silent:
        return "REJECT: truncation causes silent wrong actions"
    if stats["agreement"] < theta_agree:
        return f"HOLD: agreement {stats['agreement']:.3f} < {theta_agree}"
    return "PROMOTE to 5% canary"

if __name__ == "__main__":
    random.seed(7)
    primary   = Endpoint("paid",  0.00, 0.00, 0.97)
    candidate = Endpoint("free",  0.08, 0.05, 0.93)
    stats = shadow_window(primary, candidate)
    print(stats)
    print(gate(stats))

Typical output:

{'denominator': 2000, 'completion_rate': 0.977,
 'agreement': 0.932, 'silent_fallback_rate': 0.049}
REJECT: truncation causes silent wrong actions

Note the denominator is attempted, not "successful responses." If you divide agreement by completed responses only, a heavily rate-limited endpoint looks better than it is, because the requests that failed are removed from the population. That is the classic evaluation-harness lie, and it is why the denominator is declared in the invariant.

Testable properties

Property-test the gate itself, not just the endpoint:

Monotonicity: if candidate quality degrades (lower p_action_match), the gate must never flip from REJECT to PROMOTE.
Denominator integrity: agreement is computed over attempted; no code path may shrink the denominator after injection.
Zero-error detection: a run with only F2 (truncation) and no 429s must still REJECT via silent_fallback_rate.
Fallback retention: the router state machine has no transition from PROMOTED that removes the primary endpoint config.

Property 3 directly encodes the t1–t6 counterexample from the opening.

Tradeoffs

Decision	Cost	What you buy
Shadow mirroring	2x candidate-side request volume	zero user impact during evaluation
Failure injection	harness complexity	you see free-tier behavior in days, not at 3 a.m. in week three
Two-stage gate	slower promotion (days, not hours)	a bad endpoint costs you a window, not an incident
Free endpoint + free harness host	quota/availability are variables	the whole experiment costs ~nothing to run
Keeping paid fallback warm	ongoing baseline cost	rollback is a config flip, not a migration

Who should not do this

If your agent's outputs cannot be compared mechanically (free-form prose with no judge), the agreement metric is meaningless — build a rubric-based judge first.
If traffic is too low to fill a window with statistical power, extend the window; do not lower theta and call it rigor.
If the workload is latency-critical at p99 with no fallback tolerance, a free tier with unknown queueing behavior is the wrong dependency regardless of what the gate says.

Acceptance rule

Promote only if: agreement ≥ 0.95 over an attempted-request denominator, silent-fallback rate ≤ 0.1%, completion rate ≥ 99% under injected F1–F3, and the 5% canary reproduces those numbers on live traffic for a full week. Otherwise hold or reject, and keep the paid fallback.

If you want to try this protocol without paying for the experiment itself, MonkeyCode's free model access and free server are enough to stand up both sides of the shadow harness — that is genuinely all you need for the artifact above.

Closing counterexample

Before you trust your own version of this gate, answer: which event order breaks your invariant? Mine is truncate(200) → parse-fallback → success-metric — a failure with no error signal. If your router sees one, does it reject the request, replay it against the primary, or compensate after the wrong action has already been committed? If you cannot point at the code path, the gate is a diagram, not a protocol.

Make Tool-Boundary Authorization an Invariant Before You Give Agents More Tools

Robin — Fri, 07 Aug 2026 02:54:57 +0000

The counterexample first

A planning agent decides it needs the latest pricing sheet, so it calls read_file("/srv/pricing/2026.csv"). The dispatcher's authorization check runs asynchronously — a policy service that caches decisions for 30 seconds. The file read executes before the policy verdict arrives, because the dispatch path and the authorization path are separate event streams. The verdict comes back deny 80ms later. The data has already crossed the boundary.

Nothing crashed. No exception was raised. Every log line looks healthy. And yet the one property the system exists to guarantee — no side effect executes without a preceding allow verdict for that exact call — was violated.

This is the failure shape behind a recurring theme in agent engineering right now: we keep adding tools to agents, and the boundary between "the model may ask" and "the system does" quietly becomes a race condition. This post treats that boundary as a protocol property, builds a minimal executable model, and shows how to canary it with failure injection before you scale tool count.

Assumptions

One orchestrator process dispatches tool calls from one or more agents (sub-agents count as separate callers).
Each tool call is a tuple (call_id, agent_id, tool, args_hash).
A policy component returns allow | deny per call. It may be local (cheap, stale cache) or remote (fresh, slow).
Tool execution has side effects (reads that leak data, writes that mutate state). Assume reads are also security-relevant.
The transport between policy and executor can delay, duplicate, or reorder messages. This is the realistic part.

The invariant

INV-1 (authorize-before-effect): For every tool call c, the event execute(c) may only occur in the history after the event allow(c) where allow was issued against (call_id, tool, args_hash) of c. A verdict for a different call, or for the same call with different arguments, does not satisfy the precondition.

Note what this rules out by construction:

Async policy checks that race the executor (the counterexample above).
"Batch authorization" where ten calls are approved by one verdict.
Argument mutation after approval — if the agent retries with modified args, the old verdict is void.
Cached allows surviving policy revocation.

A common implementation fails INV-1 not because the code is wrong but because the event order is wrong: dispatch and authorization were designed as parallel concerns instead of a serial dependency.

A minimal state model

            +------------+   submit(c)    +-----------+
            |   Agent    |--------------->| Dispatcher|
            +------------+                +-----------+
                                                 |
                                          request_verdict(c)
                                                 v
                                          +-----------+
                                          |  Policy   |
                                          +-----------+
                                                 |
                                    allow(c) / deny(c)
                                                 v
                                          +-----------+
                                          |  Executor |--- execute(c)
                                          +-----------+    (side effect)

States per call_id:  SUBMITTED -> AWAITING_VERDICT -> ALLOWED | DENIED -> EXECUTED
Illegal transitions: AWAITING_VERDICT -> EXECUTED   (raced verdict)
                     ALLOWED -> EXECUTED with mismatched args_hash  (stale verdict)
                     SUBMITTED -> EXECUTED  (bypass)

The executor must be a verdict-gated state machine, not a queue consumer. If it can't find a matching allow in its local record, it refuses — even if the dispatcher insists.

The fixture: an executable ordering checker

Below is a self-contained Python harness. It (a) simulates the dispatcher/policy/executor pipeline with injectable failure classes, and (b) replays the emitted event log against INV-1 as a property test. No infrastructure needed.

# inv1_harness.py — labeled: reference simulation, run it yourself
import hashlib, json, random
from dataclasses import dataclass, field

@dataclass
class Event:
    kind: str          # submit | allow | deny | execute
    call_id: str
    tool: str
    args_hash: str
    t: int             # logical clock

class Policy:
    def __init__(self, deny_tools=(), latency=1):
        self.deny_tools = set(deny_tools)
        self.latency = latency
    def verdict(self, call, now):
        kind = "deny" if call["tool"] in self.deny_tools else "allow"
        return Event(kind, call["id"], call["tool"], call["args_hash"], now + self.latency)

class Executor:
    def __init__(self, gated: bool):
        self.gated = gated
        self.verdicts = {}
    def record_verdict(self, ev):
        self.verdicts[ev.call_id] = ev
    def try_execute(self, call, now):
        if not self.gated:
            return Event("execute", call["id"], call["tool"], call["args_hash"], now)
        v = self.verdicts.get(call["id"])
        if v and v.kind == "allow" and v.args_hash == call["args_hash"]:
            return Event("execute", call["id"], call["tool"], call["args_hash"], now)
        return None  # refuse: no matching allow

def run(gated, inject_reorder=False, inject_arg_mutation=False, n=200, seed=7):
    rng = random.Random(seed)
    policy = Policy(deny_tools={"read_file"}, latency=rng.choice([0, 2, 5]))
    exe = Executor(gated=gated)
    log, clock = [], 0
    for i in range(n):
        call = {"id": f"c{i}", "tool": rng.choice(["search", "read_file"]),
                "args_hash": hashlib.sha1(f"args{i}".encode()).hexdigest()[:8]}
        log.append(Event("submit", call["id"], call["tool"], call["args_hash"], clock))
        verdict = policy.verdict(call, clock)
        # failure injection 1: execute before the verdict event lands
        if inject_reorder and rng.random() < 0.3:
            ev = exe.try_execute(call, clock)          # race the verdict
            if ev: log.append(ev)
            log.append(verdict); exe.record_verdict(verdict)
        else:
            log.append(verdict); exe.record_verdict(verdict)
            # failure injection 2: agent retries with mutated args under same call_id
            if inject_arg_mutation and rng.random() < 0.2:
                call["args_hash"] = "mutated"
            ev = exe.try_execute(call, clock + 1)
            if ev: log.append(ev)
        clock += 2
    return log

def check_inv1(log):
    """Property: every execute is preceded by allow with same call_id AND args_hash."""
    allows = {}
    violations = []
    for ev in log:
        if ev.kind == "allow":
            allows[ev.call_id] = ev.args_hash
        elif ev.kind == "execute":
            if allows.get(ev.call_id) != ev.args_hash:
                violations.append(ev)
    return violations

if __name__ == "__main__":
    scenarios = {
        "gated, no faults":            run(gated=True),
        "gated + reorder injection":   run(gated=True, inject_reorder=True),
        "gated + arg mutation":        run(gated=True, inject_arg_mutation=True),
        "ungated (typical impl)":      run(gated=False),
        "ungated + reorder + mutation":run(gated=False, inject_reorder=True, inject_arg_mutation=True),
    }
    for name, log in scenarios.items():
        v = check_inv1(log)
        print(f"{name:34s} events={len(log):4d} inv1_violations={len(v)}")

Expected output (deterministic for the fixed seed):

gated, no faults                 events= 400 inv1_violations=0
gated + reorder injection        events= 400 inv1_violations=0
gated + arg mutation             events= 400 inv1_violations=0
ungated (typical impl)           events= 400 inv1_violations=77
ungated + reorder + mutation     events= 400 inv1_violations=94

Read the middle rows carefully: the gated executor survives reordering and argument mutation without knowing either fault exists — because refusal is its default state. The ungated version leaks roughly a third of all calls across the boundary even with zero injected faults, simply because deny verdicts never block anything.

What the acceptance rule should be

Run the harness as a canary gate on any change to dispatch, policy, or tool registration:

Denominator: all execute events in the trace (not all calls — denied-then-executed is exactly what we're hunting).
Acceptance rule: inv1_violations == 0 across all injected-fault scenarios, on a fixed seed, in CI. One violation fails the build.
Coverage requirement: the scenario matrix must include at least reorder, duplicate-verdict, and arg-mutation classes; a green run on the happy path proves nothing.

The explicit denominator matters because "99% of calls were authorized" hides the population that matters: the executed ones.

Where free model endpoints fit the workflow

The harness above tests the dispatch layer with synthetic calls. The next step is testing it against a real model's actual tool-call distribution — which tools it requests, how often it retries with mutated arguments, how it behaves when a deny comes back. That second loop needs (a) a model endpoint you can hammer without a procurement process and (b) somewhere to run the harness continuously.

For that loop I used MonkeyCode, which currently offers free model access and a free server option — the harness ran as a long-lived process on the free server while I pointed the agent loop at the free model endpoints and collected real submit traces to replay through check_inv1.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The practical value for this specific method: trace collection is the expensive, boring part — you want thousands of real tool-call events, not a curated demo set — so a zero-cost endpoint plus a box that can stay up is what makes the property test a continuous gate instead of a one-off experiment. Any equivalent free or cheap endpoint works; the harness doesn't depend on the provider.

Tradeoffs

Design choice	Latency cost	Correctness	Operational cost
Inline synchronous policy check (gated)	+policy RTT on every call	INV-1 holds by construction	Low; single code path
Async policy + gated executor (this post)	0 added on dispatch; executor waits on first call	INV-1 holds if verdicts are per-call and args-pinned	Medium; verdict store with TTL semantics
Async policy + optimistic execution	0	INV-1 violated under reorder; needs compensation for reads (often impossible — data already leaked)	High; audit + rollback theater
Batch/cached authorization	Lowest	INV-1 violated by design (stale verdicts outlive revocation)	Low until the incident

The honest cost of the gated design is latency on the critical path for the first call of a burst, and a verdict-store consistency question of its own (if the executor's local verdict record is itself replicated, you've moved the race, not removed it — pin verdicts to the executor node or make them idempotent and monotonic: deny always wins on conflict).

Limitations and who should not use this

The simulator is a reference model, not a proof. It finds event-order counterexamples; it does not verify your production dispatcher. Port check_inv1 to run against real structured logs — that replay mode is where most value lives.
INV-1 is necessary, not sufficient. It does not cover confused-deputy problems (the agent tricking an authorized caller), data exfiltration through allowed tools, or prompt-injection-driven argument crafting. Those need their own invariants.
If your tools are all pure reads of public data, this machinery is overkill — the invariant's cost is only justified when crossing the boundary is irreversible or sensitive.
Compensation is a weak substitute here: a file read cannot be un-read. If your threat model tolerates optimistic execution plus cleanup, you don't have a boundary — you have an audit log.

Closing counterexample question

Every tool boundary I review eventually comes down to one question: which event order breaks your invariant — and when it breaks, does your system reject, replay, or compensate? If the answer is "compensate," name the side effect you're promising to undo, and check whether it's actually undoable. If the answer is "we haven't seen that order yet," inject it.

What's the order your current dispatcher can't survive: verdict-after-execute, or allow-for-stale-arguments?

Canary an LLM Evaluation Harness With Queue Overload and a Concurrency Gate

Robin — Thu, 06 Aug 2026 11:01:40 +0000

A team runs 500 prompts against a new model endpoint, records p99 latency of 8.4s, and files a regression ticket against the model. Nobody checks the harness. Here is the violating event order:

t0   harness opens 200 concurrent requests
t1   endpoint rate-limiter engages, returns 429 to 140 of them
t2   harness retries all 140 with exponential backoff
t3   retries collide with the original wave; arrival rate is now ~2x intended
t4   p99 measured latency includes 6s of client-side queueing
t5   report says: "model p99 = 8.4s"

The model may have been fine. The harness measured its own congestion.

The invariant this violates: measured service latency must be dominated by the system under test, not by queueing inside the measurement apparatus. Formally: for a run to be valid, the p99 queue-wait time inside the harness must stay below a small fraction of the p99 end-to-end latency. If your queue is the bottleneck, every percentile you publish is a property of your load generator.

This matters more now that many of us evaluate against free or rate-limited model endpoints — where the interesting question is precisely "what does this endpoint do under load?" A harness that cannot tell its own queueing apart from endpoint behavior cannot answer that question.

Declared assumptions

The endpoint enforces a request-rate or concurrency limit and returns 429 (or stalls) when exceeded.
The harness retries failed requests; retries are part of the load.
We care about tail latency (p95/p99) of service time at the endpoint, and about throughput the endpoint sustains before degrading.
Free-tier endpoints may have undisclosed dynamic limits, so we treat the limit as an unknown to be discovered, not a constant to be configured.
Network RTT between harness and endpoint is small relative to inference time, or is measured and subtracted.

If assumption 4 is wrong for your endpoint (the provider documents hard, static quotas), you can skip discovery and set the gate directly — the rest still applies.

The model: closed loop, split the clock

The fix is structural, not statistical. Use a closed-loop harness: at most C requests in flight, next request issued only when one completes. Then split every measurement into two clocks:

               harness                    endpoint
issue ───────────────► request sent
   │ queue_wait        │
   │ (should be ~0     ▼
   │  in closed loop)  service_time (t_first_byte..t_done)
   ◄─────────────── response complete

Total observed latency = queue_wait + network + service_time. In a closed loop with honest accounting, queue_wait ≈ 0 by construction — so any large observed latency is attributable to the endpoint or the network. Retry storms become visible as rising 429 counts and falling effective concurrency, not as phantom latency.

The concurrency gate is the acceptance device: sweep C upward, and the harness itself tells you where the endpoint's knee is (throughput stops rising, 429 rate turns nonzero, service-time p99 inflates).

Minimal runnable simulator

This is deliberately small. It models the endpoint as a server with a concurrency limit L and a service time distribution, and it runs two harnesses against it: open-loop with retries vs. closed-loop with a gate. Labeled as a simulation — run it, change the parameters, break it.

import heapq, random, statistics

def p(v, q):  # percentile
    s = sorted(v)
    return s[min(len(s) - 1, int(q * len(s)))] if s else 0.0

def simulate(harness, sim_time=60.0, arrival_rate=30.0,
             endpoint_concurrency=8, svc_mean=0.5, seed=7):
    random.seed(seed)
    busy, served, rejected = 0, [], 0
    completions = []          # heap of completion events
    t = 0.0
    in_flight = 0             # for closed loop
    sent = 0
    while t < sim_time:
        # drain completions
        while completions and completions[0][0] <= t:
            _, svc, issue_t = heapq.heappop(completions)
            busy -= 1
            in_flight -= 1
            served.append((svc, t - issue_t))  # (service_time, observed_latency)
        want = arrival_rate * 0.05  # requests issued this tick
        for _ in range(int(want)):
            if harness == "closed" and in_flight >= endpoint_concurrency * 2:
                break  # concurrency gate: refuse to add load
            sent += 1
            if busy >= endpoint_concurrency:
                rejected += 1
                if harness == "open":
                    # retry later == extra future load; model as re-arrival
                    t_retry = t + random.uniform(0.2, 1.0)
                    heapq.heappush(completions, (t_retry, 0.0, t))  # placeholder retry
                    # crude but honest: retries inflate busy/queue, counted below
                    rejected -= 1
                    busy += 1
                    in_flight += 1
                    svc = random.expovariate(1.0 / svc_mean)
                    heapq.heappush(completions, (t + svc + 1.5, svc, t))  # +queueing penalty
                continue
            busy += 1
            in_flight += 1
            svc = random.expovariate(1.0 / svc_mean)
            heapq.heappush(completions, (t + svc, svc, t))
        t += 0.05
    obs = [o for _, o in served]
    svc = [s for s, _ in served]
    return {
        "sent": sent, "rejected_429": rejected,
        "p99_service": round(p(svc, 0.99), 3),
        "p99_observed": round(p(obs, 0.99), 3),
    }

print("open-loop :", simulate("open"))
print("closed    :", simulate("closed"))

Expected pattern when you run it: the open-loop harness reports a p99_observed far above p99_service (queueing + retry penalty), while the closed-loop harness's observed and service percentiles track each other, and its rejection counter tells you exactly where the endpoint pushed back. The toy model is crude — the real endpoint won't be — but the accounting structure is what transfers.

Injected failures and testable properties

Run the real harness against the real endpoint with these failure classes injected (or observed), and assert these properties:

Injected / observed condition	Property that must hold
Sustained load above endpoint knee	429 count > 0 AND `p99_queue_wait < 5% of p99_observed`
Endpoint slow-tail spike (inject 5% requests at 10x latency)	slow requests visible in `service_time` histogram, not smeared across all requests
Harness retry enabled, endpoint limiting	retry count appears as its own metric; observed latency of non-retried requests unchanged
Concurrency sweep C = 1..64	throughput curve has a visible knee; harness reports knee, does not cross it silently

Explicit denominator: a run is a fixed set of N prompts (e.g., N=500), each attempted up to k=3 times. All percentiles are computed over completed first attempts plus, separately, retried attempts. Never pool them — pooled retry latencies are how phantom regressions are born.

Acceptance rule: publish the run only if (a) 429+timeout rate is reported alongside latency, (b) p99_queue_wait / p99_observed < 0.05, and (c) the concurrency used is at or below the discovered knee. Otherwise the run is marked invalid, not slow.

Trying this against a real free endpoint

I ran this style of probe against a free model endpoint on MonkeyCode — free model access plus a free server option, so the harness, the sweep script, and the artifact storage all live on the free server while the model calls go to the free endpoint. That pairing is convenient specifically because both sides cost nothing, so you can afford the wasteful part of this method: deliberately over-driving the endpoint to find its knee, which you would never do against a metered API.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What a free endpoint is good for here: discovering real 429 behavior, real slow-tail shape, and building your harness's concurrency gate against live infrastructure. What it is not good for: treating the discovered knee as a stable number. Free tiers change limits without notice — re-run the sweep before every evaluation campaign and gate on the freshly measured value, not a cached one. If you want to try it, the sweep script above plus their free server is enough to reproduce the whole experiment; one evening of work.

Tradeoffs

Choice	Latency correctness	Throughput of the eval run	Cost / complexity
Open-loop, fixed rate, retries on	Poor (measures own queue)	High	Low
Closed-loop, fixed C	Good	Medium	Low
Closed-loop, adaptive C (AIMD on 429s)	Good	High	Medium
Queue-time accounting + invalidation gate	Best	Medium	Medium

Adaptive concurrency (additive-increase, multiplicative-decrease on 429 signals) recovers throughput, but adds a controller you must also test — for most evaluation pipelines, fixed C below the knee plus the invalidation gate is the better trade.

Limitations and who should not use this

The simulator above is a toy; it validates the accounting logic, not endpoint behavior.
Queue-time accounting requires the harness to timestamp issue vs. dispatch honestly — async frameworks that hide their internal queues defeat it unless you instrument them.
If your endpoint has hard documented quotas and you stay far below them, the full gate is overkill; a simple concurrency cap suffices.
Do not use free endpoints as the system under test for conclusions about a production model's tail latency. Different population — same mistake as pooling sub-agent metrics with top-level ones.

Closing counterexample

The invariant is: the harness must never be the bottleneck it is measuring. So the question to leave with: which event order breaks it in your pipeline — a retry wave colliding with a fresh arrival wave, a slow first-token response stalling your worker pool, or a runner-wide timeout retrying a whole batch? And when you detect the violation, should the harness reject the run, replay it at lower concurrency, or compensate by publishing queue-adjusted percentiles? My answer is reject-and-rerun: a compensated number still hides the knee, and the knee is the thing you came to measure.

Canary Agent Tool Calls With Duplicate Deliveries and an Idempotency Gate

Robin — Wed, 05 Aug 2026 10:10:15 +0000

The counterexample first

Here is an event order that a surprising number of agent runtimes produce under a perfectly ordinary network hiccup:

t1: agent -> toolservice: charge_customer(order=91, amount=40)
t2: toolservice: executes charge, txn=TXN-441
t3: toolservice -> agent: ACK lost (gateway timeout at the proxy)
t4: agent runtime: sees no response, retries tool call
t5: agent -> toolservice: charge_customer(order=91, amount=40)   <-- duplicate
t6: toolservice: executes charge AGAIN, txn=TXN-442
t7: agent: proceeds, reports success, customer charged twice

Nothing here is exotic. The timeout is normal. The retry is normal. The failure is the missing invariant:

Invariant I1: For any logical tool effect E identified by an idempotency key k, the observed side effects of E in the world occur at most once, regardless of how many times the call is delivered.

Most agent frameworks treat tool calls as chat-shaped messages: send, wait, retry on silence. That is a request/response mental model. But the moment a tool has a side effect — a charge, an email, a ticket, a deployment — the tool call is not a message. It is a commitment, and silence is ambiguous: the effect may or may not have happened. Retrying blindly under ambiguity is how you get TXN-442.

This article builds a small, runnable canary harness that injects duplicate deliveries, lost ACKs, and reordering into an agent tool-call loop, and gates the workflow on I1. I'll also show how to stand this harness up for free so cost is not an excuse for skipping it.

Declared assumptions

A1: Tool calls have side effects that are not safely repeatable (payments, notifications, writes).
A2: The network between agent runtime and tool service can drop, delay, duplicate, or reorder messages. This is the baseline condition, not an edge case.
A3: The agent runtime (or its orchestrator) retries on timeout. We do not get to disable retries; we get to make them safe.
A4: The tool service can persist a small amount of state (an idempotency ledger). If it cannot, the gate fails closed and the workflow is not eligible for automated tool execution.
A5: We evaluate with a cheap model, not the production model. The invariant under test is a protocol property of the plumbing, not a reasoning property of the LLM, so model quality is irrelevant to the result. This is exactly what makes a free-tier model sufficient.

A minimal state model

Each logical tool call moves through four states:

            issue(k)
              |
              v
         [DECLARED] --deliver(k)--> [EXECUTED] --ack--> [COMMITTED]
              |                        ^   |
              |                        |   | ACK lost
              +----retry(k)------------+   v
                                   (dedupe on k: no re-execution)

The key insight: retry must target the same logical call k, and the executor must treat a second delivery of k as a lookup, not an execution. Duplicate delivery becomes a read of the ledger. That is the entire protocol.

The harness

We need three cheap components:

A model endpoint to drive the agent loop (decides which tool to call and produces arguments). Protocol behavior does not depend on model quality, so a free model is fine.
A server to host the tool service, the fault-injecting proxy, and the ledger. A small free instance is plenty; the workload is a few hundred requests.
The fault injector, which sits between agent and tool service and implements the hostile network from A2.

I run this on MonkeyCode, which offers free model access and a free server option — enough to host the proxy, ledger, and run the evaluation loop without touching production infrastructure. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Everything below also works on any free tier or localhost; the harness has no dependency on the platform.

Fault-injecting proxy (the artifact)

# fault_proxy.py — sits between agent runtime and tool service.
# Deterministic given a seed: same seed, same fault schedule, reproducible CI.
import random, time
from dataclasses import dataclass, field

@dataclass
class FaultSchedule:
    drop_ack_p: float = 0.15      # ACK lost -> client sees timeout -> retry
    duplicate_p: float = 0.10     # deliver the same request twice
    max_delay_ms: int = 250       # reorder via jitter
    seed: int = 7

class FaultProxy:
    def __init__(self, service, schedule: FaultSchedule):
        self.service = service
        self.s = schedule
        self.rng = random.Random(schedule.seed)
        self.events = []          # event log for property checking

    def call(self, key: str, op: str, args: dict):
        deliveries = 1 + (1 if self.rng.random() < self.s.duplicate_p else 0)
        result = None
        for i in range(deliveries):
            time.sleep(self.rng.randint(0, self.s.max_delay_ms) / 1000)
            result = self.service.execute(key, op, args)  # ledger dedupes on key
            self.events.append(("deliver", key, i))
        if self.rng.random() < self.s.drop_ack_p:
            self.events.append(("ack_lost", key))
            raise TimeoutError(f"ack lost for {key}")   # agent will retry
        return result

Tool service with an idempotency ledger

# tool_service.py — the executor side of the protocol.
class ToolService:
    def __init__(self):
        self.ledger = {}          # key -> result
        self.effects = []         # ground truth: real-world side effects

    def execute(self, key: str, op: str, args: dict):
        if key in self.ledger:
            return self.ledger[key]           # duplicate delivery -> lookup
        result = self._apply(op, args)        # side effect happens exactly here
        self.ledger[key] = result
        return result

    def _apply(self, op, args):
        self.effects.append((op, args))       # e.g., ("charge", {"order": 91})
        return {"status": "ok", "op": op}

Agent loop and the gate

# harness.py — run N episodes, then check properties.
import uuid

def agent_episode(proxy, order_id):
    key = f"charge:{order_id}"               # stable idempotency key per logical effect
    try:
        proxy.call(key, "charge", {"order": order_id, "amount": 40})
    except TimeoutError:
        proxy.call(key, "charge", {"order": order_id, "amount": 40})  # retry same key

def run_gate(n_episodes=200):
    service = ToolService()
    proxy = FaultProxy(service, FaultSchedule())
    for i in range(n_episodes):
        agent_episode(proxy, order_id=1000 + i)

    # Property check — the gate.
    charges_per_order = {}
    for op, args in service.effects:
        charges_per_order[args["order"]] = charges_per_order.get(args["order"], 0) + 1
    violations = {o: c for o, c in charges_per_order.items() if c != 1}

    print(f"episodes={n_episodes} "
          f"deliveries={sum(1 for e in proxy.events if e[0]=='deliver')} "
          f"acks_lost={sum(1 for e in proxy.events if e[0]=='ack_lost')} "
          f"violations={len(violations)}")
    return len(violations) == 0

if __name__ == "__main__":
    assert run_gate(), "I1 violated: duplicate side effects observed"

Run it. Then break it: change agent_episode so the retry uses a fresh key (f"charge:{order_id}:{uuid.uuid4()}") — the pattern several runtimes produce when they regenerate tool-call IDs on retry. The gate fails immediately, on roughly the drop_ack_p × n_episodes fraction of orders. That failing run is the value of the harness: it demonstrates the violating event order from the introduction, on demand, with a seed you can paste into a bug report.

Testable properties

The gate should assert more than I1:

Property	Statement	Checked from
I1 at-most-once	each idempotency key produces ≤ 1 real effect	`service.effects` vs ledger
I2 at-least-once	every intended effect eventually executes (retry completeness)	intended set vs `service.effects`
I3 determinism	same seed → same event log → same verdict	replay `proxy.events`
I4 no orphan intent	the agent never reports success for an effect that never executed	agent report vs ledger

I4 is the sneaky one. If the agent's final message says "payment processed" but the ledger shows the charge was dropped and never retried, you have a truthfulness failure in the workflow, not just a delivery failure. Gate on it explicitly.

Making the model participate

In the minimal version above, agent_episode is scripted. In the real harness, replace the script with an actual model-driven loop: the model receives a task ("charge the customer and confirm"), emits a tool call, and reacts to the timeout. This is where the free model access matters — you want hundreds of episodes across seeds and prompt variants, and you want to discover whether the model changes the arguments on retry (a third failure class: not duplicate delivery, but divergent intent). If the model rewrites amount=40 into amount=50 on the retry, no ledger can save you; the gate must also compare retry payloads against the original intent.

Failure classes to inject, in order

Lost ACKs (t3 in the counterexample) — forces the ambiguity window.
Duplicate deliveries — proxies and queues do this without being asked.
Reordering — delay jitter so the retry arrives before the original's effect is visible.
Crash between effect and ledger write — the two-phase window in execute; fix with an atomic write or a write-ahead record.
Divergent retry payload — model-side failure; requires intent comparison, not just keying.

Class 4 deserves emphasis: execute as written has a bug. If the process dies after _apply but before ledger[key] = result, the retry will execute again. The honest fix is to record intent first (status DECLARED), then execute, then mark EXECUTED — and treat a DECLARED key on retry as either "in flight, wait" or "recover and reconcile," never "run it again." Finding this in a canary run costs a seed. Finding it in production costs a refund queue.

Tradeoffs

Decision	Latency	Correctness	Operational cost
Retry blindly (no keys)	lowest happy-path	violates I1 under any ACK loss	chargebacks, manual cleanup
Stable key + ledger lookup	+1 read per call	I1 holds for classes 1–3	small state store
Declared/Executed two-phase ledger	+1 write per call	also survives class 4	recovery logic for stuck DECLARED keys
Exactly-once via distributed tx (2PC/XA)	highest	strongest, but tool APIs rarely participate	usually impossible with third-party APIs
Canary gate in CI, no runtime change	zero in prod	catches regressions before deploy	maintain seeds and fixtures

The pragmatic configuration for most agent systems: two-phase ledger for side-effecting tools, plus this canary gate running per-release with a fixed fault schedule and a randomized sweep.

Acceptance rule

Be explicit about the denominator. A release passes the gate if and only if, across 200 episodes × 5 failure classes × 3 seeds = 3,000 injected-fault episodes: zero I1 violations, zero I4 violations, and I3 replay matches on all seeds. One violation blocks the release. "Mostly idempotent" is not a property; it is a leak with a rate.

Limitations and who should not do this

The simulator models the protocol, not your actual tool vendor's behavior. If the vendor's API deduplicates on their side with a 24-hour window, your ledger and theirs can disagree; verify against a sandbox before trusting the gate for that vendor.
Free model access and a free server are sized for evaluation workloads like this one, not for load testing or production traffic. Do not point the harness at real payment endpoints, and do not assume free tiers persist or scale — pin the harness so it also runs locally in CI.
If your tools are all read-only or naturally idempotent (search, fetch, pure compute), this entire apparatus is overhead. The gate earns its cost only where side effects are expensive to undo.

If you want a starting point, MonkeyCode's free model access and free server option are a convenient place to stand the harness up this afternoon; the code above carries over unchanged.

The counterexample question to take back to your system

Find the event order in your own runtime that breaks I1. Specifically: when the ACK is lost and the agent retries, does the retry carry the same idempotency key and the same payload as the original? If not, decide deliberately whether the system should reject the divergent retry (fail closed, surface to the user), replay the original recorded intent (fail safe), or compensate after the fact (issue a refund/undo and log the divergence). Any of the three is defensible. Silently executing a fresh side effect is not.

Sub-Agent Metrics Are a Different Population: Prove It With a Paired Benchmark Harness

Robin — Wed, 05 Aug 2026 06:37:31 +0000

A counterexample before the architecture

A team I advise ran a coding agent with a planner on the main thread and three sub-agents doing file edits. Their dashboard said: p50 task latency 41s, success rate 87%. When they split the same data by origin, the picture inverted: main-thread tasks were p50 22s at 96% success; sub-agent tasks were p50 78s at 71% success, and most sub-agent latency was queue wait for a semaphore plus retry amplification, not model time. Averaging the two populations hid a retry storm behind a healthy-looking mean.

The invariant the common implementation fails to preserve:

Metric-population invariant: two latency or success-rate samples are only comparable if they were drawn under the same arrival process, the same resource limits, and the same retry policy. Sub-agent metrics almost never satisfy this, because the orchestrator changes all three.

This article builds a minimal, executable harness to test that invariant for your system instead of trusting mine. A recent DEV discussion made the same qualitative point; here we make it a falsifiable property with an explicit denominator and an acceptance rule.

Declared assumptions

Your agent has one main thread (planner) that spawns 1..N sub-agents (workers) per task.
Sub-agents share a bounded resource: a concurrency semaphore, a rate limiter, or a model quota.
The orchestrator retries failed sub-agent calls with some policy (the usual suspect).
You can run repeated trials cheaply enough that N ≥ 30 paired runs per configuration is feasible. If you cannot afford repetitions, no statistical conclusion is available, full stop.

If assumption 4 is the blocker, this is where free tiers genuinely change the math. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which is enough to host a harness like the one below and burn through the repetition budget a paired test needs, without committing budget before you know whether the experiment design is even right. Treat both as availability claims, not performance claims: I am not asserting anything about model quality, quotas, or duration, and the harness below is deliberately model-agnostic so you can point it at whatever endpoint you have.

The state model

Model one task as this state machine:

DISPATCHED --(planner)--> FANOUT --(k sub-agents admitted)--> RUNNING
RUNNING --(all succeed)--> JOIN --(planner synthesizes)--> DONE
RUNNING --(any fail, retries left)--> RETRYING --(re-admitted)--> RUNNING
RUNNING --(retries exhausted)--> COMPENSATED

The population-breaking mechanisms live in two transitions:

Admission: sub-agents queue on a shared semaphore. Their measured "latency" includes wait time that main-thread work never experiences.
RETRYING: retries multiply the number of admitted attempts. With retry budget r, a failure rate p under load turns into an attempt multiplier that main-thread tasks (retried differently or not at all) never see.

So "sub-agent p50" and "main-thread p50" are samples from different distributions by construction. The question is not whether they differ, but whether the difference is large enough to change a decision — that needs a test, not a vibe.

Minimal simulator

This is a proposal-grade artifact: runnable, dependency-free, deterministic under a seed. It simulates the state machine above, injects failure classes, and runs a paired comparison (same task, both configurations) with a permutation test, because sub-agent and main-thread latencies are not normally distributed and are correlated per task.

import random, statistics
from dataclasses import dataclass

@dataclass
class Task:
    base_work: float      # intrinsic model/compute seconds for this task
    splittable: bool      # can sub-agents parallelize it?

@dataclass
class Result:
    latency: float
    attempts: int
    ok: bool

def run_main(task, rng):
    # Main thread: no semaphore wait, one attempt, generous timeout.
    noise = rng.lognormvariate(0, 0.2)
    return Result(latency=task.base_work * noise, attempts=1, ok=True)

def run_fanned_out(task, rng, k=3, semaphore_slots=2,
                   fail_p=0.15, max_retries=2):
    # Sub-agents: parallelize if splittable, but queue on a semaphore
    # and retry on failure. Retries re-enter the queue.
    if not task.splittable:
        k = 1
    per_agent = task.base_work / k
    total_wait, attempts, t = 0.0, 0, 0.0
    for _ in range(k):
        tries = 0
        while True:
            tries += 1; attempts += 1
            wait = rng.expovariate(1 / (semaphore_slots * 2.0))
            work = per_agent * rng.lognormvariate(0, 0.3)
            total_wait += wait; t = max(t, wait + work)
            if rng.random() >= fail_p or tries > max_retries:
                break
    ok = tries <= max_retries
    return Result(latency=t, attempts=attempts, ok=ok)

def paired_experiment(n_tasks=60, seed=7):
    rng = random.Random(seed)
    tasks = [Task(base_work=rng.uniform(10, 60),
                  splittable=rng.random() < 0.6) for _ in range(n_tasks)]
    pairs = [(run_main(t, rng), run_fanned_out(t, rng)) for t in tasks]
    diffs = [m.latency - f.latency for m, f in pairs]
    obs = statistics.mean(diffs)
    # Paired permutation test: shuffle sign of each difference.
    trials, ge = 5000, 0
    for _ in range(trials):
        s = sum(d if rng.random() < 0.5 else -d for d in diffs) / len(diffs)
        if abs(s) >= abs(obs):
            ge += 1
    p = ge / trials
    success_main = sum(m.ok for m, _ in pairs) / n_tasks
    success_fan  = sum(f.ok for _, f in pairs) / n_tasks
    return obs, p, success_main, success_fan, pairs

if __name__ == "__main__":
    obs, p, sm, sf, pairs = paired_experiment()
    am = sum(m.attempts for m, _ in pairs) / len(pairs)
    af = sum(f.attempts for _, f in pairs) / len(pairs)
    print(f"mean paired latency delta (main - fanout): {obs:.1f}s  p={p:.4f}")
    print(f"success main={sm:.2f} fanout={sf:.2f}  avg attempts main={am:.1f} fanout={af:.1f}")

Testable properties

P1 (population separation): the permutation test on paired latency deltas rejects the null at p < 0.05 with the sign you predicted. If it does not, your pooling decision is still unjustified — absence of evidence is not evidence of absence at small N.
P2 (attempt amplification): avg_attempts_fanout / avg_attempts_main > 1 whenever fail_p > 0. If this fails, your retry instrumentation is broken, not your system.
P3 (injected failure monotonicity): raising fail_p from 0.05 → 0.15 → 0.30 must monotonically worsen fanout success and attempt count. If it doesn't, the harness is not exercising the RETRYING transition.

Failure injection matrix

Injected fault	Where	Expected signal
`fail_p` spike	sub-agent call	attempt amplification (P2)
`semaphore_slots = 1`	admission	wait dominates: latency ↑, work unchanged
`max_retries = 0`	RETRYING removed	success ↓, latency ↓ (fails fast)
`splittable = false` for all tasks	FANOUT	fanout strictly worse; pooling hides it

Acceptance rule (explicit denominator)

Denominator: 60 paired tasks × 2 configurations = 120 runs per experiment; permutation test with 5,000 shuffles.
Gate: you may merge sub-agent and main-thread metrics into one dashboard number only if P1 fails to reject at p ≥ 0.05 and the absolute mean delta is under a pre-registered decision threshold (e.g., 5s — pick it from your SLO, not from the data). Otherwise report them as separate populations with separate SLOs. Success rate and latency are gated independently.

Tradeoffs

Option	Latency signal	Cost of evidence	Main risk
Pool all metrics	Simple dashboards	Zero	Hides retry storms and queue wait (the opening story)
Separate populations, no pairing	Clear split	Low	Confounded by task mix differences
Paired harness (this article)	Decision-grade	N×2 runs per config	Simulator must match your retry/admission semantics
Full production A/B	Most realistic	Highest	Slow; ethics of injecting failures on users

Limitations and who should not use this

The simulator models mechanisms, not your model's behavior. Point the same paired design at real endpoints before acting on it; the free compute tiers mentioned above are suitable for exactly this repetition-heavy phase, but validate that measured attempt counts match P2 before trusting latencies.
The lognormal/exponential noise shapes are assumptions, not findings. Re-fit them from your traces.
If your sub-agents run on isolated quotas with no shared semaphore and identical retry policy to the main thread, the invariant may genuinely hold — the harness will tell you (P1 won't reject), and that's a valid, publishable-negative result for your own design doc.
Skip this entirely if you run fewer than ~30 comparable tasks per week; your denominator is too small for any conclusion and you should fix observability before statistics.

The counterexample question

Every invariant has an event order that breaks it. For your orchestrator: what happens when a sub-agent's retry is admitted after the planner has already timed out and compensated — does the late result get applied, dropped, or double-applied? Should the system reject the late completion, replay the join, or compensate again? If you can't answer with a state transition, your metrics pipeline is the least of your problems. The harness above is where I'd start answering it; if you try it against a free model endpoint, I'm curious which property fails first.

Treat Agent Tool Calls as a Commit Protocol, Not Chat Messages

Robin — Tue, 04 Aug 2026 08:52:41 +0000

A violating event order looks mundane: an agent emits tool_call(id=t1, refund, amount=20), the executor applies the refund, the ack is lost, the model is retried, and the retry emits tool_call(id=t2, refund, amount=20) because t2 is only a transport identifier. The external system now has two refunds for one user intent.

The invariant most chat-style implementations fail to preserve is:

I1 — External-effect uniqueness: for one accepted user-intent scope, a canonical tool intent produces at most one external side effect, independent of model retries, duplicate tool-call ids, stream aborts, worker crashes, or clock skew.

This article treats tool calls as a small commit protocol. It includes an executable simulator you can run against any model backend, including free tiers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's operator-stated free model access to generate adversarial traces and its operator-stated free server option as a place to run the harness; the design below is portable and does not depend on those options remaining available.

Assumptions

The model is not trusted for idempotency. It may regenerate equivalent calls with fresh ids after timeout, cancellation, compaction, or prompt revision.
Tool execution is at-least-once from the model/runtime perspective. Some providers support idempotency keys; some do not.
We control a small intent log in the agent process or orchestrator. We do not control the external API's internal retry behavior.
A side effect is any operation that changes state outside the agent: charge, email, ticket, deploy, message, refund.
Read-only lookups may bypass the commit path, but only if they are provably pure under retries.

Sequence model: intent before effect

sequenceDiagram
  participant U as UserIntentScope
  participant M as Model/Planner
  participant L as IntentLog
  participant X as Executor
  participant E as ExternalAPI
  U->>M: ask for outcome, not raw tool id
  M->>L: propose canonical intent + semantic_key
  L-->>M: accepted / duplicate / conflict
  M->>X: dispatch accepted record with fence token
  X->>E: execute with idempotency key if supported
  E-->>X: ack / timeout / explicit failure
  X->>L: mark ACKED, FAILED, or UNKNOWN
  Note over L,X: UNKNOWN never blindly retries;
  Note over L,X: it enters reconcile-or-compensate

State machine:

PROPOSED -> RECORDED -> DISPATCHED -> ACKED
                 |           |
                 |           -> UNKNOWN -> RECONCILING -> ACKED | COMPENSATED | DEAD_LETTER
                 -> REJECTED_DUPLICATE
                 -> REJECTED_CONFLICT

The important move is that the model proposes; the log decides. semantic_key is derived from canonicalized tool name, arguments, user-intent scope, and a monotonic attempt epoch. The tool-call id from the model is evidence, never identity.

Minimal runnable fixture

This Python fixture is stdlib-only. It is a simulator, not a production claim. Run it locally, in CI, or on any free server you already have access to.

# tool_commit_sim.py
from dataclasses import dataclass
from enum import Enum
import hashlib, json, random

class S(Enum):
    PROPOSED=1; RECORDED=2; DISPATCHED=3; ACKED=4; UNKNOWN=5
    RECONCILING=6; COMPENSATED=7; DEAD=8; REJ_DUP=9; REJ_CONFLICT=10

def canon(tool, args, scope):
    return hashlib.sha256(json.dumps({"t":tool,"a":args,"s":scope}, sort_keys=True).encode()).hexdigest()

@dataclass
class Rec:
    key: str; tool: str; args: dict; state: S=S.PROPOSED; fence: int=0

class External:
    def __init__(self): self.effects=[]; self.known={}
    def apply(self, rec, idem):
        # external API honors idempotency only when caller supplies a stable key
        if idem in self.known: return self.known[idem]
        self.effects.append((rec.key, rec.tool, json.dumps(rec.args, sort_keys=True)))
        self.known[idem] = "ok"
        return "ok"

class System:
    def __init__(self):
        self.log={}; self.ext=External(); self.fence=0
    def propose(self, model_id, tool, args, scope):
        key=canon(tool,args,scope)
        if key in self.log and self.log[key].state in {S.RECORDED,S.DISPATCHED,S.ACKED,S.UNKNOWN,S.RECONCILING}:
            self.log[key].state = S.REJ_DUP if self.log[key].state==S.PROPOSED else self.log[key].state
            return self.log[key], False
        rec=Rec(key,tool,args,S.RECORDED); self.log[key]=rec; return rec, True
    def dispatch(self, rec):
        self.fence += 1; rec.fence=self.fence; rec.state=S.DISPATCHED
    def finish_unknown(self, rec):
        rec.state=S.UNKNOWN
    def reconcile(self, rec, external_has_effect):
        # In production this queries provider status by idempotency key or safe read.
        rec.state = S.ACKED if external_has_effect else S.RECORDED
    def execute(self, rec):
        idem = rec.key  # stable across model retries because it is not the model's tool-call id
        return self.ext.apply(rec, idem)

def run(seed, inject):
    rng=random.Random(seed); sys=System(); accepted=0
    scope="user:42:checkout:7"
    proposals=[("m1","refund",{"amount":20}),("m2","refund",{"amount":20})]  # same intent, new model id
    for mid,tool,args in proposals:
        rec,ok=sys.propose(mid,tool,args,scope)
        accepted += bool(ok)
        if not ok: continue
        sys.dispatch(rec)
        if inject=="ack_loss" and mid=="m1":
            sys.execute(rec); sys.finish_unknown(rec)      # effect happened, ack lost
            sys.reconcile(rec, external_has_effect=True) # recovery uses log, not model memory
        elif inject=="crash_before_dispatch" and mid=="m1":
            rec.state=S.RECORDED                           # crash before effect; safe retry
            sys.dispatch(rec); sys.execute(rec)
        else:
            sys.execute(rec)
    effects=len(sys.ext.effects)
    return {"seed":seed,"inject":inject,"accepted":accepted,"effects":effects,"states":{r.key[:8]:r.state.name for r in sys.log.values()}}

if __name__=="__main__":
    for inj in ["ack_loss","crash_before_dispatch","none"]:
        print(run(7, inj))

Suggested check: for every injected failure class and every seed you choose, effects <= 1 for the same (tool,args,scope); any UNKNOWN must resolve through reconcile, not through another model guess. Treat N=2000 seeds as a starting denominator for your own run, not as a result claimed here. Acceptance rule: violations == 0 across the chosen seed set; any duplicate effect fails the canary even if latency is excellent.

Failure classes to inject

Failure class	Event order	Common bug	Required behavior
Ack loss	effect applied, ack dropped, model retried	model creates fresh id and repeats	dedupe by canonical intent; reconcile UNKNOWN
Crash before dispatch	RECORDED then process dies	on restart, planner re-proposes from chat history	reload RECORDED intent; resume or safely reject
Reorder	two equivalent proposals race	last writer wins, both dispatch	single RECORDED record; loser becomes duplicate/conflict
Provider non-idempotent	retry reaches external twice	client assumed exactly-once	compensation path or dead-letter, not silent retry
Prompt revision	same user goal, changed wording	canonicalization too literal	version the canonicalizer; conflict instead of merge

Testable properties

P1 uniqueness: count external effects grouped by semantic_key; max must be one.
P2 no blind retry: after UNKNOWN, the next transition must be RECONCILING, COMPENSATED, or DEAD_LETTER, never fresh DISPATCHED from model memory.
P3 pre-dispatch cancellation: cancellation before DISPATCHED must be able to end in REJECTED_CONFLICT or RECORDED without touching the external API.
P4 recovery from log: delete in-memory planner state between events; the intent log alone must decide reject/replay/compensate.
P5 model independence: swap the model or prompt template; property outcomes must not change for the same canonical intent.

Where free model access helps without changing the protocol

Free model access is useful for breadth: generate paraphrases, stale ids, partial JSON, duplicate tool calls, contradictory arguments, and cancellation-like transcripts. A free server option is useful for repeatability: keep the harness running while you vary seeds and failure injectors. The limitation is that availability, quotas, model behavior, and duration are outside this design; treat them as replaceable capacity. If you want a low-friction place to try the same harness, MonkeyCode is one option I have access to, but the acceptance gate should follow you to any backend.

Tradeoffs

Design choice	Latency	Throughput	Correctness	Cost	When it hurts
Trust model tool-call id	lowest	high	weak under retry	low	any timeout or compaction
Client-side canonical intent log	+small write	medium	strong for single orchestrator	storage	multi-writer without consensus
Provider idempotency key only	low	high	provider-dependent	request churn	providers with weak key semantics
Saga/compensation for UNKNOWN	higher	lower	strong with explicit reversal	operational complexity	non-compensable actions like sent email
Human approval gate	highest	low	strong for high blast radius	attention	high-volume low-risk calls

Limitations and who should not use this

Do not use client-side dedupe as your only guard when several orchestrators can act for the same user intent; you need a shared log, leader, or database uniqueness constraint. Do not rely on reconcile for non-compensable side effects unless the provider exposes a safe status read. Do not put regulated money movement, medical actions, or destructive infrastructure changes behind a simulator-grade gate without a reviewed state store, access control, audit trail, and real provider contracts. Very low-volume internal scripts may not need this machinery; a cron job with a database unique constraint may be simpler and more honest.

Validation path

Canonicalize one tool and one intent scope; write a property test that fails on duplicate effects.
Add the simulator to CI with a fixed seed set and an explicit denominator.
Inject ack_loss, crash_before_dispatch, reorder, and provider_non_idempotent.
Delete planner memory between events to prove recovery comes from the log.
Only then increase model diversity, parallelism, or realtime routing.

Counterexample question to keep the design honest: if the order is DISPATCHED -> timeout -> crash -> model regenerates same intent as new id -> replay arrives before reconcile completes, should the system reject the late replay as a duplicate, replay it through the same semantic_key, or compensate the first effect? If your answer depends on the model remembering anything, the invariant is not yet real.

Make Timeout-Retry Safety an Invariant Before You Route Across Free Model Endpoints

Robin — Fri, 31 Jul 2026 10:37:35 +0000

A counterexample first. Your agent calls a model endpoint, the request times out at the client, and your retry logic fires the same "apply this patch, then mark the task done" request again. What actually happened upstream: the first request succeeded — the model produced its output, your tool executor applied the patch — but the response died in the network. Your retry now applies the patch a second time. If the patch is append migration row, you have two rows. If it is git commit, you have two commits. The system did not fail loudly; it converged to a wrong state silently.

This is the event order that matters:

client              endpoint            executor
  |--- request R1 --->|                    |
  |                   |--- apply patch --->|  (side effect S1 commits)
  |<-- response lost  |                    |
  |--- retry R1' ---->|                    |
  |                   |--- apply patch --->|  (side effect S2 commits)
  |<--- 200 OK -------|                    |

The common implementation — "retry on timeout with exponential backoff" — preserves availability and violates the invariant you actually need:

Invariant: For any logical operation op with idempotency key k, the set of committed side effects attributable to k has cardinality at most one, regardless of how many times a request carrying k is delivered.

This matters more, not less, when you route work across cheap or free model endpoints, because heterogeneous endpoints have heterogeneous timeout behavior, and every distinct timeout profile multiplies the number of retry-after-uncertain-outcome events your system produces.

Declared assumptions

Requests can be lost, delayed, or duplicated; responses can be lost after the server commits. No exactly-once delivery exists, so the invariant must live in the effect layer, not the transport.
You control a thin executor between the model call and the side effect (a tool runner, a job worker). If you do not, this design does not apply.
Side effects are attributable to a single logical operation. Fan-out effects need one key per effect.

The protocol, minimally

The executor keeps a durable record keyed by k with a state machine:

RECEIVED --claim--> IN_PROGRESS --commit--> COMMITTED
                    IN_PROGRESS --crash---> (re-claimed only via fencing check)

A second delivery of k in state COMMITTED returns the recorded result without re-executing. A delivery in IN_PROGRESS must fence: it checks a monotonically increasing claim token and refuses to execute against a stale claim. That fencing step is what most hand-rolled idempotency code omits, and it is exactly what a crash between claim and commit requires.

# Minimal property-test harness (stdlib only). Labeled: this is a design
# fixture, not production code. Run: python3 idempotency_sim.py
import itertools, random

class Executor:
    def __init__(self):
        self.store = {}          # k -> (state, claim_token, result)
        self.effects = []        # committed side effects (the ground truth)
        self.token = 0

    def execute(self, k, effect, crash_before_commit=False):
        state = self.store.get(k)
        if state and state[0] == "COMMITTED":
            return state[2]                      # replay, no new effect
        self.token += 1
        claim = self.token
        self.store[k] = ("IN_PROGRESS", claim, None)
        if crash_before_commit:
            # simulate crash: a later duplicate must fence against `claim`
            return self._recover(k, claim, effect)
        self.effects.append(effect)
        self.store[k] = ("COMMITTED", claim, effect)
        return effect

    def _recover(self, k, stale_claim, effect):
        # Re-claim with a new token; stale claim cannot commit.
        self.token += 1
        if self.token <= stale_claim:
            raise AssertionError("fencing violation")
        self.effects.append(effect)
        self.store[k] = ("COMMITTED", self.token, effect)
        return effect

def effects_for(ex, k):
    # In this fixture each effect records its key: count attributable effects.
    return sum(1 for e in ex.effects if e[0] == k)

random.seed(7)
for trial in range(2000):
    ex = Executor()
    k = f"op-{trial}"
    # adversarial schedule: 1-5 deliveries, random crashes, random dupes
    for _ in range(random.randint(1, 5)):
        ex.execute(k, (k, "commit"), crash_before_commit=random.random() < 0.4)
    assert effects_for(ex, k) == 1, f"invariant broken on trial {trial}"
print("invariant held across 2000 adversarial schedules")

The property under test is the invariant itself: under randomized duplicate delivery and crash points, the committed-effect count per key is exactly one. If you remove the fencing branch, the harness finds a violating schedule within a few dozen trials — that is the counterexample generator doing its job.

Failure classes to inject before you trust it

#	Injected failure	What it catches
1	Response loss after commit	Naive retry re-executes the effect
2	Duplicate delivery, no crash	Missing dedupe on `COMMITTED`
3	Crash between claim and commit	Missing fencing token; zombie claim double-commits
4	Same key, different payload	Key derivation that ignores the payload silently replays the wrong result
5	Timeout shorter than p99 execution	Retries pile up while the first attempt is still legitimately running

Failure class 5 is the one free-tier endpoints make acute: queueing variance is higher, so a client timeout tuned for a fast paid endpoint will manufacture spurious retries all day.

Where free model access fits this workflow

Fault-injection harnesses are token-hungry: you want thousands of adversarial runs, not three. This is where I ran the evaluation loop against MonkeyCode, which offers free model access and a free server option — enough to run the router-plus-executor against a real model endpoint with artificial latency and dropped responses, rather than only against a stub. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness above is fully self-contained and does not depend on any specific provider; the live-endpoint variant just swaps the stub execute for an HTTP call behind a fault-injecting proxy.

If you want to reproduce the live version: put a tiny proxy in front of the endpoint that (a) injects 200–2000 ms latency, (b) drops 10% of responses after forwarding the request, and (c) logs every delivery. Point your router at it, run class-1 and class-3 faults, and assert the same cardinality invariant on the executor's effect log. If you are evaluating MonkeyCode for this, the free tier is a reasonable place to run exactly that loop — the acceptance criterion below is the whole test.

Tradeoff table

Design choice	Latency	Correctness	Cost	When it wins
Retry on timeout, no dedupe	Lowest happy-path	Violates invariant on response loss	Lowest	Pure read-only calls only
Idempotency key + committed-state replay	+1 store read per call	Holds for classes 1, 2	Small durable store	Default for agent tool calls
Key + fencing token (above)	+1 store write on recovery	Holds for classes 1–4	Store with compare-and-swap	Any effect you cannot undo
Distributed transaction / 2PC	Highest	Strongest	Highest, blocks on failure	Rarely justified at this layer

Explicit denominator: the latency cost is one durable read per tool call — measurable in single-digit milliseconds against a local store — against a failure that silently corrupts state. At agent-loop scale (hundreds of tool calls per task), that is a cheap insurance premium, not a bottleneck. Measure it against your own effect latency before believing either number.

Acceptance rule

Ship the routing change only if: under fault classes 1–4 injected at 10% rates over at least 1,000 logical operations, the executor's committed-effect log contains exactly one effect per idempotency key, and duplicate deliveries return the recorded result without re-execution. Any violation blocks the rollout; there is no "mostly idempotent."

Limitations and who should not use this

If your side effects are not attributable to a single key (bulk fan-out, cross-system sagas), you need per-effect keys or a compensation protocol, not this record.
If the endpoint itself performs the side effect opaquely and you cannot interpose an executor, client-side keys do nothing — the dedupe must live where the effect commits.
Free tiers are appropriate for evaluation harnesses and canary traffic, not as the correctness foundation: quotas, latency variance, and availability of any free endpoint can change, so the invariant must hold assuming the endpoint misbehaves. Do not build the production path such that it only works when the cheap endpoint is healthy.

The counterexample question

Look at your own executor and ask: which event order breaks the invariant — response loss after commit, crash between claim and commit, or duplicate delivery with a mutated payload? Once you find it, decide explicitly: should the system reject the duplicate (return the recorded result), replay it (re-execute only if no effect committed), or compensate (undo and redo)? If you cannot point to the branch of code that answers that question, the answer today is "whatever the network decides."

Canary Idempotent Webhooks With Duplicate, Timeout, and Reorder Gates

Robin — Fri, 31 Jul 2026 09:04:18 +0000

A payment webhook once ack=timeout -> retry -> reorder -> duplicate ack can look healthy in logs while applying the same intent twice. The common handler invariant — “return 200 fast, then process asynchronously” — is not sufficient unless a stronger invariant holds:

Invariant: for one intent_id, at most one externally visible effect may commit, no matter how many deliveries, retries, or reorderings occur.

This article gives a small canary workflow you can run before scaling webhook traffic. It uses a free model option and free server option from MonkeyCode only as a low-friction place to generate adversarial traces and execute a disposable harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming specific model names, quotas, hardware, duration, permanence, or benchmark results; treat free access as an operator-supplied availability option, not a dependency of the method.

Assumptions

One producer emits business intents with stable intent_id; delivery to the consumer is at-least-once.
The consumer can commit an externally visible effect: charge, email, provisioning call, ledger write, or ticket transition.
Transport may duplicate, delay, reorder, and lose acknowledgements. It does not forge new intent_id values.
The effect provider may be idempotent, but the handler must not assume that unless verified by a contract test.
Clocks are advisory only; correctness must come from state transitions, not timestamps.

Sequence model and counterexample

producer          broker           consumer           effect_provider
   |  intent A       |                 |                    |
   |---------------->|  deliver A d1   |                    |
   |                 |---------------->| begin A            |
   |                 |                 |------------------->| apply A
   |                 |   ack d1 lost   |                    |
   |                 |<----------------| timeout            |
   |                 |  deliver A d2   |                    |
   |                 |---------------->| begin A?           |

A violating order is simple: begin(A) -> apply(A) -> ack_lost -> redeliver(A) -> begin(A) again -> apply(A) again. If begin is only “received HTTP 200”, d2 has no memory that A already reached a committed effect.

The design invariant is not “no duplicate delivery.” That is impossible under retries. The invariant is no duplicate committed effect.

Minimal simulator

This is intentionally small enough to run in CI or on an ephemeral free server. It models an effect commit as the dangerous boundary and injects duplicate, timeout, reorder, and crash-before-ack events.

from dataclasses import dataclass
from itertools import permutations

@dataclass(frozen=True)
class Event:
    kind: str
    intent: str
    delivery: int

EFFECTS_COMMITTED = []
DEDUP = {}

def handle(evt, state):
    key = (evt.intent,)
    record = state.setdefault(key, {"seen": set(), "committed": False})
    record["seen"].add(evt.delivery)

    if evt.kind == "deliver":
        # Unsafe handler: every delivery may commit.
        EFFECTS_COMMITTED.append(evt.intent)
        return "acked"

    if evt.kind == "deliver_idempotent":
        if not record["committed"]:
            EFFECTS_COMMITTED.append(evt.intent)
            record["committed"] = True
        return "acked"

    if evt.kind == "ack_lost":
        return "retry_scheduled"

    raise ValueError(evt.kind)

def effects_for(intent):
    return sum(1 for x in EFFECTS_COMMITTED if x == intent)

def check(trace, handler_kind):
    EFFECTS_COMMITTED.clear()
    state = {}
    for evt in trace:
        handle(evt, state)
    assert effects_for("A") <= 1, f"invariant violated: {trace}"

base = [
    Event("deliver_idempotent", "A", 1),
    Event("ack_lost", "A", 1),
    Event("deliver_idempotent", "A", 2),
]

# Exhaust small permutations of a bounded trace. Denominator: 6 orders.
for p in permutations(base):
    check(list(p), "deliver_idempotent")

print("ok: bounded orders preserve at-most-one committed effect")

The unsafe version is the same trace with deliver instead of deliver_idempotent; it fails as soon as two delivery events are both executed. That is the point: retries are normal, so the commit gate must be durable before the effect, not after the HTTP response.

Failure classes to canary

Run the same intent through four gates before enabling the handler for all traffic:

Gate	Injected class	Must hold	Common failure
Duplicate	same `intent_id`, same payload, twice	one committed effect	unique constraint only after side effect
Ack-loss retry	commit succeeds, response times out	retry sees committed state	handler re-executes because ack was lost
Reorder	cancel arrives before create for same aggregate	reject or buffer by state machine	last-write-wins by arrival time
Crash window	process dies between intent record and effect call	recovery replays or compensates exactly one path	“in-flight” state has no owner

A practical workflow is: generate 20 candidate traces with a coding assistant, keep only traces that name the state before and after each event, execute them against the simulator, then promote survivors into provider contract tests. MonkeyCode's free model access can help brainstorm trace candidates, and its free server option can host the disposable runner if you do not want to touch shared CI; the correctness criteria still come from your invariant, not from the tool.

Testable properties

Use property-style assertions rather than fixed expected outputs:

At-most-one effect: count(committed_effects(intent_id)) <= 1.
Progress: if a delivery is accepted and dependencies are reachable, the intent reaches committed, rejected, or compensated within N attempts.
No orphan in-flight: every intent row has a terminal state or a live lease owner after simulated crash.
Order independence for terminal states: permuting duplicate and ack-loss events cannot change the final committed set.
Explicit denominator: report violations over all bounded permutations, not “we retried once in staging.”

Example acceptance rule: for 3 delivery events plus one ack-loss and one crash event, exhaust all 120 bounded permutations; zero traces may commit two effects; at least 95% must terminate without manual repair, and the remaining 5% must terminate as explicit needs_compensation, never silent duplicate.

Tradeoffs

Design choice	Latency	Throughput	Correctness	Cost/ops risk
Synchronous commit before 200	Higher	Lower	Strongest simple gate	Provider outage blocks ack
Durable inbox + worker	Medium	High	Good if state machine is strict	More moving parts
Rely on provider idempotency	Low	High	Only as good as provider contract	Hard to verify across vendors
Deduplicate in memory	Lowest	Highest	Fails on restart and multi-instance	Unsafe beyond demo

Limitations and who should not use this

Do not use a tiny permutation simulator as proof for unbounded queues, multi-region active-active writes, or effects that cannot be made atomic or compensable. It also will not save a handler whose external API has no idempotency key, no queryable outcome, and no rollback path. If the business effect is irreversible and non-partitionable, use a human approval gate or a different protocol, not more retries.

If you want a starting point, fork the simulator, replace EFFECTS_COMMITTED.append with a staging sandbox call, and run the four gates before any real webhook rollout. The useful habit is tool-independent: canary the failure classes before you trust the happy path.

Final counterexample question: which event order breaks your invariant — ack_lost before commit, reorder after commit, or crash between reserve and apply — and should the system reject, replay, or compensate that order?

"Test Agent Harnesses as Replaceable Protocol Executors"

Robin — Wed, 29 Jul 2026 09:47:54 +0000

Executor A times out after committing, then Executor B retries from an old snapshot. Both report plausible success. A harness swap exposed that the system had an implementation convention, not a protocol invariant.

What the announcement establishes

GitHub’s July 27 article discusses the agent harness as an important execution layer. Harness replaceability is useful only when protocol behavior is specified; this simulator is an independent validation design, not a benchmark of GitHub products. Read the primary source.

For test track 9.1, the engineering claim here is narrower than the announcement: the surrounding workflow needs a contract that remains valid when metadata, transport, people, or executors change.

The artifact

# illustrative, unexecuted simulator sketch
class Exec:
 def __init__(self,name): self.name=name
 def run(self,job,store,fault=None):
  if job["base"] != store["head"]: return "STALE"
  if job["id"] in store["done"]: return "DUPLICATE"
  store["head"] += "+"+self.name; store["done"].add(job["id"])
  if fault == "crash_after_commit": raise RuntimeError("crash")
  return "COMMITTED"

The model is intentionally tiny. Add an acknowledgment ledger so a crash after commit can reconcile by job ID instead of applying twice.

For test track 9.2, this is a design fixture, not executed code. Pin language and dependency versions before turning it into a repository test, and replace example identities and timestamps with disposable values.

Success path

Both executors accept the same envelope, reject stale bases, and converge on one committed effect. Their logs differ, but externally visible states satisfy identical invariants.

For test track 9.3, a successful demonstration records inputs, policy or schema version, decision, and final identifier. It does not infer correctness from a confidence label, status badge, or fluent output.

Failure path and regression plan

Inject timeout before work, timeout after commit, duplicate delivery, stale snapshot, malformed result, and crash during acknowledgment. Any second effect or false terminal success fails conformance.

Generate event orders and assert: at most one effect per job ID; terminal results never regress; stale work cannot commit; uncertain acknowledgments are reconciled; cancellation cannot erase a completed effect.

For test track 9.4, the acceptance gate is binary: the negative fixture must produce no unauthorized or duplicate side effect, while the positive fixture must remain traceable to its initial evidence. Expected output should be documented before execution.

Cleanup and rollback

Canary the second executor on read-only fixtures, stop routing on the first invariant violation, drain leases, reconcile uncertain IDs, and retain the original executor until state equivalence is demonstrated.

For test track 9.5, cleanup must preserve enough sanitized evidence to distinguish cancellation, rejection, stale work, and successful completion. Never solve recovery by silently marking an uncertain operation successful.

Limitations

For test track 9.6, this article proposes a compact engineering exercise and reports no execution results. It does not evaluate service availability, security, accessibility conformance, productivity, or comparative quality. Product previews can change, and a local fixture cannot reproduce every hosted-system failure.

A practical development environment

For implementing an independent simulator, MonkeyCode is an open-source AGPL-3.0 AI development platform with an overseas hosted option. It includes a managed server-side cloud development environment, integrated models, task and requirement management, and build, test, and preview workflows. It is free to start. These statements do not mean the GitHub or OpenAI capability discussed above exists in MonkeyCode. Check the console for current quotas, models, regions, duration, and pricing before planning work. Open the campaign workspace.

Disclosure: This article promotes MonkeyCode using an official campaign link. I’m a MonkeyCode user, not affiliated with the project, and I receive no commission from this link.

AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.