Treat a Model Upgrade as a Replay Protocol, Not a Benchmark Score

A new open-weight model drops — this week it's MiniMax's latest release lighting up the timeline — and the same event order plays out on a hundred teams at once:

1. Someone runs five hand-picked prompts against the new model.
2. The outputs look better.
3. The router config flips: 10% of production traffic goes to the new model.
4. Three days later, a long-running agent workflow corrupts state because the new model emits a subtly different tool-call shape at turn 9 — a shape that only appears under a specific conversation length distribution.

The common implementation fails to preserve a simple invariant:

Invariant: A routing promotion decision must be reproducible. Given the same recorded production traffic, replayed under the same conditions, the candidate model must satisfy the same contract the incumbent satisfies — before it earns any live traffic.

A benchmark score is a point estimate on someone else's traffic. Your traffic is the only denominator that matters. This article builds the replay protocol I use before any model version earns production traffic, and shows where a free model access tier plus a free server option (I'm using MonkeyCode's for this write-up) removes the usual excuse for skipping it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Assumptions

• You have an incumbent model M0 serving an agent or LLM workflow, and a candidate M1 (a new release, a fine-tune, or a cheaper endpoint).
• You can record request/response traces at the model-call boundary. If you can't, that's the real first project.
• Your downstream consumers (tool parsers, validators, state machines) define a contract: schema shape, token budget, termination conditions, tool-call arity.
• Replay traffic is either synthetic or sanitized; no PII leaves your boundary.
• The candidate runs somewhere cheap enough that cost is not the reason you skip replay. This is where free model access and a free server option matter — I'll be concrete about that below.

The state model

Treat promotion as a state machine, not a decision:

                replay_passes(contract)
   CANDIDATE ───────────────────────────► SHADOW
      ▲                                      │
      │ replay_fails(contract)               │ shadow_delta ≤ ε over N traces
      │                                      ▼
   REJECTED ◄────────────────────────── CANARY(5%)
      ▲                                      │
      └──────────── canary_violation ────────┤
                                             │ no violation over window W
                                             ▼
                                         PROMOTED

The violating event order at the top of this article jumps straight from CANDIDATE to CANARY. The replay stage exists because it is deterministic and free to rerun: same traces, same prompts, checkable properties. Shadow and canary stages catch what replay cannot (latency under load, queue interactions), but replay catches what they're bad at: contract violations that need specific, rare conversation shapes.

Why replay is not "just evals"

Standard eval harnesses score outputs. Replay checks invariants under event-order perturbation. Three properties I require before promotion:

1. Contract conformance. Every response parses against the downstream schema. Not "mostly parses" — the parser is total or the promotion is dead.
2. Termination equivalence. For agent loops, M1 must reach a terminal state within the same turn budget as M0 on the same trace prefix. A model that rambles one extra turn on 3% of long conversations is a queue-depth incident at scale.
3. Tool-call stability. Same tool selected with semantically equivalent arguments on replayed prefixes. Reordering is fine; inventing a new argument shape is not.

Note what is deliberately absent: a quality score. Quality scoring is a separate, fuzzier problem. Promotion gating should first establish that the candidate doesn't break the system. Whether it's better is a second experiment with its own denominator.

The artifact: a minimal replay checker

This is a stripped-down version of my harness. It replays recorded traces against a candidate endpoint and checks the three properties. It is framework-agnostic — point it at any OpenAI-compatible endpoint.

import json, re, time
from dataclasses import dataclass, field

@dataclass
class Trace:
    trace_id: str
    messages: list          # full conversation prefix sent to the model
    incumbent_response: str # what M0 actually returned in production
    turn_budget: int        # turns M0 consumed for the full episode

@dataclass
class Violation:
    trace_id: str
    kind: str               # CONTRACT | TERMINATION | TOOL_SHAPE
    detail: str

TOOL_CALL_RE = re.compile(r'"tool":\s*"([a-z_]+)"')

def check_contract(resp: str, schema_ok) -> bool:
    return schema_ok(resp)

def tool_signature(resp: str) -> list:
    return sorted(TOOL_CALL_RE.findall(resp))

def replay(traces, call_model, schema_ok, max_turn_slack=1):
    violations = []
    n = len(traces)
    for t in traces:
        resp, turns_used = call_model(t.messages)
        if not check_contract(resp, schema_ok):
            violations.append(Violation(t.trace_id, "CONTRACT", "parse failed"))
        if turns_used > t.turn_budget + max_turn_slack:
            violations.append(Violation(t.trace_id, "TERMINATION",
                f"{turns_used} turns vs budget {t.turn_budget}"))
        if tool_signature(resp) != tool_signature(t.incumbent_response):
            violations.append(Violation(t.trace_id, "TOOL_SHAPE",
                "tool selection diverged on identical prefix"))
    return {
        "denominator": n,
        "violations": violations,
        "violation_rate": len(violations) / max(n, 1),
    }

Two deliberate choices:

• denominator is printed first. A "0.4% violation rate" is meaningless until you know whether it's 2/500 or 2/40. Every report from the harness leads with N.
• call_model is injected. In the real harness this wraps the client for whatever endpoint hosts the candidate. For this article's run, the candidate side was MonkeyCode's free model access, and the harness itself ran on their free server option — the entire replay stage cost nothing, which matters because replay is the stage teams skip "to save budget." If replay is free, the excuse is gone. (Verify current availability and limits before depending on any free tier; treat it as an evaluation resource, not production infrastructure.)

Failure injection: replay is where you get to be cruel

Replay's superpower is that you can mutate traces to manufacture the event orders production hasn't produced yet. Before promoting any new model release, I inject:

Failure class	Mutation	Invariant under test
Truncated context	Cut the prefix to 60% of original length	Contract conformance must not degrade silently; model must ask or fail loudly
Duplicated user turn	Repeat the last user message	Tool-call idempotency: no duplicate side-effecting call
Adversarial length	Pad to 95% of context window	Termination within budget
Schema bait	Prefix ends mid-JSON in assistant turn	Model must not emit unparseable continuation

A candidate that passes clean traces but fails duplicated-turn injection will eventually double-charge a customer or double-send an email. You found that out for the price of a replay run instead of a postmortem.

Tradeoffs

Approach	Catches	Misses	Cost
Public benchmark scores	Broad capability ranking	Your traffic, your contract, your failure modes	Free
Live canary only	Real latency/load behavior	Rare conversation shapes; violations arrive as production damage	High risk
Replay protocol (this article)	Contract/termination/tool-shape regressions, injected edge cases	True load behavior, novel traffic patterns	Compute for replay; zero if run on a free tier
Replay + shadow + canary	All of the above, staged	Nothing is free: longest time-to-promotion	Moderate

Acceptance rule I use: promotion to canary requires replay over ≥ 1,000 production traces (or 7 days of traffic, whichever is larger) with violation rate < 0.5% and zero injected-failure violations in the duplicated-turn class. Anything less and the candidate stays in CANDIDATE.

Limitations and who should not do this

• Replay assumes recorded traces are representative. If your traffic shifted since recording, you're validating against a ghost. Re-record.
• Deterministic replay is approximate: temperature, provider-side changes, and non-deterministic tool execution all introduce noise. Treat borderline violations as signals, not verdicts.
• If your workflow has no parseable contract at the model boundary — pure free-form text consumed by humans — contract replay buys you little; you want preference-based shadow evaluation instead.
• A free server tier is the right home for the harness and candidate endpoint during evaluation. Do not architect production traffic around a free tier; quotas and availability can change without notice.

Validate it yourself

1. Record 200 traces from your incumbent model this week (sanitized).
2. Run the replay checker against any candidate endpoint. The code above is complete enough to start; swap in your schema validator.
3. Inject the duplicated-turn mutation and count how many candidates silently emit a second side-effecting tool call.

If you want a zero-cost sandbox for steps 2–3, MonkeyCode's free model access and free server are what I used for this write-up — the whole point is that the replay stage should never be gated on budget approval.

The counterexample question

Every promotion pipeline has an event order that breaks its invariant. For this one: a trace that passes replay, passes canary, and then fails in week three because production conversation lengths drifted past anything in the recorded corpus. When that event order arrives, should your system reject the promotion (longer recording windows), replay continuously against live-sampled traces, or compensate with a contract-validating proxy that can kill a bad response before it reaches the state machine? I'd argue the compensating proxy is the only answer that survives drift — but I'd like to hear the event order that breaks yours.