Make a MiniMax H3 Claim a Replay Invariant Before It Changes Your Model Queue

A platform team ran a model router. The router used a cheap primary model and an expensive fallback. A request that timed out would escalate. A request that returned malformed JSON would escalate too. The team wanted the cheapest reliable primary. That constraint created a fragile equilibrium.

One morning a model release started circulating. To keep the example concrete, call it MiniMax H3. The announcement claimed lower cost and stronger benchmarks. The team's chat filled with screenshots. An engineer opened a pull request. It swapped the primary model. The on-call engineer stopped the change. The release was a claim, not a migration order.

The team listed assumptions before touching the router.

• Traffic is nonuniform.
• Benchmark averages hide tail behavior.
• A candidate can win on cost while losing on escalation rate.
• A release announcement is not a benchmark run.

The team needed a cost-controlled baseline. They used MonkeyCode's free model access and a free server instance to host the comparator. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That setup kept the evaluation from spending production budget before a decision.

The core model was small. Each request traversed a path.

request -> router -> primary
router -> fallback on timeout or malformed output
comparator replays trace -> baseline and candidate

A comparator replayed the same request trace through a baseline and a candidate. The baseline used the team's current primary model, exposed through the free MonkeyCode tier. The candidate pointed at the external release. The comparator recorded four values per outcome: success, latency, token count, and error type.

import asyncio
from dataclasses import dataclass

@dataclass
class Request:
    id: str
    prompt: str
    max_tokens: int

@dataclass
class Response:
    text: str
    tokens: int
    latency_ms: int

@dataclass
class Outcome:
    model: str
    ok: bool
    latency_ms: int
    tokens: int
    error: str = ''

class ReplayComparator:
    def __init__(self, baseline, candidate):
        self.baseline = baseline
        self.candidate = candidate

    async def run(self, trace, inject):
        results = []
        for req in trace:
            for name, client in [('baseline', self.baseline), ('candidate', self.candidate)]:
                try:
                    if inject.get(name) == 'timeout':
                        await asyncio.sleep(0.05)
                        raise TimeoutError('injected timeout')
                    resp = await client.complete(req)
                    results.append(Outcome(name, True, resp.latency_ms, resp.tokens))
                except Exception as exc:
                    results.append(Outcome(name, False, 0, 0, type(exc).__name__))
        return results

The next step was property checking. The team did not compare benchmark scores. They compared invariants on the same trace. A candidate could pass only if it did not increase errors, did not blow the budget, and held tail latency under the router timeout.

async def test_properties(results, candidate_timeout_ms):
    base_errors = sum(1 for r in results if r.model == 'baseline' and not r.ok)
    cand_errors = sum(1 for r in results if r.model == 'candidate' and not r.ok)
    assert cand_errors <= base_errors, 'candidate errors exceed baseline'

    base_cost = sum(r.tokens for r in results if r.model == 'baseline')
    cand_cost = sum(r.tokens for r in results if r.model == 'candidate')
    assert cand_cost <= base_cost * 1.05, 'candidate cost exceeded 5% limit'

    cand_latencies = [r.latency_ms for r in results if r.model == 'candidate' and r.ok]
    tail_latency = sorted(cand_latencies)[int(len(cand_latencies) * 0.95)]
    assert tail_latency <= candidate_timeout_ms, 'candidate tail latency exceeds router timeout'

    return {'base_errors': base_errors, 'cand_errors': cand_errors,
            'base_cost': base_cost, 'cand_cost': cand_cost,
            'candidate_tail_latency_ms': tail_latency}

The team injected failure modes before accepting any result. Each failure class had a matching invariant.

Failure class	Injected behavior	Acceptance gate
Timeout	Sleep and raise TimeoutError	Candidate tail latency stays under router timeout
Duplicate delivery	Run the same request twice	Success result must come from one logical completion
Truncated output	Return fewer tokens	Valid JSON and minimum token count still hold
Rate limit	Raise an HTTP 429	Router must escalate without adding retries
Cost overrun	Force token growth	Candidate cost stays within 5% of baseline

The tradeoff table forced the team to state what they were optimizing.

Decision	Result	Risk
Replay on free baseline	No production spend, reproducible trace	Baseline may drift from live traffic
Replay on live samples	Real traffic distribution	Cost and privacy risk, harder to reproduce
Benchmark-score comparison	Fast, seductive	Hides tail latency and escalation behavior

The open part mattered. The team attached the comparator to version control. The free server meant another engineer could run the same trace without secret budget approvals. The external model remained an environment variable. That was the open-source spirit here. It was not a slogan. It was a testable artifact that separated a vendor claim from a routing decision.

The team did not migrate on the first day. They ran the trace, injected failures, and checked the invariant. They recorded the acceptance rule in the repository. If a future release changed the queue, the same comparator would judge it.

Which event order breaks the invariant? A request times out on the candidate. The router retries the primary. The fallback completes after the retry. The system now has two success stories for one logical request. It must either reject the duplicate, replay the topic, or compensate the fallback. If the new model only wins on that duplicated traffic, it has not won the invariant.