Canary a Free-Tier Model Promotion With Rate Limits, Truncation, and a Shadow Gate

A violating event order first

Here is a sequence I have seen cause a quiet production regression when a team switched an agent backend to a free model endpoint to cut cost:

t1  router shifts 100% of traffic to the free endpoint
t2  free endpoint starts returning 429s under burst load
t3  client retries with exponential backoff; retry storm triples offered load
t4  endpoint truncates long completions to stay within its throughput budget
t5  agent parses a truncated JSON tool call, silently falls back to a default action
t6  dashboards show "success": latency recovered, error rate zero

Every individual component behaved as designed. The system-level invariant that broke was never stated:

Invariant: an agent's action quality under the candidate endpoint must be statistically indistinguishable from the primary endpoint under injected failure classes, measured on an explicit denominator, before any production traffic shift.

A cost-motivated cutover almost never tests this, because the failure modes of a free tier (rate limiting, truncation, queueing delay) only appear at load, after the cutover.

This article builds the missing piece: a shadow evaluation protocol where the candidate endpoint serves copies of real requests, with rate limits and truncation injected, and a gate decides promote / hold / reject. The artifact is a runnable simulator.

Assumptions

• The primary endpoint is paid and treated as the reference behavior, not as ground truth.
• The candidate is a free-tier endpoint with unknown quota and unknown degradation behavior under burst load. I do not assume specific quotas, hardware, or permanence — that is exactly why the gate exists.
• Agent outputs are tool calls (structured JSON), so partial equivalence is checkable mechanically: action name + argument set.
• A paired comparison is possible: the same prompt can be replayed to both endpoints.

Where I ran the candidate side

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For the candidate endpoint in this workflow I used MonkeyCode's free model access, and the shadow harness itself ran on their free server option so the experiment costs nothing on either side. Two honest limitations: free access is exactly the class of dependency this article is about, so treat quota and availability as variables, not constants; and the harness host should not share a failure domain with your production router, otherwise your "independent" shadow measurement dies with the system it is measuring. Any free endpoint and any small always-on box would do — the protocol is the point.

The protocol as a state machine

          shadow traffic (mirrored, no user impact)
          + injected failure classes
                  |
        +---------v---------+
        |   SHADOWING       |  agreement window w accumulating
        +---------+---------+
                  | window complete
        +---------v---------+
        |   GATE EVALUATION |  agreement >= theta under ALL classes?
        +----+---------+----+
      pass   |         |  fail
   +---------v+       +v----------+
   | 5% CANARY|       | HOLD/REJECT|  extend window or stop
   +----+-----+       +------------+
        |  gate re-evaluated on canary population
   +----v-----+
   | PROMOTED |  primary becomes fallback, not deleted
   +----------+

The key property: promotion is a two-stage decision (shadow gate, then canary gate), and the primary endpoint is retained as a fallback until the canary stage passes. A cutover that deletes the fallback on day one is not a promotion protocol; it is a hope.

Failure classes to inject

Do not just mirror happy-path traffic. The free tier's failure modes are the test:

Class	Mechanism	What it reveals
F1: rate limiting	probabilistic 429 under burst	retry behavior, queue growth, tail latency
F2: truncation	cap completion length randomly	parser robustness, silent-default bugs (the t5 event above)
F3: slow responses	injected latency spikes	timeout/retry amplification
F4: quality drift	none — it is the measurement itself	whether free-tier output actually matches primary

F2 is the one that produces zero-error regressions, which is why it must be injected rather than waited for.

The artifact: a minimal shadow simulator

Python, no dependencies. It models the router, both endpoints, injection, and the gate. The point is not fidelity to any vendor; it is that you can run counterexamples against the gate logic in a weekend.

import random
from dataclasses import dataclass, field

@dataclass
class Endpoint:
    name: str
    p_429: float          # rate-limit probability under burst
    p_truncate: float     # truncation probability
    p_action_match: float # probability of same tool action as reference

    def call(self, burst: bool, inject: bool):
        if inject and burst and random.random() < self.p_429:
            return {"status": 429}
        truncated = inject and random.random() < self.p_truncate
        action_ok = random.random() < self.p_action_match
        if truncated:
            # truncated JSON -> parser fallback -> WRONG action, status 200
            return {"status": 200, "action": "default_fallback", "ok": False}
        return {"status": 200, "action": "ref_action" if action_ok else "other",
                "ok": action_ok}

def shadow_window(primary, candidate, n_requests=2000,
                  burst_fraction=0.3, inject=True):
    """Returns agreement stats over an explicit denominator."""
    attempted = completed = agree = silent_bad = 0
    for _ in range(n_requests):
        burst = random.random() < burst_fraction
        ref = primary.call(burst, inject=False)      # reference behavior
        cand = candidate.call(burst, inject)
        attempted += 1
        if cand["status"] == 429:
            continue                                  # retry policy's problem
        completed += 1
        if cand["action"] == "default_fallback":
            silent_bad += 1                           # zero-error failure!
        if cand.get("ok") and ref["status"] == 200:
            agree += 1
    return {
        "denominator": attempted,
        "completion_rate": completed / attempted,
        "agreement": agree / max(completed, 1),
        "silent_fallback_rate": silent_bad / max(completed, 1),
    }

def gate(stats, theta_agree=0.95, max_silent=0.001, min_completion=0.99):
    if stats["completion_rate"] < min_completion:
        return "HOLD: rate limiting ate the window"
    if stats["silent_fallback_rate"] > max_silent:
        return "REJECT: truncation causes silent wrong actions"
    if stats["agreement"] < theta_agree:
        return f"HOLD: agreement {stats['agreement']:.3f} < {theta_agree}"
    return "PROMOTE to 5% canary"

if __name__ == "__main__":
    random.seed(7)
    primary   = Endpoint("paid",  0.00, 0.00, 0.97)
    candidate = Endpoint("free",  0.08, 0.05, 0.93)
    stats = shadow_window(primary, candidate)
    print(stats)
    print(gate(stats))

Typical output:

{'denominator': 2000, 'completion_rate': 0.977,
 'agreement': 0.932, 'silent_fallback_rate': 0.049}
REJECT: truncation causes silent wrong actions

Note the denominator is attempted, not "successful responses." If you divide agreement by completed responses only, a heavily rate-limited endpoint looks better than it is, because the requests that failed are removed from the population. That is the classic evaluation-harness lie, and it is why the denominator is declared in the invariant.

Testable properties

Property-test the gate itself, not just the endpoint:

1. Monotonicity: if candidate quality degrades (lower p_action_match), the gate must never flip from REJECT to PROMOTE.
2. Denominator integrity: agreement is computed over attempted; no code path may shrink the denominator after injection.
3. Zero-error detection: a run with only F2 (truncation) and no 429s must still REJECT via silent_fallback_rate.
4. Fallback retention: the router state machine has no transition from PROMOTED that removes the primary endpoint config.

Property 3 directly encodes the t1–t6 counterexample from the opening.

Tradeoffs

Decision	Cost	What you buy
Shadow mirroring	2x candidate-side request volume	zero user impact during evaluation
Failure injection	harness complexity	you see free-tier behavior in days, not at 3 a.m. in week three
Two-stage gate	slower promotion (days, not hours)	a bad endpoint costs you a window, not an incident
Free endpoint + free harness host	quota/availability are variables	the whole experiment costs ~nothing to run
Keeping paid fallback warm	ongoing baseline cost	rollback is a config flip, not a migration

Who should not do this

• If your agent's outputs cannot be compared mechanically (free-form prose with no judge), the agreement metric is meaningless — build a rubric-based judge first.
• If traffic is too low to fill a window with statistical power, extend the window; do not lower theta and call it rigor.
• If the workload is latency-critical at p99 with no fallback tolerance, a free tier with unknown queueing behavior is the wrong dependency regardless of what the gate says.

Acceptance rule

Promote only if: agreement ≥ 0.95 over an attempted-request denominator, silent-fallback rate ≤ 0.1%, completion rate ≥ 99% under injected F1–F3, and the 5% canary reproduces those numbers on live traffic for a full week. Otherwise hold or reject, and keep the paid fallback.

If you want to try this protocol without paying for the experiment itself, MonkeyCode's free model access and free server are enough to stand up both sides of the shadow harness — that is genuinely all you need for the artifact above.

Closing counterexample

Before you trust your own version of this gate, answer: which event order breaks your invariant? Mine is truncate(200) → parse-fallback → success-metric — a failure with no error signal. If your router sees one, does it reject the request, replay it against the primary, or compensate after the wrong action has already been committed? If you cannot point at the code path, the gate is a diagram, not a protocol.