DEV Community

Robin
Robin

Posted on

Make Escalation Rate an Invariant Before You Mix a Cheap Primary Model With an Expensive Fallback

#architecture #ai #testing #distributed

Every time a cheap model drops — this week it's DeepSeek-V4-Pro-0813 in my feed, last month it was something else — the same architecture gets sketched on a whiteboard: route everything to the cheap one, and "occasionally" escalate to the expensive one (Grok 4.6, or whatever your premium endpoint is) when the cheap one struggles. Cheap and good, with a little premium on top. What could go wrong?

Here is an event order that did go wrong for a system I reviewed:

  1. Cheap model degrades slightly (a silent provider-side change). Retryable, low-confidence responses rise from 2% to 6%.
  2. The router's escalation rule is if confidence < threshold: escalate. Escalation traffic to the premium model triples.
  3. Premium model rate-limits the account. Escalations start timing out.
  4. The retry wrapper retries the escalation — which retries the premium call — which deepens the rate limit.
  5. The monthly premium budget is consumed in 9 days. Finance asks why. Nobody can say which requests escalated or why.

The invariant the implementation failed to preserve: the escalation rate is a bounded, explainable system property, not an emergent side effect of a threshold. If you can't state your escalation budget and prove your router respects it under degradation, you don't have a routing strategy — you have a cost surprise on a timer.

Declared assumptions

  • "Cheap and good" is a claim to validate, not a fact. I have not benchmarked DeepSeek-V4-Pro-0813 or Grok 4.6; treat both names below as example endpoints. Your harness must verify quality before promotion.
  • Each request yields a primary response plus a scalar confidence/quality signal. Your signal may be a verifier, a critic model, or a heuristic — the protocol doesn't care which.
  • You have a premium budget expressed as a rate (e.g., ≤ 5% of requests may escalate sustained) and a cap (absolute count per window).
  • Escalations and retries share the same downstream rate limit — this is the coupling that creates the failure above.

The escalation protocol, as a state model

Treat each request as a state machine, not a function call:

PRIMARY_CALL ──ok, confident──▶ DONE
     │
     ├─ok, low confidence──▶ BUDGET_CHECK ──allowed──▶ ESCALATE ──▶ DONE(escalated)
     │                            │
     │                            └─denied──▶ DONE(degraded, logged, sampled for replay)
     │
     └─error──▶ RETRY_ONCE ──▶ (same states; never re-escalate a retried escalation)
Enter fullscreen mode Exit fullscreen mode

Three rules make this convergent:

  1. Budget check before escalation, not after. A token bucket gates escalations. Denied escalations degrade gracefully to the primary answer with a flag, so you can replay them later against the premium model offline.
  2. Retries never re-enter escalation. A retried escalation that fails returns degraded, not another premium call. This breaks the retry ↔ rate-limit feedback loop.
  3. Every escalation carries a reason code (low_confidence, verifier_reject, timeout_primary) written to an append-only log. If you can't explain your escalation distribution, you can't debug your bill.

Minimal simulator

This runs with the standard library only. It injects primary-model degradation and a premium rate limit, then checks whether your router preserves the escalation invariant:

import random
from dataclasses import dataclass, field

@dataclass
class Bucket:
    rate: float          # tokens per request-window
    cap: int
    tokens: float = 0.0
    def allow(self) -> bool:
        self.tokens = min(self.cap, self.tokens + self.rate)
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True
        return False

@dataclass
class Stats:
    n: int = 0
    escalated: int = 0
    degraded: int = 0
    premium_calls: int = 0
    reasons: dict = field(default_factory=dict)

def simulate(n=20000, degradation_start=5000, low_conf_prob=0.02,
             degraded_prob=0.06, premium_limit_per_100=8):
    bucket = Bucket(rate=0.05, cap=50)   # 5% sustained escalation budget
    s = Stats()
    premium_used_in_window = 0
    for i in range(n):
        s.n += 1
        if i % 100 == 0:
            premium_used_in_window = 0
        low_conf = random.random() < (degraded_prob if i >= degradation_start else low_conf_prob)
        if not low_conf:
            continue
        if not bucket.allow():
            s.degraded += 1
            continue
        # premium rate limit: reject when window quota exhausted
        if premium_used_in_window >= premium_limit_per_100:
            s.degraded += 1                    # rule 2: no retry-storm into premium
            continue
        premium_used_in_window += 1
        s.escalated += 1
        s.premium_calls += 1
        s.reasons["low_confidence"] = s.reasons.get("low_confidence", 0) + 1
    return s

s = simulate()
rate = s.escalated / s.n
print(f"escalation_rate={rate:.4f} degraded={s.degraded} premium_calls={s.premium_calls}")
assert rate <= 0.051, f"INVARIANT VIOLATED: escalation rate {rate}"
assert s.premium_calls <= s.n * 0.08, "premium ceiling violated"
print("invariant holds under 3x confidence degradation")
Enter fullscreen mode Exit fullscreen mode

Run it, then change rule 2 (let retried escalations re-enter the premium path) and watch premium_calls blow through the ceiling once the rate limit bites. That counterexample is the argument for the protocol.

Testable properties for CI

  • P1 (bounded escalation): for any simulated degradation ≤ 10× baseline, sustained escalation rate ≤ budget + ε.
  • P2 (no feedback loop): premium failures never increase subsequent premium call volume.
  • P3 (explainability): 100% of escalations carry a reason code; distribution over reasons is logged per window.
  • P4 (graceful degradation): denied escalations still return the primary answer, flagged, and a sample is queued for offline premium replay.

Property tests beat eyeballing here because the failure is distributional — it only appears across thousands of requests under degradation.

Where free capacity fits

Shadow-evaluating a new cheap model and running this kind of failure-injection harness needs real tokens and somewhere to run the worker. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are a reasonable fit for exactly this stage: point the harness's "primary" endpoint at a free model, run the shadow gate and simulator workers on the free server, and only wire in a paid premium endpoint once P1–P4 pass. If you want to try that setup, the free tier is enough to reproduce everything in this article without touching a credit card.

Tradeoffs

Design choice Latency Cost predictability Quality ceiling Debuggability
Naive threshold escalation Low (until rate-limited) Poor — emergent High Poor
Token-bucket budget gate (this article) Low; bounded tail Strong — enforced invariant Slightly lower under degradation Strong (reason codes)
Always-premium Medium Perfect but expensive Highest Trivial
Cheap-only, no fallback Lowest Perfect Lowest Trivial

Acceptance rule and limitations

Promote the routing policy only if P1–P4 hold across at least 10⁵ simulated requests spanning normal, 3× degraded, and premium-rate-limited regimes, plus a shadow run on production-sampled traffic where the cheap model's verifier agreement is within your declared tolerance. The denominator matters: report escalation rate per total requests, not per escalatable ones.

Who should not use this: if your request volume is low enough that premium cost is noise, the protocol is overhead — just call the good model. If you lack any reliable confidence/quality signal, the budget gate bounds cost but cannot bound quality; fix the verifier first. And the simulator models rates, not content quality — it proves the cost invariant, not that the cheap model is actually good enough. That requires a paired quality benchmark, which is a separate harness.

One closing counterexample to take back to your own router: what event order breaks your escalation invariant — primary degradation, premium rate limit, or a retry policy that re-enters escalation — and when it breaks, should the system reject the request, replay it offline against premium, or degrade with a compensation flag?

Top comments (0)