Reject Retry Storms Before Token Cost Beats Deadline Slack

Your pager fires on review_latency_p95 at 02:14.
Queue utilization still reads only twelve percent.
The p95 timer already sits near forty-eight seconds.

Your SLO still says fifteen seconds end to end.
Idle workers do not explain that gap.
Retries, queue age, and token burn do.

Read The Contradiction First

Do not scale replicas from utilization alone.
Retries hide the real cost of free capacity.
Each retry spends tokens, time, and queue slots.

Watch these fields on one screen:

• queue_age_ms
• retry_count
• token_in and token_out
• deadline_slack_ms
• worker_util

Utilization without queue age lies to you.
Queue age without retries also lies.
You need both before you spend money.

Lab Topology You Can Copy

Run this as a local drill, not production.
One admission proxy. One bounded queue. One worker.
Keep the blast radius inside one directory.

Declared test conditions:

• Workload: 40 review jobs in 20 seconds
• Prompt budget: 800 input units, 200 output cap
• SLO: 15s end-to-end, 5s max queue age
• Retry policy: max 3, 200ms exponential backoff
• Worker: one process, 2s mean service time

This topology is a rehearsal, not a live outage.
Expected output is labeled in a later section.
Do not quote it as a vendor benchmark.

[loadgen] -> [admission] -> [queue depth=8]
                 |                |
            fail_closed        [worker]
                 |
          metrics.jsonl

The Four Costs You Actually Pay

Free capacity is not free under retries.
You pay wall time even when invoices stay quiet.
You also pay jobs that miss the deadline.

Track unit cost with this notebook formula:

cost_per_job =
  tokens_attempt
  + queue_age_ms
  + retry_n * tokens_attempt
  + missed_slo_penalty

On a free server, cash price can look like zero.
queue_age_ms and retry_n still compound fast.
A missed SLO is the invoice you cannot ignore.

Time, tokens, retries, and queueing move together.
If one field spikes, stop enqueueing new work.
Scaling later does not refund spent retries.

When Free Capacity Is The Wrong Bet

Use this table before you enqueue another job.
Treat it as an admission checklist, not folklore.
Three matching rows means fail closed now.

Evidence	Operational action	Rationale
queue_age_ms > 5000 and retry_count >= 1	reject or shed	deadline slack is already gone
retry_count >= 2 and token_in still rising	fail closed	retries are the spend
worker_util < 30% and p95 is high	inspect retries, do not scale	idle workers mean amplification
deadline_slack_ms < service_p95	do not enqueue	you cannot recover in line
reject ratio stays above 10%	leave free capacity	the queue became the product

If those rows fire inside one minute, stop.
Free capacity is the wrong bet in that window.
Pay for isolation, or cut the feature path.

Artifact: Retry-Budget Harness

The harness below is local and disposable.
It does not call a production model endpoint.
It records tokens, retries, and queue age.

Save it as retry_budget.py and run it once.

#!/usr/bin/env python3
"""Local retry-budget drill. Synthetic worker only."""
import json, os, random, time, uuid
from collections import deque

SLO_MS = 15000
MAX_AGE_MS = 5000
MAX_RETRY = 3
DEPTH = 8
TOKEN_IN = 800
TOKEN_OUT_CAP = 200

class Job:
    def __init__(self):
        self.id = uuid.uuid4().hex[:8]
        self.enqueued = time.monotonic()
        self.retry = 0
        self.tokens = 0

def deadline_slack_ms(job):
    age = (time.monotonic() - job.enqueued) * 1000
    return SLO_MS - age

def admit(queue, job, stats):
    age = (time.monotonic() - job.enqueued) * 1000
    if age > MAX_AGE_MS or deadline_slack_ms(job) < 2000:
        stats["rejected"] += 1
        return False
    if len(queue) >= DEPTH:
        stats["rejected"] += 1
        return False
    queue.append(job)
    stats["accepted"] += 1
    return True

def worker_once(job, inject_hold_s, inject_429):
    if inject_hold_s:
        time.sleep(inject_hold_s)
    if inject_429 and job.retry % 2 == 0:
        job.retry += 1
        job.tokens += TOKEN_IN
        return "retry"
    time.sleep(0.05)  # local stand-in for 2s service
    job.tokens += TOKEN_IN + random.randint(40, TOKEN_OUT_CAP)
    return "ok"

def main():
    queue = deque()
    stats = dict(accepted=0, rejected=0, retries=0, tokens=0, done=0)
    inject = os.getenv("INJECT", "none")
    hold = 8 if inject == "hold" else 0
    nack = inject == "429"
    t0 = time.monotonic()
    ages = []
    for i in range(40):
        job = Job()
        if not admit(queue, job, stats):
            continue
        while queue:
            cur = queue.popleft()
            ages.append((time.monotonic() - cur.enqueued) * 1000)
            result = worker_once(cur, hold, nack)
            stats["tokens"] += cur.tokens
            if result == "retry" and cur.retry < MAX_RETRY:
                stats["retries"] += 1
                if not admit(queue, cur, stats):
                    break
            else:
                stats["done"] += 1
        time.sleep(0.02)
    p95 = sorted(ages)[int(0.95 * (len(ages) - 1))] if ages else 0
    fail_closed = p95 > MAX_AGE_MS or stats["retries"] >= 12
    out = dict(stats, p95_age_ms=round(p95, 1),
               elapsed_ms=round((time.monotonic() - t0) * 1000, 1),
               decision="fail_closed" if fail_closed else "keep_open")
    print(json.dumps(out, indent=2))
    open("/tmp/retry-budget.json", "w").write(json.dumps(out))

if __name__ == "__main__":
    main()

Run the baseline, then inject one fault.

python3 retry_budget.py
INJECT=hold python3 retry_budget.py
INJECT=429 python3 retry_budget.py
rm -f /tmp/retry-budget.json

Expected output (local, labeled, not production):

{
  "accepted": 28,
  "rejected": 12,
  "retries": 19,
  "tokens": 9400,
  "done": 21,
  "p95_age_ms": 6800.0,
  "decision": "fail_closed"
}

Your numbers will differ on every laptop.
The fields must still move in the same direction.
If they do not, fix the harness before any server test.

Failure Injection And Rollback

Inject one fault at a time. Never stack them.
Record the five telemetry fields after each inject.
Reject work before p95 crosses fifteen seconds.

1. Hold the worker for eight seconds.
2. Force a 429 on every second attempt.
3. Duplicate the same job id twice.
4. Zero the token counter on purpose.

Rollback path after the drill:

• Export ADMISSION=fail_closed on the proxy
• Stop the injector with kill %1 or Ctrl-C
• Drain remaining jobs with MAX_RETRY=0
• Delete /tmp/retry-budget.json and any logs

Do not leave the injector running overnight.
A forgotten 429 loop burns the next rehearsal.
Cleanup is part of the test, not a courtesy.

Pick One Threshold And Defend It

Pick one primary threshold, not five equal ones.
For this workload, trip on queue age first.
Utilization lags. Token counters lag until completion.

Rationale you can hand a reviewer:

• Queue age is visible before the SLO dies
• Deadline slack equals remaining SLO minus age
• Retries only matter after the first miss starts
• Token totals arrive too late for admission

Local tripwires for the declared 15s SLO:

• warn when queue_age_ms > 3000
• reject when queue_age_ms > 5000
• fail closed when retries are at least two and warn is already hot

Tune only after you replay the injector twice.
Do not copy these numbers into production configs.
They match this lab SLO and this queue depth only.

Where A Free Server Fits

You can stage the same admission loop against a free model endpoint and a free server after the harness fail-closes cleanly.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access and a free server option you can use as a rehearsal lane. Use that lane to practice reject-and-drain, not to hide retry storms. If the local drill already fail-closes, dedicated capacity is the cheaper operational bet.

Confirm live quotas and server terms on the product page before you schedule a drill. Those numbers change. This article does not freeze them.

If you want that rehearsal, run this harness against the free server after local fail-closed stays green.

Limitations

This harness does not model GPU contention.
It does not model multi-tenant fairness rules.
It does not prove any vendor SLO or invoice.

Token units here are synthetic counters only.
They are not billed statements or capacity pledges.
Do not quote them in a budget review.

Short operational sentences skip distributed design.
Leave evaluation architecture to another owner.
You are diagnosing a deployed admission path.

Who Should Not Use This

Skip this if jobs have no deadline at all.
Skip this if the work is overnight batch only.
Skip this if you cannot shed or reject load.

Also skip it if you need named models or quotas.
This article does not claim hardware, duration, or permanence.
Read those from primary product docs at drill time.

What You Do At 02:14

You do not scale from twelve percent utilization.
You read queue age, retries, tokens, and slack.
Then you reject, drain, or pay for isolation.

Free capacity is a rehearsal tool under quiet load.
It is the wrong bet once retries amplify token cost.
Measure the four costs before the next page fires.