Reject Blind Retries Before Free Worker Idle Beats Deadline Slack

You get paged at 02:14 for the inference gateway.
CPU on the free worker sits at twelve percent.
Queue age already reads forty-seven seconds against a thirty-second deadline.

Utilization still looks like spare serving room on the panel.
Queue age says the completion SLO is already dead.
You retry the same completion three extra times anyway.

Idle capacity just multiplied the unpaid work.
Free worker idle time is the wrong bet here.
Reject the retry before leftover slack hits zero.

Read The Alert Before You Retry

Do not trust a green utilization panel alone.
Idle time is not the same as spare budget.
Retries clone work that already missed the deadline slack.
A free replica cannot mint a new deadline.

Ask one operational question before the next POST.
Which signal should gate that extra retry attempt.
Choose queue age, utilization, or leftover deadline slack.
Write the winning threshold into the admission config.

Evidence You Should Capture First

Pull these fields from one scrape window:

• queue_age_ms at the admission edge
• worker_util on the serving replica
• deadline_slack_ms on the inbound request
• retry_count on the client or sidecar
• token_units already spent on this id
• http_status of the last upstream attempt

If queue age already exceeds leftover slack, stop.
Do not spend idle cores on a late job.
A free replica does not restore lost wall time.

Put age first, slack second, and utilization last.
Utilization is a capacity hint, not a permit.
Late work does not become cheap when cores look idle.

Topology For The Local Drill

Keep the lab small and fully local.
You do not need a paid GPU cluster.

client -> retry_proxy -> admission -> fake_worker
                 |              |
            retry_log      queue_age gauge

The fake worker sleeps, then returns token units.
The proxy retries on timeout or HTTP 503.
Admission rejects when age beats leftover slack.
That closed loop is the entire control plane.

Declared Test Conditions

State the workload before you run anything:

• One synthetic prompt with a 256-token target
• Client deadline set to 3000 milliseconds
• Worker sleep of 2500 milliseconds on the happy path
• Injected extra sleep of 4000 milliseconds for the fault
• Uncapped client retries set to four attempts
• Admission rule: reject when age is greater than slack
• Local process only, with no production endpoints

These numbers are lab knobs, not product benchmarks.
Do not treat them as vendor throughput claims.
Relabel the knobs if your SLO uses seconds instead.

Why Free Idle Time Is The Wrong Bet

Free serving often feels like unused monthly budget.
Retries convert that idle time into extra token units.
Each clone still waits behind the same admission queue.
Wall clock does not reset because the replica is free.

You pay three ways at once during the storm:

1. Token units on duplicate forwards of the same id
2. Queue age growth on every waiting replica
3. Operator time spent chasing a healthy-looking worker

Utilization can stay low through all three costs.
That contradiction is why the 02:14 page fired.
Idle cores plus late age means reject, not retry.

Decision Table: Age Versus Idle Versus Slack

Use this table at the admission edge.

queue_age vs slack	worker_util	retry_count	action
age under half of slack	any	0	admit once
age still under slack	util under 0.30	0	admit once, no retry
age at or past slack	idle or busy	any	reject, do not retry
any late age	any	1 or more	reject and drain

Idle utilization never overrides a late queue age.
Write that rule in the proxy, not a wiki page.

Say the threshold rationale out loud before merge.
Why does age beat utilization for this SLO.
The caller already spent the remaining deadline slack.
Idle cores cannot mint a replacement client deadline.

Reproducible Local Artifact

The script below is a labeled lab example.
It is not a production measurement run at all.
Expected output below is marked as expected only.

#!/usr/bin/env python3
# Local retry-admission drill. Unexecuted until you run it.
import json
import time
from dataclasses import dataclass

DEADLINE_MS = 3000
WORKER_SLEEP_MS = 2500
FAULT_SLEEP_MS = 4000
MAX_CLIENT_RETRIES = 4

@dataclass
class Probe:
    req_id: str
    retry_count: int
    enqueued_at: float
    deadline_ms: int

def queue_age_ms(p: Probe) -> float:
    return (time.monotonic() - p.enqueued_at) * 1000.0

def deadline_slack_ms(p: Probe) -> float:
    return p.deadline_ms - queue_age_ms(p)

def admit(p: Probe) -> str:
    age = queue_age_ms(p)
    slack = deadline_slack_ms(p)
    if p.retry_count >= 1 and slack <= 0:
        return 'reject_retry_late'
    if age >= p.deadline_ms:
        return 'reject_age_beats_slack'
    return 'admit_once'

def fake_worker(fault: bool) -> dict:
    sleep_ms = FAULT_SLEEP_MS if fault else WORKER_SLEEP_MS
    time.sleep(sleep_ms / 1000.0)
    return {'ok': not fault, 'sleep_ms': sleep_ms, 'token_units': 64}

def run_client(fault: bool, cap_retries: bool) -> dict:
    p = Probe('lab-1', 0, time.monotonic(), DEADLINE_MS)
    events = []
    attempts = 1 if cap_retries else MAX_CLIENT_RETRIES
    for i in range(attempts):
        p.retry_count = i
        decision = admit(p)
        events.append({
            'attempt': i,
            'decision': decision,
            'queue_age_ms': round(queue_age_ms(p), 1),
            'deadline_slack_ms': round(deadline_slack_ms(p), 1),
            'worker_util_hint': 0.12,
        })
        if decision.startswith('reject'):
            break
        result = fake_worker(fault)
        events.append({'worker': result})
        if result['ok']:
            break
    return {'fault': fault, 'cap_retries': cap_retries, 'events': events}

if __name__ == '__main__':
    print(json.dumps({
        'uncapped_fault': run_client(True, False),
        'capped_fault': run_client(True, True),
    }, indent=2))

Save it as retry_admission_drill.py on your laptop.
Run it only against this local fake worker.
Do not point it at a shared production route.

Expected Output Shape

You should see rejects after slack hits zero.
Uncapped faults keep cloning the same probe id.
Capped faults stop at the first late age sample.

# expected, not a production scrape
decision=reject_age_beats_slack
queue_age_ms>=4000
deadline_slack_ms<=0
worker_util_hint=0.12

If uncapped still admits at negative slack, you failed.
Fix the proxy before you touch any remote server.
Do not borrow a free worker to hide the bug.

Shell Commands To Run The Drill

python3 retry_admission_drill.py | tee /tmp/retry-drill.json
wc -l /tmp/retry-drill.json
grep -c reject /tmp/retry-drill.json

You want rejects on the uncapped fault path.
You want a short event list on the capped path.
If both dumps look identical, the cap never engaged.

Minute Timeline Of The Failure Drill

Walk the clock so the page matches logs.

• T+0 ms: client enqueues id lab-1 with 3000 ms slack
• T+2500 ms: happy worker would return; fault path still sleeps
• T+3000 ms: slack hits zero; utilization still reads 0.12
• T+4000 ms: worker returns 503 or a late payload
• T+4001 ms: blind client schedules retry number one
• T+4001 ms: admission must reject; queue age already beat slack

If your proxy admits at T+4001, the drill failed.
Rollback the retry flag before you page anyone else.
Idle twelve percent CPU is not a reason to clone work.

429 Is Not A Permit To Replay

A 429 means the worker is protecting itself.
A 503 means the path is unsafe for more copies.
Neither status refunds the deadline slack you spent.

Blind replay after 429 clones spend on a full queue.
Blind replay after 503 clones work onto a dying replica.
Cap both with the same age-versus-slack rule.

Log the status beside retry_count for the postmortem.
Do not collapse them into a single retry bucket.
Status without age will send you back to idle CPU.

Token Units Versus Wall Clock

Token counters move even when the user got nothing.
Retries that miss slack still increment token_units.
Cost ops should sum tokens per id, not per HTTP 200.

If one id shows four times the token_units, you cloned it.
Idle utilization will not show that clone.
Age and retry_count will show it immediately.

Billable or free, duplicate forwards still steal slack.
Free capacity does not erase a four-copy storm.
Reject the second copy when slack is already gone.

Failure Injection And Rollback

Inject one fault at a time in this order.

1. Stretch worker sleep past the declared deadline.
2. Leave utilization painted at twelve percent idle.
3. Watch queue_age_ms cross leftover slack.
4. Confirm admission rejects every further retry.
5. Drain ids whose age already beat the deadline.

Rollback path if this proxy already fronts real traffic:

• Set RETRY_MAX=0 on the client sidecar immediately
• Flip admission to reject_all for the affected route
• Drain in-flight ids older than leftover slack
• Restore the last known good file from version control
• Unblock only after age stays under slack for two scrapes

Do not scale the free worker to absorb retries.
Scaling idle replicas hides the age signal you need.
Add replicas only after rejects stop the clone loop.

Staging Lane Without Burning Paid Tokens

You may want a cheap place to prove the cap.
MonkeyCode offers free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use that path only as a staging lane for the drill.
It is not a production capacity plan or an SLO.
Free idle time still cannot beat a missed deadline.
Keep paid traffic behind the same retry cap.

If you run the drill there, keep the same telemetry fields.
Log queue age, retry count, and leftover slack together.
Do not raise retries because the replica happens to be free.

Who Should Not Use This Approach

Skip this admission rule in a few cases.

• Hard real-time paths that never form a queue
• Batch jobs whose deadline sits in overnight windows
• Idempotent fan-out that already deduplicates request ids
• Teams that cannot scrape queue_age_ms today

If you cannot measure age, do not guess the number.
Guessing turns free capacity into a silent retry storm.
Instrument the edge before you copy this reject rule.

Limitations

This lab does not measure real model quality.
It does not claim throughput for any hosted vendor.
Sleep stands in for serving delay, nothing else.
Token units are counters in logs, not invoices.

Free model access can throttle, queue, or disappear.
Do not build a user-facing SLO on unpaid headroom.
Declare the retry cap in config, not in chat.

The decision table ignores fairness across shared tenants.
Add per-key budgets if you share one worker pool.
This drill also ignores cache hits and streaming tokens.

Cleanup

Remove the lab files after you capture the JSON.

rm -f /tmp/retry-drill.json retry_admission_drill.py
unset RETRY_MAX ADMISSION_MODE

If you exported fake gauges to a local collector, drop them.
Leave no scrape target pointing at the fake worker.
Confirm no cron still launches the uncapped client.

What You Do Next

Write the threshold in one small config block.

admission:
  reject_if_queue_age_ms_gte: 3000
  reject_if_retry_count_gte: 1
  ignore_worker_util_when_slack_ms_lte: 0

Then page on age, not on idle cores.
When utilization is low and age is high, reject.
Free worker idle time remains the wrong cost bet.

Which threshold will you enforce in the proxy first.
Queue age, retry count, or leftover deadline slack.
Pick one, then make the local drill prove the reject.