Reject Sync Completions Before Free Queue Age Beats Deadline Slack

The pager fired at 09:14 during a quiet deploy window.
Your interactive p99 sat at 8.4 seconds.
Host CPU never left the idle band.

Queue age told a different story than utilization.
A free shared worker still chewed a nightly eval batch.
User chats waited behind unlabeled jobs with no class header.

Which operational action follows from that evidence?
You reject new sync completions on that pool immediately.
You drain the batch slot, then restore interactive traffic.

What you are actually measuring

Utilization is a trap on shared free capacity.
A quiet CPU can hide a long queue age.
Deadline slack dies while dashboards stay green.

Track three signals on every admission path:

• queue_age_ms: wait of the oldest admitted job
• deadline_slack_ms: budget left before client timeout
• token_burn_per_admit: estimated tokens for the next completion

You act on slack, not on pretty utilization charts.
Tokens matter only after the request still has time to finish.

Topology under test

Keep the topology small and local.
One admission proxy. One fake worker. One metric file.

client --> admission.py --> worker_stub.py
                 |
                 +--> /tmp/free_pool_metrics.json

Declared conditions for this drill:

• Client timeout: 4000 ms
• Max sync queue age: 1500 ms
• Interactive class: sync
• Batch class: eval
• Worker concurrency: 1
• No production models and no cloud credentials

Label every number below as fixture output.
This is a local fault drill, not a hosted benchmark.
Do not treat these figures as capacity of any vendor pool.

Artifact: an admission gate you can run

Save admission.py next to the metric file.
It rejects sync work when slack cannot cover queue age.

#!/usr/bin/env python3
"""Local admission gate for a shared free worker pool."""
import json, time, sys

METRICS = "/tmp/free_pool_metrics.json"
CLIENT_TIMEOUT_MS = 4000
MAX_QUEUE_AGE_MS = 1500
SYNC_CLASSES = {"sync", "interactive", "chat"}

def load_metrics():
    with open(METRICS) as f:
        return json.load(f)

def admit(job):
    m = load_metrics()
    now = int(time.time() * 1000)
    slack = (now + CLIENT_TIMEOUT_MS) - now
    age = int(m["queue_age_ms"])
    inflight = int(m["inflight"])
    klass = job["class"]
    reasons = []
    if klass in SYNC_CLASSES and age > MAX_QUEUE_AGE_MS:
        reasons.append("queue_age_gt_sync_threshold")
    if slack <= age:
        reasons.append("slack_lte_queue_age")
    if klass in SYNC_CLASSES and inflight > 0 and m["worker_class"] == "eval":
        reasons.append("sync_blocked_by_eval_drain")
    return {
        "decision": "reject" if reasons else "admit",
        "reasons": reasons,
        "queue_age_ms": age,
        "deadline_slack_ms": slack,
        "token_burn_per_admit": job.get("est_tokens", 0),
        "worker_class": m["worker_class"],
        "retry_hint": "dedicated_pool" if reasons else "none",
    }

if __name__ == "__main__":
    job = json.loads(sys.argv[1])
    print(json.dumps(admit(job), indent=2))

Seed metrics before the first call.

cat > /tmp/free_pool_metrics.json <<'EOF'
{
  "queue_age_ms": 6100,
  "inflight": 1,
  "worker_class": "eval",
  "token_burn_rate": 0
}
EOF

Run a sync job against the dirty pool.

python3 admission.py '{"class":"sync","est_tokens":800}'

Expected fixture output, labeled as expected, not observed production:

{
  "decision": "reject",
  "reasons": [
    "queue_age_gt_sync_threshold",
    "slack_lte_queue_age",
    "sync_blocked_by_eval_drain"
  ],
  "queue_age_ms": 6100,
  "deadline_slack_ms": 4000,
  "token_burn_per_admit": 800,
  "worker_class": "eval",
  "retry_hint": "dedicated_pool"
}

You rejected the interactive completion on purpose.
Queue age already consumed the entire client timeout.
Free capacity was the wrong bet for that traffic class.

Why free capacity fails interactive paths

Free shared pools optimize for availability, not isolation.
Eval suites, replays, and chat often share one worker slot.
Retries look cheap until they occupy that only slot.

Cost here is time, then tokens, then retries.
Each timed-out user waits, retries, then waits again.
Token burn rises while deadline slack collapses to zero.

Ask these threshold questions before you place traffic:

1. Is queue age growing faster than deadline slack?
2. Is utilization low while age stays high?
3. Are eval jobs unlabeled on the same worker?
4. Would a reject-plus-redirect beat a blind retry?

If (1) is true, reject sync admissions now.
If (2) is true, ignore CPU charts during the incident.
If (3) is true, drain eval before you reopen chat.
If (4) is true, return retry_hint=dedicated_pool instead of 200.

Failure injection: occupy the free worker

Add worker_stub.py so you can inject a stuck eval.
The stub publishes rising queue age every 200 ms.

#!/usr/bin/env python3
"""Occupy the local worker and publish queue age."""
import json, time, signal

METRICS = "/tmp/free_pool_metrics.json"
stop = False

def handle(sig, frame):
    global stop
    stop = True

signal.signal(signal.SIGTERM, handle)
started = int(time.time() * 1000)
while not stop:
    payload = {
        "queue_age_ms": int(time.time() * 1000) - started,
        "inflight": 1,
        "worker_class": "eval",
        "token_burn_rate": 12
    }
    with open(METRICS, "w") as f:
        json.dump(payload, f)
    time.sleep(0.2)

json.dump(
    {"queue_age_ms": 0, "inflight": 0, "worker_class": "idle", "token_burn_rate": 0},
    open(METRICS, "w"),
)

Start the stuck eval, then watch age climb.

python3 worker_stub.py &
echo $! > /tmp/worker_stub.pid
sleep 3
python3 admission.py '{"class":"sync","est_tokens":800}'
python3 admission.py '{"class":"eval","est_tokens":4000}'

Expected behavior under this injection:

• sync rejects once queue_age_ms exceeds 1500
• eval may still admit if slack covers age
• Killing the stub returns age to zero
• A second sync admit should pass only after idle metrics

That split is the cost-ops rule.
Batch can wait on free capacity without a user timeout.
Interactive traffic cannot survive a single occupied slot.

Label the job at the edge

You cannot protect slack without a class label.
Add a header or field before the worker sees the body.

# Reject unlabeled work the same way you reject dirty sync.
python3 admission.py '{"class":"unlabeled","est_tokens":800}'

Extend admit() so unlabeled jobs never enter the free slot.
Unlabeled work is how nightly eval stealthily becomes user latency.
Fix the contract first, then measure age.

Suggested edge contract:

• X-Job-Class: sync for user-facing completions
• X-Job-Class: eval for replay, drain, and nightly suites
• X-Idempotency-Key on every sync retry
• Retry-After on rejects, pointing at a dedicated pool

Without the idempotency key, a rejected client retries the same pool.
That retry is how free capacity inverts unit cost.
You pay tokens twice and still miss the deadline.

When free capacity is the wrong bet

Use free shared capacity for work that can wait.
Do not use it for work that has a user-visible timeout.

Wrong bet, reject at admit time:

• Interactive chat and other sync completions
• Webhook-triggered replies with a 2–5 second client timeout
• Health flows that page humans on p99
• Any path where a retry hits the same unlabeled worker

Acceptable bet, after you measure age:

• Nightly eval and prompt-replay jobs
• Drain tests and admission-gate CI
• Backfill that can pause when queue_age_ms rises

Time is the first cost. Tokens are the second.
Retries are the multiplier that makes both worse.
Queueing is the signal that tells you to stop.

A free model pool is still a single slot until proven otherwise

You may already have a free model path for experiments.
MonkeyCode offers free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat that pool like the stub worker in this drill.
Send unlabeled eval, replay, and drain tests there on purpose.
Keep user-facing sync completions off that shared slot until age stays under your cap.

Do not assume isolation you have not measured.
Do not publish an interactive SLO against unlabeled free capacity.
Run the admission gate before you point clients at it.

Rollback and cleanup

Drain is a first-class rollback, not an afterthought.
You stop sync admissions, finish or kill eval, then reopen.

# 1. Prove the gate still rejects sync on a dirty pool
python3 admission.py '{"class":"sync","est_tokens":1}'

# 2. Stop the occupying eval
kill "$(cat /tmp/worker_stub.pid)"
sleep 1

# 3. Confirm idle metrics
python3 - <<'PY'
import json
print(json.load(open("/tmp/free_pool_metrics.json")))
PY

# 4. Re-admit a sync probe only after age is under 1500 ms
python3 admission.py '{"class":"sync","est_tokens":800}'

# 5. Remove fixture files
rm -f /tmp/free_pool_metrics.json /tmp/worker_stub.pid

Expected post-drain fixture:

{
  "queue_age_ms": 0,
  "inflight": 0,
  "worker_class": "idle",
  "token_burn_rate": 0
}

If age does not fall, you still have a stuck reader.
Do not reopen sync traffic.
Trace inflight jobs before you lift the reject.

Rollback order you can paste into a runbook:

1. Flip admission to reject sync and unlabeled
2. Snapshot queue_age_ms and inflight
3. Stop eval producers, then SIGTERM the worker
4. Wait until age is under the sync cap
5. Admit one synthetic sync probe
6. Only then restore real interactive clients

Operational threshold and rationale

Pick queue age over utilization for this failure class.
Utilization stays low when one long eval holds the slot.
Queue age tracks user-visible wait directly.

Compare three candidate thresholds:

Signal	Reject sync when	Why it works	Why it lies
Utilization	greater than 70%	Easy to chart	Misses single-slot occupancy
Queue age	greater than 1500 ms	Matches wait	Needs a correct clock
Deadline slack	slack less than or equal to age	Matches timeouts	Noisy if clients vary

Use queue age as the primary reject.
Use slack as the safety latch.
Ignore utilization unless it contradicts age.

Token burn matters after you survive the wait.
A rejected sync job burns zero tokens on that worker.
A retried sync job can burn twice after a timeout.

So you reject early.
You convert a timeout into an immediate reject with a retry hint.
The client routes to a dedicated pool instead of retrying blindly.

Test plan you can run in CI

Declare the workload, then assert the gate.
Do not skip cleanup.

1. Write idle metrics and admit a sync job. Expect admit.
2. Start worker_stub.py. Wait 3 seconds. Expect queue_age_ms above 1500.
3. Admit sync again. Expect reject and retry_hint=dedicated_pool.
4. Admit eval. Expect admit only if slack still covers age.
5. SIGTERM the stub. Expect idle metrics.
6. Admit sync once more. Expect admit.
7. Delete /tmp/free_pool_metrics.json and the pid file.

If step 3 admits, the gate is wrong.
If step 6 rejects, drain is wrong.
Fix those before any client sees the pool.

Limitations

This drill uses a single-slot stub.
It does not model preemption or multi-tenant fairness.
Clocks are local and metrics live in a JSON file.

The 1500 ms age cap is a fixture, not a universal SLO.
Your client timeouts may be shorter than 4000 ms.
Recalculate slack from the real timeout, not from this article.

Labeled outputs are expected fixture values.
They are not production measurements.
Do not cite them as capacity of any hosted free pool.

Who should not use this approach

Do not use this gate as your only production auth layer.
It is an admission heuristic, not a security control.

Skip this pattern if you already isolate interactive workers.
Skip it if every job class has its own concurrency budget.
Skip it if you cannot identify sync versus eval at admit time.

If you cannot label the job, you cannot protect slack.
Fix the labels first.
Then measure age.

What you do on the next quiet morning

Wire queue_age_ms next to your completion latency chart.
Add reject reasons to the admission log line.
Run the stub drain in CI before you trust a free pool with batch work.

Free capacity is useful for eval and replay.
It is the wrong bet for interactive completions.
Reject those jobs before queue age eats your deadline slack.

If you park batch jobs on a free model server, run this drain drill first and keep the sync class off that worker.