Reject Probe Jobs Before Free Queue Age Beats Slack

You get paged at 09:14 during an eval night.
Probe dashboards stay green across every synthetic check.
Production first-token wait still blows a four-second slack budget.

CPU on the free worker sits near twelve percent.
Admission queue age sits at forty-seven seconds instead.
Those two numbers do not describe the same failure.

Which operational action follows from that evidence?
Do you scale the free worker under low CPU?
Or do you reject the probe class before production slack dies?

What You Observed Versus What You Inferred

You observed healthy probes and broken production slack.
You observed high queue age beside idle CPU.
You did not observe a compute saturation event.

Inference is cheap here and usually wrong.
Idle CPU does not mean spare serving capacity.
Probe jobs can occupy the only slot without burning cores.

Treat queue age as the scarce signal tonight.
Treat utilization as a supporting field only.
Do not page from CPU until age and slack agree.

Topology You Can Reproduce Locally

Keep one producer, one admission loop, two job classes.
Class prod carries a deadline and remaining slack.
Class probe covers eval, canary, and synthetic traffic.

client --> admission.py --> memq:prod
                        --> memq:probe
worker_free <-- pop prod first
worker_free <-- pop probe only if age and slack allow
paid_path   <-- never used in this local drill

Pin the free worker to production when slack is thin.
Keep paid overflow off this drill on purpose.
That split is policy, not a latency benchmark.

Declared Lab Conditions

Run this on one local machine only.
Do not point the worker at production credentials.
Treat every number below as a lab fixture.

Declared workload for one drill:

• 20 production jobs, 800 ms fake work each
• 40 probe jobs, 400 ms fake work each
• One probe burst at t=0 with production mixed in
• Production deadline: 4000 ms after enqueue
• Probe deadline: none, best effort only
• Worker concurrency: 1
• Admission tick: 50 ms

These sleeps are not model latency measurements.
They exist so queue age becomes visible without GPUs.
If your fake work is faster, raise the probe burst size.

Telemetry Fields To Log On Every Decision

Log one JSON line per admit, reject, or complete.
Keep field names stable so diffs stay readable.
You need age and slack on the same line.

• ts_ms
• job_id
• job_class (prod or probe)
• enqueued_at_ms
• queue_age_ms
• deadline_slack_ms
• cpu_pct (optional, often misleading)
• action (admit, reject_probe, reject_prod, complete)
• reason

Queue age is now minus enqueue time.
Deadline slack is deadline minus now minus remaining work.
If slack is missing, you cannot reject on evidence.

Decision Table: Age First, CPU Second

Use this table before you add hardware.
The reject target is probe work, not production retries.
Production still needs an explicit slack floor.

queue_age_ms	prod slack_ms	probe action	prod action
< 500	> 1500	admit	admit
>= 500	> 1500	reject	admit
>= 500	1 to 1500	reject	admit if slack > remaining work
any	<= 0	reject	reject and page

Why queue age instead of utilization?
A blocked slot raises age without raising CPU.
Probe success can stay green while slack burns.

Why not reject production first?
Production still owns the only customer deadline.
Probes are optional measurement traffic tonight.

Local Admission Artifact

The script below is a proposed local drill.
It never calls a hosted model endpoint.
It only shows reject order under a declared burst.

#!/usr/bin/env python3
"""Reject probe jobs when queue age steals prod slack."""
from __future__ import annotations

import json
import threading
import time
import uuid
from dataclasses import dataclass, field
from typing import Literal

PROD_DEADLINE_MS = 4000
AGE_REJECT_PROBE_MS = 500
PROD_WORK_MS = 800
PROBE_WORK_MS = 400


@dataclass(order=True)
class Job:
    sort_key: int
    job_id: str = field(compare=False)
    job_class: Literal["prod", "probe"] = field(compare=False)
    enqueued_at_ms: int = field(compare=False)
    work_ms: int = field(compare=False)
    deadline_ms: int | None = field(compare=False, default=None)


class MemQueue:
    def __init__(self) -> None:
        self._lock = threading.Lock()
        self._items: list[Job] = []

    def push(self, job: Job) -> None:
        with self._lock:
            self._items.append(job)

    def peek_age_ms(self, now_ms: int) -> int:
        with self._lock:
            if not self._items:
                return 0
            oldest = min(self._items, key=lambda j: j.enqueued_at_ms)
            return now_ms - oldest.enqueued_at_ms

    def pop_class(self, job_class: str) -> Job | None:
        with self._lock:
            for i, job in enumerate(self._items):
                if job.job_class == job_class:
                    return self._items.pop(i)
            return None

    def depth(self) -> int:
        with self._lock:
            return len(self._items)


def now_ms() -> int:
    return time.time_ns() // 1_000_000


def log_line(**fields: object) -> None:
    print(json.dumps(fields, sort_keys=True), flush=True)


def slack_ms(job: Job, current_ms: int) -> int:
    if job.deadline_ms is None:
        return 10**9
    return job.deadline_ms - current_ms - job.work_ms


def admit_decision(q: MemQueue, job: Job, current_ms: int) -> str:
    age = q.peek_age_ms(current_ms)
    slack = slack_ms(job, current_ms)
    if job.job_class == "probe" and age >= AGE_REJECT_PROBE_MS:
        return "reject_probe"
    if job.job_class == "prod" and slack <= 0:
        return "reject_prod"
    return "admit"


def worker(q: MemQueue, stop: threading.Event) -> None:
    while not stop.is_set():
        current = now_ms()
        prod = q.pop_class("prod")
        job = prod or q.pop_class("probe")
        if job is None:
            time.sleep(0.05)
            continue
        action = admit_decision(q, job, current)
        age = current - job.enqueued_at_ms
        log_line(
            action=action,
            job_class=job.job_class,
            job_id=job.job_id,
            queue_age_ms=age,
            deadline_slack_ms=slack_ms(job, current),
            reason="age_vs_slack",
            ts_ms=current,
        )
        if action.startswith("reject"):
            continue
        time.sleep(job.work_ms / 1000)
        done = now_ms()
        log_line(
            action="complete",
            job_class=job.job_class,
            job_id=job.job_id,
            queue_age_ms=done - job.enqueued_at_ms,
            deadline_slack_ms=slack_ms(job, done),
            reason="work_done",
            ts_ms=done,
        )


def enqueue_burst(q: MemQueue) -> None:
    start = now_ms()
    for i in range(20):
        q.push(
            Job(
                sort_key=i,
                job_id=f"prod-{uuid.uuid4().hex[:8]}",
                job_class="prod",
                enqueued_at_ms=start,
                work_ms=PROD_WORK_MS,
                deadline_ms=start + PROD_DEADLINE_MS,
            )
        )
    for i in range(40):
        q.push(
            Job(
                sort_key=100 + i,
                job_id=f"probe-{uuid.uuid4().hex[:8]}",
                job_class="probe",
                enqueued_at_ms=start,
                work_ms=PROBE_WORK_MS,
                deadline_ms=None,
            )
        )


def main() -> None:
    q = MemQueue()
    stop = threading.Event()
    t = threading.Thread(target=worker, args=(q, stop), daemon=True)
    t.start()
    enqueue_burst(q)
    # Drain window for the declared burst only.
    time.sleep(25)
    stop.set()
    t.join(timeout=2)
    log_line(action="cleanup", depth=q.depth(), reason="stop", ts_ms=now_ms())


if __name__ == "__main__":
    main()

Run it with a short, boring command:

python3 admit_probe_reject.py | tee /tmp/admit.jsonl
python3 - <<'PY'
import json
from collections import Counter
c = Counter()
with open("/tmp/admit.jsonl") as f:
    for line in f:
        row = json.loads(line)
        c[(row.get("action"), row.get("job_class"))] += 1
print(c)
PY

Labeled Expected Output

This is expected lab output, not a live capture.
Your exact job ids will differ on every run.
The shape of actions should stay stable.

{"action": "admit", "job_class": "prod", "queue_age_ms": 12, "deadline_slack_ms": 3188, ...}
{"action": "reject_probe", "job_class": "probe", "queue_age_ms": 812, "reason": "age_vs_slack"}
{"action": "complete", "job_class": "prod", "deadline_slack_ms": 2380, ...}
{"action": "cleanup", "depth": 0, "reason": "stop"}

Count reject_probe after the first production job starts.
Count reject_prod only if slack already hit zero.
If probes complete while prod slack is negative, the drill failed.

Failure Injection You Should Repeat

Inject one change at a time. Keep the workload declared.

1. Disable the age gate and rerun the same burst.
2. Watch production completions land after 4000 ms slack.
3. Restore the gate and confirm probe rejects rise first.
4. Raise AGE_REJECT_PROBE_MS to 5000 and watch slack collapse again.

The broken run is the useful one.
You should see prod deadline_slack_ms cross zero.
That is the signal you would have missed behind green probes.

Optional shell check for the broken run:

python3 - <<'PY'
import json
bad = 0
with open("/tmp/admit.jsonl") as f:
    for line in f:
        row = json.loads(line)
        if row.get("action") == "complete" and row.get("job_class") == "prod":
            if row.get("deadline_slack_ms", 0) <= 0:
                bad += 1
print({"prod_completed_after_slack_zero": bad})
PY

Expected after a disabled gate: bad greater than zero.
Expected after the restored gate: bad equal to zero.
Do not treat that as a model quality score.

Free Capacity Is The Wrong Bet Here

Free model access looks costless on a dashboard.
Free server option looks like spare queue depth.
Eval traffic still occupies the same worker slot.

If you already prototype against MonkeyCode free model access, keep probe batches off that worker once queue age rises. The free server option is useful lab capacity, not a dump for overnight evals. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Cost shows up as wasted retries after missed slack.
Time shows up as queue age, not as invoice line items.
Rejecting probes is cheaper than replaying production work.

Do not move failed production jobs onto the free path.
That failover hides first-token wait behind a “free” label.
This drill stops before that second failure mode.

Rollback And Cleanup

Ship the reject as a flag, not a silent filter.
Turn probes back on with one config change.
Keep the log field so you can prove the rollback.

admission:
  reject_probe_when_queue_age_ms: 500
  reject_prod_when_slack_ms: 0
  probe_class_enabled: true

Cleanup for the local drill:

pkill -f admit_probe_reject.py || true
rm -f /tmp/admit.jsonl

If this logic already sits in a deployed worker, rollback means:

1. Set probe_class_enabled to false.
2. Drain memq:probe without starting new evals.
3. Watch deadline_slack_ms for class prod only.
4. Re-enable probes after age stays under 500 ms.

Do not drain by raising concurrency on the free worker.
That hides the gate and trains the next eval burst.
Fix the class policy, then restore volume.

Who Should Not Use This Gate

Skip this if you run a single-user notebook.
Skip this if eval never overlaps production jobs.
Skip this if you have no probe class to reject.

Also skip it for offline batch with no deadline.
A nightly job without slack cannot use this table.
You would only add rejects without a customer signal.

Limits You Should State Out Loud

Local sleeps are not hosted tail latency.
500 ms age is a starting threshold, not an SLO.
This drill does not claim free capacity will remain.

The script admits in memory, not across nodes.
It does not model preemption, cold start, or disk.
Those are separate failure drills with other signals.

Do not publish the lab counts as product throughput.
Do not copy the sleeps into a vendor comparison.
The only honest result is the reject order under load.

The Threshold Question To Answer Next

Pick one threshold and write the rationale down.
Queue age, utilization, or deadline slack?
You need one primary signal on the page.

Use queue age when slots block without CPU heat.
Use slack when the customer deadline is already known.
Use utilization only to disprove a saturation story.

If your next eval night still shares a free worker, add the reject log line before the burst, not after the page.