Debug Free-Slot Stalls With Low CPU And Queue Age

Your on-call pager lights up at 02:14 UTC.
The scoring batch still holds ninety seconds of slack.
The free worker shows CPU at eighteen percent.

You admit the job on that idle-looking signal.
Six minutes later the deadline is already gone.
One long request held the only free slot.

Utilization looked healthy during the entire stall.
Wall-clock wait told a completely different story.
Token cost stayed zero while the SLO burned.

Split the three signals before you admit

You treated free tokens as a capacity grant.
That hid the only clock your users feel.
Calendar time still moves on a free worker.

Watch these three fields in one panel:

• worker_cpu_util at 0.18, which looks safe
• queue_age_ms at 240000, which is not
• deadline_slack_ms at 90000, already fatal

Admit on utilization and you miss the stall.
Admit because tokens are free and you miss it.
Admit only when wait plus retry still fits slack.

Low CPU plus rising queue age means head-of-line blocking.
The occupant is not burning a CPU budget.
It is holding the only runnable slot.

When free capacity is the wrong bet

Shared free workers fail in slow, quiet ways.
They rarely fail like dedicated billed replicas do.
You should reject the free path in these cases.

Condition	Free bet	Action
Expected wait plus one retry exceeds slack	Wrong	Shed or route paid
Queue age rises while CPU stays under 30%	Wrong	Suspect HOL; reject
Timeout retry would resend the full prompt	Wrong	Cap retries at zero
Slack covers estimated wait and one retry	Maybe	Admit with a kill switch
Work is interactive with unbounded wait	Wrong	Do not use free capacity

Queue age is a wait proxy, not a proof.
Treat it as an operational estimate only.
Replace it with a histogram when you have one.

Free tokens do not cancel occupant hold time.
A stuck slot makes every later job late.
That is opportunity cost, not a billing line.

Local topology for the drill

Keep every process on loopback for this drill.
Do not aim this drill at production.
You need one slot, one occupant, one probe.

Run these three pieces together:

1. A single-slot worker that can sleep on command
2. A FIFO listener in front of that worker
3. An admission gate using slack and queue age

Declared workload for this labeled exercise:

• Worker concurrency equals one
• Occupant request sleeps for 25 seconds
• Probe job is given 12 seconds of slack
• Retry budget equals one attempt of eight seconds
• Success means completion before slack hits zero

This is a local fault injection, not a benchmark.
Do not publish these timings as capacity numbers.
They only show the admit inequality.

Artifact: wait-versus-slack admission gate

The gate estimates wait from current queue age.
It adds one retry timeout as calendar cost.
It rejects free capacity when that sum beats slack.

# local_drill/admit_free.py
# Labeled local example. Not production. Not a benchmark.

from dataclasses import dataclass

@dataclass
class GateInput:
    now_ms: int
    deadline_ms: int
    queue_age_ms: int
    inflight: int
    concurrency: int
    retry_timeout_ms: int
    retry_budget: int
    cpu_util: float

def remaining_slack_ms(inp: GateInput) -> int:
    return inp.deadline_ms - inp.now_ms

def expected_wait_ms(inp: GateInput) -> int:
    # Architectural inference: age grows with occupancy.
    if inp.concurrency <= 0:
        return 10**9
    extra = max(inp.inflight, 1 if inp.queue_age_ms > 0 else 0)
    return inp.queue_age_ms + extra * 1000

def admit_free(inp: GateInput) -> dict:
    slack = remaining_slack_ms(inp)
    wait = expected_wait_ms(inp)
    retry_cost = inp.retry_timeout_ms * max(inp.retry_budget, 0)
    hol = inp.cpu_util < 0.30 and inp.queue_age_ms > 5000
    need = wait + retry_cost
    decision = "admit_free"
    reason = "slack_covers_wait_and_retry"
    if slack <= 0:
        decision, reason = "shed", "deadline_already_past"
    elif hol:
        decision, reason = "reject_free", "head_of_line_with_low_cpu"
    elif need >= slack:
        decision, reason = "reject_free", "wait_plus_retry_beats_slack"
    return {
        "decision": decision,
        "reason": reason,
        "slack_ms": slack,
        "expected_wait_ms": wait,
        "retry_cost_ms": retry_cost,
        "need_ms": need,
        "hol_suspect": hol,
    }

Feed the contradiction case into the gate.

# local_drill/case_hol.py
from admit_free import GateInput, admit_free

probe = GateInput(
    now_ms=1_700_000_000_000,
    deadline_ms=1_700_000_000_000 + 12_000,
    queue_age_ms=25_000,
    inflight=1,
    concurrency=1,
    retry_timeout_ms=8_000,
    retry_budget=1,
    cpu_util=0.18,
)
print(admit_free(probe))

Expected output from the gate math follows.
These values are labeled expected output, not samples.
They are not sampled from a production cluster.

{
  "decision": "reject_free",
  "reason": "head_of_line_with_low_cpu",
  "slack_ms": 12000,
  "expected_wait_ms": 26000,
  "retry_cost_ms": 8000,
  "need_ms": 34000,
  "hol_suspect": true
}

You reject the free worker on that result.
Tokens would have cost nothing on the retry.
The deadline would still have been missed.

Inject the occupant and read telemetry

Start a one-slot worker that can hold the line.
The occupant path sleeps for twenty-five seconds.
The probe path sleeps two seconds after dequeue.

python3 - <<'PY'
import socket, threading, time

lock = threading.Lock()
state = {"busy": False, "started": time.time()}

def handle(conn):
    with lock:
        state["busy"] = True
    try:
        data = conn.recv(64) or b"probe"
        hold = 25 if data.startswith(b"occupant") else 2
        time.sleep(hold)
        conn.sendall(b"done")
    finally:
        with lock:
            state["busy"] = False

srv = socket.socket()
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", 9408))
srv.listen(8)
print("worker_listen 127.0.0.1:9408")
while True:
    conn, _ = srv.accept()
    threading.Thread(target=handle, args=(conn,), daemon=True).start()
PY

Then fire occupant and probe with wall timestamps.

python3 - <<'PY' &
import socket, time
s = socket.create_connection(("127.0.0.1", 9408))
t0 = time.time()
s.sendall(b"occupant")
s.recv(16)
print(f"occupant_done_s {time.time()-t0:.1f}")
s.close()
PY

sleep 0.4

python3 - <<'PY'
import socket, time
deadline = time.time() + 12
s = socket.create_connection(("127.0.0.1", 9408))
s.settimeout(12)
t0 = time.time()
try:
    s.sendall(b"probe")
    s.recv(16)
    print(f"probe_ok slack_left_s {deadline-time.time():.1f}")
except Exception as e:
    print(f"probe_miss class={type(e).__name__} waited_s {time.time()-t0:.1f}")
finally:
    s.close()
PY

Expected local behavior under the declared workload:

• Occupant holds the slot for about 25 seconds
• Probe hits the 12 second socket timeout
• Process CPU stays low during the sleep
• Queue wait, not tokens, consumes the slack

Log these fields on every admit decision:

• deadline_slack_ms
• queue_age_ms
• inflight
• worker_cpu_util
• admit_decision
• retry_attempt
• wall_wait_ms

If queue age rises while CPU stays low, freeze admits.
That field pair is your head-of-line signature.
Remaining free tokens do not override that signal.

Retry cost on a free path

A timed-out free call is still expensive.
You repay the wait and resend the full prompt.
You also spend tokens on the second attempt.

Allow a retry only with leftover slack remaining.
The rule below is for this drill.
It is not a universal retry policy yet.

def allow_retry(slack_ms: int, timeout_ms: int, attempt: int) -> bool:
    if attempt >= 1:
        return False
    return (2 * timeout_ms) < slack_ms

If the first wait already ate the slack, stop.
A second attempt cannot buy the deadline back.
It can only pile more work on the stuck slot.

Count retry tokens as real spend even here.
Free model access does not make resends free.
Wall-clock and tokens both move on retry.

Rollback and cleanup

Stop admission the moment head-of-line is suspected.
Let the occupant finish or kill it by deadline.
Do not retry the probe on the same slot.

pkill -f "worker_listen 127.0.0.1:9408" || true
ss -ltnp | grep 9408 || echo "9408_clear"

Rollback sequence for a real free-tier drain:

1. Freeze free-tier admission on HOL or slack breach
2. Finish inflight work or kill it at the deadline
3. Route new work to a billed worker, or shed it
4. Re-enable free admission after queue age falls

Do not raise concurrency on the contended free host.
That packing failure belongs to a different drill.
This note only answers wait versus slack.

Confirm cleanup with a port check and a quiet queue.
Leave no occupant thread after the drill ends.
Reset the admit flag only after both are clear.

Where a free model path still fits

Some lab jobs have no user deadline.
You can park those jobs on free model access.
You can park them on a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Use that path only for deadline-free rehearsal jobs.

Keep the wait gate even on rehearsal traffic.
A lab job can still pin the only free slot.
Occupant holds hurt the next person in line.

Limitations

This gate uses queue age as a wait proxy.
That choice is inference, not a trace proof.
A real wait histogram should replace it later.

Who should skip this approach entirely:

• Interactive chat with unbounded human wait tolerance
• Multi-tenant production without an isolation boundary
• Teams that do not attach deadlines to jobs
• Anyone treating local sleeps as capacity ratings

Free capacity is a lab and overflow tool.
It is a weak backbone for a hard SLO.
Time stays scarce after tokens become free.

Do not export the 25 second sleep as a rating.
Do not claim a cluster size from loopback.
Do not skip paid capacity for deadline work.

Pick an operational threshold

Write one threshold and the rationale beside it.
Do not leave the admit bit as tribal knowledge.
Start with these numbers for the local drill.

• Reject free admission when queue_age_ms + retry_timeout_ms >= 0.5 * deadline_slack_ms
• Reject free admission when CPU is under 30% and queue age exceeds 5s
• Never retry when remaining slack is under two timeouts

Why spend only half the slack on waiting?
Dequeue, model load, and tail latency still need room.
Spending all slack on wait leaves no recovery path.

Ask on-call one question before the next free pack.
Does queue age, utilization, or slack own admit?
If utilization owns it, rerun this loopback drill.