Reject Calendar-Wait Work Before Queue Age Breaks Slack

PagerDuty fires at 02:14 UTC on the batch lane.
Token counters for the hour still read zero.
The oldest job already shows negative deadline slack.

You stare at a green cost dashboard.
You also stare at a red SLO panel.
Free capacity did not fail on money today.
It failed on wall-clock time instead of spend.

Read the contradiction first

Cheap tokens do not buy you calendar time.
A free worker that is not yours yet is a wait.
Wait burns slack even when spend stays at zero.

You should treat queue age as a first-class cost.
You should not treat a zero invoice as a healthy lane.
Those two signals still answer different operational questions.

Ask one operational question before you admit more work.
Does remaining slack still cover wait plus p95 runtime?
If the answer is no, reject the job on purpose.
Do not let a free lane turn wait into an outage.

Lab topology

Run this as a local fault drill.
Do not point it at production traffic.

Proposed topology, labeled as a lab fixture:

• Redis list lane.free as the admission queue
• One worker process with a configurable hold
• A sidecar that records slack and spend
• A hard deadline per job, not a soft TTL

# docker-compose.lab.yml — proposed local topology
services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
  worker:
    build: .
    environment:
      QUEUE_KEY: lane.free
      HOLD_MS: "0"
      P95_RUNTIME_MS: "8000"
    depends_on: [redis]

Declared workload for this drill:

• Twenty synthetic inference tasks in one burst
• Deadline at 45 seconds after enqueue time
• Expected warm p95 runtime of 8 seconds
• Free-lane token price treated as 0.00
• Wall-clock penalty of 1.0 per extra wait second

Those numbers are lab fixtures, not production measurements.
Replace them with your own SLO math before copying the gate.

What free actually costs you

Token ledgers still hide idle-wait from you.
Retries then hide that same wait twice.
Queueing hides it until the deadline is already dead.

You pay in four currencies on a free lane:

1. Queue age before a worker claims the job
2. Runtime after claim, including any cold start
3. Retry amplification if the claim later drops
4. Operator time spent explaining a green invoice

Only item two looks like model spend.
The other three still break the customer SLO.
A zero token price only deletes one column.

Artifact: price wait at admission

The gate below is a proposed local checker.
It is not a production admission controller.
It reads queue age, remaining slack, and a p95 hint.
It rejects admission when wait would spend the deadline.

#!/usr/bin/env python3
"""lab_admit.py — proposed free-lane admission checker."""
import json
import time
import redis

QUEUE = "lane.free"
P95_RUNTIME_MS = 8_000
WALLCLOCK_WEIGHT = 1.0  # slack-units per extra wait second


def slack_ms(job):
    return int(job["deadline_unix"] * 1000 - time.time() * 1000)


def queue_age_ms(r):
    raw = r.lindex(QUEUE, 0)
    if not raw:
        return 0
    head = json.loads(raw)
    return int(time.time() * 1000 - head["enqueued_ms"])


def effective_cost(wait_ms, token_cost):
    extra_s = max(0.0, wait_ms / 1000.0)
    return token_cost + extra_s * WALLCLOCK_WEIGHT


def decide(r, job, token_cost=0.0):
    wait = queue_age_ms(r)
    remain = slack_ms(job)
    need = wait + P95_RUNTIME_MS
    cost = effective_cost(wait, token_cost)
    admit = remain > need and remain > 0
    return {
        "job_id": job["id"],
        "queue_age_ms": wait,
        "slack_ms": remain,
        "need_ms": need,
        "token_cost": token_cost,
        "wallclock_cost": cost,
        "admit": admit,
        "reason": "admit" if admit else "reject_wait_beats_slack",
    }


if __name__ == "__main__":
    r = redis.Redis(decode_responses=True)
    raw = r.lindex(QUEUE, 0)
    if not raw:
        print(json.dumps({"reason": "empty_queue"}))
    else:
        print(json.dumps(decide(r, json.loads(raw)), indent=2))

Seed the overdue head job with Python, not shell date math.
That keeps the fixture reproducible on Linux and macOS.

#!/usr/bin/env python3
"""lab_seed.py — enqueue one tight-deadline lab job."""
import json
import time
import redis

now = time.time()
job = {
    "id": "lab-014",
    "enqueued_ms": int((now - 12) * 1000),
    "deadline_unix": now + 18,
}
r = redis.Redis(decode_responses=True)
r.delete("lane.free")
r.lpush("lane.free", json.dumps(job))
print(json.dumps(job, indent=2))

Expected output after you seed that overdue job:

{
  "job_id": "lab-014",
  "queue_age_ms": 12000,
  "slack_ms": 18000,
  "need_ms": 20000,
  "token_cost": 0.0,
  "wallclock_cost": 12.0,
  "admit": false,
  "reason": "reject_wait_beats_slack"
}

Token cost stays at zero in this fixture.
The admission gate still rejects the job.
That is the whole point of the drill.
Wait plus p95 already exceeds remaining slack.

Seed, inject, and watch

Start Redis, then seed, then ask the gate.

docker compose -f docker-compose.lab.yml up -d redis
python3 lab_seed.py
python3 lab_admit.py

Inject a free-lane stall in the worker next.
Sleep inside the worker instead of running inference.

#!/usr/bin/env python3
"""worker.py — lab worker with an injected hold."""
import os
import time
import json
import redis

HOLD_MS = int(os.environ.get("HOLD_MS", "0"))
r = redis.Redis(decode_responses=True)
raw = r.lindex("lane.free", 0)
if not raw:
    raise SystemExit("empty_queue")
job = json.loads(raw)
time.sleep(HOLD_MS / 1000.0)
print(json.dumps({"held_ms": HOLD_MS, "job_id": job["id"]}))

HOLD_MS=30000 python3 worker.py
python3 lab_admit.py

Watch these telemetry fields on every decision:

• queue_age_ms
• slack_ms
• need_ms
• token_cost
• wallclock_cost
• admit
• reason

If CPU is idle and queue_age_ms climbs, you lack spare capacity.
In that case you only have an admission lie.
Idle cores behind a long free queue still miss the deadline.

Thresholds you must declare

Do not copy a magic number from this lab.
Write the reject threshold beside the SLO.
Use this admission decision order every time:

1. If slack_ms <= 0, reject and page. The job is already late.
2. If queue_age_ms + p95_runtime_ms >= slack_ms, reject admission.
3. If token_cost == 0 but wallclock_cost exceeds burn budget, reject.
4. Only then admit the job onto the free lane.

Rationale sits in the deadline, not in the invoice.
Queue age should beat utilization in this drill.
A busy-looking worker can still meet slack.
An empty free worker behind a long queue cannot.

You should compare these three signals every time:

• Queue age: how long the head job has waited
• Utilization: whether the worker is actually busy
• Deadline slack: time left before the SLO breaks

You should trust remaining deadline slack first.
Utilization without slack is a vanity metric.
A cheap lane that cannot finish is not cheap.

When free capacity is the wrong bet

Free lanes win for catch-up batch with loose deadlines.
They lose for anything that still has a customer clock.
Write that split down before the next incident.

Skip the free lane when you see any of these:

• Interactive or near-interactive SLOs under two minutes
• Deadlines tighter than observed queue age plus p95
• Retry storms that multiply wait, not just tokens
• Drain tests that must finish before a release window
• Incident work where calendar time is the scarce resource

Keep the free lane when the job can survive delay:

• Backfill, eval, and replay jobs with hour-scale slack
• Load you can drop without paging a human
• Experiments where a reject is cheaper than a miss

A zero token price does not change that table.
It only removes the token spend column.
Time, retries, and queueing still sit on the books.

Rehearse on a free experiment host

You can rehearse this gate before you touch prod.
MonkeyCode offers free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the same slack math on that host.
Do not treat the free server as spare production capacity.
Use it to inject holds and practice reject-before-miss.
If the free host queue ages past your slack, reject there too.
The drill is the policy; the vendor is just a box.

Failure handling and rollback

When the gate starts rejecting, follow this order:

1. Stop enqueueing onto lane.free
2. Snapshot queue_age_ms and slack_ms for the head ten jobs
3. Drain or dead-letter anything already negative
4. Move only jobs with remain > need to a dedicated worker
5. Leave the rest rejected; do not blind-retry onto the same list

Rollback the drill itself with a short cleanup path:

redis-cli DEL lane.free
unset HOLD_MS
docker compose -f docker-compose.lab.yml down -v

If you changed an alert, restore the previous threshold.
Do not leave a lab HOLD_MS in a shared compose file.

Cleanup checklist before you walk away:

• Delete seeded jobs from lane.free
• Drop the lab Redis volume
• Clear any synthetic pages you fired
• Record the threshold you used, with units

Limitations

This lab gate does not schedule GPUs.
It also does not predict worker preemption.
It does not replace a real autoscaler.

It also assumes you know p95 runtime.
If that hint is a lie, the reject is a lie.
Measure runtime on the same class of worker you will use.

Who should not use this approach:

• Teams without a numeric deadline per job
• Platforms that must run every enqueue, regardless of slack
• Workloads where wait is the product, such as long research sweeps
• Anyone hoping a free invoice will hide a missed ship time

If you cannot name slack in milliseconds, stop.
Fix the deadline instrumentation before any gate.

After the alert

Now go back to that 02:14 UTC alert.
Zero spend was not a healthy signal.
Negative slack was the only real signal.

Reject calendar-wait work before queue age breaks slack.
Price wait in the same record as tokens.
Keep free capacity for jobs that can survive the queue.
If you rehearse on a free server, keep rejects loud.
A quiet zero-cost miss is still a miss.