Reject Incident Debug On Free Capacity Before Queue Age Beats Page Slack

The on-call phone lights up at 02:17.
Completion p95 now sits at 8.4 seconds.
Your error budget still assumed 2.5 seconds only.

Queue age on replica-staging-free reads 41 seconds.
A nightly eval job still occupies every worker slot.
Your incident debug prompt waits behind two hundred synthetic items.

Which operational action follows from that evidence?
You do not raise worker concurrency on that box.
You do not retry the debug completion against the same queue.

You reject page-path work from free shared capacity.
Then you drain the eval flood with a deadline.
Then you fail the page path over to dedicated capacity.

This opening is a drill you can reproduce

Treat those timestamps as a tabletop, not a vendor postmortem.
The metric shape is what you must recognize under stress.
Later commands recreate the same queue-age inversion locally.

Free capacity looks cheap until a page needs a worker now.
Token invoices stay quiet while MTTA burns minutes.
That is the cost contradiction this note isolates.

Topology for the lab

Run one admission process and one worker process together.
Keep a FIFO queue in memory for the drill.
Do not introduce a cluster. Localhost is enough.

• admitd: classifies work and records queue age
• workerd: pops items and sleeps a declared service time
• flood.py: enqueues nightly eval jobs
• page_debug.py: enqueues one incident completion

Backing flag is an explicit string, not autodetection.
Use free_shared or dedicated in the config file.
You will flip that flag and rerun the same flood.

Config

# admit-lab.yaml
listen: 127.0.0.1:8087
backing: free_shared
page_queue_age_ms: 2000
eval_queue_age_ms: 30000
worker_slots: 2
service_time_eval_ms: 50
service_time_page_ms: 80
deadline_slack_page_ms: 2500
drain_wall_clock_s: 60

Declared conditions stay in that file.
Change one field per run.
Write the filename into every log line.

Scratch endpoint for the flood only

You need some completion URL for the flood script.
Do not paste production API keys into the lab shell.
MonkeyCode's free model access and free server option can stand in as that scratch URL while you learn the reject path.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Leave page_debug pointed at dedicated capacity anyway.
The free server is for eval floods and classifier practice.
That is the cost lesson, not a migration plan.

Declared workload

You will run three waves in order.

1. Prime two eval jobs so both worker slots go busy.
2. Flood two hundred eval jobs with no pacing.
3. Submit one page_debug item with a 2.5s deadline.

Do not mix production traffic into this lab.
Do not point the flood at a shared team endpoint.
Stay on loopback until the reject path is obvious.

Telemetry fields you actually log

Log these fields on every admit decision.

• ts
• work_class
• backing
• queue_depth
• queue_age_ms
• deadline_slack_ms
• util_slots
• admit_result
• reject_reason

Watch queue_age_ms and deadline_slack_ms first.
Slot utilization lies during mixed completion work.
One long eval can hold 50% utilization and still kill pages.

Ask the threshold question before you pick a number.
Is the SLO bound queue age, utilization, or deadline slack?
For page-path completions, prefer deadline slack and queue age.

Utilization can sit at 100% on a healthy batch box.
That same 100% is fatal for a 2.5s page deadline.
Queue age is the signal that matches the user wait.

Admission rule

Page-path classes are page_debug and slo_probe.
Eval classes are nightly_eval and batch_score.
Canary traffic is staging_canary and never shares the page path.

Reject page_debug when any condition below is true.

• backing == free_shared
• queue_age_ms > page_queue_age_ms
• deadline_slack_ms < queue_age_ms + service_time_page_ms

Reject more evals when queue_age_ms > eval_queue_age_ms.
That second gate protects the drain, not the page path.
The page path should already be elsewhere.

Original artifact: the admit lab

Save this as page_path_admit.py.
It is a labeled lab, not production control software.
Expected lines are marked in the final printout.

#!/usr/bin/env python3
"""Local work-class admission drill. Labeled lab only."""
from __future__ import annotations

import json
import time
from collections import deque
from dataclasses import dataclass, field
from typing import Deque, Dict, Optional

PAGE_CLASSES = {"page_debug", "slo_probe"}
EVAL_CLASSES = {"nightly_eval", "batch_score"}


@dataclass
class Cfg:
    backing: str = "free_shared"
    page_queue_age_ms: int = 2000
    eval_queue_age_ms: int = 30000
    worker_slots: int = 2
    service_time_eval_ms: int = 50
    service_time_page_ms: int = 80
    deadline_slack_page_ms: int = 2500
    drain: bool = False


@dataclass
class Item:
    work_class: str
    enqueued_mono: float
    deadline_slack_ms: int


@dataclass
class Lab:
    cfg: Cfg
    q: Deque[Item] = field(default_factory=deque)
    busy: int = 0
    decisions: list = field(default_factory=list)

    def queue_age_ms(self) -> int:
        if not self.q:
            return 0
        return int((time.monotonic() - self.q[0].enqueued_mono) * 1000)

    def util_slots(self) -> float:
        return self.busy / float(self.cfg.worker_slots)

    def admit(self, work_class: str) -> Dict:
        age = self.queue_age_ms()
        slack = self.cfg.deadline_slack_page_ms
        svc = (
            self.cfg.service_time_page_ms
            if work_class in PAGE_CLASSES
            else self.cfg.service_time_eval_ms
        )
        reason = None
        if work_class in PAGE_CLASSES and self.cfg.backing == "free_shared":
            reason = "backing_not_for_page_path"
        elif work_class in PAGE_CLASSES and age > self.cfg.page_queue_age_ms:
            reason = "queue_age_exceeds_page_slo"
        elif work_class in PAGE_CLASSES and slack < age + svc:
            reason = "deadline_slack_exhausted"
        elif work_class in EVAL_CLASSES and self.cfg.drain:
            reason = "drain_blocks_eval"
        elif work_class in EVAL_CLASSES and age > self.cfg.eval_queue_age_ms:
            reason = "eval_queue_age_exceeded"
        rec = {
            "ts": time.time(),
            "work_class": work_class,
            "backing": self.cfg.backing,
            "queue_depth": len(self.q),
            "queue_age_ms": age,
            "deadline_slack_ms": slack,
            "util_slots": round(self.util_slots(), 3),
            "admit_result": "reject" if reason else "admit",
            "reject_reason": reason,
        }
        self.decisions.append(rec)
        if reason is None:
            self.q.append(
                Item(work_class, time.monotonic(), slack)
            )
        return rec

    def start_workers_for_head(self) -> None:
        while self.q and self.busy < self.cfg.worker_slots:
            self.q.popleft()
            self.busy += 1


def run_declared_workload(backing: str, drain: bool = False) -> None:
    lab = Lab(Cfg(backing=backing, drain=drain))
    for _ in range(2):
        lab.admit("nightly_eval")
    lab.start_workers_for_head()
    # Busy slots, empty visible queue: age starts at 0 until flood waits.
    time.sleep(0.05)
    for _ in range(200):
        lab.admit("nightly_eval")
    page = lab.admit("page_debug")
    print(json.dumps({"expected_label": True, "page_decision": page}, indent=2))


if __name__ == "__main__":
    print("# wave free_shared")
    run_declared_workload("free_shared")
    print("# wave dedicated after flood (age should trip)")
    run_declared_workload("dedicated")
    print("# wave drain")
    run_declared_workload("free_shared", drain=True)

Commands

python3 -c "import yaml,sys; print('py ok')"
cp admit-lab.yaml admit-lab.yaml.bak
python3 page_path_admit.py | tee /tmp/admit-lab.log
grep backing_not_for_page_path /tmp/admit-lab.log

Flip only the backing string on the second mental run.
Do not change worker slots to "make the page fit."
If you need more slots, you needed dedicated capacity.

Expected output (labeled)

Do not treat these lines as a production measurement.
They are the script's own expected printout for the declared YAML.

# wave free_shared
page_decision.reject_reason = backing_not_for_page_path
page_decision.admit_result = reject
page_decision.backing = free_shared

# wave dedicated after flood
page_decision.reject_reason = queue_age_exceeds_page_slo
  OR deadline_slack_exhausted
# (age climbs once 200 evals sit behind 2 busy slots)

# wave drain
nightly_eval later items -> drain_blocks_eval
page_debug on free_shared -> backing_not_for_page_path

If page_debug admits on free_shared, the classifier is wrong.
Stop there. Do not proceed to a cluster flag change.
Fix the class map first.

Metric contradiction to name out loud

Deadline slack shrinking is the second signal after age.
If slack is 2500ms and age is 41000ms, reject immediately.
Retries will not create a free worker. Drain will.

Token cost of the eval flood can still look excellent.
Time cost of the page is the bill that actually moved.
Write both numbers in the incident ticket, not only tokens.

When free capacity is the wrong bet

Use this decision table during capacity reviews.

Work class	Free shared backing	Dedicated backing	Cut when
nightly_eval	Admit until eval age	Admit	eval age > 30s
staging_canary	Admit, isolated queue	Optional	mixed with pages
slo_probe	Reject always	Admit	backing is free_shared
page_debug	Reject always	Admit	age > 2s or slack < age + service
customer interactive	Reject always	Admit	any shared free replica

Free capacity is a lab and batch bet.
It is the wrong bet for pages, probes, and incident debug.
Token price going to zero does not move MTTA.

Time cost during a page dominates leftover free tokens.
A twelve minute incident debug queue is not a discount.
It is an availability charge you pay in attention.

Failure injection

Inject these faults one at a time.

1. Eval flood with backing: free_shared. Expect page_debug reject backing_not_for_page_path.
2. Same flood with backing: dedicated and age already over 2s. Expect reject queue_age_exceeds_page_slo.
3. Mark drain: true. Expect eval admits to stop. Expect page path still rejected on free backing.
4. Raise worker_slots while keeping free_shared. Expect the page reject to remain. Slots do not change work class.

If reject reasons disagree with the table, stop the drill.
Fix the classifier before you touch production flags.
Do not hot-patch by raising worker slots.

Failure handling during a real page

Follow this order. Do not skip isolation.

1. Mark the free replica backing=free_shared in the service catalog.
2. Block page_debug and slo_probe at admission. Return 503 with Retry-After and a dedicated base URL.
3. Stop nightly eval producers. Do not SIGKILL in-flight evals yet.
4. Drain with a wall clock, for example 60 seconds.
5. If drain misses the wall clock, shed evals and keep slots for dedicated failover.
6. Open the page path only on dedicated backing.
7. File the cost note: incident minutes versus eval tokens saved.

Return 503, not 429, for class mismatch.
429 invites clients to retry the same poisoned queue.
503 plus an alternate base URL breaks that loop.

# sketched gate in an existing proxy, not a new mesh
curl -sS -D - http://127.0.0.1:8087/complete \
  -H 'X-Work-Class: page_debug' \
  -H 'X-Backing: free_shared' \
  --max-time 3 \
  -o /tmp/body.txt || true
# expected: HTTP/1.1 503, Retry-After: 0, X-Reject-Reason: backing_not_for_page_path

Cleanup and rollback

After the lab, undo every local mutation.

pkill -f page_path_admit.py || true
rm -f /tmp/admit-lab.queue /tmp/admit-lab.log
cp admit-lab.yaml.bak admit-lab.yaml

Rollback in a real cluster follows the same shape.
Revert the admission ConfigMap to the last tagged file.
Confirm work_class=page_debug hits dedicated backing only.

Keep the eval flood script off PATH in production images.
A leftover injector is an incident factory.
Delete it in the same change that disables the gate override.

Who should not use this approach

Do not use free shared capacity as your only model path.
This drill assumes a dedicated failover exists.
If you have one box, you do not have a page-path choice.

Do not apply these rejects to offline batch windows.
Nightly evals can wait. That is their contract.
Starving them on dedicated GPUs wastes the expensive pool.

Do not copy the in-memory queue into production.
You need persistence, authn, and a real request class header.
This file teaches the decision, not a new mesh.

Limitations

Service times here are sleeps, not decoder traces.
They show queue math. They do not show GPU kernels.
Do not publish these sleeps as throughput benchmarks.

The 2.5s page SLO is a declared lab budget.
Pick your own number from your customer contract.
Write that number into the YAML before the first flood.

Free model access and a free server do not create isolation.
Isolation is an admission rule plus a second backing.
Without that pair, cheap capacity is just a longer queue.