DEV Community: Odd_Background_328

Shed Stale Compaction Before Queue Age Misses the Turn

Odd_Background_328 — Thu, 24 Sep 2026 18:17:52 +0000

You catch the latency page at 02:14 UTC.
Chat p95 sits past the four-second turn budget.
The serving GPU is idle, so saturation is not the cause.

A background compaction job still occupies the free path.
Its transcript version is already behind the live turn.
The caller moved on six seconds before this page.

Free-path queue age reads eleven seconds in your collector.
Remaining slack on that turn is four seconds.
Another summary cannot land before the next token is due.

Read the contradiction first

Do not open this incident with a model swap.
The idle GPU and the late queue disagree.
Idle compute does not cancel a queued generation.

You pay tokens for work the turn will never read.
Separate what you measured from what you infer.
You measured queue age, slack, and version skew.

You infer that a zero price hid a deadline miss.
That inference is not a metric.
Keep it out of the alert name.

Name the topology

Keep this drill on one laptop you control.
You need four boxes and one shared clock.

An API process owns the turn deadline.
A compaction worker reads only a versioned transcript.
A local queue can age on purpose.
Two adapters exist: a free path and a paid path.

MonkeyCode states free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Treat both as availability notes, not as latency promises.

Do not hard-code a model name from memory.
Read the live console before you bind a path.
Neither availability claim is a latency SLO you can page on.

Grants, hold times, and model lists change without your deploy.
A number copied from an old post is not a control.
If the console is silent, leave the free path dark.

Declare the workload

This is a local drill, not a production trace.
You inject queue age instead of sampling production.
Label every number below as declared drill input.

Use these conditions, and do not widen them mid-run.

One compaction job per turn, never a fan-out.
Versions increase only when the user sends text.
Estimated generation time is a constant you pass in.
Token estimate is characters divided by four, rounded up.
Retry count starts at zero and stops at two.
The token ceiling is a budget you choose.

Do not paste a vendor grant into this file.
A grant is not your ceiling unless you measured spend.
Your ceiling should be smaller than the turn can afford.

Gate the job

Admit compaction only when all three checks pass.
Reject stale work before you touch the queue.
Reject late work before you touch the free path.
Reject over-budget work before a retry multiplies tokens.

from dataclasses import dataclass

@dataclass(frozen=True)
class TurnBudget:
    slack_ms: int
    queue_age_ms: int
    est_gen_ms: int
    est_tokens: int
    retry_count: int
    token_ceiling: int
    transcript_version: int
    latest_version: int

def admit_compaction(b: TurnBudget) -> str:
    if b.transcript_version != b.latest_version:
        return "reject_stale"
    projected_ms = b.queue_age_ms + b.est_gen_ms
    if projected_ms >= b.slack_ms:
        return "reject_slack"
    spend = b.est_tokens * (b.retry_count + 1)
    if spend > b.token_ceiling:
        return "reject_tokens"
    return "admit"

The first check is a fact about versions.
The second check compares queue wait with slack.
The third check enforces a ceiling you set.

A free price does not skip any of these checks.
The paid path uses the same three checks.
Price changes the bill, not the deadline math.

Decision table

Signal you hold	Comparison	Action
Version skew	job version != latest	`reject_stale`
Time	queue age + est gen >= slack	`reject_slack`
Tokens	est tokens × (retries + 1) > ceiling	`reject_tokens`
All three clear	current, inside slack, under ceiling	`admit`

Queue age beats utilization as the shed signal.
A cool GPU can still sit behind an old queue.
Deadline slack is the budget that actually expires.

Run the drill

Save that function as admit.py in a scratch directory.
Then run the three cases declared in this section.

python - <<'PY'
from admit import TurnBudget, admit_compaction

cases = [
    TurnBudget(4000, 11000, 1500, 800, 0, 5000, 7, 9),
    TurnBudget(9000, 1200, 1500, 800, 2, 2000, 9, 9),
    TurnBudget(9000, 1200, 1500, 800, 0, 5000, 9, 9),
]
for case in cases:
    print(admit_compaction(case))
PY

The expected lines below come from this function.
They are not a vendor benchmark and not production.

reject_stale
reject_tokens
admit

Case one is stale, so age never matters.
Case two is current, but retries blow the ceiling.
Case three is current, inside slack, and under budget.

If your printout differs, stop and diff the function.
Do not tune thresholds until the labels match.
A mismatched label means the gate is not the one you reviewed.

What each input stands for

Case one mirrors the page you opened.
Slack is 4000 ms and queue age is 11000 ms.
Version 7 is behind version 9, so you shed first.

Case two is on time and on the current version.
Two retries turn 800 estimated tokens into 2400.
Your ceiling is 2000, so you stop before submit.

Case three is the only admit in this drill.
Queue age plus estimate is 2700 ms.
Slack is 9000 ms, and spend stays at 800.

Change one field and rerun before you trust a new threshold.
Write the new expected line next to the change.
If you cannot predict the label, you do not understand the gate.

When the free path is the wrong bet

Use the free path only for work that can expire.
Compaction can expire cleanly.
A user-visible answer cannot.

Shed the job when any line below is true.

Queue age plus estimated generation meets slack.
The transcript version moved after enqueue.
Retries would push estimated tokens over your ceiling.
You cannot cancel the generation after submit.
The result must return inside this same turn.
You have no fresh console reading for the free path.

A zero price is not a reason to wait.
Waiting spends wall time the caller already lost.
Retries then spend tokens on a summary nobody reads.

That is the cost failure, not a model quality debate.
You lost slack first.
You burned tokens second.

Do not failover an interactive turn onto the free server.
The free server option is still a queue with unknown age.
Unknown age is a reason to shed, not a reason to hope.

If hold time is unpublished, do not invent one.
If the token grant is unpublished, do not invent one.
Schedule only against numbers you just read.

Failure handling

On reject_stale, drop the job and ack the queue.
Do not requeue it under a new idempotency key.
A new key hides a duplicate spend in your bill.

On reject_slack, leave the last good summary in place.
Serve the current turn with the uncompacted window.
Page only if this reject rate crosses your own threshold.

Pick that threshold from missed turns, not from pride.
A starting page rule is five slack rejects in five minutes.
Replace it once you have a week of your own series.

On reject_tokens, stop retries for that version.
Emit compaction_rejected_total with the reason label attached.

Suggested fields, not a required vendor schema:

queue_age_ms
slack_ms
est_tokens
retry_count
transcript_version
reason

If you cannot cancel after submit, skip that path.
Uncancelable free work becomes silent token debt later.
Debt without a cancel handle is an ops bug.

A tiny collector check

After a reject, your log line should carry the reason.
Grep the scratch log before you call the drill done.

python - <<'PY'
print("reason=reject_slack queue_age_ms=11000 slack_ms=4000")
PY
grep -E 'reason=reject_(stale|slack|tokens)' /tmp/compaction-drill.log || true

The print is a format sample, not a caught incident.
The grep proves your eye can find the label.
If the label is missing, the gate is not operable.

Cleanup and rollback

The drill queue is local.
Drain it before you leave the shell.

# scratch only — do not point this at production
rm -f /tmp/compaction-drill.queue /tmp/compaction-drill.log
python -c "from admit import admit_compaction; print('rollback-ok')"

Roll the gate back with one config flag.

compaction:
  enabled: false
  free_path: disabled
  token_ceiling: 0
  max_retries: 0

Disabling the worker is the rollback, not a rewrite.
The chat path must keep serving without summaries.
If chat fails when compaction is off, fix that coupling first.

Re-enable only after three clean drill labels.
Then watch reject rate, queue age, and token spend together.
A green GPU graph alone is not a reopen signal.

Who should not use this gate

Skip this pattern if you lack a turn deadline.
Skip it when transcript versions are not monotonic.
Skip it if free capacity is your only generation path.

This gate sheds work.
It does not create capacity.
Teams that need every summary will see more full windows.

That is a product choice, not a bug in the check.
Do not use this drill as a capacity certificate.
One laptop queue does not prove a hosted server.

Do not publish these case numbers as latency results.
Do not treat a free server as a standby SLO.
Do not let a retry loop compact the same version twice.

What you should do next

Read queue age beside slack on the next late page.
If age already exceeds slack, reject the compaction job.
Confirm the live free-path limits in your console first.

If those limits are missing, keep the free path dark.
A missing hold time is not a zero hold time.
Point this gate at a MonkeyCode free path only after a fresh console check.

Reject Interactive Leases Before Free-Server Hold Time Beats Slack

Odd_Background_328 — Wed, 23 Sep 2026 16:04:32 +0000

02:14, two metrics disagree

Picture this on-call scene before you touch replicas.
Your pager fires during an otherwise quiet batch window.
The oldest summarizer job has waited 740 seconds already.

Your service objective still demands completion within 480 seconds.
The free pool reports only 19 percent CPU busy.
You almost add replicas to chase that age number.

That scale-out would hide the fault and waste money.
Do not scale on this metric contradiction just yet.
Ask which operational action the evidence actually supports.

Low utilization with rising age means stolen worker leases.
The scarce resource is hold time, not processor cycles.
Treat CPU as a contradiction check, not the page.

What you are actually running

You share one serving pool across two work classes.
Batch summarization carries a hard completion deadline tonight.
Interactive debug sessions carry no deadline of their own.

Use this proposed topology for the local drill.
Keep it on one box so blast radius stays tiny.

The gateway handles admission decisions and lease tracking.
The free-server process runs shared model serving locally.
The batch-worker submits deadline-bound jobs with an 8-minute SLO.
The debug-client holds notebooks, curl loops, and open streams.

Declare these test conditions before any fault injection.
Copy them into the drill script header before you run.

Run one serving replica on the shared local pool.
Fix maximum concurrent leases at two worker slots.
Keep batch compute near 25 seconds for each job.
Fix the batch deadline at 480 seconds wall clock.
Let the interactive session hold one slot for 180 seconds.
Keep interactive CPU load near five percent on purpose.
Allow three retries with a 10-second backoff budget.
Arrive six batch jobs within a 40-second window.

You will measure hold time, not CPU, as the scarce resource.
Utilization will lie while a debug client holds the slot.

Why free capacity is the wrong bet

Free model access looks cheap on a quiet dashboard.
A free server often looks idle under debug sessions.
Idle CPU does not mean idle workers on that box.

A lease occupies a slot until the client hangs up.
Interactive sessions hold connections far beyond useful compute.
Streaming responses stay open while you reread the prompt.

Retries clone the same hold without adding progress.
Queue age grows while the processor mostly sleeps.
You should treat hold time as the scarce resource now.

Queue age versus deadline slack beats utilization here.
Reject interactive work when slack cannot absorb another hold.
Do not wait for CPU to look busy first.

Time, tokens, retries, and queueing move together under a hold.
A cheap pool still spends finite tokens on duplicated retries.
Free capacity is the wrong bet when hold time stays unbounded.

Telemetry you need on the box

Export these fields from the gateway and server.
Without them you will page on vanity CPU again.

lease_hold_seconds tracks how long a slot stays checked out.
lease_class labels the holder as batch or interactive.
queue_age_seconds records the oldest waiting batch job.
deadline_slack_seconds equals deadline minus now minus remaining compute.
retry_count captures debug storms and batch copies together.
token_in and token_out stay finite even when unmarked free.
cpu_busy_ratio exists only for contradiction checks on-call.

Keep these PromQL sketches labeled as unexecuted examples.
Do not paste them into production without a recording rule.

deadline_slack_seconds
  = batch_deadline_seconds
  - queue_age_seconds
  - expected_compute_seconds

reject_interactive
  = deadline_slack_seconds < 90
    and lease_hold_seconds{class="interactive"} > 60

If slack falls under 90 seconds, reject new interactive leases.
You should not wait for CPU saturation as confirmation.

Local fault-injection drill

Run this drill on your laptop before touching production.
The script occupies one lease with almost no CPU.
It then watches batch age while the slot stays held.

# lease_hold_drill.py — proposed local drill, not production telemetry
import json
import os
from collections import deque
from dataclasses import dataclass, field

BATCH_DEADLINE_S = 480
MAX_LEASES = 2
INTERACTIVE_HOLD_S = 180
BATCH_COMPUTE_S = 25
SLACK_REJECT_S = 90


@dataclass
class Lease:
    cls: str
    t0: float
    job_id: str


@dataclass
class Pool:
    leases: list = field(default_factory=list)
    batch_q: deque = field(default_factory=deque)
    log: list = field(default_factory=list)

    def slack(self, now):
        if not self.batch_q:
            return BATCH_DEADLINE_S
        age = now - self.batch_q[0][1]
        return BATCH_DEADLINE_S - age - BATCH_COMPUTE_S

    def admit(self, cls, job_id, now, reject_interactive):
        slack = self.slack(now)
        if cls == 'interactive' and reject_interactive and slack < SLACK_REJECT_S:
            self.log.append({
                't': now, 'event': 'reject_interactive',
                'job': job_id, 'slack': round(slack, 1),
            })
            return False
        if len(self.leases) >= MAX_LEASES:
            if cls == 'batch':
                self.batch_q.append((job_id, now))
                self.log.append({
                    't': now, 'event': 'enqueue_batch',
                    'job': job_id, 'q': len(self.batch_q),
                })
            else:
                self.log.append({
                    't': now, 'event': 'reject_interactive_full',
                    'job': job_id,
                })
            return False
        self.leases.append(Lease(cls, now, job_id))
        self.log.append({
            't': now, 'event': 'lease', 'cls': cls,
            'job': job_id, 'hold_slots': len(self.leases),
        })
        return True

    def release(self, job_id, now):
        self.leases = [lease for lease in self.leases if lease.job_id != job_id]
        while self.batch_q and len(self.leases) < MAX_LEASES:
            jid, enq = self.batch_q.popleft()
            age = now - enq
            self.leases.append(Lease('batch', now, jid))
            self.log.append({
                't': now, 'event': 'dequeue_batch', 'job': jid,
                'age': round(age, 1),
                'slack': round(BATCH_DEADLINE_S - age - BATCH_COMPUTE_S, 1),
            })


def run_drill(reject_interactive=True):
    pool = Pool()
    pending_batch = []
    arrivals = {5, 10, 15, 22, 30, 40}
    for t in range(0, 220):
        now = float(t)
        if t == 0:
            pool.admit('interactive', 'dbg-1', now, reject_interactive)
        if t == 8 and not reject_interactive:
            pool.admit('interactive', 'dbg-2', now, reject_interactive)
        if t in arrivals:
            ok = pool.admit('batch', f'b-{t}', now, reject_interactive)
            if ok:
                pending_batch.append((f'b-{t}', now + BATCH_COMPUTE_S))
        finished = [job for job, end in pending_batch if end <= now]
        pending_batch = [(job, end) for job, end in pending_batch if end > now]
        for job in finished:
            pool.release(job, now)
        if t == INTERACTIVE_HOLD_S:
            pool.release('dbg-1', now)
        if t == INTERACTIVE_HOLD_S + 5 and not reject_interactive:
            pool.release('dbg-2', now)
    return pool.log


if __name__ == '__main__':
    mode = os.environ.get('DRILL_MODE', 'reject')
    log = run_drill(reject_interactive=(mode == 'reject'))
    print(json.dumps(log, indent=2))

Run both admission modes and compare queue age.
Save the logs so you can diff hold time later.

python3 lease_hold_drill.py > /tmp/reject.json
DRILL_MODE=admit python3 lease_hold_drill.py > /tmp/admit.json

python3 - <<'PY'
import json
for name in ('reject', 'admit'):
    rows = json.load(open('/tmp/%s.json' % name))
    ages = [row['age'] for row in rows if row.get('event') == 'dequeue_batch']
    rejects = sum(1 for row in rows if row['event'].startswith('reject'))
    print(name, 'dequeued', len(ages), 'max_age', max(ages, default=0),
          'rejects', rejects)
PY

Expected output, clearly labeled unexecuted

On a quiet laptop the shape should look like this.
Treat the block as a target shape, not a benchmark claim.

reject dequeued 6 max_age 25.0 rejects 1
admit  dequeued 6 max_age 180.0 rejects 0

Your exact numbers will move with scheduler timing.
The shape should not move if the leak is real.
A 180-second hold on two slots starves batch slack.

Slack-based admission keeps max age near compute time.
The admit mode spends hold time, retries, and tokens together.
That is the cost story hiding under a calm CPU chart.

Decision table: keep free capacity or reject

Use this table during the 02:14 argument.
Pick one primary signal before you debate replicas.

Signal	Threshold	Action	Rationale
Interactive lease while slack is under 90s	90s slack	Reject debug	Deadline work owns the slots
CPU under 0.3 and queue age over 60s	60s age	Do not scale	This is a hold leak
Interactive retry count over 2	2 retries	Shed the client	Retries clone hold time
Tokens grow with no batch progress	Any growth	Kill the lease	Free tokens are still finite
Slack over 180s and a slot free	180s slack	Admit debug	Spare slack is the only window

Ask which threshold you would defend at 02:14.
Choose among queue age, utilization, and deadline slack.
Pick deadline slack as the primary reject signal.

Utilization lags and flatters a quietly held worker.
Queue age without remaining compute still lies to you.
Slack folds remaining compute into the same reject line.

Where a free isolated box still helps

You still need a place to run the drill.
Production serving is the wrong blast radius tonight.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode provides free model access and a free server option.
Use that isolation for the lease drill described above.

Do not park deadline-bound production batch on that pool.
A free shared pool is a lab, not spare production capacity.
Treat it as a blast-radius box you can throw away.

Inject holds, watch slack, then destroy the box.
If you already isolate drills on a free server, keep both signals.
Place deadline_slack_seconds beside lease_hold_seconds on that board.

Failure handling

Inject these faults on the laptop only.
Stop if any step escapes the local process namespace.

Hold one streaming client open for 180 seconds.
Retry the same debug prompt three times on timeout.
Start six batch jobs during that interactive hold.
Watch deadline slack cross the 90-second line.
Confirm new interactive leases return 429 with reason.

Use this proposed gateway body for the reject path.
Keep the reason field stable so clients can fail closed.

{
  "error": "interactive_lease_rejected",
  "reason": "deadline_slack_seconds below 90",
  "queue_age_seconds": 410,
  "retry_after_seconds": 120
}

Do not convert that 429 into a client retry loop.
Debug clients must fail closed on admission rejects.
Batch clients may retry with jitter and a budget.

Rollback and cleanup

Tear the drill down before you walk away.
Leftover leases are how tomorrow's 02:14 page is born.

kill %1 2>/dev/null || true
curl -sf -X POST localhost:8080/admin/drain || true
rm -f /tmp/reject.json /tmp/admit.json

Rollback if admission shipped wider than you intended.
Keep metrics hot even while the reject flag is off.

Flip the reject_interactive feature flag to off.
Keep emitting slack metrics so you are not blind.
Cap interactive leases at one until you re-enable.
Page on slack, never on CPU busy ratio alone.

Leave the pool empty after the last batch drain.
Do not leave the notebook holding a production-like slot.

Who should not use this approach

Skip this pattern if you have dedicated debug hardware.
Skip it when every job already has a hard lease timeout.
Skip it if you do not hold a batch deadline at all.

Do not mix paid production traffic onto a free shared server.
Noisy neighbors and preemptible capacity will break the story.
Free capacity is the wrong bet under unbounded hold time.

Limitations

The script is a discrete-time toy, not a load test.
It does not model tokenizer stalls or network jitter.
It does not name models, quotas, or hardware SKUs.

Your real threshold will differ from ninety seconds.
Treat 90 seconds as a starting argument, not law.
Measure expected_compute_seconds from traces, never from hope.

Token counters still matter when the invoice reads zero.
Retries spend both tokens and hold time together.
Count both before you call the pool cheap.

What you do at 02:14

You do not scale the quiet pool under this alert.
You kill the debug lease that stole the slot.
You reject new interactive work until slack recovers.

File hold time as the cause, not low CPU.
Rerun the local drill after the reject flag stays on.
Admit debug sessions only when slack is honestly large.

Which page threshold will you defend tomorrow morning?
Will it be slack, age, or utilization under lease contention?

Reject Tool-Call Fanout Before Token Burn Beats Slack

Odd_Background_328 — Tue, 22 Sep 2026 14:28:33 +0000

You get paged at 02:14 UTC tonight.
Queue age already sits at 47 seconds.
Interactive CPU still looks almost idle.

You open the token meter next.
Spend is climbing at 3x the planned rate.
Deadline slack on the batch window is 90 seconds.

Which operational action follows that evidence?
Scale the free worker, or reject the fanout class?
This note argues for reject, then prove it locally.

The contradiction you should not ignore

Queue age is high. Utilization is low.
That pairing is not a capacity shortage.
It is usually a work-shape problem under metering.

Tool-calling jobs hide nested HTTP work.
Each nested call can enqueue sibling calls.
Retries then multiply tokens without moving the head job.

Free capacity makes that shape look cheap.
Wall-clock slack still expires on the original deadline.
Tokens still leave the shared pool either way.

Topology for the local drill

Keep the lab topology tiny and explicit.

admitd: admission proxy with class rules
workq: FIFO queue with age gauges
worker: one process, one in-flight job
meter: token and retry counters
deadline: remaining slack per job class

Do not share this worker with interactive traffic.
Label every number below as a lab expectation.
Do not treat it as a production observation.

Proposed config, saved as admitd.yaml:

classes:
  tool_fanout:
    max_depth: 1
    max_siblings: 2
    reject_when:
      queue_age_ms: 15000
      slack_ms_below: 60000
      tokens_per_success_est: 8000
  interactive:
    max_depth: 0
    reserved: true
queue:
  visibility_timeout_s: 30
  max_inflight: 1
meter:
  window_s: 60
rollback:
  drain_timeout_s: 45
  fail_closed: true

Declared lab workload:

20 parent jobs, each with two tool calls.
Each tool call may spawn one retry on 429.
Parent deadline is 120 seconds after enqueue.
Worker concurrency stays at one.
No other publishers touch workq.

Start from the alert, not the model card

You should bind alerts to evidence fields.
Do not page on “tokens used” alone.
Page on slack versus age versus burn together.

Proposed telemetry fields:

queue_age_ms
deadline_slack_ms
tokens_in_window
retries_in_window
tool_depth
class
admit_decision

Expected lab log line, clearly labeled expected:

# expected lab output, not a live incident
ts=02:14:07 class=tool_fanout queue_age_ms=47012
deadline_slack_ms=90000 tokens_in_window=24110
retries_in_window=14 tool_depth=2 admit_decision=reject
reason=token_burn_vs_slack

If CPU is idle while that line prints, you do not scale.
You reject the class that fans out.
Then you drain what already entered.

Artifact: a fail-closed admission check

The checker below is a local, unexecuted example.
Run it against recorded metrics, not guesswork.
Keep it fail-closed when slack is unknown.

from dataclasses import dataclass

@dataclass
class Sample:
    queue_age_ms: int
    deadline_slack_ms: int
    tokens_in_window: int
    successes_in_window: int
    tool_depth: int
    retries_in_window: int

class Reject(Exception):
    pass


def tokens_per_success(sample: Sample) -> float:
    if sample.successes_in_window <= 0:
        return float("inf")
    return sample.tokens_in_window / sample.successes_in_window


def admit_tool_fanout(sample: Sample, cfg: dict) -> str:
    rules = cfg["classes"]["tool_fanout"]["reject_when"]
    if sample.deadline_slack_ms < rules["slack_ms_below"]:
        raise Reject("slack_below_floor")
    if sample.queue_age_ms > rules["queue_age_ms"]:
        raise Reject("queue_age_over_limit")
    if sample.tool_depth > cfg["classes"]["tool_fanout"]["max_depth"]:
        raise Reject("tool_depth_over_limit")
    est = tokens_per_success(sample)
    if est > rules["tokens_per_success_est"]:
        raise Reject("token_burn_vs_success")
    if sample.retries_in_window >= 10 and sample.successes_in_window == 0:
        raise Reject("retry_amplification")
    return "admit"

Wire a one-shot probe. Do not loop this in production yet.

python3 - <<'PY'
from admit_check import Sample, admit_tool_fanout, Reject
import yaml
cfg = yaml.safe_load(open("admitd.yaml"))
sample = Sample(
    queue_age_ms=47012,
    deadline_slack_ms=90000,
    tokens_in_window=24110,
    successes_in_window=1,
    tool_depth=2,
    retries_in_window=14,
)
try:
    print(admit_tool_fanout(sample, cfg))
except Reject as e:
    print("reject", e)
PY

Expected lab output:

reject tool_depth_over_limit

You now have a binary operational action.
Reject beats adding another free worker here.
The extra worker would only clone the fanout.

When free capacity is the wrong bet

Free model access can still be useful.
A free server option can isolate the drill.
Neither one repairs a bad work shape.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
If you need an isolated box for the same admission drill, MonkeyCode's free model access and free server option can host the worker without mixing it into paid interactive traffic.

Use this decision table before you bet on free capacity.

Evidence	Free capacity bet	Operational action
Queue age high, CPU idle, depth=1	Maybe	Keep one worker, watch slack
Depth >= 2, retries rising	Wrong	Reject class, fail closed
Tokens/success unknown	Wrong	Reject until meter exists
Slack < 60s, age > 15s	Wrong	Drain, do not enqueue more
Interactive class shares the pool	Wrong	Split queues before any scale
Successes move, retries stay flat	Acceptable	Keep free worker, cap depth

Ask for one threshold and one rationale.
Prefer deadline slack over utilization.
Utilization lies when nested calls wait on I/O.

Rationale you can defend in a review:

Queue age over 15s with one inflight job means head-of-line delay.
Slack under 60s leaves no drain window.
Tokens per success over 8k, in this lab, means nested waste.
Retry count without successes means amplification, not load.

Change those numbers only after a declared rerun.
Do not copy them into production blindly.

Failure injection you can run locally

Inject one fault at a time.
Record age, slack, tokens, and retries after each.
Stop if slack crosses the floor.

# 1) nested fanout
python3 inject_fanout.py --parents 20 --depth 2 --siblings 2

# 2) retry storm on 429
python3 inject_fanout.py --parents 20 --depth 1 --force-429 8

# 3) slow tool, no extra tokens
python3 inject_fanout.py --parents 5 --depth 1 --tool-sleep-ms 4000

# 4) meter blackout
python3 inject_fanout.py --parents 5 --drop-meter

Expected lab pattern, labeled expected:

Case 1: reject on tool_depth_over_limit.
Case 2: reject on retry_amplification.
Case 3: admit if age stays under 15s.
Case 4: reject because tokens per success is infinite.

Case 3 is the trap.
Slow tools raise age without burning tokens.
You still reject if age beats slack, not if tokens look fine.

Drain, rollback, cleanup

Rejection without drain still burns the in-flight parent.
Practice the rollback on the lab queue.

# stop publishers for tool_fanout only
curl -X POST localhost:8080/classes/tool_fanout/reject

# wait for inflight, then snapshot meters
sleep 45
curl -s localhost:8080/metrics > /tmp/meter.after-reject

# fail closed if drain exceeds timeout
if [ "$(curl -s localhost:8080/queue/age_ms)" -gt 15000 ]; then
  curl -X POST localhost:8080/queue/purge?class=tool_fanout
fi

# restore only after slack recovers
curl -X POST localhost:8080/classes/tool_fanout/enable \
  --data 'max_depth=1&max_siblings=2'

Cleanup the lab files after the drill.

rm -f /tmp/meter.after-reject
rm -rf ./workq-data ./meter-wal

If you pointed the worker at a free server, tear that worker down too.
Do not leave a preemptible box holding visibility locks.
Stale locks look like queue age the next morning.

What this drill does not claim

This is an admission workflow, not a model benchmark.
It does not name models, quotas, or hardware SKUs.
It does not claim free capacity is faster or slower.

It also does not replace tracing of each tool HTTP call.
You still need span ids on nested calls.
Without spans, depth is only an inference.

Separate observed fields from architectural guesses.
queue_age_ms is observed.
“The model is slow” is not observed here.

Who should not use this approach

Skip this reject path if you lack a token meter.
Skip it if parents have unknown tool graphs.
Skip it for user-facing streaming with sub-second SLOs.

Do not use a shared free worker for mixed classes.
Interactive work will inherit batch fanout delays.
Paid isolation is cheaper than a missed deadline then.

Do not tune thresholds during an active incident.
Run the injectors on a laptop first.
Promote the rule only after two clean lab passes.

The action that follows the 02:14 evidence

You had high age, idle CPU, and rising tokens.
That is reject-and-drain, not scale-on-free.
Free capacity is the wrong bet once fanout starts.

Keep depth at one on cheap workers.
Keep a slack floor of 60 seconds in this lab.
Keep retries from counting as progress.

Then rerun the four injectors after every parser change.
Tool schemas change more often than queue code.
Your reject rule must track the schema, not the slogan.

Reject Tool-Call Fanout Before Token Burn Beats Slack

Odd_Background_328 — Tue, 22 Sep 2026 14:20:32 +0000

You get paged at 02:14 UTC tonight.
Queue age already sits at 47 seconds.
Interactive CPU still looks almost idle.

You open the token meter next.
Spend is climbing at 3x the planned rate.
Deadline slack on the batch window is 90 seconds.

Which operational action follows that evidence?
Scale the free worker, or reject the fanout class?
This note argues for reject, then prove it locally.

The contradiction you should not ignore

Queue age is high. Utilization is low.
That pairing is not a capacity shortage.
It is usually a work-shape problem under metering.

Tool-calling jobs hide nested HTTP work.
Each nested call can enqueue sibling calls.
Retries then multiply tokens without moving the head job.

Free capacity makes that shape look cheap.
Wall-clock slack still expires on the original deadline.
Tokens still leave the shared pool either way.

Topology for the local drill

Keep the lab topology tiny and explicit.

admitd: admission proxy with class rules
workq: FIFO queue with age gauges
worker: one process, one in-flight job
meter: token and retry counters
deadline: remaining slack per job class

Do not share this worker with interactive traffic.
Label every number below as a lab expectation.
Do not treat it as a production observation.

Proposed config, saved as admitd.yaml:

classes:
  tool_fanout:
    max_depth: 1
    max_siblings: 2
    reject_when:
      queue_age_ms: 15000
      slack_ms_below: 60000
      tokens_per_success_est: 8000
  interactive:
    max_depth: 0
    reserved: true
queue:
  visibility_timeout_s: 30
  max_inflight: 1
meter:
  window_s: 60
rollback:
  drain_timeout_s: 45
  fail_closed: true

Declared lab workload:

20 parent jobs, each with two tool calls.
Each tool call may spawn one retry on 429.
Parent deadline is 120 seconds after enqueue.
Worker concurrency stays at one.
No other publishers touch workq.

Start from the alert, not the model card

You should bind alerts to evidence fields.
Do not page on “tokens used” alone.
Page on slack versus age versus burn together.

Proposed telemetry fields:

queue_age_ms
deadline_slack_ms
tokens_in_window
retries_in_window
tool_depth
class
admit_decision

Expected lab log line, clearly labeled expected:

# expected lab output, not a live incident
ts=02:14:07 class=tool_fanout queue_age_ms=47012
deadline_slack_ms=90000 tokens_in_window=24110
retries_in_window=14 tool_depth=2 admit_decision=reject
reason=token_burn_vs_slack

If CPU is idle while that line prints, you do not scale.
You reject the class that fans out.
Then you drain what already entered.

Artifact: a fail-closed admission check

The checker below is a local, unexecuted example.
Run it against recorded metrics, not guesswork.
Keep it fail-closed when slack is unknown.

from dataclasses import dataclass

@dataclass
class Sample:
    queue_age_ms: int
    deadline_slack_ms: int
    tokens_in_window: int
    successes_in_window: int
    tool_depth: int
    retries_in_window: int

class Reject(Exception):
    pass


def tokens_per_success(sample: Sample) -> float:
    if sample.successes_in_window <= 0:
        return float("inf")
    return sample.tokens_in_window / sample.successes_in_window


def admit_tool_fanout(sample: Sample, cfg: dict) -> str:
    rules = cfg["classes"]["tool_fanout"]["reject_when"]
    if sample.deadline_slack_ms < rules["slack_ms_below"]:
        raise Reject("slack_below_floor")
    if sample.queue_age_ms > rules["queue_age_ms"]:
        raise Reject("queue_age_over_limit")
    if sample.tool_depth > cfg["classes"]["tool_fanout"]["max_depth"]:
        raise Reject("tool_depth_over_limit")
    est = tokens_per_success(sample)
    if est > rules["tokens_per_success_est"]:
        raise Reject("token_burn_vs_success")
    if sample.retries_in_window >= 10 and sample.successes_in_window == 0:
        raise Reject("retry_amplification")
    return "admit"

Wire a one-shot probe. Do not loop this in production yet.

python3 - <<'PY'
from admit_check import Sample, admit_tool_fanout, Reject
import yaml
cfg = yaml.safe_load(open("admitd.yaml"))
sample = Sample(
    queue_age_ms=47012,
    deadline_slack_ms=90000,
    tokens_in_window=24110,
    successes_in_window=1,
    tool_depth=2,
    retries_in_window=14,
)
try:
    print(admit_tool_fanout(sample, cfg))
except Reject as e:
    print("reject", e)
PY

Expected lab output:

reject tool_depth_over_limit

You now have a binary operational action.
Reject beats adding another free worker here.
The extra worker would only clone the fanout.

When free capacity is the wrong bet

Free model access can still be useful.
A free server option can isolate the drill.
Neither one repairs a bad work shape.

Use this decision table before you bet on free capacity.

Evidence	Free capacity bet	Operational action
Queue age high, CPU idle, depth=1	Maybe	Keep one worker, watch slack
Depth >= 2, retries rising	Wrong	Reject class, fail closed
Tokens/success unknown	Wrong	Reject until meter exists
Slack < 60s, age > 15s	Wrong	Drain, do not enqueue more
Interactive class shares the pool	Wrong	Split queues before any scale
Successes move, retries stay flat	Acceptable	Keep free worker, cap depth

Ask for one threshold and one rationale.
Prefer deadline slack over utilization.
Utilization lies when nested calls wait on I/O.

Rationale you can defend in a review:

Queue age over 15s with one inflight job means head-of-line delay.
Slack under 60s leaves no drain window.
Tokens per success over 8k, in this lab, means nested waste.
Retry count without successes means amplification, not load.

Change those numbers only after a declared rerun.
Do not copy them into production blindly.

Failure injection you can run locally

Inject one fault at a time.
Record age, slack, tokens, and retries after each.
Stop if slack crosses the floor.

# 1) nested fanout
python3 inject_fanout.py --parents 20 --depth 2 --siblings 2

# 2) retry storm on 429
python3 inject_fanout.py --parents 20 --depth 1 --force-429 8

# 3) slow tool, no extra tokens
python3 inject_fanout.py --parents 5 --depth 1 --tool-sleep-ms 4000

# 4) meter blackout
python3 inject_fanout.py --parents 5 --drop-meter

Expected lab pattern, labeled expected:

Case 1: reject on tool_depth_over_limit.
Case 2: reject on retry_amplification.
Case 3: admit if age stays under 15s.
Case 4: reject because tokens per success is infinite.

Case 3 is the trap.
Slow tools raise age without burning tokens.
You still reject if age beats slack, not if tokens look fine.

Drain, rollback, cleanup

Rejection without drain still burns the in-flight parent.
Practice the rollback on the lab queue.

# stop publishers for tool_fanout only
curl -X POST localhost:8080/classes/tool_fanout/reject

# wait for inflight, then snapshot meters
sleep 45
curl -s localhost:8080/metrics > /tmp/meter.after-reject

# fail closed if drain exceeds timeout
if [ "$(curl -s localhost:8080/queue/age_ms)" -gt 15000 ]; then
  curl -X POST localhost:8080/queue/purge?class=tool_fanout
fi

# restore only after slack recovers
curl -X POST localhost:8080/classes/tool_fanout/enable \
  --data 'max_depth=1&max_siblings=2'

Cleanup the lab files after the drill.

rm -f /tmp/meter.after-reject
rm -rf ./workq-data ./meter-wal

If you pointed the worker at a free server, tear that worker down too.
Do not leave a preemptible box holding visibility locks.
Stale locks look like queue age the next morning.

What this drill does not claim

This is an admission workflow, not a model benchmark.
It does not name models, quotas, or hardware SKUs.
It does not claim free capacity is faster or slower.

It also does not replace tracing of each tool HTTP call.
You still need span ids on nested calls.
Without spans, depth is only an inference.

Separate observed fields from architectural guesses.
queue_age_ms is observed.
“The model is slow” is not observed here.

Who should not use this approach

Skip this reject path if you lack a token meter.
Skip it if parents have unknown tool graphs.
Skip it for user-facing streaming with sub-second SLOs.

Do not use a shared free worker for mixed classes.
Interactive work will inherit batch fanout delays.
Paid isolation is cheaper than a missed deadline then.

Do not tune thresholds during an active incident.
Run the injectors on a laptop first.
Promote the rule only after two clean lab passes.

The action that follows the 02:14 evidence

You had high age, idle CPU, and rising tokens.
That is reject-and-drain, not scale-on-free.
Free capacity is the wrong bet once fanout starts.

Keep depth at one on cheap workers.
Keep a slack floor of 60 seconds in this lab.
Keep retries from counting as progress.

Then rerun the four injectors after every parser change.
Tool schemas change more often than queue code.
Your reject rule must track the schema, not the slogan.

Reject Probe Jobs Before Free Queue Age Beats Slack

Odd_Background_328 — Mon, 21 Sep 2026 12:33:30 +0000

You get paged at 09:14 during an eval night.
Probe dashboards stay green across every synthetic check.
Production first-token wait still blows a four-second slack budget.

CPU on the free worker sits near twelve percent.
Admission queue age sits at forty-seven seconds instead.
Those two numbers do not describe the same failure.

Which operational action follows from that evidence?
Do you scale the free worker under low CPU?
Or do you reject the probe class before production slack dies?

What You Observed Versus What You Inferred

You observed healthy probes and broken production slack.
You observed high queue age beside idle CPU.
You did not observe a compute saturation event.

Inference is cheap here and usually wrong.
Idle CPU does not mean spare serving capacity.
Probe jobs can occupy the only slot without burning cores.

Treat queue age as the scarce signal tonight.
Treat utilization as a supporting field only.
Do not page from CPU until age and slack agree.

Topology You Can Reproduce Locally

Keep one producer, one admission loop, two job classes.
Class prod carries a deadline and remaining slack.
Class probe covers eval, canary, and synthetic traffic.

client --> admission.py --> memq:prod
                        --> memq:probe
worker_free <-- pop prod first
worker_free <-- pop probe only if age and slack allow
paid_path   <-- never used in this local drill

Pin the free worker to production when slack is thin.
Keep paid overflow off this drill on purpose.
That split is policy, not a latency benchmark.

Declared Lab Conditions

Run this on one local machine only.
Do not point the worker at production credentials.
Treat every number below as a lab fixture.

Declared workload for one drill:

20 production jobs, 800 ms fake work each
40 probe jobs, 400 ms fake work each
One probe burst at t=0 with production mixed in
Production deadline: 4000 ms after enqueue
Probe deadline: none, best effort only
Worker concurrency: 1
Admission tick: 50 ms

These sleeps are not model latency measurements.
They exist so queue age becomes visible without GPUs.
If your fake work is faster, raise the probe burst size.

Telemetry Fields To Log On Every Decision

Log one JSON line per admit, reject, or complete.
Keep field names stable so diffs stay readable.
You need age and slack on the same line.

ts_ms
job_id
job_class (prod or probe)
enqueued_at_ms
queue_age_ms
deadline_slack_ms
cpu_pct (optional, often misleading)
action (admit, reject_probe, reject_prod, complete)
reason

Queue age is now minus enqueue time.
Deadline slack is deadline minus now minus remaining work.
If slack is missing, you cannot reject on evidence.

Decision Table: Age First, CPU Second

Use this table before you add hardware.
The reject target is probe work, not production retries.
Production still needs an explicit slack floor.

queue_age_ms	prod slack_ms	probe action	prod action
< 500	> 1500	admit	admit
>= 500	> 1500	reject	admit
>= 500	1 to 1500	reject	admit if slack > remaining work
any	<= 0	reject	reject and page

Why queue age instead of utilization?
A blocked slot raises age without raising CPU.
Probe success can stay green while slack burns.

Why not reject production first?
Production still owns the only customer deadline.
Probes are optional measurement traffic tonight.

Local Admission Artifact

The script below is a proposed local drill.
It never calls a hosted model endpoint.
It only shows reject order under a declared burst.

#!/usr/bin/env python3
"""Reject probe jobs when queue age steals prod slack."""
from __future__ import annotations

import json
import threading
import time
import uuid
from dataclasses import dataclass, field
from typing import Literal

PROD_DEADLINE_MS = 4000
AGE_REJECT_PROBE_MS = 500
PROD_WORK_MS = 800
PROBE_WORK_MS = 400


@dataclass(order=True)
class Job:
    sort_key: int
    job_id: str = field(compare=False)
    job_class: Literal["prod", "probe"] = field(compare=False)
    enqueued_at_ms: int = field(compare=False)
    work_ms: int = field(compare=False)
    deadline_ms: int | None = field(compare=False, default=None)


class MemQueue:
    def __init__(self) -> None:
        self._lock = threading.Lock()
        self._items: list[Job] = []

    def push(self, job: Job) -> None:
        with self._lock:
            self._items.append(job)

    def peek_age_ms(self, now_ms: int) -> int:
        with self._lock:
            if not self._items:
                return 0
            oldest = min(self._items, key=lambda j: j.enqueued_at_ms)
            return now_ms - oldest.enqueued_at_ms

    def pop_class(self, job_class: str) -> Job | None:
        with self._lock:
            for i, job in enumerate(self._items):
                if job.job_class == job_class:
                    return self._items.pop(i)
            return None

    def depth(self) -> int:
        with self._lock:
            return len(self._items)


def now_ms() -> int:
    return time.time_ns() // 1_000_000


def log_line(**fields: object) -> None:
    print(json.dumps(fields, sort_keys=True), flush=True)


def slack_ms(job: Job, current_ms: int) -> int:
    if job.deadline_ms is None:
        return 10**9
    return job.deadline_ms - current_ms - job.work_ms


def admit_decision(q: MemQueue, job: Job, current_ms: int) -> str:
    age = q.peek_age_ms(current_ms)
    slack = slack_ms(job, current_ms)
    if job.job_class == "probe" and age >= AGE_REJECT_PROBE_MS:
        return "reject_probe"
    if job.job_class == "prod" and slack <= 0:
        return "reject_prod"
    return "admit"


def worker(q: MemQueue, stop: threading.Event) -> None:
    while not stop.is_set():
        current = now_ms()
        prod = q.pop_class("prod")
        job = prod or q.pop_class("probe")
        if job is None:
            time.sleep(0.05)
            continue
        action = admit_decision(q, job, current)
        age = current - job.enqueued_at_ms
        log_line(
            action=action,
            job_class=job.job_class,
            job_id=job.job_id,
            queue_age_ms=age,
            deadline_slack_ms=slack_ms(job, current),
            reason="age_vs_slack",
            ts_ms=current,
        )
        if action.startswith("reject"):
            continue
        time.sleep(job.work_ms / 1000)
        done = now_ms()
        log_line(
            action="complete",
            job_class=job.job_class,
            job_id=job.job_id,
            queue_age_ms=done - job.enqueued_at_ms,
            deadline_slack_ms=slack_ms(job, done),
            reason="work_done",
            ts_ms=done,
        )


def enqueue_burst(q: MemQueue) -> None:
    start = now_ms()
    for i in range(20):
        q.push(
            Job(
                sort_key=i,
                job_id=f"prod-{uuid.uuid4().hex[:8]}",
                job_class="prod",
                enqueued_at_ms=start,
                work_ms=PROD_WORK_MS,
                deadline_ms=start + PROD_DEADLINE_MS,
            )
        )
    for i in range(40):
        q.push(
            Job(
                sort_key=100 + i,
                job_id=f"probe-{uuid.uuid4().hex[:8]}",
                job_class="probe",
                enqueued_at_ms=start,
                work_ms=PROBE_WORK_MS,
                deadline_ms=None,
            )
        )


def main() -> None:
    q = MemQueue()
    stop = threading.Event()
    t = threading.Thread(target=worker, args=(q, stop), daemon=True)
    t.start()
    enqueue_burst(q)
    # Drain window for the declared burst only.
    time.sleep(25)
    stop.set()
    t.join(timeout=2)
    log_line(action="cleanup", depth=q.depth(), reason="stop", ts_ms=now_ms())


if __name__ == "__main__":
    main()

Run it with a short, boring command:

python3 admit_probe_reject.py | tee /tmp/admit.jsonl
python3 - <<'PY'
import json
from collections import Counter
c = Counter()
with open("/tmp/admit.jsonl") as f:
    for line in f:
        row = json.loads(line)
        c[(row.get("action"), row.get("job_class"))] += 1
print(c)
PY

Labeled Expected Output

This is expected lab output, not a live capture.
Your exact job ids will differ on every run.
The shape of actions should stay stable.

{"action": "admit", "job_class": "prod", "queue_age_ms": 12, "deadline_slack_ms": 3188, ...}
{"action": "reject_probe", "job_class": "probe", "queue_age_ms": 812, "reason": "age_vs_slack"}
{"action": "complete", "job_class": "prod", "deadline_slack_ms": 2380, ...}
{"action": "cleanup", "depth": 0, "reason": "stop"}

Count reject_probe after the first production job starts.
Count reject_prod only if slack already hit zero.
If probes complete while prod slack is negative, the drill failed.

Failure Injection You Should Repeat

Inject one change at a time. Keep the workload declared.

Disable the age gate and rerun the same burst.
Watch production completions land after 4000 ms slack.
Restore the gate and confirm probe rejects rise first.
Raise AGE_REJECT_PROBE_MS to 5000 and watch slack collapse again.

The broken run is the useful one.
You should see prod deadline_slack_ms cross zero.
That is the signal you would have missed behind green probes.

Optional shell check for the broken run:

python3 - <<'PY'
import json
bad = 0
with open("/tmp/admit.jsonl") as f:
    for line in f:
        row = json.loads(line)
        if row.get("action") == "complete" and row.get("job_class") == "prod":
            if row.get("deadline_slack_ms", 0) <= 0:
                bad += 1
print({"prod_completed_after_slack_zero": bad})
PY

Expected after a disabled gate: bad greater than zero.
Expected after the restored gate: bad equal to zero.
Do not treat that as a model quality score.

Free Capacity Is The Wrong Bet Here

Free model access looks costless on a dashboard.
Free server option looks like spare queue depth.
Eval traffic still occupies the same worker slot.

If you already prototype against MonkeyCode free model access, keep probe batches off that worker once queue age rises. The free server option is useful lab capacity, not a dump for overnight evals. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Cost shows up as wasted retries after missed slack.
Time shows up as queue age, not as invoice line items.
Rejecting probes is cheaper than replaying production work.

Do not move failed production jobs onto the free path.
That failover hides first-token wait behind a “free” label.
This drill stops before that second failure mode.

Rollback And Cleanup

Ship the reject as a flag, not a silent filter.
Turn probes back on with one config change.
Keep the log field so you can prove the rollback.

admission:
  reject_probe_when_queue_age_ms: 500
  reject_prod_when_slack_ms: 0
  probe_class_enabled: true

Cleanup for the local drill:

pkill -f admit_probe_reject.py || true
rm -f /tmp/admit.jsonl

If this logic already sits in a deployed worker, rollback means:

Set probe_class_enabled to false.
Drain memq:probe without starting new evals.
Watch deadline_slack_ms for class prod only.
Re-enable probes after age stays under 500 ms.

Do not drain by raising concurrency on the free worker.
That hides the gate and trains the next eval burst.
Fix the class policy, then restore volume.

Who Should Not Use This Gate

Skip this if you run a single-user notebook.
Skip this if eval never overlaps production jobs.
Skip this if you have no probe class to reject.

Also skip it for offline batch with no deadline.
A nightly job without slack cannot use this table.
You would only add rejects without a customer signal.

Limits You Should State Out Loud

Local sleeps are not hosted tail latency.
500 ms age is a starting threshold, not an SLO.
This drill does not claim free capacity will remain.

The script admits in memory, not across nodes.
It does not model preemption, cold start, or disk.
Those are separate failure drills with other signals.

Do not publish the lab counts as product throughput.
Do not copy the sleeps into a vendor comparison.
The only honest result is the reject order under load.

The Threshold Question To Answer Next

Pick one threshold and write the rationale down.
Queue age, utilization, or deadline slack?
You need one primary signal on the page.

Use queue age when slots block without CPU heat.
Use slack when the customer deadline is already known.
Use utilization only to disprove a saturation story.

If your next eval night still shares a free worker, add the reject log line before the burst, not after the page.

Reject Orphan Retries Before Token Waste Beats Slack

Odd_Background_328 — Sun, 20 Sep 2026 10:41:16 +0000

You open the 02:17 pager with two contradictory graphs. Queue depth looks calm under the admission cap. Token spend still climbs after clients already cancelled.

Which operational action follows from that split evidence? You do not scale the free overflow pool yet. You hunt orphan in-flight work before adding capacity.

Topology you can reproduce

You run mixed inference behind a single admission queue. Paid workers hold leases only for hard deadline work. A free overflow pool takes quiet-hour batch overflow only.

Name every hop in the runbook before the drill. Missing hops are how cancel events disappear.

The gateway records client cancel, deadline, and retry-id.
The admission queue exposes age, depth, and in-flight count.
Each worker heartbeats a lease keyed by task_id.
The token ledger accumulates cost per task_id only.
Overflow workers stay isolated from the paid lease pool.

That overflow pool is optional capacity, not an SLO. Treat it as a scratch replica during this rehearsal.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project you can use for that isolated replica. It provides free model access and a free server option.

Park the drill there, then tear the replica down. Do not attach production SLOs to that pool.

Declared workload

You must freeze the workload before you inject cancels. Changing payload size mid-drill invalidates the token ledger.

Write these knobs on the incident whiteboard first. Keep them unchanged until the drill ends.

Eight synthetic tasks share one twelve-second deadline each.
Each prompt is a fixed eight-hundred-token fixture file.
One client cancel arrives at three-and-a-half seconds.
Retry policy allows one retry with the same retry-id.
Overflow workers heartbeat their leases every four seconds.
Paid workers stay capped at two concurrent leases.

Do not treat these numbers as production benchmarks. They are drill knobs, not capacity claims.

Failure mode: orphan in-flight work

Clients cancel while the gateway still shows healthy depth. Workers keep decoding because leases never saw the cancel. Tokens still accrue against task_ids you already abandoned.

That is orphan in-flight work, not a capacity shortage. Free overflow makes the miss cheap to ignore at first. It becomes expensive when retries spawn a second decoder.

Watch three signals together, never one graph alone. One green graph is how this page stays open.

queue_age_ms can look healthy while decoders still run.
in_flight_by_retry_id should drop to zero after cancel.
tokens_by_task_id must freeze within one cancel grace.

If the third signal keeps moving, you have an orphan. Scaling workers will multiply the waste, not clear it.

Artifact: cancel-and-shed probe

You can reproduce the miss with a local probe. The script below is a rehearsal harness, not production code.

Save it as cancel_shed_probe.py beside a fixture prompt.

#!/usr/bin/env python3
"""Local cancel-and-shed rehearsal. Not a production worker."""
from __future__ import annotations

import argparse
import json
import threading
import time
from dataclasses import dataclass
from typing import Dict, Optional

DEADLINE_S = 12.0
CANCEL_AT_S = 3.5
CANCEL_GRACE_S = 1.0
DECODE_TOKENS_PER_S = 40  # drill knob only
PROMPT_TOKENS = 800


@dataclass
class Task:
    task_id: str
    retry_id: str
    deadline_mono: float
    canceled: bool = False
    in_flight: int = 0
    shed_reason: Optional[str] = None
    tokens_at_cancel: int = 0
    tokens_at_end: int = 0


class Ledger:
    def __init__(self) -> None:
        self._by_task: Dict[str, int] = {}
        self._lock = threading.Lock()

    def add(self, task_id: str, n: int) -> None:
        with self._lock:
            self._by_task[task_id] = self._by_task.get(task_id, 0) + n

    def get(self, task_id: str) -> int:
        with self._lock:
            return self._by_task.get(task_id, 0)


class Worker(threading.Thread):
    def __init__(
        self,
        task: Task,
        ledger: Ledger,
        cancel_event: threading.Event,
        orphan_mode: bool,
    ) -> None:
        super().__init__(daemon=True)
        self.task = task
        self.ledger = ledger
        self.cancel_event = cancel_event
        self.orphan_mode = orphan_mode

    def run(self) -> None:
        self.task.in_flight += 1
        self.ledger.add(self.task.task_id, PROMPT_TOKENS)
        start = time.monotonic()
        while True:
            now = time.monotonic()
            slack = self.task.deadline_mono - now
            if not self.orphan_mode:
                if self.cancel_event.is_set() or self.task.canceled:
                    self.task.shed_reason = 'client_cancel'
                    break
                if slack < CANCEL_GRACE_S:
                    self.task.shed_reason = 'deadline_slack'
                    break
            if now - start > DEADLINE_S * 1.5:
                self.task.shed_reason = 'drill_bound'
                break
            time.sleep(0.25)
            self.ledger.add(self.task.task_id, int(DECODE_TOKENS_PER_S * 0.25))
        self.task.in_flight -= 1
        self.task.tokens_at_end = self.ledger.get(self.task.task_id)


def run_one(mode: str, idx: int) -> dict:
    orphan_mode = mode == 'orphan'
    task = Task(
        task_id=f't-{idx}',
        retry_id=f'r-{idx}',
        deadline_mono=time.monotonic() + DEADLINE_S,
    )
    ledger = Ledger()
    cancel_event = threading.Event()
    worker = Worker(task, ledger, cancel_event, orphan_mode)
    worker.start()
    time.sleep(CANCEL_AT_S)
    task.canceled = True
    cancel_event.set()
    task.tokens_at_cancel = ledger.get(task.task_id)
    time.sleep(0.3)
    in_flight_after = task.in_flight
    worker.join(timeout=DEADLINE_S * 2)
    return {
        'mode': mode,
        'task': task.task_id,
        'retry_id': task.retry_id,
        'cancel_at_s': CANCEL_AT_S,
        'deadline_s': DEADLINE_S,
        'in_flight_after_cancel': in_flight_after,
        'tokens_at_cancel': task.tokens_at_cancel,
        'tokens_at_deadline': task.tokens_at_end,
        'shed_reason': task.shed_reason,
    }


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument('--mode', choices=['orphan', 'shed'], required=True)
    parser.add_argument('--tasks', type=int, default=1)
    args = parser.parse_args()
    rows = [run_one(args.mode, i) for i in range(args.tasks)]
    print(json.dumps(rows, indent=2))
    leaked = [r for r in rows if r['tokens_at_deadline'] > r['tokens_at_cancel'] + 20]
    if args.mode == 'orphan':
        print('VERDICT: orphan decode after cancel' if leaked else 'VERDICT: unexpected shed')
    else:
        print('VERDICT: shed within grace' if not leaked else 'VERDICT: cancel missed')


if __name__ == '__main__':
    main()

How you run it

Create a venv so the drill stays off the host Python. Install nothing beyond the standard library for this probe.

python3 -m venv .drill
source .drill/bin/activate
python3 cancel_shed_probe.py --mode orphan --tasks 8
python3 cancel_shed_probe.py --mode shed --tasks 8

Compare the two modes on the same fixture. Do not mix timestamps across the two runs.

Expected output (labeled, not observed production)

Orphan mode should look roughly like this example.

mode=orphan task=t-0 retry_id=r-0
cancel_at_s=3.50 deadline_s=12.00
in_flight_after_cancel=1
tokens_at_cancel=140
tokens_at_deadline=480
shed_reason=drill_bound
VERDICT: orphan decode after cancel

Shed mode should freeze tokens near the cancel point.

mode=shed task=t-0 retry_id=r-0
cancel_at_s=3.50 deadline_s=12.00
in_flight_after_cancel=0
tokens_at_cancel=140
tokens_at_deadline=152
shed_reason=client_cancel
VERDICT: shed within grace

Those figures are fixture math, not vendor benchmarks. Recalculate them if you change the decode knob.

Inject the cancel miss

You now break the cancel path on purpose. That is the drill, not a production experiment.

Run these faults one at a time. Combining faults hides which hop dropped cancel.

Drop the cancel event on overflow workers only.
Heartbeat leases slower than the remaining deadline slack.
Retry without forwarding the original retry-id header field.
Attribute tokens to worker_id instead of task_id.
Leave a second decoder running after the client left.

After each fault, capture these four telemetry fields.

deadline_slack_ms
lease_age_ms
orphan_token_delta
duplicate_in_flight

You promote a fault only when orphan_token_delta stays zero. Utilization is the last signal you should trust here.

Decision table: shed versus retry

Ask yourself this operational question before you ship. Do you shed on queue age, utilization, or deadline slack?

Write the rule in the runbook with a rationale.

Signal	Trust it for	Do not use it for
queue age	Backup shed when depth lies	Free-pool idle that hides orphans
utilization	Paid pool saturation only	Overflow CPU while decoding abandoned tokens
deadline slack	Primary cancel and shed trigger	Long batch jobs without a client deadline
orphan token delta	Proof the cancel path worked	Capacity planning or vendor comparison

The primary trigger is deadline slack, not utilization. Queue age is the backup when depth and spend disagree. Orphan token delta is the proof, not the actuator.

Proposed shed rule for this rehearsal only:

Shed if lease_age + cancel_grace > deadline_slack.
Shed if in_flight_by_retry_id > 1 after one retry.
Reject new overflow admission if orphan_token_delta > 0.
Never retry onto the free pool without the same retry-id.

Label that rule as a proposal until you measure it. Then store the measured grace on the service page.

When free overflow is the wrong bet

Free capacity is the wrong bet under several conditions. You should fail closed to paid workers, or reject the work.

Reject overflow admission when any line below is true. One true line is enough to fail closed.

Cancel propagation time exceeds the remaining deadline slack.
The worker has no cancel channel and no lease kill.
Retries lack a stable retry-id for duplicate detection.
The ledger keys cost by host instead of task.
Heartbeat period is larger than one third of the deadline.
A second decoder can start before the first sheds.

Time, tokens, retries, and queueing all stack here. A free slot that cannot shed still burns wall-clock slack. It also burns tokens the client will never read.

That is a cost incident, not a savings story. Paid path is cheaper when slack is already thin. Reject the work before the queue looks busy.

Waiting for utilization to spike is too late. Scale only after the orphan counter returns to zero.

Cleanup and rollback

You end the drill with an explicit teardown. Do not leave replica workers holding open leases.

pkill -f cancel_shed_probe.py || true
rm -f /tmp/orphan-ledger.json
deactivate

Rollback the admission rule if the shed was too aggressive. Restore the previous retry limit and heartbeat period. Record the change in the same incident ticket.

If production ever copied the fault, follow this order. Keep the steps in this exact rollback order.

Stop overflow admission at the gateway.
Cancel all leases older than one grace period.
Freeze the token ledger for affected task_ids.
Drain in-flight decoders before reopening overflow.
Reopen only after orphan_token_delta stays at zero.

Who should skip this drill

Skip this approach if you have no client deadlines. Batch jobs without slack cannot use shed-on-slack. Use a calendar drain window instead, with a human gate.

Skip it if you cannot key work by retry-id. Without that key you cannot detect duplicate decoders. Fix the identity model before you tune overflow.

Skip it if legal or safety work cannot be cancelled. Those tasks need a paid, interrupt-safe worker pool. Do not park them on optional free capacity.

Limitations

This probe does not speak any vendor model protocol. It fakes decode cost with a constant token knob. Real streaming cancel behavior will differ per gateway.

It also ignores prompt-cache effects and tokenizer drift. Treat the ledger as an ops counter, not a billing replica. Do not publish these fixture numbers as capacity results.

The topology assumes one retry and one overflow pool. Multi-region failovers need a different lease story. Leave that design work outside this runbook.

What you write in the runbook

Put one threshold on the service page tonight. State the signal, the number, and the rationale.

Example line you can adapt after the drill:

Shed overflow when lease_age + 1s > deadline_slack; utilization on the free pool is not an SLO.

Then ask the oncall the same question you opened with. Queue looks calm, tokens still move, clients already left. Do you scale the pool, retry the task, or shed?

You already know the answer from the split graphs. Shed first, then scale only after orphan_token_delta is zero.

Reject Paid-to-Free Failover Before First-Byte Wait Beats Slack

Odd_Background_328 — Sat, 19 Sep 2026 08:58:14 +0000

Your pager fires at 02:14 UTC during batch. Token spend sits near zero on the cost panel. Job p99 sits at forty-seven seconds against fifteen.

Free failover queue age reads thirty-eight seconds already. Remaining deadline slack is only nine seconds now. Which operational action follows from that evidence?

You do not enlarge the free pool tonight. You reject paid-to-free failover for that class.

Zero spend is not a recovery signal

A green cost panel can hide a dead SLO. Zero token spend often means jobs waited on free capacity. You paid with wall-clock time, not with tokens.

Deadline slack died while finance still looked healthy. Keep spend and latency on two separate ledgers. Never let a cheap dashboard veto a red latency page.

Topology for the local failover drill

Treat this as a labeled lab, not folklore. You need one admission process and two backends.

client -> admission (slack gate)
            |-- paid_backend   (reserved, metered)
            |-- free_backend   (best-effort FIFO)
            |-- reject_429     (fail closed)

Keep both backends on separate local ports. Do not hide them behind one reverse proxy.

You must record which path each job took. Log backend, queue_age_ms, first_byte_ms, and remaining_slack_ms together.

Declared workload

Write the conditions down before any fault injection.

Job class: 200-word summary, one shot, no tools
Arrival: eight jobs per second for ninety seconds
Deadline: fifteen seconds from enqueue timestamp
Paid concurrency: two workers, 1200ms p95 first-byte
Free concurrency: one worker after the FIFO wait
Free first-byte p95 after queue: 4000ms
Free queue: FIFO, no preemption, no reservation
Retry: none on timeout; reject the job instead
Success: first byte before slack falls under 2000ms

These figures are lab defaults only, not production. They are not product benchmarks or fleet observations.

Change them if your serving path differs. Keep every number declared in the runbook header.

When free capacity is the wrong bet

Paid-to-free failover looks smart during a burst. Overflow onto a free queue looks fine on invoices. All three fail when wait age exceeds remaining slack.

You are not buying inference in that window. You are buying a lottery ticket on queue delay.

Timeouts make the free-capacity bet worse fast. A timed-out job retries into the same cold FIFO.

Token spend stays near zero on every retry. Deadline miss rate climbs with each requeue.

That is still a cost incident for the SLO. The invoice is not the only bill you pay.

Artifact: slack-gated paid-to-free failover

Label this helper as a proposed local gate. It is a sketch, not a vendor client.

# slack_gate.py — proposed local admission helper
import json
import time
from dataclasses import dataclass

DEADLINE_MS = 15_000
RESERVE_MS = 2_000
PAID_P95_FIRST_BYTE_MS = 1_200
FREE_P95_FIRST_BYTE_MS = 4_000
FREE_QUEUE_AGE_CUT_MS = 3_000

@dataclass
class Snapshot:
    paid_inflight: int
    paid_limit: int
    free_queue_age_ms: int
    free_inflight: int
    enqueued_at_ms: int

def remaining_slack_ms(now_ms: int, snap: Snapshot) -> int:
    waited = now_ms - snap.enqueued_at_ms
    return DEADLINE_MS - waited - RESERVE_MS

def admit(now_ms: int, snap: Snapshot) -> str:
    slack = remaining_slack_ms(now_ms, snap)
    if slack <= 0:
        return "reject_deadline"
    paid_ok = snap.paid_inflight < snap.paid_limit
    if paid_ok and slack > PAID_P95_FIRST_BYTE_MS:
        return "paid"
    free_need = snap.free_queue_age_ms + FREE_P95_FIRST_BYTE_MS
    age_ok = snap.free_queue_age_ms < FREE_QUEUE_AGE_CUT_MS
    if free_need < slack and age_ok:
        return "free"
    return "reject_no_slack"

if __name__ == "__main__":
    now = int(time.time() * 1000)
    snap = Snapshot(
        paid_inflight=2,
        paid_limit=2,
        free_queue_age_ms=8_000,
        free_inflight=1,
        enqueued_at_ms=now - 4_000,
    )
    decision = admit(now, snap)
    print(json.dumps({
        "decision": decision,
        "slack_ms": remaining_slack_ms(now, snap),
        "free_need_ms": snap.free_queue_age_ms + FREE_P95_FIRST_BYTE_MS,
    }))

Run the helper once with that snapshot. Expected output for this labeled snapshot follows.

{"decision": "reject_no_slack", "slack_ms": 9000, "free_need_ms": 12000}

Paid workers sit full in this snapshot. Free path needs twelve seconds of wait plus infer.

You only hold nine seconds of remaining slack. You reject the failover instead of waiting cheaper.

Runbook table for the on-call

paid inflight	free queue age	remaining slack	action
below limit	any	greater than paid p95 plus reserve	send paid
at limit	under 3s	greater than free p95 plus age	send free
at limit	3s or more	any	reject 429
any	any	at or below zero	reject 429

Pin the three-second free queue age cut. Rationale: age already spent the cheap part of slack.

Free worker utilization can still read twenty percent. One blocked first-byte hides behind that low CPU.

Queue age and slack beat utilization here. Do not add hosts because CPU looks idle.

Failure injection on the free backend

Do not wait for production to teach failover cost. Inject three faults against the local free stub.

Sleep six seconds before the first response byte.
Hold one extra FIFO job so age passes three seconds.
Return HTTP 200 with an empty body after eight seconds.

Local stub

Proposed stub for fault one:

python3 - <<'PY'
from http.server import BaseHTTPRequestHandler, HTTPServer
import time

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        time.sleep(6)
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b'{"ok":true,"bytes":0}')
    def log_message(self, *args):
        pass

HTTPServer(("127.0.0.1", 8088), H).serve_forever()
PY

Drive the gate after the stub is up:

python3 slack_gate.py
# keep arrival at eight jobs per second for ninety seconds

What you should see

Watch these fields instead of the token counter:

decision as paid, free, or reject_*
queue_age_ms on the free path only
first_byte_ms by path
remaining_slack_ms at admit time
deadline_miss after the handler returns
token_spend as a secondary ledger only

Expected observation under faults one and two follows. Reject count rises while deadline misses stay flat.

Token spend stays near zero on both paths. The useful page is reject_no_slack, not spend.

If misses rise with rejects, raise RESERVE_MS. Do not reopen unbounded paid-to-free failover tonight.

Telemetry fields to emit

Ship counters and histograms without blending them together.

job_decision{path="paid|free|reject"}
job_first_byte_ms{path="paid|free"}
job_queue_age_ms{path="free"}
job_remaining_slack_ms{stage="admit"}
job_deadline_miss{path="paid|free|reject"}
job_token_spend{path="paid|free"}

Alert on this contradiction and nothing else. Token spend falls while deadline misses rise together.

That pattern is zero-spend failover eating slack. Page on-call with queue age and slack samples. Leave the cost panel off the paging policy.

Use free access for the drill, not the SLO

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use that pair to stage this failover drill.

Burn retries and reject paths there without an invoice. Keep the same slack gate on every request.

If free queue age plus first-byte exceeds slack, reject. Do not promote the free path onto SLO traffic.

Free capacity is a lab and an overflow probe. It is the wrong bet for hard deadline work.

Rollback and cleanup

Keep the gate behind an environment flag.

export ADMIT_FREE=0
# or force reject by cutting age to zero
export FREE_QUEUE_AGE_CUT_MS=0
systemctl reload admission

Drain free inflight before you touch paid workers. Do not SIGKILL paid workers during the drain.

curl -s localhost:9090/metrics | grep job_inflight
kill $(lsof -t -i:8088)

Revert lab p95 constants to paid measurements only. Delete the delay stub from staging DNS and unit files.

Record reject ratio in the incident note. Record that spend was not the recovery signal.

Who should not use this gate

Skip this gate for chat with no deadline. Skip it if you cannot measure first-byte wait. Skip it when a late answer beats a hard reject.

Batch research jobs can wait on free capacity. User-facing summaries with a fifteen-second SLO cannot.

If your only backend is free, you cannot fail over. Buy a reserved paid path before you add failover. Then place the slack gate in front of it.

Limits of this note

The p95 constants are declared lab values. They are not observations from a named fleet.

First-byte wait is not full generation time. A long completion can still miss after a fast first byte. Add a completion budget if the SLO covers the full answer.

This note ignores auth, prompt size, and cache hits. Those belong in a later operational drill instead. Do not copy the three-second cut without your own age histogram.

Threshold question for the next review

Which signal trips reject in your runbook today? Is it queue age, utilization, or remaining slack?

Utilization lies when one worker blocks on first byte. Queue age tells you wait already spent. Remaining slack tells you wait you can still afford.

Trip on slack versus queue age plus p95 first-byte. Write that comparison in the runbook before the next burst.

Reject Calendar-Wait Work Before Queue Age Breaks Slack

Odd_Background_328 — Fri, 18 Sep 2026 06:59:13 +0000

PagerDuty fires at 02:14 UTC on the batch lane.
Token counters for the hour still read zero.
The oldest job already shows negative deadline slack.

You stare at a green cost dashboard.
You also stare at a red SLO panel.
Free capacity did not fail on money today.
It failed on wall-clock time instead of spend.

Read the contradiction first

Cheap tokens do not buy you calendar time.
A free worker that is not yours yet is a wait.
Wait burns slack even when spend stays at zero.

You should treat queue age as a first-class cost.
You should not treat a zero invoice as a healthy lane.
Those two signals still answer different operational questions.

Ask one operational question before you admit more work.
Does remaining slack still cover wait plus p95 runtime?
If the answer is no, reject the job on purpose.
Do not let a free lane turn wait into an outage.

Lab topology

Run this as a local fault drill.
Do not point it at production traffic.

Proposed topology, labeled as a lab fixture:

Redis list lane.free as the admission queue
One worker process with a configurable hold
A sidecar that records slack and spend
A hard deadline per job, not a soft TTL

# docker-compose.lab.yml — proposed local topology
services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
  worker:
    build: .
    environment:
      QUEUE_KEY: lane.free
      HOLD_MS: "0"
      P95_RUNTIME_MS: "8000"
    depends_on: [redis]

Declared workload for this drill:

Twenty synthetic inference tasks in one burst
Deadline at 45 seconds after enqueue time
Expected warm p95 runtime of 8 seconds
Free-lane token price treated as 0.00
Wall-clock penalty of 1.0 per extra wait second

Those numbers are lab fixtures, not production measurements.
Replace them with your own SLO math before copying the gate.

What free actually costs you

Token ledgers still hide idle-wait from you.
Retries then hide that same wait twice.
Queueing hides it until the deadline is already dead.

You pay in four currencies on a free lane:

Queue age before a worker claims the job
Runtime after claim, including any cold start
Retry amplification if the claim later drops
Operator time spent explaining a green invoice

Only item two looks like model spend.
The other three still break the customer SLO.
A zero token price only deletes one column.

Artifact: price wait at admission

The gate below is a proposed local checker.
It is not a production admission controller.
It reads queue age, remaining slack, and a p95 hint.
It rejects admission when wait would spend the deadline.

#!/usr/bin/env python3
"""lab_admit.py — proposed free-lane admission checker."""
import json
import time
import redis

QUEUE = "lane.free"
P95_RUNTIME_MS = 8_000
WALLCLOCK_WEIGHT = 1.0  # slack-units per extra wait second


def slack_ms(job):
    return int(job["deadline_unix"] * 1000 - time.time() * 1000)


def queue_age_ms(r):
    raw = r.lindex(QUEUE, 0)
    if not raw:
        return 0
    head = json.loads(raw)
    return int(time.time() * 1000 - head["enqueued_ms"])


def effective_cost(wait_ms, token_cost):
    extra_s = max(0.0, wait_ms / 1000.0)
    return token_cost + extra_s * WALLCLOCK_WEIGHT


def decide(r, job, token_cost=0.0):
    wait = queue_age_ms(r)
    remain = slack_ms(job)
    need = wait + P95_RUNTIME_MS
    cost = effective_cost(wait, token_cost)
    admit = remain > need and remain > 0
    return {
        "job_id": job["id"],
        "queue_age_ms": wait,
        "slack_ms": remain,
        "need_ms": need,
        "token_cost": token_cost,
        "wallclock_cost": cost,
        "admit": admit,
        "reason": "admit" if admit else "reject_wait_beats_slack",
    }


if __name__ == "__main__":
    r = redis.Redis(decode_responses=True)
    raw = r.lindex(QUEUE, 0)
    if not raw:
        print(json.dumps({"reason": "empty_queue"}))
    else:
        print(json.dumps(decide(r, json.loads(raw)), indent=2))

Seed the overdue head job with Python, not shell date math.
That keeps the fixture reproducible on Linux and macOS.

#!/usr/bin/env python3
"""lab_seed.py — enqueue one tight-deadline lab job."""
import json
import time
import redis

now = time.time()
job = {
    "id": "lab-014",
    "enqueued_ms": int((now - 12) * 1000),
    "deadline_unix": now + 18,
}
r = redis.Redis(decode_responses=True)
r.delete("lane.free")
r.lpush("lane.free", json.dumps(job))
print(json.dumps(job, indent=2))

Expected output after you seed that overdue job:

{
  "job_id": "lab-014",
  "queue_age_ms": 12000,
  "slack_ms": 18000,
  "need_ms": 20000,
  "token_cost": 0.0,
  "wallclock_cost": 12.0,
  "admit": false,
  "reason": "reject_wait_beats_slack"
}

Token cost stays at zero in this fixture.
The admission gate still rejects the job.
That is the whole point of the drill.
Wait plus p95 already exceeds remaining slack.

Seed, inject, and watch

Start Redis, then seed, then ask the gate.

docker compose -f docker-compose.lab.yml up -d redis
python3 lab_seed.py
python3 lab_admit.py

Inject a free-lane stall in the worker next.
Sleep inside the worker instead of running inference.

#!/usr/bin/env python3
"""worker.py — lab worker with an injected hold."""
import os
import time
import json
import redis

HOLD_MS = int(os.environ.get("HOLD_MS", "0"))
r = redis.Redis(decode_responses=True)
raw = r.lindex("lane.free", 0)
if not raw:
    raise SystemExit("empty_queue")
job = json.loads(raw)
time.sleep(HOLD_MS / 1000.0)
print(json.dumps({"held_ms": HOLD_MS, "job_id": job["id"]}))

HOLD_MS=30000 python3 worker.py
python3 lab_admit.py

Watch these telemetry fields on every decision:

queue_age_ms
slack_ms
need_ms
token_cost
wallclock_cost
admit
reason

If CPU is idle and queue_age_ms climbs, you lack spare capacity.
In that case you only have an admission lie.
Idle cores behind a long free queue still miss the deadline.

Thresholds you must declare

Do not copy a magic number from this lab.
Write the reject threshold beside the SLO.
Use this admission decision order every time:

If slack_ms <= 0, reject and page. The job is already late.
If queue_age_ms + p95_runtime_ms >= slack_ms, reject admission.
If token_cost == 0 but wallclock_cost exceeds burn budget, reject.
Only then admit the job onto the free lane.

Rationale sits in the deadline, not in the invoice.
Queue age should beat utilization in this drill.
A busy-looking worker can still meet slack.
An empty free worker behind a long queue cannot.

You should compare these three signals every time:

Queue age: how long the head job has waited
Utilization: whether the worker is actually busy
Deadline slack: time left before the SLO breaks

You should trust remaining deadline slack first.
Utilization without slack is a vanity metric.
A cheap lane that cannot finish is not cheap.

When free capacity is the wrong bet

Free lanes win for catch-up batch with loose deadlines.
They lose for anything that still has a customer clock.
Write that split down before the next incident.

Skip the free lane when you see any of these:

Interactive or near-interactive SLOs under two minutes
Deadlines tighter than observed queue age plus p95
Retry storms that multiply wait, not just tokens
Drain tests that must finish before a release window
Incident work where calendar time is the scarce resource

Keep the free lane when the job can survive delay:

Backfill, eval, and replay jobs with hour-scale slack
Load you can drop without paging a human
Experiments where a reject is cheaper than a miss

A zero token price does not change that table.
It only removes the token spend column.
Time, retries, and queueing still sit on the books.

Rehearse on a free experiment host

You can rehearse this gate before you touch prod.
MonkeyCode offers free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the same slack math on that host.
Do not treat the free server as spare production capacity.
Use it to inject holds and practice reject-before-miss.
If the free host queue ages past your slack, reject there too.
The drill is the policy; the vendor is just a box.

Failure handling and rollback

When the gate starts rejecting, follow this order:

Stop enqueueing onto lane.free
Snapshot queue_age_ms and slack_ms for the head ten jobs
Drain or dead-letter anything already negative
Move only jobs with remain > need to a dedicated worker
Leave the rest rejected; do not blind-retry onto the same list

Rollback the drill itself with a short cleanup path:

redis-cli DEL lane.free
unset HOLD_MS
docker compose -f docker-compose.lab.yml down -v

If you changed an alert, restore the previous threshold.
Do not leave a lab HOLD_MS in a shared compose file.

Cleanup checklist before you walk away:

Delete seeded jobs from lane.free
Drop the lab Redis volume
Clear any synthetic pages you fired
Record the threshold you used, with units

Limitations

This lab gate does not schedule GPUs.
It also does not predict worker preemption.
It does not replace a real autoscaler.

It also assumes you know p95 runtime.
If that hint is a lie, the reject is a lie.
Measure runtime on the same class of worker you will use.

Who should not use this approach:

Teams without a numeric deadline per job
Platforms that must run every enqueue, regardless of slack
Workloads where wait is the product, such as long research sweeps
Anyone hoping a free invoice will hide a missed ship time

If you cannot name slack in milliseconds, stop.
Fix the deadline instrumentation before any gate.

After the alert

Now go back to that 02:14 UTC alert.
Zero spend was not a healthy signal.
Negative slack was the only real signal.

Reject calendar-wait work before queue age breaks slack.
Price wait in the same record as tokens.
Keep free capacity for jobs that can survive the queue.
If you rehearse on a free server, keep rejects loud.
A quiet zero-cost miss is still a miss.

Debug Free-Slot Stalls With Low CPU And Queue Age

Odd_Background_328 — Thu, 17 Sep 2026 05:05:03 +0000

Your on-call pager lights up at 02:14 UTC.
The scoring batch still holds ninety seconds of slack.
The free worker shows CPU at eighteen percent.

You admit the job on that idle-looking signal.
Six minutes later the deadline is already gone.
One long request held the only free slot.

Utilization looked healthy during the entire stall.
Wall-clock wait told a completely different story.
Token cost stayed zero while the SLO burned.

Split the three signals before you admit

You treated free tokens as a capacity grant.
That hid the only clock your users feel.
Calendar time still moves on a free worker.

Watch these three fields in one panel:

worker_cpu_util at 0.18, which looks safe
queue_age_ms at 240000, which is not
deadline_slack_ms at 90000, already fatal

Admit on utilization and you miss the stall.
Admit because tokens are free and you miss it.
Admit only when wait plus retry still fits slack.

Low CPU plus rising queue age means head-of-line blocking.
The occupant is not burning a CPU budget.
It is holding the only runnable slot.

When free capacity is the wrong bet

Shared free workers fail in slow, quiet ways.
They rarely fail like dedicated billed replicas do.
You should reject the free path in these cases.

Condition	Free bet	Action
Expected wait plus one retry exceeds slack	Wrong	Shed or route paid
Queue age rises while CPU stays under 30%	Wrong	Suspect HOL; reject
Timeout retry would resend the full prompt	Wrong	Cap retries at zero
Slack covers estimated wait and one retry	Maybe	Admit with a kill switch
Work is interactive with unbounded wait	Wrong	Do not use free capacity

Queue age is a wait proxy, not a proof.
Treat it as an operational estimate only.
Replace it with a histogram when you have one.

Free tokens do not cancel occupant hold time.
A stuck slot makes every later job late.
That is opportunity cost, not a billing line.

Local topology for the drill

Keep every process on loopback for this drill.
Do not aim this drill at production.
You need one slot, one occupant, one probe.

Run these three pieces together:

A single-slot worker that can sleep on command
A FIFO listener in front of that worker
An admission gate using slack and queue age

Declared workload for this labeled exercise:

Worker concurrency equals one
Occupant request sleeps for 25 seconds
Probe job is given 12 seconds of slack
Retry budget equals one attempt of eight seconds
Success means completion before slack hits zero

This is a local fault injection, not a benchmark.
Do not publish these timings as capacity numbers.
They only show the admit inequality.

Artifact: wait-versus-slack admission gate

The gate estimates wait from current queue age.
It adds one retry timeout as calendar cost.
It rejects free capacity when that sum beats slack.

# local_drill/admit_free.py
# Labeled local example. Not production. Not a benchmark.

from dataclasses import dataclass

@dataclass
class GateInput:
    now_ms: int
    deadline_ms: int
    queue_age_ms: int
    inflight: int
    concurrency: int
    retry_timeout_ms: int
    retry_budget: int
    cpu_util: float

def remaining_slack_ms(inp: GateInput) -> int:
    return inp.deadline_ms - inp.now_ms

def expected_wait_ms(inp: GateInput) -> int:
    # Architectural inference: age grows with occupancy.
    if inp.concurrency <= 0:
        return 10**9
    extra = max(inp.inflight, 1 if inp.queue_age_ms > 0 else 0)
    return inp.queue_age_ms + extra * 1000

def admit_free(inp: GateInput) -> dict:
    slack = remaining_slack_ms(inp)
    wait = expected_wait_ms(inp)
    retry_cost = inp.retry_timeout_ms * max(inp.retry_budget, 0)
    hol = inp.cpu_util < 0.30 and inp.queue_age_ms > 5000
    need = wait + retry_cost
    decision = "admit_free"
    reason = "slack_covers_wait_and_retry"
    if slack <= 0:
        decision, reason = "shed", "deadline_already_past"
    elif hol:
        decision, reason = "reject_free", "head_of_line_with_low_cpu"
    elif need >= slack:
        decision, reason = "reject_free", "wait_plus_retry_beats_slack"
    return {
        "decision": decision,
        "reason": reason,
        "slack_ms": slack,
        "expected_wait_ms": wait,
        "retry_cost_ms": retry_cost,
        "need_ms": need,
        "hol_suspect": hol,
    }

Feed the contradiction case into the gate.

# local_drill/case_hol.py
from admit_free import GateInput, admit_free

probe = GateInput(
    now_ms=1_700_000_000_000,
    deadline_ms=1_700_000_000_000 + 12_000,
    queue_age_ms=25_000,
    inflight=1,
    concurrency=1,
    retry_timeout_ms=8_000,
    retry_budget=1,
    cpu_util=0.18,
)
print(admit_free(probe))

Expected output from the gate math follows.
These values are labeled expected output, not samples.
They are not sampled from a production cluster.

{
  "decision": "reject_free",
  "reason": "head_of_line_with_low_cpu",
  "slack_ms": 12000,
  "expected_wait_ms": 26000,
  "retry_cost_ms": 8000,
  "need_ms": 34000,
  "hol_suspect": true
}

You reject the free worker on that result.
Tokens would have cost nothing on the retry.
The deadline would still have been missed.

Inject the occupant and read telemetry

Start a one-slot worker that can hold the line.
The occupant path sleeps for twenty-five seconds.
The probe path sleeps two seconds after dequeue.

python3 - <<'PY'
import socket, threading, time

lock = threading.Lock()
state = {"busy": False, "started": time.time()}

def handle(conn):
    with lock:
        state["busy"] = True
    try:
        data = conn.recv(64) or b"probe"
        hold = 25 if data.startswith(b"occupant") else 2
        time.sleep(hold)
        conn.sendall(b"done")
    finally:
        with lock:
            state["busy"] = False

srv = socket.socket()
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", 9408))
srv.listen(8)
print("worker_listen 127.0.0.1:9408")
while True:
    conn, _ = srv.accept()
    threading.Thread(target=handle, args=(conn,), daemon=True).start()
PY

Then fire occupant and probe with wall timestamps.

python3 - <<'PY' &
import socket, time
s = socket.create_connection(("127.0.0.1", 9408))
t0 = time.time()
s.sendall(b"occupant")
s.recv(16)
print(f"occupant_done_s {time.time()-t0:.1f}")
s.close()
PY

sleep 0.4

python3 - <<'PY'
import socket, time
deadline = time.time() + 12
s = socket.create_connection(("127.0.0.1", 9408))
s.settimeout(12)
t0 = time.time()
try:
    s.sendall(b"probe")
    s.recv(16)
    print(f"probe_ok slack_left_s {deadline-time.time():.1f}")
except Exception as e:
    print(f"probe_miss class={type(e).__name__} waited_s {time.time()-t0:.1f}")
finally:
    s.close()
PY

Expected local behavior under the declared workload:

Occupant holds the slot for about 25 seconds
Probe hits the 12 second socket timeout
Process CPU stays low during the sleep
Queue wait, not tokens, consumes the slack

Log these fields on every admit decision:

deadline_slack_ms
queue_age_ms
inflight
worker_cpu_util
admit_decision
retry_attempt
wall_wait_ms

If queue age rises while CPU stays low, freeze admits.
That field pair is your head-of-line signature.
Remaining free tokens do not override that signal.

Retry cost on a free path

A timed-out free call is still expensive.
You repay the wait and resend the full prompt.
You also spend tokens on the second attempt.

Allow a retry only with leftover slack remaining.
The rule below is for this drill.
It is not a universal retry policy yet.

def allow_retry(slack_ms: int, timeout_ms: int, attempt: int) -> bool:
    if attempt >= 1:
        return False
    return (2 * timeout_ms) < slack_ms

If the first wait already ate the slack, stop.
A second attempt cannot buy the deadline back.
It can only pile more work on the stuck slot.

Count retry tokens as real spend even here.
Free model access does not make resends free.
Wall-clock and tokens both move on retry.

Rollback and cleanup

Stop admission the moment head-of-line is suspected.
Let the occupant finish or kill it by deadline.
Do not retry the probe on the same slot.

pkill -f "worker_listen 127.0.0.1:9408" || true
ss -ltnp | grep 9408 || echo "9408_clear"

Rollback sequence for a real free-tier drain:

Freeze free-tier admission on HOL or slack breach
Finish inflight work or kill it at the deadline
Route new work to a billed worker, or shed it
Re-enable free admission after queue age falls

Do not raise concurrency on the contended free host.
That packing failure belongs to a different drill.
This note only answers wait versus slack.

Confirm cleanup with a port check and a quiet queue.
Leave no occupant thread after the drill ends.
Reset the admit flag only after both are clear.

Where a free model path still fits

Some lab jobs have no user deadline.
You can park those jobs on free model access.
You can park them on a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Use that path only for deadline-free rehearsal jobs.

Keep the wait gate even on rehearsal traffic.
A lab job can still pin the only free slot.
Occupant holds hurt the next person in line.

Limitations

This gate uses queue age as a wait proxy.
That choice is inference, not a trace proof.
A real wait histogram should replace it later.

Who should skip this approach entirely:

Interactive chat with unbounded human wait tolerance
Multi-tenant production without an isolation boundary
Teams that do not attach deadlines to jobs
Anyone treating local sleeps as capacity ratings

Free capacity is a lab and overflow tool.
It is a weak backbone for a hard SLO.
Time stays scarce after tokens become free.

Do not export the 25 second sleep as a rating.
Do not claim a cluster size from loopback.
Do not skip paid capacity for deadline work.

Pick an operational threshold

Write one threshold and the rationale beside it.
Do not leave the admit bit as tribal knowledge.
Start with these numbers for the local drill.

Reject free admission when queue_age_ms + retry_timeout_ms >= 0.5 * deadline_slack_ms
Reject free admission when CPU is under 30% and queue age exceeds 5s
Never retry when remaining slack is under two timeouts

Why spend only half the slack on waiting?
Dequeue, model load, and tail latency still need room.
Spending all slack on wait leaves no recovery path.

Ask on-call one question before the next free pack.
Does queue age, utilization, or slack own admit?
If utilization owns it, rerun this loopback drill.

Reject Free-Server Preempts Before Restart Tokens Beat Slack

Odd_Background_328 — Wed, 16 Sep 2026 03:29:31 +0000

You get paged at 02:14 local time.
GPU utilization sits at only four percent now.
Queue age already reads eleven full minutes.
Deadline slack on the batch is eight minutes.
Which operational action now follows from that evidence?

Do not add workers from the idle reading.
Idle GPU often means a recent preempt event.
The free shared host reclaimed your decode slot.
Your generation died without a usable checkpoint.
Restart tokens will now consume remaining slack.

Why idle GPU lies during free-server drain

Free shared servers do not promise reserved time.
Your process can vanish between two decoder steps.
The collector still reports low utilization after that.
That number describes an empty seat, not spare capacity.
You must watch queue age against deadline slack.

Restart cost is not a simple HTTP retry.
It is a full prompt replay plus partial output.
That replay burns tokens you already spent once.
It also burns wall clock you no longer have left.
Reject new work before that replay loop starts.

A four percent GPU graph can look healthy.
The queue behind it is already past halfway.
That contradiction is your admission signal tonight.
Scale-out copies the same preempt onto new hosts.
Drain and reject instead of packing more tasks.

Topology you can run on a laptop

Keep the drill local and fully declared.
Use one admission process and one fake worker.
Inject preempts with a timer inside the lab.

Declared topology:

admit.py handles admission and the token ledger
the worker is a fake decoder with coarse checkpoints
SIMULATE_PREEMPT=1 kills progress after two ticks
one laptop process, with no remote model host

Declared workload:

one task with 800 prompt tokens attached
200 output tokens as the completion target
120 second deadline for the whole task
preempt after two decode ticks in this lab
no production traffic and no shared cluster

This is a labeled lab, not a production claim.
Expected JSON below comes from the script only.
Do not read those numbers as a billed invoice.

Admission rule before you enqueue anything

You need one threshold with a written rationale.
Use restart-token waste versus remaining deadline slack.
Do not use GPU utilization as an admission gate.

Rationale sits in one operational sentence here.
If a restart cannot finish inside slack, reject it.

Concrete threshold for this declared drill:

Reject enqueue when queue age exceeds half the deadline
Drain the worker when restart tokens exceed 20 percent
Abandon free-server mode after two recorded preempts
Fail the task instead of starting a third full replay

Queue age beats utilization for this decision.
Slack is the scarce resource, not GPU percent.
Two preempts already imply an unstable slot.
Write that comparison beside the alert in git.

Artifact: admission plus a preempt loop

Save the next block as admit.py locally.
It is a simulator, not a vendor client SDK.

#!/usr/bin/env python3
"""Local admission drill for free-server preempt waste."""
from __future__ import annotations

import json
import os
import time
from dataclasses import dataclass, field
from pathlib import Path

LEDGER = Path(os.environ.get("LEDGER_PATH", "/tmp/preempt_ledger.jsonl"))
DEADLINE_MS = int(os.environ.get("DEADLINE_MS", "120000"))
TASK_BUDGET = int(os.environ.get("TASK_BUDGET", "1000"))
QUEUE_AGE_FRAC = float(os.environ.get("QUEUE_AGE_FRAC", "0.5"))
RESTART_FRAC = float(os.environ.get("RESTART_FRAC", "0.2"))
MAX_PREEMPTS = int(os.environ.get("MAX_PREEMPTS", "2"))


@dataclass
class Task:
    task_id: str
    prompt_tokens: int
    output_tokens: int = 0
    restart_tokens: int = 0
    preempts: int = 0
    enqueued_ms: int = field(default_factory=lambda: int(time.time() * 1000))
    checkpoint_tokens: int = 0


def now_ms() -> int:
    return int(time.time() * 1000)


def queue_age_ms(task: Task) -> int:
    return now_ms() - task.enqueued_ms


def deadline_slack_ms(task: Task) -> int:
    return DEADLINE_MS - queue_age_ms(task)


def emit(event: dict) -> None:
    event["ts_ms"] = now_ms()
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    with LEDGER.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(event) + "\n")
    print(json.dumps(event), flush=True)


def admit(task: Task) -> str:
    slack = deadline_slack_ms(task)
    age = queue_age_ms(task)
    if slack <= 0:
        emit({
            "event": "reject",
            "reason": "slack_zero",
            "task_id": task.task_id,
            "queue_age_ms": age,
            "deadline_slack_ms": slack,
        })
        return "reject_slack_zero"
    if age > QUEUE_AGE_FRAC * DEADLINE_MS:
        emit({
            "event": "reject",
            "reason": "queue_age",
            "task_id": task.task_id,
            "queue_age_ms": age,
            "deadline_slack_ms": slack,
        })
        return "reject_queue_age"
    if task.preempts >= MAX_PREEMPTS:
        emit({
            "event": "reject",
            "reason": "preempt_budget",
            "task_id": task.task_id,
            "preempts": task.preempts,
            "restart_tokens": task.restart_tokens,
        })
        return "reject_preempt_budget"
    if task.restart_tokens > RESTART_FRAC * TASK_BUDGET:
        emit({
            "event": "reject",
            "reason": "restart_tokens",
            "task_id": task.task_id,
            "restart_tokens": task.restart_tokens,
            "task_budget": TASK_BUDGET,
        })
        return "reject_restart_tokens"
    emit({
        "event": "admit",
        "task_id": task.task_id,
        "queue_age_ms": age,
        "deadline_slack_ms": slack,
        "preempts": task.preempts,
    })
    return "admit"


def on_preempt(task: Task) -> None:
    wasted = task.prompt_tokens + task.output_tokens - task.checkpoint_tokens
    task.restart_tokens += max(wasted, 0)
    task.preempts += 1
    task.output_tokens = task.checkpoint_tokens
    emit({
        "event": "preempt",
        "task_id": task.task_id,
        "wasted_tokens": wasted,
        "restart_tokens": task.restart_tokens,
        "preempts": task.preempts,
        "deadline_slack_ms": deadline_slack_ms(task),
        "gpu_util_lie": 0.04,
    })


def decode_tick(task: Task) -> None:
    task.output_tokens += 20
    if task.output_tokens % 100 == 0:
        task.checkpoint_tokens = task.output_tokens
        emit({
            "event": "checkpoint",
            "task_id": task.task_id,
            "checkpoint_tokens": task.checkpoint_tokens,
        })


def main() -> None:
    task = Task(task_id="batch-0214", prompt_tokens=800)
    decision = admit(task)
    if not decision.startswith("admit"):
        return
    ticks = 0
    while task.output_tokens < 200:
        if os.environ.get("SIMULATE_PREEMPT") == "1" and ticks == 2:
            on_preempt(task)
            decision = admit(task)
            if not decision.startswith("admit"):
                emit({
                    "event": "drain",
                    "task_id": task.task_id,
                    "action": "fail_task_not_replay",
                })
                return
        decode_tick(task)
        ticks += 1
        time.sleep(0.05)
    emit({
        "event": "complete",
        "task_id": task.task_id,
        "output_tokens": task.output_tokens,
        "restart_tokens": task.restart_tokens,
    })


if __name__ == "__main__":
    main()

Run the happy path first without a preempt.

export LEDGER_PATH=/tmp/preempt_ledger.jsonl
rm -f "$LEDGER_PATH"
DEADLINE_MS=120000 TASK_BUDGET=1000 python3 admit.py

Expected output, labeled as lab output only:

{"event": "admit", "task_id": "batch-0214", "queue_age_ms": 0, "deadline_slack_ms": 120000, "preempts": 0}
{"event": "checkpoint", "task_id": "batch-0214", "checkpoint_tokens": 100}
{"event": "complete", "task_id": "batch-0214", "output_tokens": 200, "restart_tokens": 0}

Now inject the preempt with the environment flag.

SIMULATE_PREEMPT=1 DEADLINE_MS=120000 TASK_BUDGET=1000 python3 admit.py

Expected preempt path from the same script:

{"event": "preempt", "task_id": "batch-0214", "wasted_tokens": 840, "restart_tokens": 840, "preempts": 1, "gpu_util_lie": 0.04}
{"event": "reject", "reason": "restart_tokens", "task_id": "batch-0214", "restart_tokens": 840, "task_budget": 1000}
{"event": "drain", "task_id": "batch-0214", "action": "fail_task_not_replay"}

Read the ledger after the injected run completes.

python3 - <<'PY'
import json
from pathlib import Path
rows = [json.loads(line) for line in Path("/tmp/preempt_ledger.jsonl").read_text().splitlines()]
for row in rows:
    print(
        f"{row['event']:12} "
        f"slack={row.get('deadline_slack_ms')} "
        f"restarts={row.get('restart_tokens')}"
    )
PY

Compute slack from the ledger, not dashboards

GPU graphs lag behind the preempt by minutes.
The ledger is the source for the reject call.
Parse the last event for each task_id value.

# parse_slack.py — lab helper, declared expected fields only
import json
from pathlib import Path

last = {}
for line in Path("/tmp/preempt_ledger.jsonl").read_text().splitlines():
    row = json.loads(line)
    last[row["task_id"]] = row

for task_id, row in last.items():
    action = "reject" if row["event"] in {"reject", "drain"} else "hold"
    print(task_id, action, row.get("reason"), row.get("restart_tokens"))

If restart_tokens already exceeds 20 percent, stop.
A third replay cannot buy back lost slack.
Fail that task and keep the remaining queue moving.

Failure injection beyond the environment flag

Kill a worker only on hosts you already operate.
Do not point this drill at unknown remote machines.

# Lab only: freeze then kill a local worker pid.
WORKER_PID=$(pgrep -f admit.py | head -n 1)
kill -STOP "$WORKER_PID"
sleep 2
kill -KILL "$WORKER_PID"

Watch these telemetry fields during the reject call:

queue_age_ms as the wait you cannot hide
deadline_slack_ms as the budget that pages you
restart_tokens as waste from replayed prompts
preempts as the stability vote for that slot
checkpoint_tokens as the only reusable output
gpu_util_lie as the metric you must ignore

Ignore host CPU when making the reject call.
CPU and GPU idle recover after an eviction.
Deadline slack will not recover after that.

Checkpoint cadence versus restart waste

Checkpoint every 100 output tokens in this lab.
That cadence is a declared test condition only.
A longer gap raises wasted tokens on each preempt.
A shorter gap raises fsync cost and log volume.

You pick the gap from slack, not from comfort.
If slack is thirty seconds, checkpoint more often.
If slack is fifteen minutes, checkpoint less often.
Measure wasted tokens after a forced worker kill.

Prompt-heavy tasks lose more on every restart.
An 800 token prompt plus 40 decoded tokens wastes 840.
That already exceeds a 20 percent restart budget.
Fail the task. Do not replay the same prompt.

Alerting that names the next action

Wire the reject to an alert that names the action.
A graph without an action will not drain the queue.

# lab_alerts.yaml — local threshold, not a vendor rule
groups:
  - name: free_server_preempt
    rules:
      - alert: RestartTokensBeatSlack
        expr: restart_tokens > 0.2 * task_budget
        for: 0s
        labels:
          action: drain_and_reject
        annotations:
          summary: reject replay; slack cannot absorb another preempt
      - alert: QueueAgeHalfDeadline
        expr: queue_age_ms > 0.5 * deadline_ms
        labels:
          action: reject_enqueue

The alert must tell you to reject, not to scale.
Scaling onto another free host copies the preempt.
You then own two unstable queues, not one.

When free capacity is the wrong bet

Free model access looks cheap on the invoice line.
Free servers look idle on the utilization graph.
Both can still lose a deadline with one preempt.

Do not bet free capacity in these cases:

The page has a sub-minute latency SLO attached.
The task cannot checkpoint any partial output.
Prompt tokens dwarf the remaining output tokens.
You already recorded two preempts on that host.
Queue age already consumed half of the deadline.

Paid burst can beat a third full prompt replay.
Time is the budget that actually pages on-call.
Tokens are the budget that silently compounds overnight.
Replays without checkpoints multiply both budgets together here.

You may need a scratch host for this drain drill.
MonkeyCode offers free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
That option can host the same admission loop.
You still avoid opening a paid burst for the lab.
Keep the ledger either way during the run.
Free slots still vanish under a preempt.

Rollback and cleanup after the reject

Fail the in-flight task. Do not replay it.
Flip admission to reject_preempt_budget immediately.
Drain the queue with a short local script.

# Stop enqueue. Drain. Then tear the lab down.
export ADMIT_MODE=reject_preempt_budget
sleep 5
rm -f /tmp/preempt_ledger.jsonl
unset SIMULATE_PREEMPT ADMIT_MODE DEADLINE_MS TASK_BUDGET

Rollback order matters under page pressure:

Reject new work before touching any worker.
Checkpoint or fail in-flight work second.
Kill workers only after the drain flag.
Restore the old threshold from version control.

Do not scale out onto another free host yet.
That copies the preempt onto a second waiting queue.

Who should not use this approach

Skip this drill if you lack a per-task ledger.
Skip it if your model host is already reserved.
Skip it for interactive chat with tight p95 limits.
Skip it if policy forbids local token ledger files.

This method will not size a production cluster.
It will not prove a vendor capacity SLA.
It only answers one operational question tonight.
Can this task finish after one more preempt?

Limits you should write on the runbook

The simulator does not talk to a live GPU.
Token counts are declared, not billed invoices.
SIGKILL is not the only real failure mode.
Disk full and NIC loss need separate later drills.
Do not treat expected JSON as a capacity benchmark.

Revisit the half-deadline queue-age fraction quarterly.
Tight SLOs need a smaller age fraction written down.
Long batch windows can tolerate more queue age.
Write the rationale next to the threshold in git.

You now have a reject path, not a hope path.
Idle GPU is not permission to enqueue more work.
Restart tokens are the signal that pages you later.

If you repeat this drain, send ledger pairs.
Paste queue_age_ms versus restart_tokens from the ledger.
Do not paste a screenshot of idle GPU.

Free Token Budgets Hide Retry Cost: Keep a Per-Task LLM Ledger

Odd_Background_328 — Tue, 15 Sep 2026 03:07:50 +0000

Free Token Budgets Hide Retry Cost: Keep a Per-Task LLM Ledger

It is 02:40 and your nightly enrichment job is 40 minutes behind. The page fires on queue age. No error rate moved.

You moved that job to free capacity last week. Spend went to zero. Throughput went with it. Now you must pick: revert, or wait it out.

This is the ordinary shape of a free-tier incident. The invoice is a terrible control signal. Tokens and seconds per successful task are the real ones.

Free capacity changes cost, not work

A retry is real work. A 503 still burned your wall clock. It may also have burned tokens before failing.

So two numbers move in opposite directions. Cost per token drops to zero. Cost per successful task can rise without bound.

You cannot see that gap without per-task accounting. Aggregate dashboards will not show it either. They average the failures away.

Declare the workload before you decide anything

Write these five things down first. Comparisons without them are noise.

Task shape: prompt tokens, completion tokens, output format, and strictness of parsing.
Concurrency: workers in flight and any per-key rate limits you already know about.
Retry policy: max attempts, backoff curve, jitter, and which errors are retryable at all.
Deadline: the SLO in minutes, plus the current queue age at decision time.
Failure taxonomy: 429, 5xx, connect timeout, read timeout, malformed output, refusal.

Step 5 is where most teams lose money. Retrying a malformed output or a refusal is pure burn. It can never succeed without changing the prompt.

A per-task ledger you can run today

The script below wraps your call site. It appends one JSON line per task. It is provider-agnostic and talks to an OpenAI-compatible endpoint.

#!/usr/bin/env python3
"""ledger.py - per-task attempt accounting for AI batch work."""
import json, os, time, urllib.error, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ.get("LLM_MODEL", "default")
LEDGER = os.environ.get("LEDGER_PATH", "ledger.jsonl")
RETRYABLE = {408, 429, 500, 502, 503, 504, 599}


def est_tokens(text: str) -> int:
    """Coarse estimate. Swap in your tokenizer before trusting totals."""
    return max(1, len(text) // 4)


def call(prompt: str, timeout: float = 45.0):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        payload = json.loads(resp.read())
    return payload["choices"][0]["message"]["content"], payload.get("usage", {})


def run_task(task_id, prompt, queue_wait_s=0.0, deadline_s=900, max_attempts=3):
    t0 = time.monotonic()
    rec = {
        "task_id": task_id, "ts": time.time(), "queue_wait_s": queue_wait_s,
        "attempts": 0, "prompt_tokens": 0, "completion_tokens": 0,
        "reported_tokens": 0, "attempt_latency_s": [], "status": "ok",
        "last_error": None, "deadline_s": deadline_s,
    }
    for n in range(1, max_attempts + 1):
        if time.monotonic() - t0 + queue_wait_s > deadline_s:
            rec["status"] = "deadline_exceeded"
            rec["last_error"] = "deadline_slack_exhausted"
            break
        rec["attempts"] = n
        rec["prompt_tokens"] += est_tokens(prompt)
        a0 = time.monotonic()
        try:
            text, usage = call(prompt)
            rec["attempt_latency_s"].append(round(time.monotonic() - a0, 3))
            rec["completion_tokens"] += usage.get("completion_tokens") or est_tokens(text)
            rec["reported_tokens"] += usage.get("total_tokens") or 0
            break
        except urllib.error.HTTPError as e:
            rec["attempt_latency_s"].append(round(time.monotonic() - a0, 3))
            rec["last_error"] = f"http_{e.code}"
            if e.code not in RETRYABLE:
                rec["status"] = "failed"
                break
        except (urllib.error.URLError, TimeoutError, OSError) as e:
            rec["attempt_latency_s"].append(round(time.monotonic() - a0, 3))
            rec["last_error"] = type(e).__name__
        time.sleep(min(2 ** n, 20) * (0.5 + 0.5 * (time.time() % 1)))
    else:
        rec["status"] = "failed"
    rec["elapsed_s"] = round(time.monotonic() - t0, 3)
    rec["total_tokens"] = rec["prompt_tokens"] + rec["completion_tokens"]
    with open(LEDGER, "a") as fh:
        fh.write(json.dumps(rec) + "\n")
    return rec

Run it against your own batch

export LLM_BASE_URL="https://your-endpoint.example/v1"
export LLM_API_KEY="..."
export LEDGER_PATH="ledger.jsonl"

python -c "from ledger import run_task; run_task('t-0001', 'summarize: <text>')"

Aggregate retry amplification

jq -r '[.attempts, .total_tokens, .queue_wait_s, .elapsed_s, .status] | @tsv' ledger.jsonl \
| awk '{a+=$1; tok+=$2; q+=$3; e+=$4; n++; if ($5=="ok") ok++}
       END {printf "tasks=%d ok=%d attempts/success=%.2f tokens/task=%.0f queue_p50~=%.1fs elapsed_p50~=%.1fs\n",
                    n, ok, a/ok, tok/n, q/n, e/n}'

Add a status breakdown, because failures still burn budget:

jq -r '.status' ledger.jsonl | sort | uniq -c | sort -rn
jq -r 'select(.attempts>1) | "\(.last_error)"' ledger.jsonl | sort | uniq -c | sort -rn

Example output (labeled, not a measurement)

The block below is illustrative formatting only. It is not a benchmark and your numbers will differ.

tasks=500 ok=482 attempts/success=1.31 tokens/task=2140 queue_p50~=38.0s elapsed_p50~=21.5s
    482 ok
     18 failed
   22 http_429
   11 http_503

Read 1.31 as the tax. Three out of every ten successes bought an extra attempt.

Three thresholds that flip the decision

1. Attempts per success

Keep this under your own tolerance. Many teams pick 1.2 because retry overhead plus queue growth eats the savings fast. Above it, free capacity is not cheaper. It is just slower.

2. Queue age against deadline slack

Define slack as deadline_s - elapsed_p95. Then compare queue_p95 to it.

If queue age exceeds roughly a quarter of your slack, you have no room for one retry storm. Move the critical path back to capacity you control.

3. Token burn against the allowance

Treat an allowance as a burn-rate budget, not a target. Samples every five minutes:

jq -r 'select(.ts > (now - 300)) | .total_tokens' ledger.jsonl | awk '{s+=$1} END {print s/300, "tokens/sec"}'

If that rate reaches a meaningful fraction of the allowance while attempts/success stays elevated, cap concurrency. Do not add workers.

Decision table

Observed signal	Likely cause	Action
attempts/success 1.0, queue age rising	concurrency too low	raise workers, watch 429s
attempts/success > 1.3, 429 dominant	admission control missing	add token-bucket limiter, lower workers
attempts/success > 1.3, 5xx dominant	provider-side instability	keep batch on paid path, retry later
failures with `malformed_output`	prompt or parser, not capacity	fix prompt, do not retry
queue_p95 > 25% of slack	deadline risk	revert critical path, keep tail on free
token rate near allowance, low attempts	genuinely cheap workload	keep it there, record the reason

That last row matters. Some workloads belong on free capacity. You need evidence, not a hunch.

Failure handling and rollback

Ship a kill switch and a drain path before you migrate anything.

Cap max_attempts at 3 in code, configurable by env, never by caller.
Route exhausted tasks to a dead-letter file with the full ledger record.
Add FREE_TIER_ENABLED=0 to bypass the new path without a deploy.
Shadow first: run both paths for one window and compare attempts/success.
Drain: stop dequeuing, let in-flight tasks finish, then flip the flag.
Keep the old endpoint config for one week after cutover.

Where free model access and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator states that MonkeyCode provides free model access and a free server option, with a 10M token allowance. Treat those as availability claims from the vendor, not as measurements. Substituting them into this ledger produces comparable numbers, which is the point.

A free server option removes one objection: you can run the ledger collector and a shadow batch without a new invoice. Run the shadow window there first. If attempts/success stays near 1.0 and queue age fits your slack, promote the workload. If it does not, you learned that for the cost of a JSON file.

Keep the ledger provider-neutral. The file format above does not name a vendor for a reason. When a cheaper path appears next quarter, you only change the endpoint.

Limitations and who should not use this

Token counts here are estimates. Use a real tokenizer before you forecast spend.
Single-process file appends will not survive high concurrency. Ship records to a log collector or database instead.
A per-task ledger is not distributed tracing. Pair it with span IDs if you need causality.
If your SLO has hard, contractually visible deadlines, free capacity is the wrong bet regardless of price. Keep the critical path where you control latency.
If you cannot instrument the call site, do not migrate. Measure first.
If your payloads are regulated, review the data path before sending anything to a new endpoint.
If your workload is a single interactive request with a human waiting, retry budgets rarely pay off. Latency is the product.

What to do next

Pick one batch job. Wrap it with the ledger for one week. Then compare attempts/success, queue age against slack, and token burn rate.

Those three numbers will tell you whether free capacity is a saving or a deferral. Which one of them is currently invisible in your dashboards?

Reject Shared-Host Burst Packing Before Swap-In Beats Token Budget

Odd_Background_328 — Sun, 13 Sep 2026 19:16:43 +0000

At 04:12 UTC your pager fires on inference p99.
Queue age still looks green on the dashboard.
Host CPU idle still sits near forty percent.

You now hold a contradiction, not spare capacity.
Which operational action follows from that evidence?
You do not add replicas from that idle chart.

The scene you actually walked into

A batch packer stuffed eight long contexts onto one box.
That box was a shared free host with no isolation.
The model process stayed runnable while pages left RAM.

p99 jumped while utilization stayed flattering and quiet.
Tokens still completed, just after the deadline window.
Cost looked free. Tail latency was not free.

You are operating deployed inference, not designing raft.
Measure reclaim, then admit work, then spend tokens.
Free capacity is a bet with a hard stop condition.

Topology under test

Keep the drill on one host you can destroy.
Do not point this at a customer-facing endpoint.
Label every number below as a lab expectation.

Proposed lab topology:

One shared host running a single inference worker
One Redis list as the admission and drain queue
One sidecar that samples PSI, swap, and queue age
One packer that can emit oversized context jobs
cgroup memory max set below the unpacked working set

[packer] -> redis:jobs -> [admitd] -> [worker]
                              |
                              +-> [sidecar: psi, si, queue_age_ms]
worker cgroup: memory.max = 2G, memory.swap.max = 0

Declared workload, not a marketing benchmark:

Job type: offline summarization with a hard deadline
Context: 8 files, about 24k tokens of prompt text
Concurrency: 8 packed jobs on one worker process
Deadline: 8 seconds from enqueue to first useful token
Host: 4 vCPU, 2G memory.max, swap disabled in cgroup

If your box differs, rewrite the thresholds, not the story.
Do not copy these numbers onto a production runbook.

Signals that beat a green CPU chart

CPU idle cannot see anonymous pages leaving the working set.
Queue age cannot see a runnable worker waiting on reclaim.
You need host pressure fields beside the job counters.

Collect at least these telemetry fields every second:

queue_age_ms from Redis LINDEX plus enqueue time
deadline_slack_ms as deadline minus now minus age
psi_mem_some_avg10 from /proc/pressure/memory
si from vmstat 1 as swap-in pages per second
cg_mem_current from memory.current in the cgroup
tokens_out and retry_count from the worker log

# labeled lab commands, not production scrape config
while true; do
  date --iso-8601=seconds
  awk '/some/ {print "psi_mem_some", $0}' /proc/pressure/memory
  grep -E '^(si|so)' <(vmstat 1 2 | tail -1 | awk '{print "si", $7, "so", $8}')
  cat /sys/fs/cgroup/infer/memory.current
  redis-cli LPUSH_SENTINEL >/dev/null 2>&1 || redis-cli LLEN jobs
  sleep 1
done

Expected lab pattern, clearly labeled, not observed prod:

CPU idle stays above thirty percent during the pack
psi_mem_some_avg10 climbs above 15.00 within 20s
queue_age_ms stays under 500 while p99 crosses 8s
tokens_out still increases, so the queue looks healthy
deadline_slack_ms goes negative before retries start

That is the contradiction you page on.
Idle CPU plus shrinking slack means reclaim, not underload.

Local fault injection before any shared free host

Inject packing on localhost first.
Do not discover swap-in during a customer batch window.
Keep memory.swap.max at zero so the failure is loud.

# create a tight cgroup; rollback later with cgdelete
sudo mkdir -p /sys/fs/cgroup/infer
echo "+memory +cpu" | sudo tee /sys/fs/cgroup/cgroup.subtree_control
echo 2147483648 | sudo tee /sys/fs/cgroup/infer/memory.max
echo 0 | sudo tee /sys/fs/cgroup/infer/memory.swap.max

# start a fake worker that touches a 3G anonymous map
echo $$ | sudo tee /sys/fs/cgroup/infer/cgroup.procs
python3 - <<'PY'
import time, mmap, os
print("pid", os.getpid(), flush=True)
buf = mmap.mmap(-1, 3 * 1024 * 1024 * 1024)
for off in range(0, len(buf), 4096):
    buf[off] = 1
print("packed", flush=True)
time.sleep(30)
PY

Watch the sidecar while that map fills.
You should see PSI rise before the process is killed.
If the kernel OOM-kills the worker, admission was already late.

Failure handling you want in the worker, not in chat:

Read deadline_slack_ms before loading the next context.
Reject the job when slack is under 1500 ms.
Reject the job when psi_mem_some_avg10 exceeds 10.00.
Nack with reason=host_pressure, never a blind retry.
Cap in-flight packed tokens, not just in-flight jobs.

# proposed admitd gate; unexecuted example, not a benchmark
PSI_LIMIT = 10.0
SLACK_LIMIT_MS = 1500
PACKED_TOKEN_LIMIT = 32000

def admit(job, psi_mem_some, slack_ms, packed_tokens):
    if psi_mem_some > PSI_LIMIT:
        return "reject_host_pressure"
    if slack_ms < SLACK_LIMIT_MS:
        return "reject_deadline_slack"
    if packed_tokens + job.prompt_tokens > PACKED_TOKEN_LIMIT:
        return "reject_packed_tokens"
    return "admit"

Retries belong on a different host class.
A nack into the same packed worker repeats the reclaim.
That is how free capacity burns wall clock twice.

When free capacity is the wrong bet

Ask for one operational threshold, with a rationale.
Do not mix queue age, utilization, and slack as equals.
They fail in different orders on a shared host.

Signal	Cheap on a free shared host?	Reject when	Why that beats CPU idle
CPU idle	Often yes	Never use alone	Misses reclaim waits
Queue age	Until packing starts	Age > 2s with slack < 1.5s	Age can lag runnable stalls
PSI memory some	No, shared noisy neighbors	avg10 > 10.00	Shows stall time, not idle
Swap-in `si`	No	si > 0 with swap disabled	Means the cgroup leaked
Packed prompt tokens	No	In-flight > 32k on 2G	Working set, not job count
Token price	Yes, until retries	Retry_count > 0 after nack	Retries double wall clock

Use queue age to see waiting work.
Use PSI and packed tokens to see host steal.
Use deadline slack to decide reject versus run.

Free hosts win for rehearsal, soak, and dead-letter replay.
They lose when p99 is the product, not the invoice.
They also lose when neighbors can allocate anonymous maps.

You should refuse the free-host bet when:

The job has a user-facing first-token deadline
The prompt working set can exceed memory.max
You cannot disable swap inside the worker cgroup
Retries reenter the same packed process
You lack PSI and si on the same scrape interval

Token, time, and retry math you can audit

Do not price a packed job by input tokens alone.
Wall clock during reclaim still holds the worker slot.
A retry after a late nack spends tokens without slack.

cost_proxy = prompt_tokens
            + completion_tokens
            + retry_count * prompt_tokens
            + stall_ms * workers_held / 1000

reject if deadline_slack_ms < stall_ms_p99 + 500

stall_ms_p99 comes from PSI, not from the tokenizer.
If you cannot measure stall, you cannot spend free capacity.
That is an ops gate, not a model-quality debate.

Run a one-hour drain math check before any cutover:

# unexecuted checklist; fill from your sidecar, not guesses
redis-cli LLEN jobs
redis-cli LRANGE jobs 0 0   # inspect enqueue_ts on the head
journalctl -u admitd --since "-10 min" | grep reject_

Expected output shape from admitd during a pack storm:

reject_packed_tokens job=7 packed=36012 limit=32000
reject_host_pressure job=8 psi=12.40
reject_deadline_slack job=9 slack_ms=420

If you instead see retry worker_idle=1, the gate failed.
Idle workers can still be stuck in reclaim.
Believe PSI over worker idle every time.

Rollback and cleanup

Leave the host quieter than you found it.
A forgotten 3G map is the next person's incident.
Roll back admission to deny-all if slack stays negative.

# drain, then unpin, then delete the cgroup
redis-cli PAUSE jobs || redis-cli RENAME jobs jobs_drain
sudo kill -TERM "$(pgrep -f packed)" || true
sleep 2
echo 0 | sudo tee /sys/fs/cgroup/infer/cgroup.procs || true
sudo rmdir /sys/fs/cgroup/infer
redis-cli LLEN jobs_drain

Rollback path for the serving path:

Flip admitd to reject_all with reason host_pressure.
Stop the packer. Do not stop Redis until drain completes.
Move the drain list to a dedicated host class.
Re-enable admit only after PSI avg10 stays under 1.00.
Keep swap disabled until the next declared drill.

If rmdir fails, a task is still in the cgroup.
Find it with cat cgroup.procs before you walk away.
Do not reboot to hide a packing bug.

Limitations, and who should skip this

This drill assumes you can set cgroup memory.max.
It assumes you can read PSI on the same host.
It assumes jobs carry a real deadline, not best effort.

Skip this approach when:

You run fully managed inference with no host scrapes
Your batch has no deadline and no token budget
You cannot fail closed without losing required work
You need multi-region consensus design, not host ops

Do not treat lab PSI numbers as a vendor benchmark.
Do not keep swap disabled if your platform requires it.
Do not run the 3G map on a laptop you cannot lock.

The useful rule is narrow.
Reject packed work before reclaim steals the token budget.
Green CPU on a shared free host is not an SLO.

A rehearsal slot, not a production bet

You can rehearse admitd against a scratch model endpoint.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option, which is enough to run this packing drill off the production path.

Keep customer deadlines off that shared box.
Use it to prove reject reasons, then throw the cgroup away.
If the sidecar cannot see PSI, do not promote the worker.