DEV Community: Odd_Background_328

Your Free AI Lane Is Slow in the Queue, Not in the Model

Odd_Background_328 — Tue, 20 Oct 2026 09:17:08 +0000

You moved a batch to free capacity. The token bill dropped to zero. The deadline still slipped.

Nothing in the model got slower. The jobs sat in a queue you never charted.

Most AI cost dashboards start counting at the first token. That is the wrong start line for a free lane. On free capacity, the wall clock usually belongs to scheduling, not to generation.

This post is a drill, not a vendor pitch. You record two clocks per job. Then you run one script and get a verdict: hold the batch, or spill it to paid capacity before the cutoff.

Start the clock at enqueue, not at first token

Your job record needs five timestamps. Anything less and you cannot separate waiting from working.

enqueue_ts — the job was accepted by your scheduler.
start_ts — the first request hit the serving path.
accept_ts — the artifact passed your acceptance check.
deadline_ts — the wall-clock deadline for that batch.
attempt — the retry index, so rework stays visible.

From those you get three derived numbers:

wait = start_ts - enqueue_ts
gen = accept_ts - start_ts
wait_share = wait / (accept_ts - enqueue_ts)

Gate on wait_share. A lane with p95 wait share of 0.7 is a scheduling problem wearing a model costume. Buying faster tokens there changes almost nothing.

The two-clock ledger in one script

The script below reads JSON Lines and prints a verdict. It is deliberately dumb. No dashboards, no agent, no database.

#!/usr/bin/env python3
# two_clock.py — lane verdict from job events (JSON Lines)
import json, sys

def load(path):
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                yield json.loads(line)

def pct(values, q):
    if not values:
        return 0.0
    s = sorted(values)
    i = min(len(s) - 1, int(round(q * (len(s) - 1))))
    return s[i]

def main(path, wait_share_max=0.60, margin_frac=0.15):
    rows = list(load(path))
    waits, gen, share = [], [], []
    for r in rows:
        total = r["accept_ts"] - r["enqueue_ts"]
        if total <= 0:
            continue
        wait = r["start_ts"] - r["enqueue_ts"]
        waits.append(wait)
        gen.append(total - wait)
        share.append(wait / total)

    drain = max(r["accept_ts"] for r in rows)
    first = min(r["enqueue_ts"] for r in rows)
    deadline = max(r["deadline_ts"] for r in rows)
    slack = deadline - drain
    margin = (deadline - first) * margin_frac

    print(f"jobs={len(rows)} wait_p50={pct(waits,0.5):.0f}s wait_p95={pct(waits,0.95):.0f}s")
    print(f"gen_p50={pct(gen,0.5):.0f}s wait_share_p95={pct(share,0.95):.2f}")
    print(f"slack={slack:.0f}s margin_budget={margin:.0f}s")

    if pct(share, 0.95) > wait_share_max:
        print("VERDICT: spill — wait dominates, throughput is not your lever")
    elif slack < margin:
        print("VERDICT: spill — projected drain lands inside the safety margin")
    else:
        print("VERDICT: hold — the lane is inside budget")

if __name__ == "__main__":
    main(sys.argv[1])

Run it against a finished batch:

python3 two_clock.py lane-free.jsonl

Expected output shape (illustrative, not a measurement from your system):

jobs=412 wait_p50=38s wait_p95=611s gen_p50=22s wait_share_p95=0.81
slack=1740s margin_budget=1080s
VERDICT: spill — wait dominates, throughput is not your lever

Note what the verdict ignores: tokens. They live in the same record if you need them, but they do not move this decision. Free capacity is a scheduling bet first and a billing bet second.

Why those two thresholds

wait_share_max = 0.60 says: if most of a job's life is spent queued, the model is not the constraint. Tighten it if your acceptance check is slow. Loosen it only with data.

margin_frac = 0.15 is a reserve against your own estimate. Drain forecasts are optimistic at the tail. Keep a slice of the window unspent so a late retry still fits.

Inject the wait locally before you trust the gate

Do not tune thresholds on a quiet lane. Congest it yourself, on purpose, in your own shell.

#!/usr/bin/env bash
# fake_lane.sh — synthetic congestion, zero production impact
set -euo pipefail
HOLD=${HOLD_SECONDS:-90}
DEADLINE=$(( $(date +%s) + ${WINDOW_SECONDS:-3600} ))
while read -r id; do
  ENQ=$(date +%s)
  sleep "$HOLD"
  echo "{\"job_id\":\"$id\",\"enqueue_ts\":$ENQ,\"start_ts\":$(date +%s),\"accept_ts\":$(date +%s),\"deadline_ts\":$DEADLINE}"
done

Generate job ids, then run the same 40 jobs twice:

seq 1 40 > ids.txt
HOLD_SECONDS=5  ./fake_lane.sh < ids.txt > lane-fast.jsonl
HOLD_SECONDS=90 ./fake_lane.sh < ids.txt > lane-slow.jsonl
python3 two_clock.py lane-fast.jsonl
python3 two_clock.py lane-slow.jsonl

The fast file should print hold. The slow file should print spill. If both print the same verdict, your gate is dead code and you should fix it before touching production routing.

Decision table: when free capacity is the wrong bet

Signal	Threshold	Action
p95 wait share	> 0.60	Spill the deadline-critical slice to paid capacity
Slack vs margin	slack < margin	Spill now, not at the next cron tick
Retry rate	rising while wait share rises	Fix admission, not the model
Rework per accepted artifact	above your baseline	Free tokens are buying wasted artifacts
Interactive p99 latency	any dependency	Do not route interactive traffic here at all

Read the table top-down. The first row that trips decides. You do not average these signals into one score.

Where the free tier fits into the drill

You can rehearse this whole loop without a paid commitment. MonkeyCode is an open-source project, and its operator states that it offers free model access, a free server option, and a free allowance of 10M tokens. Verify the current limits on the project page yourself before you plan any capacity around them; treat them as a starting condition, not a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the free lane genuinely buys you here is rehearsal time. You get real request paths, real timestamps, and a real queue to measure. What it does not buy you is a guarantee about wait distributions under your deadline. That number only exists after you run the ledger on your own workload.

One practical pairing: keep the free server as the lane under test, and keep the spill path wired to whatever paid capacity you already trust. The drill is only useful if the spill branch is reachable in one config change.

Rollback and cleanup

Keep the escape hatch boring and tested.

Gate routing behind one flag, for example FREE_LANE_ENABLED=0.
Flip the flag and stop new admission into the free lane.
Let in-flight jobs finish; do not kill them mid-acceptance.
Keep the ledger files. They are the evidence for the next threshold change.
Re-run two_clock.py on the spill window and compare wait shares.

If the spill window shows the same p95 wait share, the queue was never the free lane's fault. Look upstream at your own admission logic.

Who should not use this

Teams without timestamped job events. You cannot reconstruct two clocks from logs you never wrote.
Interactive or latency-critical endpoints. Queue-shaped capacity belongs behind batch work.
Work where a rework costs more than the tokens you saved.
Anyone planning to hard-code today's free allowance into a capacity plan for next quarter.

If you already run a batch queue, this is a one-afternoon exercise. Write four timestamps, run one script, and let the verdict argue with your assumptions.

Stop Lane Thrash: Pin Jobs After the First Ambiguous Retry

Odd_Background_328 — Mon, 19 Oct 2026 07:52:45 +0000

02:14 — the token spend alert fires.

Spend is up 90% over the same window yesterday. Accepted artifacts are up 2%. Nothing in the error budget explains the gap.

The queue has two lanes: a free lane for model work, and a paid fallback. Jobs bounce between them. Every bounce looks like a cheap retry in the dashboard.

This post is about the bill you cannot see in the token graph. It is the rework caused by a job that succeeds twice.

What Lane Thrash Looks Like

Lane thrash is a job that crosses lanes more than once per artifact. The usual cause is an ambiguous outcome.

The free lane times out after it already generated output. Your client sees a failure. Your retry sees a fresh job.

Two artifacts arrive later with the same inputs. Both look valid. Both get accepted.

You now pay tokens twice and review work twice. The metric that moves is not spend. It is duplicate artifacts per hundred jobs.

Topology: One Queue, Two Lanes

Keep one queue. Do not build two queues for two lanes.

A single admission point assigns a lane per attempt.
Every attempt writes one ledger row before it calls a model.
The ledger row carries an Idempotency-Key derived from job inputs.
Artifact writes are keyed by that same value.

That last bullet is the control that actually stops duplicates. Lane pinning only reduces how often you need it.

Config You Can Copy

# lane.yaml
lanes:
  free:
    max_attempts: 2
    pin_on: [ambiguous, permanent]
  paid:
    max_attempts: 1
reconcile:
  late_completion_window_s: 900
  dedup_key: Idempotency-Key
alerts:
  cross_lane_retries_per_job: 0.20
  duplicate_artifacts_per_100_jobs: 1
free_lane_enabled: true

The late_completion_window_s value matters. A slow free-lane response that lands after your client timed out is a duplicate unless you reconcile inside that window.

A Ledger That Answers the Right Question

CREATE TABLE job_attempt (
  job_id      TEXT NOT NULL,
  idem_key    TEXT NOT NULL,
  attempt_no  INTEGER NOT NULL,
  lane        TEXT NOT NULL CHECK (lane IN ('free','paid')),
  started_at  TEXT NOT NULL,
  ended_at    TEXT,
  outcome     TEXT,   -- ok | retryable | ambiguous | permanent
  tokens_in   INTEGER DEFAULT 0,
  tokens_out  INTEGER DEFAULT 0,
  artifact_id TEXT,
  PRIMARY KEY (job_id, attempt_no)
);

CREATE INDEX idx_artifact ON job_attempt(artifact_id);

CREATE VIEW job_rollup AS
SELECT job_id,
       MAX(idem_key)                   AS idem_key,
       SUM(lane = 'free')              AS free_attempts,
       SUM(lane = 'paid')              AS paid_attempts,
       SUM(tokens_in + tokens_out)     AS tokens,
       COUNT(DISTINCT artifact_id)     AS artifacts
FROM job_attempt
GROUP BY job_id;

Two rollup queries carry the whole decision:

-- tokens per unique artifact, plus cross-lane pressure
SELECT ROUND(SUM(tokens) * 1.0 / SUM(artifacts), 1)   AS tokens_per_artifact,
       ROUND(SUM(free_attempts > 0 AND paid_attempts > 0) * 1.0
             / COUNT(*), 3)                          AS cross_lane_per_job
FROM job_rollup;

-- the duplicates themselves
SELECT idem_key, COUNT(DISTINCT artifact_id) AS n
FROM job_attempt
WHERE artifact_id IS NOT NULL
GROUP BY idem_key
HAVING n > 1;

The boolean-sum form avoids FILTER, so it runs on older SQLite builds. Check your version before you paste.

The Router: Pin on Ambiguous, Retry on Retryable

The failure class decides the lane. Not the retry count alone.

# lane_router.py — stdlib only
PIN_ON = {"ambiguous", "permanent"}
MAX_FREE_ATTEMPTS = 2

def next_lane(free_attempts: int, pinned: bool, last_outcome: str) -> tuple[str, bool]:
    """Return (lane, pinned) for the next attempt."""
    if pinned:
        return "paid", True
    if last_outcome in PIN_ON:
        return "paid", True
    if free_attempts >= MAX_FREE_ATTEMPTS:
        return "paid", True
    return "free", False

retryable means 429 or 503 with no work done. Retry it on the free lane once. It is cheap and safe.

ambiguous means a timeout after partial streaming. The work may exist. Pin that job to paid and reconcile by idem_key.

Three tests pin the behavior down:

def test_pin_on_ambiguous():
    assert next_lane(1, False, "ambiguous") == ("paid", True)

def test_retryable_stays_free_once_more():
    assert next_lane(1, False, "retryable") == ("free", False)

def test_budget_exhausted_pins():
    assert next_lane(2, False, "retryable") == ("paid", True)

Fault-Inject the Free Lane Locally

Do not discover ambiguity handling in production. Point the router at a local fake lane.

# fake_free_lane.py — port 8099, stdlib only
import itertools, time
from http.server import BaseHTTPRequestHandler, HTTPServer

SCRIPT = itertools.cycle(["ok", "429", "503", "slow"])

class FreeLane(BaseHTTPRequestHandler):
    def do_POST(self):
        step = next(SCRIPT)
        if step == "slow":
            time.sleep(35)          # exceeds the 30s client timeout
            step = "ok"
        self.send_response({"ok": 200, "429": 429, "503": 503}[step])
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b'{"artifact": "synthetic"}')

    def log_message(self, *args):
        pass

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8099), FreeLane).serve_forever()

Run it, replay 300 recorded jobs against it, then read the rollup. The slow branch is the one that manufactures duplicates.

Run the 300-Job Drill

Declared workload for the numbers below:

300 synthetic jobs, one artifact each.
Free-lane outcomes: 25% retryable, 5% ambiguous, 2% permanent.
Synthetic token constants: 800 per free attempt, 1,200 per paid attempt.
Reconciliation window 900s, dedup on idem_key.

Expected output from the synthetic model, not a measurement of any vendor:

Metric	Pin + dedup	Retry free, no pin
Unique accepted artifacts	300	300
Free attempts	375	411
Paid attempts	46	6
Total tokens	355,200	336,000
Tokens per accepted artifact	1,184	1,120
Duplicate artifacts	0	15
Cross-lane retries per job	0.15	0.02

Read the Rollup, Not the Invoice

Look at the token row first. The two policies differ by about 5%.

That gap will not justify an on-call change. The duplicate row will.

Fifteen duplicates at two minutes of review each is thirty minutes of avoidable human work. At volume, that dwarfs the token delta.

So the operational question is not "which lane is cheaper per token." It is "which lane produces fewer unrecoverable artifacts."

Thresholds and Why They Are These Numbers

cross_lane_retries_per_job > 0.20 — above one in five jobs crossing lanes, the reconciliation window is probably too short.
duplicate_artifacts_per_100_jobs > 1 — one duplicate per hundred is already a review tax you can measure.
Free-lane p95 completion above half your deadline slack — the lane cannot drain before cutoff.

Set the slack threshold from your own p95, not from a blog number. Recompute it after every model change.

When the Free Lane Is the Wrong Bet

Condition	Route
Deadline slack p95 < 2× free-lane p95 latency	Stay on paid
Ambiguous rate > 2% and no idempotency key at the boundary	Fix the key first
Sustained arrival above measured free throughput for 30 min	Split by priority
Token delta between lanes < 10%	Decide on rework and latency
Batch deadline ≥ 6h and outputs idempotent	Free lane first, pin on ambiguous

That fourth row is the one most teams get wrong. When tokens converge, the cost argument disappears and the recovery argument takes over.

Where the Free Capacity Comes From

I run this drill against MonkeyCode's free model access, with the free server option hosting the queue worker and ledger. The operator lists 10M free tokens and a free server option. Verify the current terms on the project page before you route production work.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is open source, so the ledger and router above stay portable. Nothing in the drill depends on vendor-specific APIs.

Run the 300-job drill against the free tier before you move real traffic. The duplicate count is the number you actually care about.

Rollback and Cleanup

Rollback takes three steps, in this order:

Set free_lane_enabled: false and let the queue drain. Do not kill in-flight jobs.
Reconcile the last 900s of attempts by idem_key before deleting anything.
Replay pinned jobs on paid with the original keys.

Cleanup after the drill:

pkill -f fake_free_lane.py
rm -f drill.db
ss -ltnp | grep 8099   # confirm the port is free

Keep the ledger for 30 days. It is the only record that proves how many duplicates you produced.

Who Should Not Use This

Teams without an idempotency key at the API boundary.
Workloads with strict p99 latency SLOs and no slack to spend.
Jobs whose partial output cannot be reconciled, such as side-effectful writes.
Multi-tenant systems that need hard fairness between tenants.
Anyone who needs a permanence guarantee on free capacity.

If you are in the first group, stop here. Build the key first. The lane router is worthless without it.

Gate Jobs on Tokens per Accepted Artifact Before Retries Eat the Budget

Odd_Background_328 — Sun, 18 Oct 2026 06:47:59 +0000

Last Tuesday our batch lane burned 3.1 million tokens before 09:00.
Only forty-nine of sixty jobs produced an artifact we could ship.

Queue utilization looked healthy at 41%. The token graph went up 4x.
So which operational action follows from that evidence? Not a bigger queue. A budget gate.

Here is the workflow I now run before adding any free capacity to a lane.

The contradiction metric: tokens per accepted artifact

Utilization measures occupancy. It does not measure yield.
A retry loop can keep a lane busy while producing nothing.

Track one ratio instead:

Numerator: every token you spend, including failed calls and rejected outputs.
Denominator: artifacts that passed your checks and shipped.

Call it TPA (tokens per accepted artifact). Put it next to wall time.
When TPA rises while utilization stays flat, you are paying for retries, not throughput.

Two-phase reservation: price retries before you send them

Most teams cap tokens per request. That cap is useless under retries.
Three attempts each under the cap still triples your spend.

So reserve for the worst case, then settle on reality.
The gate below is plain Python with no dependencies. It is a runnable example, not a production library.

# budget_gate.py - two-phase token reservation for batch jobs
import json, time
from dataclasses import dataclass, field


class BudgetExceeded(Exception):
    pass


@dataclass
class Reservation:
    job_id: str
    lane: str
    reserved: int
    spent: int = 0
    attempts: int = 0
    opened_at: float = field(default_factory=time.time)


class TokenLedger:
    def __init__(self, cap_per_job: int, path: str = "ledger.jsonl"):
        self.cap = cap_per_job
        self.path = path
        self.open = {}

    def reserve(self, job_id: str, lane: str, estimate: int) -> Reservation:
        if estimate > self.cap:
            raise BudgetExceeded(f"{job_id}: estimate {estimate} > cap {self.cap}")
        res = Reservation(job_id=job_id, lane=lane, reserved=estimate)
        self.open[job_id] = res
        return res

    def charge(self, job_id: str, tokens: int) -> None:
        res = self.open[job_id]
        res.attempts += 1
        res.spent += tokens
        if res.spent > res.reserved:
            self.close(job_id, accepted=False, reason="over_reservation")
            raise BudgetExceeded(f"{job_id}: spent {res.spent} > reserved {res.reserved}")

    def close(self, job_id: str, accepted: bool, reason: str) -> None:
        res = self.open.pop(job_id, None)
        if res is None:
            return
        record = {
            "job_id": res.job_id, "lane": res.lane,
            "reserved": res.reserved, "spent": res.spent,
            "attempts": res.attempts, "accepted": accepted,
            "settle_reason": reason,
            "wall_ms": int((time.time() - res.opened_at) * 1000),
        }
        with open(self.path, "a") as fh:
            fh.write(json.dumps(record) + "\n")

The reservation covers retries. That is the whole point.

# runner.py - pre-pay for max_attempts, settle once
import time


def backoff(attempt: int) -> float:
    return min(2 ** attempt * 0.5, 8.0)


def run_job(ledger, job, client, max_attempts: int = 3):
    per_call = job.prompt_tokens + job.max_output_tokens
    ledger.reserve(job.id, lane=job.lane, estimate=per_call * max_attempts)
    last = None
    for attempt in range(1, max_attempts + 1):
        try:
            out = client.complete(job.prompt, max_tokens=job.max_output_tokens)
            # Assumption: the provider bills prompt tokens even on a failed call.
            # Verify your provider's billing; drop this charge if it does not.
            ledger.charge(job.id, out.usage_total)
            if passes_checks(out.text):
                ledger.close(job.id, accepted=True, reason="accepted")
                return out.text
            ledger.close(job.id, accepted=False, reason="rejected_by_checks")
            return None
        except Transient as exc:
            ledger.charge(job.id, job.prompt_tokens)
            last = exc
            time.sleep(backoff(attempt))
    ledger.close(job.id, accepted=False, reason=f"retries_exhausted:{last}")
    return None

Two properties matter more than the code.
Every job has a pre-paid ceiling, and every settled job leaves one JSONL record.

Read the ledger in one command

jq -s '
  {spent: (map(.spent) | add // 0),
   accepted: (map(select(.accepted)) | length),
   retried: (map(select(.attempts > 1)) | length)}
  | . + {tokens_per_accepted:
         (if .accepted == 0 then null else (.spent / .accepted) end)}' \
  ledger.jsonl

Expected output on a healthy lane, clearly labeled as expected, not measured by me on your workload:

{"spent": 1840000, "accepted": 49, "retried": 7, "tokens_per_accepted": 37551}

Now you have a number you can put in a review, an alert, or a capacity plan.

Run the shadow week on the cheapest box you have

You do not need production capacity to get this number.
I run the gate in shadow mode on a small free instance first, because shadow mode only needs to observe.

The MonkeyCode project advertises free model access, with the operator citing a 10M-token allowance, plus a free server option. I have not verified today's quotas, model list, or hardware.
Treat both as current-terms claims and read the project page before you size anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Shadow mode rules, and they are strict:

The gate computes reservations and logs them. It never blocks a job.
The real client still runs where it runs today. Nothing moves.
After seven days, compare tokens_per_accepted against your paid baseline.
Only then decide whether the free lane is worth enforcing.

Do not put a latency-critical path on a free box to "see how it goes".
Free capacity is a measurement instrument here, not a serving tier.

Fault injection before you trust the gate

A gate you never failed is a guess. Force failures locally.

# test_gate.py - run with: python -m pytest -q test_gate.py
import pytest
from budget_gate import BudgetExceeded, TokenLedger


class Transient(Exception):
    pass


class FlakyClient:
    def __init__(self, fail_first: int):
        self.fail_first = fail_first
        self.calls = 0

    def complete(self, prompt, max_tokens):
        self.calls += 1
        if self.calls <= self.fail_first:
            raise Transient("429 from upstream")
        return type("Out", (), {"text": "ok", "usage_total": 900})()


def test_retries_are_pre_paid(tmp_path):
    ledger = TokenLedger(cap_per_job=10000, path=str(tmp_path / "l.jsonl"))
    job = type("Job", (), {"id": "j1", "lane": "free",
                            "prompt_tokens": 800, "max_output_tokens": 400,
                            "prompt": "x"})()
    out = run_job(ledger, job, FlakyClient(fail_first=2), max_attempts=3)
    assert out == "ok"
    assert ledger.open == {}


def test_runaway_is_rejected(tmp_path):
    ledger = TokenLedger(cap_per_job=10000, path=str(tmp_path / "l.jsonl"))
    job = type("Job", (), {"id": "j2", "lane": "free",
                            "prompt_tokens": 800, "max_output_tokens": 400,
                            "prompt": "x"})()
    out = run_job(ledger, job, FlakyClient(fail_first=99), max_attempts=3)
    assert out is None
    assert ledger.open == {}

Assert three things after every run: no open reservations, spend never exceeds reservation, and the settle reason is populated.
An empty open dict is your leak detector.

Telemetry fields to emit

Six fields carry the whole diagnosis. Ship them on every settle.

lane - which capacity class ran the job.
reserved vs spent - reservation accuracy over time.
attempts - retry distribution, not just the mean.
accepted and settle_reason - why the tokens vanished.
wall_ms - p95 against your deadline slack.
tokens_per_accepted - the ratio you alert on.

Alert on the ratio, not on raw spend. Raw spend rises with useful work too.

When free capacity is the wrong bet

This is the part most write-ups skip. Free capacity has a cost, and it is usually wall time.
Use this table as a starting threshold set, then tune with your own ledger.

Signal over 7 days	Free lane is fine	Move work off the free lane
Retry rate	under 15% and flat	over 30%, or rising week over week
TPA vs paid baseline	within 1.2x	above 2x
p95 wall time	under 40% of deadline slack	approaching 70% of slack
Human review load	sampled review	every artifact needs review
Job side effects	idempotent, cheap to re-run	writes, payments, sends
Traffic shape	batch, tolerant of delay	interactive or user-facing

Two rows kill more free-lane plans than the rest. Non-idempotent writes and interactive latency.
No amount of saved tokens pays for a duplicate side effect.

Also price the human time. If a reviewer spends four minutes per artifact, TPA is not your biggest cost.

Limitations and who should not use this

Hard deadline SLOs under five minutes. Reservation adds a write and a settle.
Data that cannot leave your network. A hosted free tier is not the place to test it.
Teams without per-call usage numbers. You cannot settle what you cannot measure.
Jobs whose failures have side effects. Fix idempotency before you optimize spend.
Workloads where output length is unbounded. Reserve a hard max_output_tokens first.

This gate also cannot tell you whether an artifact is good. It only prices what you already accepted.

Rollback and cleanup

Ship the gate with a mode switch, and know the switch before you need it.

# shadow: log only, never block
BUDGET_MODE=shadow python runner.py
# enforce: block over-cap jobs
BUDGET_MODE=enforce BUDGET_CAP_PER_JOB=12000 python runner.py

Rollback is one environment change: set BUDGET_MODE=shadow and restart.
No code revert, no ledger migration, no data loss.

Then clean up:

Rotate ledger.jsonl daily. Sum the ratio, then archive.
Add a reaper for reservations older than your job timeout. Dead workers leak reservations.
Keep the fault-injection tests in CI. A retry change that skips the charge line is silent theft.

The one-line takeaway

Cap tokens per accepted artifact, not per request.
Then free capacity becomes a measured bet instead of a hopeful one.

If you want to reproduce the shadow week with no spend, the free model access and free server option are enough to collect the ledger above. Just verify the current terms yourself before you plan around them.

Price Free AI Capacity in Deadline Slack, Not in Tokens

Odd_Background_328 — Fri, 16 Oct 2026 04:46:24 +0000

It is 02:14. Your nightly batch window closes at 02:30. The free lane is still running, and the token meter reads zero.

Last week that felt like a win. Tonight the window is gone. Which operational action follows from that evidence?

Not delete the free lane. Price it.

Two numbers, one decision

Most teams measure the wrong number.

Token spend. Cheap, often zero on a free tier.
Deadline slack. Seconds you can spend before an SLO breaks.

Free capacity converts the first into the second. That conversion has a rate. Measure the rate, not the sticker price.

Define slack use per job class:

slack_used = free_p95_duration * (1 + retry_ratio) - paid_p50_duration

Three terms matter. The free lane p95 per attempt. The extra attempts per completed job. The paid baseline you gave up.

What the retry tax actually costs

A retry is not a retry. It is wall-clock time plus a new queue position plus a fresh attempt.

Preemption returns the worker mid-attempt. The partial work is gone.
Throttling returns a rate-limit signal. The partial work is gone.
Worker loss returns nothing at all. You wait for a timeout to notice.

Each event resets the attempt clock. Each event consumes slack, not tokens. That is how a free lane stays cheap and still blows the window.

If your provider returns 429, stop and back off. Do not retry around the limit.

The artifact: a break-even router

Here is a small script. It reads measured job-class statistics from one fixed window and prints a routing decision.

It does not call a model. It prices time.

#!/usr/bin/env python3
# breakeven.py - route batch job classes by deadline slack, not list price.
import json
import sys
from dataclasses import dataclass


@dataclass
class ClassStats:
    name: str
    attempts: int          # free-lane attempt starts in the window
    completed: int         # jobs that finished inside the class deadline
    free_wall_p50: float   # seconds per attempt, free lane
    free_wall_p95: float   # seconds per attempt, free lane
    paid_wall_p50: float   # seconds per job, reserved lane
    paid_cost: float       # currency units per completed job, reserved lane
    slack_budget: float    # seconds of negative slack this class may spend
    miss_penalty: float    # currency units charged when the deadline slips


def retry_ratio(s):
    if s.completed == 0:
        return None
    return max(s.attempts - s.completed, 0) / s.completed


def free_duration_p95(s):
    r = retry_ratio(s)
    return None if r is None else s.free_wall_p95 * (1.0 + r)


def slack_used(s):
    d = free_duration_p95(s)
    return None if d is None else d - s.paid_wall_p50


def miss_proxy(s):
    # Crude upper-bound proxy, not a fitted probability model.
    used = slack_used(s)
    if used is None:
        return 1.0
    if used <= s.slack_budget:
        return 0.0
    return min(1.0, (used - s.slack_budget) / max(used, 1e-9))


def route(s):
    used = slack_used(s)
    if used is None:
        return 'hold', 'no completed jobs in window'
    if used > s.slack_budget:
        return 'reserved', 'p95 slack use exceeds budget'
    risk = miss_proxy(s) * s.miss_penalty
    if risk >= s.paid_cost:
        return 'reserved', 'risk-adjusted free cost >= paid cost'
    return 'free', 'slack within budget and risk below paid cost'


def main(path):
    with open(path) as fh:
        rows = [ClassStats(**json.loads(l)) for l in fh if l.strip()]
    print('class             slack_used  budget  miss_p  free_risk  decision')
    for s in sorted(rows, key=lambda r: r.name):
        used = slack_used(s)
        decision, why = route(s)
        risk = miss_proxy(s) * s.miss_penalty
        used_s = 'n/a' if used is None else '%ds' % used
        print('%-17s %9s %6ds %7.2f %10.2f  %s' % (
            s.name, used_s, s.slack_budget, miss_proxy(s), risk, decision))
        print('%-17s reason: %s' % ('', why))


if __name__ == '__main__':
    main(sys.argv[1] if len(sys.argv) > 1 else 'jobs.jsonl')

Declared workload for the sample fixture

State the conditions before you quote any result.

Two job classes, 1,000 and 900 completed jobs in one 24h window.
Free lane: one worker pool, no reserved capacity, timing measured per attempt.
Paid baseline: p50 per job from the previous week on reserved capacity.
No model-quality comparison. This drill prices time, not accuracy.

Run it locally in under a minute

Create the fixture. These are sample values, not published measurements.

{"name":"nightly-embed","attempts":1180,"completed":1000,"free_wall_p50":38.0,"free_wall_p95":96.0,"paid_wall_p50":41.0,"paid_cost":0.42,"slack_budget":120.0,"miss_penalty":180.0}
{"name":"ocr-backfill","attempts":960,"completed":900,"free_wall_p50":22.0,"free_wall_p95":150.0,"paid_wall_p50":24.0,"paid_cost":0.11,"slack_budget":60.0,"miss_penalty":40.0}

Run the router:

python3 breakeven.py jobs.jsonl

Expected output on the fixture above. Label it expected until your own window reproduces it.

class             slack_used  budget  miss_p  free_risk  decision
nightly-embed            72s    120s    0.00       0.00  free
                  reason: slack within budget and risk below paid cost
ocr-backfill            136s     60s    0.56      22.36  reserved
                  reason: p95 slack use exceeds budget

The second class is the interesting one. Its per-attempt p95 is slower, and its retry ratio adds about 7 percent. Slack use lands at 136 seconds against a 60 second budget.

Free is the expensive lane here, and tokens never entered the calculation.

Failure injection: prove the ledger counts

A routing rule you never broke is a guess. Break the lane first.

# worker.py is a throwaway harness, not a model client.
python3 worker.py --class ocr-backfill --concurrency 4 &
WPID=$!
for i in 1 2 3; do
  sleep 60
  pkill -f 'worker.py --class ocr-backfill' -n
  sleep 5
  python3 worker.py --class ocr-backfill --concurrency 4 &
  WPID=$!
done
kill $WPID
python3 breakeven.py jobs.jsonl

Watch one thing: does a killed attempt show up as attempt=fail or does it vanish?

If it vanishes, your attempts counter is wrong. Your retry ratio is then wrong. Your routing decision is then wrong.

A minimal harness so the smoke test runs:

# worker.py - throwaway harness for the ledger drill only.
import argparse, random, time

p = argparse.ArgumentParser()
p.add_argument('--class', dest='cls', required=True)
p.add_argument('--concurrency', type=int, default=1)
p.add_argument('--attempt-seconds', type=float, default=30.0)
p.add_argument('--fail-rate', type=float, default=0.15)
a = p.parse_args()

while True:
    time.sleep(a.attempt_seconds)
    ok = random.random() > a.fail_rate
    print('%s attempt=%s' % (a.cls, 'ok' if ok else 'fail'), flush=True)

This harness proves your ledger counts. It proves nothing about any model. Swap in your real worker for that.

Telemetry you must emit

Without these fields the ledger is unreadable a week later.

job_class and route at admission time
attempt_index per job, starting at 0
attempt_started_at, attempt_ended_at
attempt_outcome: ok, preempted, timeout, throttled, worker_lost
deadline_at and slack_remaining_s at completion
tokens_in, tokens_out, even on a free tier
worker_id, so a bad host is visible

Keep tokens_in on free runs. You need it later to compute the exchange rate between free capacity and reserved capacity.

Decision table and threshold rationale

Evidence in the window	Route	Why
p95 slack use under 50 percent of budget	free	headroom absorbs the retry tax
p95 slack use between 50 and 100 percent	free, alert at 70 percent	one bad hour breaks it
p95 slack use over budget	reserved	measured breach, not a guess
risk-adjusted free cost at or above paid cost	reserved	free is the expensive lane
zero completed jobs	hold	no data, no route

Why 50 percent as a default? Because retry ratio is usually noisier than wall time. If your retry ratio p95 moves more than your duration p95, tighten the threshold.

If the miss penalty is contractual, do not run the free lane at any threshold. Slack budgets are for engineering tradeoffs, not for contracts.

Cleanup and rollback

Run these in order. Write them down before the drill, not after.

Drain the class. Set free-lane route weight to 0.
Wait for in-flight attempts, or cancel them and record worker_lost explicitly.
Restore the reserved weight from your config store, never from memory.
Keep the window JSONL file. It is the evidence for the next review.
Revert alert thresholds only after two clean windows.

Step 3 matters most. A rollback typed by hand at 02:20 is a second incident.

Where free capacity fits in this drill

The drill needs a free lane before you can price one. MonkeyCode is an open-source project with free model access; the operator states 10M free tokens and a free server option.

I treat both availability claims as operator-supplied. Limits and terms change, so confirm them on the project page before you plan around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In this workflow the free tier is exactly the lane whose retry ratio you are measuring. Point the harness at one small class, run one window, read the ledger. Start with one class and one window on the free tier, then decide.

Limitations and who should not use this

Be honest about the edges.

Not for synchronous user traffic with a hard p99. Free lanes are variable by nature.
Not for classes where a single miss is contractually unacceptable. Keep those reserved.
Not reliable under about 200 completed jobs per window. The p95 is noise below that; widen the window.
miss_proxy is a proxy, not a fitted model. Replace it with your own estimator if you have one.
Do not use this to justify free capacity against a regulated deadline without a named risk owner.

The threshold question

Every number here is a threshold with a rationale, and yours will differ. The useful question is not whether free capacity is cheap.

It is: how many seconds of slack does a retry buy, and who pays when the budget runs out?

Answer that with a measured window, and routing stops being a mood.

Measure Rework per Completed Job Before a Free Lane Ships

Odd_Background_328 — Thu, 15 Oct 2026 11:08:30 +0000

Measure Rework per Completed Job Before a Free Lane Ships

Last quarter a 400-job batch finished with a smaller token bill than the paid lane. It also missed the delivery window by 41 minutes.

Both numbers were true. The free lane spent fewer tokens per attempt. It also restarted roughly one job in nine.

Every restart paid queue time twice. Tokens got cheaper. Wall clock did not.

That is the trap with free capacity. You price the token bill and ignore rework amplification. This post gives you a ledger for the second number.

The topology I am describing

Three pieces, all reproducible on one host:

A router that assigns each job to a free or paid lane.
A worker that checkpoints every job and resumes after a restart.
A telemetry sink that writes one JSON line per attempt.

The router config decides more than the model choice does.

router:
  lanes:
    - name: free
      max_attempts: 2
      checkpoint: required
      idempotency_key: "{{ job_id }}"
    - name: paid
      max_attempts: 1
  admission:
    require_slack_s: 0     # set from measurement, not from vibes
    require_checkpoint: true

The idempotency_key is the whole safety story. Without it, a resumed job repeats its side effects.

The telemetry contract

One line per attempt, not per job. Attempts are what you pay for.

{"job_id":"b-1042","lane":"free","attempt":1,"enqueue_s":0,"start_s":812,"end_s":934,"outcome":"preempted","tokens_in":9100,"tokens_out":2100,"interventions":1}
{"job_id":"b-1042","lane":"free","attempt":2,"enqueue_s":0,"start_s":1204,"end_s":1330,"outcome":"ok","tokens_in":9100,"tokens_out":2050,"interventions":0}

Required fields: job_id, lane, attempt, enqueue_s, start_s, end_s, outcome, tokens_in, tokens_out, interventions.

interventions counts human touches. A retry that needs no operator is cheap. A retry that wakes someone at 02:00 is not.

The artifact: a break-even ledger

Save this as break_even.py. It answers one question: can this lane absorb the remaining time before the deadline?

#!/usr/bin/env python3
"""break_even.py - decide if a free lane can absorb a deadline.

Usage:
  python break_even.py telemetry.jsonl --slack-s 3600 --margin-s 300
"""
import argparse, json
from collections import defaultdict


def load(path):
    rows = []
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def p95(values):
    if not values:
        return 0.0
    ordered = sorted(values)
    idx = min(len(ordered) - 1, int(round(0.95 * (len(ordered) - 1))))
    return ordered[idx]


def summarize(rows, lane):
    attempts = [r for r in rows if r["lane"] == lane]
    if not attempts:
        return None
    jobs = {r["job_id"] for r in attempts}
    done = {r["job_id"] for r in attempts if r["outcome"] == "ok"}
    wall = sum(max(0.0, r["end_s"] - r["start_s"]) for r in attempts)
    waits = [max(0.0, r["start_s"] - r["enqueue_s"]) for r in attempts]
    runs = [max(0.0, r["end_s"] - r["start_s"]) for r in attempts]
    tokens = sum(r["tokens_in"] + r["tokens_out"] for r in attempts)
    return {
        "lane": lane,
        "completed": len(done),
        "completion_rate": len(done) / max(1, len(jobs)),
        "rework_multiplier": len(attempts) / max(1, len(done)),
        "s_per_completed_job": wall / max(1, len(done)),
        "p95_wait_s": p95(waits),
        "p95_run_s": p95(runs),
        "tokens_per_completed_job": tokens / max(1, len(done)),
        "interventions_per_100": 100.0 * sum(r.get("interventions", 0) for r in attempts) / max(1, len(done)),
    }


def required_slack_s(stats, margin_s):
    """Worst-case wait plus rework-amplified run time plus a margin."""
    return stats["p95_wait_s"] + stats["rework_multiplier"] * stats["p95_run_s"] + margin_s


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("telemetry")
    ap.add_argument("--slack-s", type=float, required=True, help="seconds left before the deadline")
    ap.add_argument("--margin-s", type=float, default=300.0)
    args = ap.parse_args()

    rows = load(args.telemetry)
    print(f"{'lane':6}{'done':>6}{'rework':>8}{'s/job':>8}{'p95 wait':>10}{'tok/job':>10}{'need':>9}  verdict")
    for lane in ("free", "paid"):
        s = summarize(rows, lane)
        if not s:
            continue
        need = required_slack_s(s, args.margin_s)
        verdict = "OK" if args.slack_s >= need else "TOO TIGHT"
        print(f"{lane:6}{s['completed']:>6}{s['rework_multiplier']:>8.2f}"
              f"{s['s_per_completed_job']:>8.1f}{s['p95_wait_s']:>10.0f}"
              f"{s['tokens_per_completed_job']:>10.0f}{need:>8.0f}s  {verdict}")


if __name__ == "__main__":
    main()

The formula is the point. A lane that reworks 1.3 attempts per job needs 30 percent more run time, not 30 percent more tokens.

Declared workload for the example run

State the workload or the numbers mean nothing. The example below assumes:

400 jobs, each producing 2 KB of text output.
Job sizes split 60/30/10 across small, medium, and large prompts.
One local worker per lane, no GPU, checkpoint store on local disk.
Deadline slack measured from batch start to the promised handoff.

What the output looks like

This is expected output from the declared workload. It is not a benchmark of any provider.

lane    done  rework   s/job  p95 wait   tok/job     need  verdict
free     364    1.28   118.4       905    41200    2116s  TOO TIGHT
paid     400    1.01    92.7       180    38900     499s  OK

Read the two rows together. The free lane is cheaper per token. It still needs four times the slack.

Failure drill: kill the worker mid-job

Never trust a checkpoint you have not broken. Run this drill locally first.

python runner.py --lane free --jobs jobs.jsonl --telemetry t.jsonl &
RUNNER=$!
sleep 25
kill -9 $RUNNER                    # simulate a preemption, not a graceful stop
python runner.py --lane free --jobs jobs.jsonl --telemetry t.jsonl --resume

Then verify three things:

The job restarts from its last checkpoint, not from zero.
The idempotency key stops duplicate writes. Query your dedupe table for the test key.
break_even.py shows rework above 1.0 for the free lane, and interventions_per_100 above zero.

If step 2 fails, the free lane is the wrong bet. Duplicate side effects cost more than the tokens you saved.

Observed versus inferred

Keep the two apart in your own notes.

Observed: rework multiplier, p95 wait, tokens per completed job, interventions.
Inferred: why the restarts happened, and whether a paid lane would also fail under the same load.

The ledger only settles the first category. Do not put the second one in an incident review as fact.

Decision table

Tune the thresholds with your own telemetry. The left column is the condition you check.

Condition	Route to	Reason
Slack >= required, checkpoint present	free lane	rework absorbs into spare time
Rework multiplier above 1.5	paid lane	each retry repays the queue
Job not idempotent	neither, fix first	duplicates are unrecoverable cost
Interventions per 100 jobs above 5	paid lane	operator time dwarfs token savings
Deadline under 30 minutes, no checkpoint	paid lane	no recovery path exists

Cleanup and rollback

A free lane needs a kill switch before it needs a budget.

export FREE_LANE_MAX_ATTEMPTS=2
export FREE_LANE_BUDGET_TOKENS=5000000
curl -sX POST localhost:8080/admin/drain -d '{"lane":"free"}'   # finish in-flight only

After the drill: drain the lane, wait for in-flight jobs, delete the test telemetry, and reset the dedupe table for the test keys. Roll back by setting require_slack_s back to the paid-lane value in the router config.

Where a free lane comes from

MonkeyCode is an open-source project that offers free model access and a free server option, per its operator. I am describing it here as the free lane under test, because the checkpoint-and-resume path is what you actually need to break before production. It does not change the method above, and it does not replace your own telemetry. Check the project's current limits before you plan a batch around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not use this approach

Teams with a hard real-time SLO. Rework of any size breaks the contract.
Jobs without checkpoints. You will repay the entire job after every restart.
Work with non-idempotent side effects, such as payments or emails without dedupe keys.
Batch pipelines where one operator hour costs less than one hour of duplicated compute.

The ledger is only worth building if you can act on the verdict. If you cannot route to a second lane, measure the rework anyway and negotiate the deadline instead.

If you want a lane to run this drill against, MonkeyCode's free model access and free server option are a reasonable starting point for the experiment above.

Test Free-Path Dwell Locally Before the Batch Cutoff

Odd_Background_328 — Mon, 12 Oct 2026 22:56:07 +0000

The page hits during a quiet overnight batch window. Your free worker still shows spare memory and idle cores. Queue age has already climbed to forty-one seconds.

The cutoff sits ninety seconds from the original enqueue. Two unused retries remain inside the job policy. Raising concurrency looks calm, and that reading is wrong.

Spare memory is not a promise of service time. That slot is shared, and nobody reserved it for you. Wait, service, and one retry can still miss the cutoff.

Read the contradiction first

Host utilization said the machine still had room. Queue age said the batch was already slipping. You need one figure that can reconcile both views.

Call that figure free-path dwell for this batch. Add queue age, expected service, and a retry tax. Compare the sum with remaining deadline slack first.

Admit on the free path only when dwell still fits. If dwell exceeds slack, free capacity is the wrong bet. Defer the job, or hand it to a reserved worker.

Do not buy another retry on a slot you do not own. A cheap token price cannot repay a missed cutoff. Wall-clock slack is the budget that actually expires.

Declare the drill before you touch a host

Write the conditions down before you start the process. This drill runs in a local process, not your cluster. Lines marked expected are illustrations, not your metrics.

Use these declared inputs for the first pass. Change them if your cutoff differs, and record the change. Do not tune them after you see a passing printout.

Batch size is forty jobs, one attempt each at first.
Each job estimates twelve hundred tokens of completion.
Expected service on the free path is eight seconds.
Deadline slack at enqueue starts at ninety seconds.
Retry tax is one extra full service interval.
Max queue age you will tolerate is thirty seconds.

The token count is your estimate, not a vendor quota. Do not treat it as a measured bill from any provider. You are testing admission math, not model quality.

Keep the topology on one page

Keep the pieces small so the failure stays local. One gate sits between enqueue and every worker choice. It may admit, defer, or shift to reserved.

enqueue -> dwell_gate -> free_path
                 |-> defer_queue
                 |-> reserved_path

The gate reads four fields and writes one decision. Log those fields on every attempt, including the rejects.

queue_age_s records seconds since the job was enqueued.
expected_service_s stores the service time you already declared.
retry_tax_s prices one unused attempt in wall seconds.
deadline_slack_s is cutoff time minus elapsed wall time.
The decision field is admit_free, defer, or reserved.

Utilization stays on the dashboard, not inside the gate. It explains a hot host, not a late queue. Slack is the threshold, because the cutoff is absolute.

Queue age gets a veto, not the whole decision. A young queue can still lose if retry tax is fat. That split is the point of pricing dwell at all.

Run the gate on a laptop

Save the script below as dwell_gate.py on a laptop. Run it with Python 3, and keep it off the cluster. It does not call a model and does not open a port.

#!/usr/bin/env python3
# Local admission drill. Expected lines are comments, not your metrics.

from dataclasses import dataclass

@dataclass
class Job:
    queue_age_s: float
    expected_service_s: float
    retry_tax_s: float
    deadline_slack_s: float
    max_queue_age_s: float = 30.0

def dwell(job: Job) -> float:
    return job.queue_age_s + job.expected_service_s + job.retry_tax_s

def decide(job: Job) -> str:
    if job.queue_age_s > job.max_queue_age_s:
        return 'defer'
    if dwell(job) > job.deadline_slack_s:
        return 'reserved'
    return 'admit_free'

def decide_safe(job: Job) -> str:
    if job.expected_service_s <= 0 or job.deadline_slack_s <= 0:
        return 'defer'
    return decide(job)

samples = [
    Job(12, 8, 8, 90),   # expected admit_free
    Job(41, 8, 8, 90),   # expected defer, age cap
    Job(20, 8, 8, 30),   # expected reserved, dwell over slack
    Job(20, 8, 0, 30),   # expected admit_free, retry tax removed
    Job(10, 0, 8, 90),   # expected defer, non-positive service
    Job(10, 8, 8, -1),   # expected defer, negative slack
]

if __name__ == '__main__':
    for i, job in enumerate(samples, start=1):
        choice = decide_safe(job)
        print(
            'job=%s dwell=%.0f slack=%.0f decision=%s'
            % (i, dwell(job), job.deadline_slack_s, choice)
        )

Run the file with one local Python command. Stay in the directory where you saved the script. A missing file is a drill error, not a capacity signal.

python3 dwell_gate.py

Expected lines

Treat the next lines as expected output, not a measurement.

job=1 dwell=28 slack=90 decision=admit_free
job=2 dwell=57 slack=90 decision=defer
job=3 dwell=36 slack=30 decision=reserved
job=4 dwell=28 slack=30 decision=admit_free
job=5 dwell=18 slack=90 decision=defer
job=6 dwell=26 slack=-1 decision=defer

Job one fits, so the free path is an acceptable bet. Job two is young in slack but old in queue age. You defer it instead of hoping the shared slot clears.

Job three still has queue age under the cap. Dwell overruns slack, so you leave the free path. Job four drops the retry tax and fits again.

That last case is the trap inside a retry policy. A free retry looks cheaper than a reserved attempt. It is only cheaper when the extra seconds still fit.

Job five has a zero service guess, so the guard defers it. Job six has negative slack, so you do not admit it. Both guards fail closed, which is the point of the drill.

Use this table when the printout and the page disagree. The why column is the operational action, not a host guess.

queue_age_s	dwell_s	slack_s	decision	why
12	28	90	admit_free	dwell fits and age is young
41	57	90	defer	age cap fired before slack
20	36	30	reserved	age is fine, dwell is not
20	28	30	admit_free	retry tax removed, dwell fits
10	18	90	defer	service guess is not positive
10	26	-1	defer	cutoff already passed

Inject one boring failure

Add a stuck attempt before you trust the printed gate. Set queue age to fifty and slack to forty. Force retry tax to a full service interval of eight.

Dwell becomes sixty-six against forty seconds of slack. The age cap fires first, so the printed decision is defer. If the script still prints admit_free, stop the rollout.

A second injection zeros expected service by mistake. Dwell then under-counts, and the gate admits too early. Reject a non-positive service guess before you compare.

Negative slack means the cutoff has already passed. Defer, alert, and do not hide that miss inside a retry. Cleanup is deleting the local script and stopping the process.

rm -f ./dwell_gate.py

If a unit or cron copied the drill, disable that unit first. Then confirm the process table no longer shows the script. Leave production flags untouched until the canary section below.

Price tokens by the seconds they occupy

A free completion still spends wall time you cannot refund. Multiply estimated tokens by dwell seconds for a pressure gauge. That product is not an invoice and not a benchmark.

Twelve hundred tokens over sixty-six seconds is a heavy wait. The same tokens on an eight-second reserved turn weigh less. You are buying clock time, not only the completion text.

Count retries as extra token copies, not as free do-overs. Two attempts can double the token load and the dwell. Your ledger should store both, or the cost story splits.

If the model hop has its own queue, add that age too. A zero price does not make the second queue instant. Another wait can erase the slack you just protected.

Where a free path is allowed to help

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator notes for this draft mention two availability claims. One is free model access, and one is a free server option.

Neither claim states a quota, a chip, or an expiry here. Confirm those facts in current product docs before you plan. A free path can still be the wrong bet for this batch.

Use it only when the dwell gate returns admit_free. A free server is useful for this local class of work. Use it to replay the gate, not to absorb a late queue.

If the option disappears tomorrow, your SLO must still hold. Do not place the fallback on that same free host. Reserved means a worker you can drain and roll back.

Shared free capacity cannot be that rollback target. Keep any longer trial on jobs you can defer safely. Point only slack-rich jobs at the free model path.

Leave cutoff work on the reserved path you already trust. Free access can host the drill, not the promise. Your cutoff still belongs to a worker you control.

Thresholds worth arguing about in review

Pick deadline slack as the primary admit threshold. The cutoff is a wall-clock fact, not a utilization gauge. Queue age is the leading indicator you watch beside it.

Utilization is third, and it should not cast a vote. A cold GPU can still sit behind someone else's queue. A hot GPU can still serve you if your dwell fits.

Start with these caps for the declared workload above. Treat them as review numbers, not as universal law.

Defer the job when queue age exceeds thirty seconds.
Leave the free path when dwell exceeds remaining slack.
Price every unused retry as one full service interval.
Refuse a service guess that is zero or negative.

Revisit the thirty-second cap after one real week of logs. If deferrals cluster near twenty seconds, tighten that cap. If reserved shifts dominate, your service guess is too timid.

Alert on the miss, not on the idle host

Page when an admit_free decision later records a slack miss. Do not page only because memory looks full or empty. The miss is the user-facing failure you set out to avoid.

Use the sketch below, and do not treat it as a live alert. Names must match your metrics before you paste anything.

decision=admit_free AND slack_miss=true

Route that alert to the same channel as queue-age pages. Attach dwell, slack, and retry count to the incident note. Without those three fields, the on-call will chase utilization again.

If misses exceed one percent of free admits, freeze the flag. That one percent is a starting bar for this drill only. Replace it with your error budget before production use.

Roll the gate back without draining the wrong queue

Ship the gate behind a flag, defaulting to off. Admit a canary slice of five percent of batch jobs only. Watch slack misses, defer rate, and reserved shift rate.

If the miss rate rises, turn the feature flag off. Drain jobs already admitted on the previous stable path. Do not flush the defer queue into the free worker.

Paste these steps into the runbook before the canary starts. Do not start the slice until the steps are in the ticket.

Disable the feature flag that is named dwell_gate_enabled.
Confirm new jobs skip the gate within one minute.
Let in-flight reserved jobs finish, then stop canary hosts.
Delete local drill files so a cron cannot revive them.

Rollback is complete when new decisions stop appearing in logs. Keep the reserved pool warm until that log line goes quiet. A half-disabled gate is worse than no gate at all.

Who should skip this approach

Skip it when the work cannot wait and cannot move. A pager path with no reserved fallback needs another design. This gate only chooses among admit, defer, and reserved.

Skip it for interactive leases that need sub-second slack. That failure mode needs a different hold-time control. Do not mix that control into this batch cutoff drill.

Skip it if you need a benchmark of model quality. The script never sends a prompt and never scores answers. It will not tell you which model is cheaper per token.

Skip it when service time is still an unknown distribution. A single guessed interval will lie under a long tail. Measure a p95 service time before you automate the shift.

What you should log before the next review

Keep one row per decision, not one row per host. Store queue age, dwell, slack, decision, and retry count. Drop raw prompts from this log so the cost file stays small.

Review three ratios when the week of logs ends. Keep the window fixed so the ratios stay comparable.

Count free admits that later missed their slack budget.
Count defers that would have fit on a reserved worker.
Count reserved shifts whose dwell would have fit free.

The first ratio means the service guess was too kind. The second means you deferred work a reserved turn could save. The third means the age cap is stricter than the cutoff.

Adjust one cap at a time, then rerun the local samples. Changing age and slack rules together hides the cause. Write the new cap beside the old expected output lines.

Free capacity stays a tool, not a standing reservation. Use the free model path when dwell fits your slack. When it does not, pay for a worker you can actually drain.

If those options are available in your account, replay the samples. Run that replay on the free server before any batch attach. Check current docs first, because availability is not an SLO.

Debug Free-Slot Waste With a Retry-Tax Ledger

Odd_Background_328 — Sat, 10 Oct 2026 16:15:45 +0000

Your nightly embed job pages you at 02:14 local time.
The free slot still reports spare capacity on the board.
The batch deadline has already slipped by eleven minutes.

Treat this page as a composite drill, not a named outage.
Which action follows from that evidence, not from the green badge?
You do not add workers just because the badge stays green.

You price the free slot before you admit the next batch.
Spare capacity is the bait, not the decision itself.

What spare capacity hid

Spare capacity is not the same thing as a cheap path.
A free token price can still waste your nightly window.
Retries duplicate work you already spent on a dead attempt.

This note is an ops cost drill, not a pricing brochure.
You will run a local ledger against a declared workload.
You will reject free-path work when retry tax breaks budget.

Topology before any code

Put four boxes on the diagram before you write a script.

The batch client submits each job with a deadline and budget.
The admission ledger scores that job before any enqueue call.
The free slot is one route, not the only route available.
The reserved worker is the rollback path when tax runs high.

Declare the workload in config, not in a chat thread later.

window_seconds: 900
jobs: 40
input_tokens_per_job: 1200
max_attempts: 3
inject_fail_after_attempt: 1
fail_fraction: 0.25
budget_units: 60000
queue_wait_unit_per_second: 200
seed: 7
service_seconds: 1.5

Those numbers are declared test inputs, not live measurements.
Change them to match your window before you trust a reject.
Keep the seed fixed so a teammate can replay your rows.

Fields that earn the page

Record these fields on every attempt, including the killed one.

job_id ties the attempt back to one batch item.
attempt starts at one and climbs only on retry.
queue_wait_ms is delay before that attempt actually starts.
tokens_in and tokens_out come from your own meter.
outcome is ok, killed, or rejected for that row.
deadline_slack_ms is remaining window time, not host utilization.
retry_tax is the admission number you will compare to budget.

Separate what you observed from what you infer next.
Observed means the ledger row the script just wrote down.
Inferred means a vendor story you did not measure here.

How retry tax is defined

Retry tax is the waste a zero price tag can hide.
Count tokens already spent on attempts that did not finish.
Add a time charge for queue wait inside this window.

Use this definition only inside the drill, and label it local.

retry_tax = spent_tokens_on_failed_attempts
          + queue_wait_seconds * queue_wait_unit_per_second
admit if projected_tax + new_job_tokens <= budget_units

Do not treat that formula as a cloud invoice.
It is an admission signal for this declared workload only.
A zero token price does not zero the time charge.

That weight is declared so wait can rival a token chunk.
It is not a dollar rate and not a vendor price.

Why not utilization

CPU idle time can sit next to a blown token budget.
A free worker can look empty while your window is gone.
Utilization answers a different question than retry tax does.

The local ledger

This script is a proposal you execute on your own laptop.
It does not call a remote model and does not spend quota.
Failure injection is a coin flip you control in process.

#!/usr/bin/env python3
"""Local retry-tax ledger. Proposal until you run it."""
import json
import random
from pathlib import Path

CFG = {
    'window_seconds': 900,
    'jobs': 40,
    'input_tokens_per_job': 1200,
    'output_tokens_on_ok': 180,
    'max_attempts': 3,
    'inject_fail_after_attempt': 1,
    'fail_fraction': 0.25,
    'budget_units': 60000,
    'queue_wait_unit_per_second': 200,
    'seed': 7,
    'service_seconds': 1.5,
}

def retry_tax(spent_failed, queue_wait_s):
    wait_units = int(queue_wait_s * CFG['queue_wait_unit_per_second'])
    return spent_failed + wait_units

def row(job_id, attempt, queue_wait_s, tokens_in, tokens_out, outcome, clock, tax):
    return {
        'job_id': job_id,
        'attempt': attempt,
        'queue_wait_ms': int(queue_wait_s * 1000),
        'tokens_in': tokens_in,
        'tokens_out': tokens_out,
        'outcome': outcome,
        'deadline_slack_ms': int((CFG['window_seconds'] - clock) * 1000),
        'retry_tax': tax,
    }

def main():
    rng = random.Random(CFG['seed'])
    rows = []
    window_spent = 0
    clock = 0.0
    rejected = 0
    finished = 0
    killed_rows = 0
    abandoned = 0
    for job_id in range(CFG['jobs']):
        spent_failed = 0
        done = False
        for attempt in range(1, CFG['max_attempts'] + 1):
            queue_wait_s = rng.uniform(0.4, 6.0)
            tax = retry_tax(spent_failed, queue_wait_s)
            projected = tax + CFG['input_tokens_per_job']
            slack_ms = int((CFG['window_seconds'] - clock) * 1000)
            remaining = CFG['budget_units'] - window_spent
            if projected > remaining or slack_ms <= 0:
                rows.append(row(
                    job_id, attempt, queue_wait_s, 0, 0, 'rejected', clock, tax,
                ))
                rejected += 1
                done = True
                break
            clock += queue_wait_s + CFG['service_seconds']
            fail_now = (
                attempt == CFG['inject_fail_after_attempt']
                and rng.random() < CFG['fail_fraction']
            )
            if fail_now:
                spent_failed += CFG['input_tokens_per_job']
                window_spent += CFG['input_tokens_per_job']
                killed_rows += 1
                tax_after = retry_tax(spent_failed, queue_wait_s)
                rows.append(row(
                    job_id, attempt, queue_wait_s,
                    CFG['input_tokens_per_job'], 0, 'killed', clock, tax_after,
                ))
                continue
            window_spent += (
                CFG['input_tokens_per_job'] + CFG['output_tokens_on_ok']
            )
            finished += 1
            rows.append(row(
                job_id, attempt, queue_wait_s,
                CFG['input_tokens_per_job'], CFG['output_tokens_on_ok'],
                'ok', clock, tax,
            ))
            done = True
            break
        if not done:
            abandoned += 1
    payload = {
        'config': CFG,
        'finished': finished,
        'killed_rows': killed_rows,
        'rejected': rejected,
        'abandoned': abandoned,
        'window_spent': window_spent,
        'budget_left': CFG['budget_units'] - window_spent,
        'rows': rows,
    }
    out = Path('/tmp/retry-tax-ledger.json')
    out.write_text(json.dumps(payload, indent=2))
    print(
        'finished=%s killed_rows=%s rejected=%s abandoned=%s budget_left=%s'
        % (finished, killed_rows, rejected, abandoned, payload['budget_left'])
    )
    print('wrote %s' % out)

if __name__ == '__main__':
    main()

Run the file only after you read the config block above.

python3 retry_tax_ledger.py

Expected output is labeled, because this draft did not execute it.
You should see one summary line plus a JSON ledger file.
The file path is /tmp/retry-tax-ledger.json on that laptop.

# expected shape, not a measured production result
finished=<int> killed_rows=<int> rejected=<int> abandoned=<int> budget_left=<int>
wrote /tmp/retry-tax-ledger.json

If rejected stays at zero, your budget is too loose.
Tighten budget_units and run the same seed again.
If every job is rejected, your time charge is too harsh.

What the injection is for

The drill kills a fraction of first attempts on purpose.
A killed row still records tokens_in for that attempt.
The next attempt must pay that spend inside retry_tax.

That behavior is the point of the injection, not a bug.
A free slot that dies mid-job is not a free retry.
You already burned input tokens a later attempt will repeat.

On a killed row, retry_tax includes tokens from that failed attempt.
The admit check used the tax from before that spend.
Watch deadline_slack_ms on the killed row before you retry.

Negative slack means the window is already lost for that job.
Reject it instead of spending a second attempt on a corpse.
Build that sample by hand so the fields stay obvious.

The next block is an illustration, not seed output.
Use the numbers below to check your own addition only.

job_id: 3
attempt: 1
queue_wait_ms: 2100
tokens_in: 1200
tokens_out: 0
outcome: killed
deadline_slack_ms: 840000
retry_tax: 1620

Wait units are 2.1 seconds times the declared weight of 200.
Add the killed input tokens and you get 1620.
If your hand total disagrees, stop and fix the formula first.

When the free path is the wrong bet

Use this table before you route the nightly window.

Signal you can read	Free path bet	Action you take
Retry tax under budget and slack still positive	Reasonable for this batch	Admit to the free slot
First attempt killed and tax near the budget	Wrong bet tonight	Reject or switch path
Queue wait alone consumes the remaining window	Wrong bet tonight	Shed new jobs now
An interactive caller is waiting on this queue	Wrong bet always	Do not use this drill

Read the table as admission policy, not as a benchmark.
No latency percentile is claimed from any vendor here.
You measured only the local rows your script wrote down.

Where a free path still fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode exposes free model access and a free server option.
Those two routes can sit in the diagram as the free slot.
You still reject the job when projected tax breaks budget.

Do not hard-code a token grant, a duration, or a hardware size.
Those limits were not measured in this drill and can change.
Meter the path you actually call, then apply the same threshold.

A free label does not replace the admission check above.
If the meter cannot see killed-attempt tokens, do not route there.
Missing tokens_in is how a free path looks cheaper than it is.

If you already meter each attempt, try that check on the free server option.

The threshold you defend on the call

Ask for one threshold before the nightly window opens.
This drill uses budget_units, not a utilization target.
Queue age alone misses tokens you are about to pay twice.

Deadline slack alone misses budget you already spent once.
You would rather reject a job than duplicate a killed attempt.
The budget is the cap on that duplication for this window.

Pick the number from your window, not from the example config.
Start from input tokens times jobs, then cut for fail fraction.
Write that rationale beside the config so the next on-call sees it.

If you cannot explain the cut, you are not ready to reject.

Actions after the page

Follow this order when the badge is green and the window is not.

Read the latest ledger rows with outcome=killed before you scale.
Sum retry_tax for this window, not for the whole day.
Compare that sum with budget_units you declared at the start.
Reject new free-path jobs when the next projection exceeds budget.
Drain in-flight jobs only while their slack is still positive.
Switch the route to the reserved worker if the window must finish.

Do not restart the free slot as your first move tonight.
A restart hides the killed rows you needed for the tax math.
Capture the JSON file, then decide on admit or reject.

Cleanup and rollback

Cleanup is part of the drill, not a follow-up ticket later.
Confirm the process name before you run the stop command.
Then remove the ledger file and stop only that local script.

rm -f /tmp/retry-tax-ledger.json
pkill -f retry_tax_ledger.py || true

Rollback the route with a config flag, not a redeploy guess.

admission_mode: reserved_path
free_path_enabled: false

Apply that flag, then stop admitting new work to the free slot.
Let in-flight reserved jobs finish inside the same declared window.
Re-enable the free path only after a clean ledger hour.

A clean hour means zero killed rows and tax under budget.
If either check fails, leave free_path_enabled set to false.
Write the reason in the alert comment so the next shift sees it.

Limits to say out loud

This drill is a single process with a fixed random seed.
It does not model multi-tenant noise on a shared free slot.
It does not read a real invoice or a live quota counter.

Free access terms can change, so do not hard-code a promise.
The script invents queue wait with random.Random, not a trace.
Service time is fixed at one and a half seconds in this script.

Replace that constant with a trace before you trust a reject.
A fixed service time will lie once your real calls vary.

Who should skip this approach entirely:

Do not put interactive user traffic on this admission path.
Do not use it where a missed job is a safety incident.
Do not use it if you cannot meter tokens on each attempt.
Do not treat the example budget as your real production cap.
Do not use the drill to justify a cost number on an invoice.

If you lack per-attempt token counts, stop and fix metering first.
A ledger without tokens_in on killed rows will under-count tax.
That under-count is how spare capacity fools the nightly page.

What you keep with the alert

Keep the script, the config, and the rollback flag in one folder.
Run the seed once before the window, not while you are paging.
Store the JSON beside the alert so the next person sees rows.

If the projection exceeds budget, reject the job before enqueue.
Leave the rollback flag in the alert until a clean hour passes.
That is the whole loop from measured tax to cleanup.

Count Retry Tokens Before You Trust a Free Worker

Odd_Background_328 — Thu, 08 Oct 2026 09:18:30 +0000

A Thursday page with an idle host

Your pager fires at 02:14 on a Thursday. A batch summarizer has waited nine minutes in queue. The free worker still looks idle on the host graph.

Token counters climb because each retry resends the full prompt. Your deadline slack has fallen to forty seconds. You need an action that follows from that evidence.

Public posts this week cheer fast AI demos. Your batch queue does not share that mood. You still have to price each retry in time.

This opening is a drill scenario, not a production incident report. The numbers above are inputs you can change. Do not treat them as measured fleet telemetry.

The contradiction in the graphs

The host graph says the worker has spare CPU. The queue graph says age is already past slack. Those two pictures do not describe the same risk.

CPU idle does not mean the job will finish on time. A free invoice does not mean the retry is cheap. Tokens leave the budget on every resend, even at zero dollars.

You are buying time with a hidden token meter. Queue age is the clock you should trust first. Utilization is only a hint about the host.

Topology you can sketch on one page

Put four boxes on the diagram before you tune anything. Number them so the runbook matches the code. No second region appears in this drill.

The producer stamps each job with a deadline.
The queue stores age, retries, and the idempotency key.
The gate chooses stay, shed, or fail closed.
The worker calls the model only on stay.

No cache tier hides a slow first token. You are testing admission, not a full platform.

Declared workload for the local run

Fix the workload before you compare any result. Each job carries a prompt of eighteen hundred tokens. The deadline slack starts at ninety seconds flat.

You allow two retries after the first attempt. A retry resends the same prompt with the same key. The local token budget for this class is eight thousand.

You run four scripted cases, not a live customer mix. You do not claim a throughput number from this script. You only claim the branch the gate selects.

When free capacity is the wrong bet

Free capacity is the wrong bet in four cases. Read them as policy, not as vendor facts. Time, tokens, and retries decide the branch together.

Queue age has reached deadline slack, so you shed the pull.
Retry count has hit the cap, so you fail closed.
Estimated tokens have crossed the remaining class budget.
The caller cannot shed, so this gate does not fit.

Use this table when you review a sample by hand.

Check	Stay when	Leave when
Queue age	Age is under slack	Age meets slack, then shed
Retry count	Count is under the cap	Cap is hit, then fail closed
Token estimate	Estimate is under budget	Budget is hit, then shed
Shed allowed	Caller can accept a skip	Caller cannot skip, do not use

Stay only when age is under slack and budget remains. A zero dollar line does not override those checks. That is the cost decision this note is about.

Fields you should log

Log the fields that justify the branch you took. Guesses about the model do not belong in the alert. Separate the sample you saw from the rule you applied.

Record queue age in seconds at the pull.
Record deadline slack in seconds at the pull.
Record the retry count beside the idempotency key.
Record prompt tokens and the estimated resend total.
Record the branch name and the threshold that fired.

The script prints a branch from your inputs. It does not observe a remote vendor queue. Label that print as computed, not collected.

Config you pin before the run

Pin the class name, the budget, and the flag default. The snippet below is a drill config, not a cluster manifest. Keep units in the key names so a reviewer cannot mix them.

class: batch-summarizer
prompt_tokens: 1800
deadline_slack_s: 90
max_retries: 2
token_budget: 8000
primary_signal: queue_age_s
secondary_signal: token_estimate
flag: cost_gate_enabled
flag_default: false

A gate you can run locally

The following program is a labeled local drill. It is not a client for any hosted control plane. Run it on your laptop before you touch production.

Use Python 3.9 or newer for this file. You need only the standard library for this file. Do not add network calls while you learn the branches.

#!/usr/bin/env python3
'''Local cost gate. Drill file, not a vendor meter.'''

import os
from dataclasses import dataclass
from typing import List, Optional


@dataclass
class Sample:
    queue_age_s: float
    deadline_slack_s: float
    retry_count: int
    prompt_tokens: int
    max_retries: int
    token_budget: int


def token_estimate(sample: Sample) -> int:
    # Model only: each attempt resends the full prompt.
    return sample.prompt_tokens * (sample.retry_count + 1)


def decide(sample: Sample) -> str:
    estimate = token_estimate(sample)
    if sample.queue_age_s >= sample.deadline_slack_s:
        return 'shed'
    if sample.retry_count >= sample.max_retries:
        return 'fail_closed'
    if estimate >= sample.token_budget:
        return 'shed'
    return 'stay'


def emit(sample: Sample) -> None:
    estimate = token_estimate(sample)
    branch = decide(sample)
    print(f'{branch} estimate={estimate} age={sample.queue_age_s:g}')


def from_env() -> Optional[Sample]:
    raw_age = os.getenv('COST_AGE')
    if raw_age is None:
        return None
    return Sample(
        queue_age_s=float(raw_age),
        deadline_slack_s=float(os.getenv('COST_SLACK', '90')),
        retry_count=int(os.getenv('COST_RETRIES', '0')),
        prompt_tokens=int(os.getenv('COST_PROMPT_TOKENS', '1800')),
        max_retries=int(os.getenv('COST_MAX_RETRIES', '2')),
        token_budget=int(os.getenv('COST_BUDGET', '8000')),
    )


def built_in_cases() -> List[Sample]:
    return [
        Sample(120, 40, 3, 1800, 2, 8000),
        Sample(10, 90, 0, 1800, 2, 8000),
        Sample(15, 90, 4, 1800, 2, 8000),
        Sample(20, 90, 2, 1800, 5, 5000),
    ]


def main() -> None:
    single = from_env()
    cases = [single] if single else built_in_cases()
    for case in cases:
        emit(case)


if __name__ == '__main__':
    main()

Save the file as cost_gate.py beside your notes. If COST_AGE is set, the file prints one sample only. If COST_AGE is absent, the file prints four built-in cases.

Expected output, labeled as expected

This block is expected output from the inputs above. It is not a benchmark from a shared cluster. If your print differs, trust your run and diff the inputs.

shed estimate=7200 age=120
stay estimate=1800 age=10
fail_closed estimate=9000 age=15
shed estimate=5400 age=20

Case one sheds because age already beat slack. Case two stays because age, retries, and tokens fit. Case three fails closed because retries passed the cap.

Case four sheds because the resend estimate crossed budget. That fourth case is the free-capacity trap in miniature. The invoice can read zero while the budget is already gone.

Age wins when two gates would fire together. The function checks slack before retries and tokens. Read that order as policy, not as physics.

Inject the failure on purpose

You inject delay by editing queue age in the sample. You do not need a chaos tool for this lesson. Change only one field, then rerun the same file.

Commands

The shell lines below pass those injections as environment variables. They call the same file you already saved. Run them one at a time so the branch stays obvious.

python3 cost_gate.py | tee /tmp/cost-gate-cases.txt
COST_AGE=90 python3 cost_gate.py
COST_AGE=10 COST_RETRIES=2 python3 cost_gate.py
COST_AGE=10 COST_BUDGET=1000 python3 cost_gate.py

What those commands should print

Set queue age to the slack value and expect shed. Set retry count to the cap and expect fail closed. Set the budget under the estimate and expect shed.

The first injection sets age equal to slack and sheds. The second injection hits the retry cap and fails closed. The third injection sets a budget under the estimate and sheds.

shed estimate=1800 age=90
fail_closed estimate=5400 age=10
shed estimate=1800 age=10

Write the three results into your runbook as examples. Mark those results expected, not observed from production traffic. Keep the command output next to the config snippet.

A future reader should reproduce the branch without guessing. Leave the scratch file until you finish that diff. Remove that scratch file in the cleanup step below.

Thresholds and the reason you pick them

Prefer queue age against deadline slack for this class. Utilization can stay low while age still misses the turn. A busy CPU percent will not page the right owner.

Why slack beats utilization

Use token estimate against a class budget as the second gate. Retries multiply the prompt, so the next attempt is not free. A cap on retry count stops a loop that never converges.

Pick slack as the primary threshold for batch summarizers. The job is late when age meets slack, even if the host is idle. That rationale beats a utilization alert for this failure.

Do not copy these numbers into another class without a review. Interactive work needs a different clock than batch age. Probe traffic needs a different budget than summarizers.

Failure handling when the gate fires

Shed means you cancel this attempt and emit a terminal skip. Fail closed means you stop retries and surface the idempotency key. Stay means you pull once and send the prompt a single time.

On shed, write a skip event with age and slack. On fail closed, write a stop event with the retry count. On stay, write a start event before the model call.

If the caller cannot accept a skip, do not use this gate. Forcing a stay on a late job hides the miss. Page a human when the skip rate crosses the class budget.

Cleanup and rollback

Cleanup is part of the drill, not an afterthought. Delete the local sample output when the review ends. Clear any scratch queue you created for the injection.

rm -f /tmp/cost-gate-cases.txt

Roll back the gate if it sheds healthy jobs. Widen slack only after you show the false shed samples. Ship the old pull path behind the same config flag.

Keep the flag default off until the runbook is reviewed. A reviewed flag is your rollback, not a redeploy scramble. Name an owner for the flag before the first enable.

Where a free endpoint fits

You can point the stay branch at a free model endpoint. MonkeyCode is the open source project named in this outreach. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The outreach brief cites a free token allowance of ten million tokens. It also cites a free server option for hands-on runs. Treat both lines as claims you must re-check today.

Open the current project docs before you budget a class. Grants change, and this note does not freeze a quota. Do not paste a token grant into an alert threshold.

Use that project here only as a place to exercise stay. Remove the name and the gate still teaches the same decision. Do not invent install flags from this article.

Follow the current project docs for setup and cleanup. If the project docs disagree with the brief, believe the docs. A free worker is optional substrate, not the decision itself.

Who should skip this approach

Skip this gate if the job cannot be shed. Skip it if you cannot log tokens per attempt. Skip it if a human must read every summary.

Skip it for interactive chat with a hard latency target. A free worker can miss that clock without a loud CPU graph. This note does not claim a latency number for any model.

Skip it if your team will treat a free grant as a contract. Free capacity is a lab bet, not a capacity promise. When that bet is wrong, paid capacity is the honest path.

Limits of the drill

The script models resend cost on your laptop. It does not measure provider latency or cold start. It does not prove that a free server stays up.

The story numbers are fixtures, not fleet observations. Architectural guesses stay outside the printed branch text. This draft does not include a hosted queue run.

This draft does not time a free server or count a grant. You should run the file and compare the expected block. If the project docs disagree with the brief, believe the docs.

Update your budget from those docs, not from this article. Then rerun the injection cases before you enable the flag. That order keeps a free endpoint from becoming a silent queue.

What you do next

Replay last week's queue samples through the same decide function. Compare shed counts with the skips your users can accept. Enable the flag only for one batch class.

If you want a free endpoint for the stay branch, stop here. Read the current MonkeyCode docs and confirm the live grant. Only then should you point stay at that endpoint.

Reject Unreserved Jobs Before Queue Age Breaks Slack

Odd_Background_328 — Thu, 08 Oct 2026 06:32:38 +0000

You get paged at 02:14 because completions stop landing.
Idle CPU on the free server looks healthy.
The projected wait is already thirty six seconds.

Your interactive slack was only thirty seconds long.
That idle capacity did not buy you time.
Replay that page locally before you trust the chart.

Read the contradiction before you touch the model

Do not start this page by swapping models.
Start with the clock that already failed you.
Queue age ate the slack before any token returned.

Utilization stayed low because the work sat unscheduled.
That split is the signal you act on.
Act on age versus slack before you touch retries.

Name the cost you actually pay

Free model access feels cheap until retries stack.
A late free server still burns your token budget.
You pay twice when the client retries blind.

You pay in wall time even when tokens are free.
Free capacity is the wrong bet without a bound.
A grant is not a seat with your name on it.

Time, tokens, and queue age move on different clocks.
A cheap token can still miss a hard deadline.
A fast CPU can still hide a stuck reservation.

Put a gate in front of the free slot

Put one admission gate in front of the worker.
Keep a local reservation ledger beside the queue.
Do not enqueue a job without a hold.

Release the hold on success, cancel, or expiry.
The free server must stay behind that gate.
One slow slot should not accept unbounded company.

Local topology

Run a single-process gate bound to port 8080.
Keep an in-memory ledger for every token hold.
Add a fake free-slot worker with injected delay.
Retry from the client only with the same key.

This is a lab drill, not a production trace.
Expected lines below are labeled, not measured live.
Nothing here is a vendor latency claim.

Declared workload

Send forty jobs, each asking for eight hundred tokens.
Keep the deadline slack fixed at thirty seconds.
Inject a free-slot service delay of twelve seconds.

Cap the client retry at one replay only.
Set the reservation TTL to twenty seconds flat.
One free slot serves one held job at a time.

Defend this threshold in the review, not a hunch.
Reject the job when projected wait exceeds remaining slack.
Do not reject work on CPU utilization alone.

Queue age tracks waiting while utilization tracks running.
Slack is the user promise you still have left.
The rationale stays simple and strictly operational here.

A busy CPU can still finish inside slack.
An idle CPU can still miss if the queue is old.
Age versus slack is the only admission test.

Ask which number you will page on tonight.
Write that number down before the next deploy.
Page on slack breach, not on a pretty CPU chart.

Reserve before the completion call

The gate below is a proposal you can run locally.
It uses only the Python standard library here.
No model name is hardcoded in this gate.

No remote quota is assumed by this process.
Projected wait equals held jobs plus one, times delay.
That formula is declared, not observed in production.

#!/usr/bin/env python3
"""Local admission gate. Proposal, not a live benchmark."""

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import time
import uuid

SLACK_S = 30.0
SERVICE_DELAY_S = 12.0
HOLD_TTL_S = 20.0
RETRY_CAP = 1
jobs = {}
holds = {}


def now():
    return time.time()


def held_count():
    return sum(1 for job in jobs.values() if job["state"] == "held")


def project_wait():
    # Declared model: one slot, each hold blocks SERVICE_DELAY_S.
    return (held_count() + 1) * SERVICE_DELAY_S


class Gate(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        return

    def _send(self, code, payload):
        raw = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_POST(self):
        length = int(self.headers.get("Content-Length", "0"))
        body = json.loads(self.rfile.read(length) or b"{}")
        if self.path == "/admit":
            self._admit(body)
        elif self.path == "/complete":
            self._complete(body)
        elif self.path == "/expire":
            self._expire()
        else:
            self._send(404, {"error": "unknown"})

    def do_GET(self):
        if self.path != "/stats":
            self._send(404, {"error": "unknown"})
            return
        spent = sum(job.get("tokens_spent", 0) for job in jobs.values())
        wasted = sum(job.get("tokens_wasted", 0) for job in jobs.values())
        self._send(200, {
            "held": held_count(),
            "projected_wait_s": project_wait(),
            "slack_s": SLACK_S,
            "tokens_spent": spent,
            "tokens_wasted": wasted,
        })

    def _admit(self, body):
        key = body.get("idempotency_key") or str(uuid.uuid4())
        tokens = int(body.get("tokens", 0))
        attempt = int(body.get("attempt", 0))
        existing = jobs.get(key)
        if existing and existing["state"] in ("held", "done"):
            self._send(409, {
                "action": "reject_duplicate",
                "idempotency_key": key,
                "state": existing["state"],
                "tokens_held": existing["tokens"],
            })
            return
        if attempt > RETRY_CAP:
            self._send(429, {
                "action": "reject_retry_cap",
                "attempt": attempt,
                "retry_cap": RETRY_CAP,
            })
            return
        wait_s = project_wait()
        if wait_s > SLACK_S:
            jobs[key] = {
                "idempotency_key": key,
                "tokens": tokens,
                "attempt": attempt,
                "enqueued_at": now(),
                "state": "rejected",
                "tokens_spent": 0,
                "tokens_wasted": 0,
            }
            self._send(429, {
                "action": "reject_unreserved",
                "projected_wait_s": wait_s,
                "slack_s": SLACK_S,
                "held": held_count(),
                "reason": "projected_wait_exceeds_slack",
            })
            return
        holds[key] = {"tokens": tokens, "expires_at": now() + HOLD_TTL_S}
        jobs[key] = {
            "idempotency_key": key,
            "tokens": tokens,
            "attempt": attempt,
            "enqueued_at": now(),
            "state": "held",
            "tokens_spent": 0,
            "tokens_wasted": 0,
        }
        self._send(202, {
            "action": "reserve",
            "idempotency_key": key,
            "tokens_held": tokens,
            "projected_wait_s": wait_s,
            "hold_ttl_s": HOLD_TTL_S,
        })

    def _complete(self, body):
        key = body["idempotency_key"]
        job = jobs.get(key)
        if not job or job["state"] != "held":
            self._send(404, {"action": "missing_hold", "idempotency_key": key})
            return
        if holds.get(key, {}).get("expires_at", 0) < now():
            job["state"] = "expired"
            job["tokens_wasted"] = job["tokens"]
            holds.pop(key, None)
            self._send(409, {"action": "hold_expired", "idempotency_key": key})
            return
        spent = holds.pop(key)["tokens"]
        job["state"] = "done"
        job["tokens_spent"] = spent
        self._send(200, {
            "action": "commit",
            "idempotency_key": key,
            "tokens_spent": spent,
            "held_remaining": held_count(),
        })

    def _expire(self):
        released = 0
        wasted = 0
        stamp = now()
        for key, hold in list(holds.items()):
            if hold["expires_at"] <= stamp:
                job = jobs[key]
                job["state"] = "expired"
                job["tokens_wasted"] = hold["tokens"]
                wasted += hold["tokens"]
                holds.pop(key, None)
                released += 1
        self._send(200, {
            "action": "expire",
            "released": released,
            "tokens_wasted": wasted,
        })


def main():
    server = ThreadingHTTPServer(("127.0.0.1", 8080), Gate)
    server.serve_forever()


if __name__ == "__main__":
    main()

Run the gate in one local shell first.
Keep a second shell for the drill commands.
Do not point this process at production traffic.

Inject the late slot you already felt

Start the gate, then send three scripted calls.
The first call should reserve inside the slack.
The second call replays the same idempotency key.

The third call should find two holds already waiting.
Each hold stands in for twelve seconds of slot time.
Two holds project thirty six seconds, which misses slack.

python3 gate.py

curl -s -X POST http://127.0.0.1:8080/admit -H 'Content-Type: application/json' -d '{"idempotency_key":"job-17","tokens":800,"attempt":0}'
curl -s -X POST http://127.0.0.1:8080/admit -H 'Content-Type: application/json' -d '{"idempotency_key":"job-18","tokens":800,"attempt":0}'
curl -s -X POST http://127.0.0.1:8080/admit -H 'Content-Type: application/json' -d '{"idempotency_key":"job-17","tokens":800,"attempt":1}'
curl -s -X POST http://127.0.0.1:8080/admit -H 'Content-Type: application/json' -d '{"idempotency_key":"job-19","tokens":800,"attempt":0}'
curl -s http://127.0.0.1:8080/stats

Expected output for the first call if delay stays at twelve seconds.
Treat the numbers as labeled output from this declared delay.

{"action":"reserve","idempotency_key":"job-17","tokens_held":800,"projected_wait_s":12.0,"hold_ttl_s":20.0}

Expected output for the replay of job seventeen is a duplicate reject.
The hold must stay at eight hundred tokens.

{"action":"reject_duplicate","idempotency_key":"job-17","state":"held","tokens_held":800}

Expected output for job nineteen is a slack reject.
The held count in that body should still read two.

{"action":"reject_unreserved","projected_wait_s":36.0,"slack_s":30.0,"held":2,"reason":"projected_wait_exceeds_slack"}

Label that body expected, not a vendor benchmark.
Your clock and JSON spacing can shift tiny fields.
The action code is the field you should trust.

Watch fields that separate time from tokens

Emit these fields on every admission decision you make.
Skip vanity charts until these counters exist in logs.

Record action for reserve, reject, commit, or expiry.
Record projected_wait_s before you touch the server.
Record slack_s as the promise you still owe.
Record tokens_held apart from tokens already spent.
Record tokens_wasted when a hold expires unused.
Record the idempotency_key so retries cannot hide spend.

Alert when reject rate climbs while CPU stays idle.
That pairing means the free slot is late, not saturated.
Page on queue age over slack, not on token price alone.

A small spend check

Save stats JSON, then compute waste with jq.
This check is a drill helper, not a bill.
It shows amplification before finance sees a surprise.

curl -s http://127.0.0.1:8080/stats | tee stats.json
jq '{amplification: ((.tokens_spent + .tokens_wasted) / (if .tokens_spent == 0 then 1 else .tokens_spent end))}' stats.json

Expected shape after one commit and no expiry is amplification near one.
If expiry runs first, wasted tokens lift that ratio.
A ratio above one means retries or dead holds.

Stop the client loop before you raise the cap.
Commit the useful hold before you call expire.
Otherwise the ratio blames a cleanup you chose.

Decide when the free path is the wrong bet

Use free capacity only for work that can wait.
Shed it when any line below is true.

The projected wait is greater than remaining slack.
A replay would reserve the same tokens again.
Hold TTL is shorter than the free-slot delay.
You cannot name the neighbor jobs ahead of you.
The caller will retry without the same key.

Interactive work should not sit on an unbounded free slot.
Batch work can wait if the ledger still has room.
A free token grant does not create a latency SLO.

Idle CPU is not evidence that your turn will start.
Unknown neighbors can consume the slot you counted on.
That is why a shared free server is a lab bet.

Decision table you can paste into the runbook

Signal	Below threshold	Action
Projected wait versus slack	Wait is under thirty seconds	Reserve and call
Same key seen again	State is held or done	Reject duplicate
Hold age versus TTL	Hold is past twenty seconds	Expire and waste
CPU versus queue age	CPU idle, age over slack	Shed, do not scale on CPU

Read the row before you add capacity.
Scaling an idle slot does not shrink queue age.
Shedding the late job does shrink that age.

Where a free assistant fits this drill

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source coding assistant you can run in this drill.
The operator states free model access and a free server option.

The operator note states an allowance of ten million tokens.
Treat that allowance as a grant, not a reserved capacity contract.
Terms can change before you plan a quarter on them.

This draft does not claim a measured quota, duration, or hardware shape.
Use the free server to rehearse this gate.
Do not use it as the only path for a deadline.

If the free slot stalls, your ledger should already have said no.
Offers change, so re-read the terms on the day you drill.
Do not paste a screenshot of idle CPU into the budget review.

Bring the reject reason and the projected wait instead.
Those two fields survive a model swap.
A chart of idle CPU will not.

Fail, clean up, and roll back

When the gate rejects, do not blind-retry on the client.
Return the 429 to the caller with the reason code.
Keep the idempotency key so a later retry stays single-spend.

Expire holds on a timer so a crash cannot leak budget.
Call the expire path before you trust stats after a stall.
A leaked hold looks like load and spends nothing useful.

Follow these rollback steps after every local drill.
Do not skip the closed-port check after you stop.

Stop gate.py with Ctrl-C in the drill shell.
Drop the in-memory ledger by exiting the process.
Set SERVICE_DELAY_S back to the declared twelve seconds.
Confirm port 8080 is closed before the next run.
Do not leave a retry loop pointed at a shared endpoint.

If you later front a real queue, ship the gate beside it.
Feature-flag admission so you can fail open only for batch.
Fail closed for interactive work when slack is already gone.

Rollback is the flag, not a model swap at 02:14.
Practice the flag flip in staging before you need it.
A runbook line you have not drilled is a wish.

Who should not use this gate

Skip this gate if your jobs have no deadline.
Skip it if every call is unique and never retried.
Skip it if you already reserve capacity with a hard quota.

Skip it if you need multi-node fairness this week.
This ledger is local and dies with the process.
A restart forgets holds unless you add durable storage.

Do not use it to hide spend from finance.
Do not use it to bypass provider rate limits.
Do not treat expected JSON as a production SLO.

A single process will not save a region-wide stall.
Clock skew between shells can fake a late hold.
Declare one clock source before you trust expiry.

Close on the action, not the model

You saw idle CPU and a missed deadline together.
The action is reject unreserved work, not add retries.
Queue age versus slack is the threshold to write down.

Token holds stop a replay from spending the grant twice.
Free capacity stays a lab bet until that gate holds.
Confirm current free-server terms, then run this laptop drill.

Shed Stale Compaction Before Queue Age Misses the Turn

Odd_Background_328 — Thu, 24 Sep 2026 18:17:52 +0000

You catch the latency page at 02:14 UTC.
Chat p95 sits past the four-second turn budget.
The serving GPU is idle, so saturation is not the cause.

A background compaction job still occupies the free path.
Its transcript version is already behind the live turn.
The caller moved on six seconds before this page.

Free-path queue age reads eleven seconds in your collector.
Remaining slack on that turn is four seconds.
Another summary cannot land before the next token is due.

Read the contradiction first

Do not open this incident with a model swap.
The idle GPU and the late queue disagree.
Idle compute does not cancel a queued generation.

You pay tokens for work the turn will never read.
Separate what you measured from what you infer.
You measured queue age, slack, and version skew.

You infer that a zero price hid a deadline miss.
That inference is not a metric.
Keep it out of the alert name.

Name the topology

Keep this drill on one laptop you control.
You need four boxes and one shared clock.

An API process owns the turn deadline.
A compaction worker reads only a versioned transcript.
A local queue can age on purpose.
Two adapters exist: a free path and a paid path.

MonkeyCode states free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Treat both as availability notes, not as latency promises.

Do not hard-code a model name from memory.
Read the live console before you bind a path.
Neither availability claim is a latency SLO you can page on.

Grants, hold times, and model lists change without your deploy.
A number copied from an old post is not a control.
If the console is silent, leave the free path dark.

Declare the workload

This is a local drill, not a production trace.
You inject queue age instead of sampling production.
Label every number below as declared drill input.

Use these conditions, and do not widen them mid-run.

One compaction job per turn, never a fan-out.
Versions increase only when the user sends text.
Estimated generation time is a constant you pass in.
Token estimate is characters divided by four, rounded up.
Retry count starts at zero and stops at two.
The token ceiling is a budget you choose.

Do not paste a vendor grant into this file.
A grant is not your ceiling unless you measured spend.
Your ceiling should be smaller than the turn can afford.

Gate the job

Admit compaction only when all three checks pass.
Reject stale work before you touch the queue.
Reject late work before you touch the free path.
Reject over-budget work before a retry multiplies tokens.

from dataclasses import dataclass

@dataclass(frozen=True)
class TurnBudget:
    slack_ms: int
    queue_age_ms: int
    est_gen_ms: int
    est_tokens: int
    retry_count: int
    token_ceiling: int
    transcript_version: int
    latest_version: int

def admit_compaction(b: TurnBudget) -> str:
    if b.transcript_version != b.latest_version:
        return "reject_stale"
    projected_ms = b.queue_age_ms + b.est_gen_ms
    if projected_ms >= b.slack_ms:
        return "reject_slack"
    spend = b.est_tokens * (b.retry_count + 1)
    if spend > b.token_ceiling:
        return "reject_tokens"
    return "admit"

The first check is a fact about versions.
The second check compares queue wait with slack.
The third check enforces a ceiling you set.

A free price does not skip any of these checks.
The paid path uses the same three checks.
Price changes the bill, not the deadline math.

Decision table

Signal you hold	Comparison	Action
Version skew	job version != latest	`reject_stale`
Time	queue age + est gen >= slack	`reject_slack`
Tokens	est tokens × (retries + 1) > ceiling	`reject_tokens`
All three clear	current, inside slack, under ceiling	`admit`

Queue age beats utilization as the shed signal.
A cool GPU can still sit behind an old queue.
Deadline slack is the budget that actually expires.

Run the drill

Save that function as admit.py in a scratch directory.
Then run the three cases declared in this section.

python - <<'PY'
from admit import TurnBudget, admit_compaction

cases = [
    TurnBudget(4000, 11000, 1500, 800, 0, 5000, 7, 9),
    TurnBudget(9000, 1200, 1500, 800, 2, 2000, 9, 9),
    TurnBudget(9000, 1200, 1500, 800, 0, 5000, 9, 9),
]
for case in cases:
    print(admit_compaction(case))
PY

The expected lines below come from this function.
They are not a vendor benchmark and not production.

reject_stale
reject_tokens
admit

Case one is stale, so age never matters.
Case two is current, but retries blow the ceiling.
Case three is current, inside slack, and under budget.

If your printout differs, stop and diff the function.
Do not tune thresholds until the labels match.
A mismatched label means the gate is not the one you reviewed.

What each input stands for

Case one mirrors the page you opened.
Slack is 4000 ms and queue age is 11000 ms.
Version 7 is behind version 9, so you shed first.

Case two is on time and on the current version.
Two retries turn 800 estimated tokens into 2400.
Your ceiling is 2000, so you stop before submit.

Case three is the only admit in this drill.
Queue age plus estimate is 2700 ms.
Slack is 9000 ms, and spend stays at 800.

Change one field and rerun before you trust a new threshold.
Write the new expected line next to the change.
If you cannot predict the label, you do not understand the gate.

When the free path is the wrong bet

Use the free path only for work that can expire.
Compaction can expire cleanly.
A user-visible answer cannot.

Shed the job when any line below is true.

Queue age plus estimated generation meets slack.
The transcript version moved after enqueue.
Retries would push estimated tokens over your ceiling.
You cannot cancel the generation after submit.
The result must return inside this same turn.
You have no fresh console reading for the free path.

A zero price is not a reason to wait.
Waiting spends wall time the caller already lost.
Retries then spend tokens on a summary nobody reads.

That is the cost failure, not a model quality debate.
You lost slack first.
You burned tokens second.

Do not failover an interactive turn onto the free server.
The free server option is still a queue with unknown age.
Unknown age is a reason to shed, not a reason to hope.

If hold time is unpublished, do not invent one.
If the token grant is unpublished, do not invent one.
Schedule only against numbers you just read.

Failure handling

On reject_stale, drop the job and ack the queue.
Do not requeue it under a new idempotency key.
A new key hides a duplicate spend in your bill.

On reject_slack, leave the last good summary in place.
Serve the current turn with the uncompacted window.
Page only if this reject rate crosses your own threshold.

Pick that threshold from missed turns, not from pride.
A starting page rule is five slack rejects in five minutes.
Replace it once you have a week of your own series.

On reject_tokens, stop retries for that version.
Emit compaction_rejected_total with the reason label attached.

Suggested fields, not a required vendor schema:

queue_age_ms
slack_ms
est_tokens
retry_count
transcript_version
reason

If you cannot cancel after submit, skip that path.
Uncancelable free work becomes silent token debt later.
Debt without a cancel handle is an ops bug.

A tiny collector check

After a reject, your log line should carry the reason.
Grep the scratch log before you call the drill done.

python - <<'PY'
print("reason=reject_slack queue_age_ms=11000 slack_ms=4000")
PY
grep -E 'reason=reject_(stale|slack|tokens)' /tmp/compaction-drill.log || true

The print is a format sample, not a caught incident.
The grep proves your eye can find the label.
If the label is missing, the gate is not operable.

Cleanup and rollback

The drill queue is local.
Drain it before you leave the shell.

# scratch only — do not point this at production
rm -f /tmp/compaction-drill.queue /tmp/compaction-drill.log
python -c "from admit import admit_compaction; print('rollback-ok')"

Roll the gate back with one config flag.

compaction:
  enabled: false
  free_path: disabled
  token_ceiling: 0
  max_retries: 0

Disabling the worker is the rollback, not a rewrite.
The chat path must keep serving without summaries.
If chat fails when compaction is off, fix that coupling first.

Re-enable only after three clean drill labels.
Then watch reject rate, queue age, and token spend together.
A green GPU graph alone is not a reopen signal.

Who should not use this gate

Skip this pattern if you lack a turn deadline.
Skip it when transcript versions are not monotonic.
Skip it if free capacity is your only generation path.

This gate sheds work.
It does not create capacity.
Teams that need every summary will see more full windows.

That is a product choice, not a bug in the check.
Do not use this drill as a capacity certificate.
One laptop queue does not prove a hosted server.

Do not publish these case numbers as latency results.
Do not treat a free server as a standby SLO.
Do not let a retry loop compact the same version twice.

What you should do next

Read queue age beside slack on the next late page.
If age already exceeds slack, reject the compaction job.
Confirm the live free-path limits in your console first.

If those limits are missing, keep the free path dark.
A missing hold time is not a zero hold time.
Point this gate at a MonkeyCode free path only after a fresh console check.

Reject Interactive Leases Before Free-Server Hold Time Beats Slack

Odd_Background_328 — Wed, 23 Sep 2026 16:04:32 +0000

02:14, two metrics disagree

Picture this on-call scene before you touch replicas.
Your pager fires during an otherwise quiet batch window.
The oldest summarizer job has waited 740 seconds already.

Your service objective still demands completion within 480 seconds.
The free pool reports only 19 percent CPU busy.
You almost add replicas to chase that age number.

That scale-out would hide the fault and waste money.
Do not scale on this metric contradiction just yet.
Ask which operational action the evidence actually supports.

Low utilization with rising age means stolen worker leases.
The scarce resource is hold time, not processor cycles.
Treat CPU as a contradiction check, not the page.

What you are actually running

You share one serving pool across two work classes.
Batch summarization carries a hard completion deadline tonight.
Interactive debug sessions carry no deadline of their own.

Use this proposed topology for the local drill.
Keep it on one box so blast radius stays tiny.

The gateway handles admission decisions and lease tracking.
The free-server process runs shared model serving locally.
The batch-worker submits deadline-bound jobs with an 8-minute SLO.
The debug-client holds notebooks, curl loops, and open streams.

Declare these test conditions before any fault injection.
Copy them into the drill script header before you run.

Run one serving replica on the shared local pool.
Fix maximum concurrent leases at two worker slots.
Keep batch compute near 25 seconds for each job.
Fix the batch deadline at 480 seconds wall clock.
Let the interactive session hold one slot for 180 seconds.
Keep interactive CPU load near five percent on purpose.
Allow three retries with a 10-second backoff budget.
Arrive six batch jobs within a 40-second window.

You will measure hold time, not CPU, as the scarce resource.
Utilization will lie while a debug client holds the slot.

Why free capacity is the wrong bet

Free model access looks cheap on a quiet dashboard.
A free server often looks idle under debug sessions.
Idle CPU does not mean idle workers on that box.

A lease occupies a slot until the client hangs up.
Interactive sessions hold connections far beyond useful compute.
Streaming responses stay open while you reread the prompt.

Retries clone the same hold without adding progress.
Queue age grows while the processor mostly sleeps.
You should treat hold time as the scarce resource now.

Queue age versus deadline slack beats utilization here.
Reject interactive work when slack cannot absorb another hold.
Do not wait for CPU to look busy first.

Time, tokens, retries, and queueing move together under a hold.
A cheap pool still spends finite tokens on duplicated retries.
Free capacity is the wrong bet when hold time stays unbounded.

Telemetry you need on the box

Export these fields from the gateway and server.
Without them you will page on vanity CPU again.

lease_hold_seconds tracks how long a slot stays checked out.
lease_class labels the holder as batch or interactive.
queue_age_seconds records the oldest waiting batch job.
deadline_slack_seconds equals deadline minus now minus remaining compute.
retry_count captures debug storms and batch copies together.
token_in and token_out stay finite even when unmarked free.
cpu_busy_ratio exists only for contradiction checks on-call.

Keep these PromQL sketches labeled as unexecuted examples.
Do not paste them into production without a recording rule.

deadline_slack_seconds
  = batch_deadline_seconds
  - queue_age_seconds
  - expected_compute_seconds

reject_interactive
  = deadline_slack_seconds < 90
    and lease_hold_seconds{class="interactive"} > 60

If slack falls under 90 seconds, reject new interactive leases.
You should not wait for CPU saturation as confirmation.

Local fault-injection drill

Run this drill on your laptop before touching production.
The script occupies one lease with almost no CPU.
It then watches batch age while the slot stays held.

# lease_hold_drill.py — proposed local drill, not production telemetry
import json
import os
from collections import deque
from dataclasses import dataclass, field

BATCH_DEADLINE_S = 480
MAX_LEASES = 2
INTERACTIVE_HOLD_S = 180
BATCH_COMPUTE_S = 25
SLACK_REJECT_S = 90


@dataclass
class Lease:
    cls: str
    t0: float
    job_id: str


@dataclass
class Pool:
    leases: list = field(default_factory=list)
    batch_q: deque = field(default_factory=deque)
    log: list = field(default_factory=list)

    def slack(self, now):
        if not self.batch_q:
            return BATCH_DEADLINE_S
        age = now - self.batch_q[0][1]
        return BATCH_DEADLINE_S - age - BATCH_COMPUTE_S

    def admit(self, cls, job_id, now, reject_interactive):
        slack = self.slack(now)
        if cls == 'interactive' and reject_interactive and slack < SLACK_REJECT_S:
            self.log.append({
                't': now, 'event': 'reject_interactive',
                'job': job_id, 'slack': round(slack, 1),
            })
            return False
        if len(self.leases) >= MAX_LEASES:
            if cls == 'batch':
                self.batch_q.append((job_id, now))
                self.log.append({
                    't': now, 'event': 'enqueue_batch',
                    'job': job_id, 'q': len(self.batch_q),
                })
            else:
                self.log.append({
                    't': now, 'event': 'reject_interactive_full',
                    'job': job_id,
                })
            return False
        self.leases.append(Lease(cls, now, job_id))
        self.log.append({
            't': now, 'event': 'lease', 'cls': cls,
            'job': job_id, 'hold_slots': len(self.leases),
        })
        return True

    def release(self, job_id, now):
        self.leases = [lease for lease in self.leases if lease.job_id != job_id]
        while self.batch_q and len(self.leases) < MAX_LEASES:
            jid, enq = self.batch_q.popleft()
            age = now - enq
            self.leases.append(Lease('batch', now, jid))
            self.log.append({
                't': now, 'event': 'dequeue_batch', 'job': jid,
                'age': round(age, 1),
                'slack': round(BATCH_DEADLINE_S - age - BATCH_COMPUTE_S, 1),
            })


def run_drill(reject_interactive=True):
    pool = Pool()
    pending_batch = []
    arrivals = {5, 10, 15, 22, 30, 40}
    for t in range(0, 220):
        now = float(t)
        if t == 0:
            pool.admit('interactive', 'dbg-1', now, reject_interactive)
        if t == 8 and not reject_interactive:
            pool.admit('interactive', 'dbg-2', now, reject_interactive)
        if t in arrivals:
            ok = pool.admit('batch', f'b-{t}', now, reject_interactive)
            if ok:
                pending_batch.append((f'b-{t}', now + BATCH_COMPUTE_S))
        finished = [job for job, end in pending_batch if end <= now]
        pending_batch = [(job, end) for job, end in pending_batch if end > now]
        for job in finished:
            pool.release(job, now)
        if t == INTERACTIVE_HOLD_S:
            pool.release('dbg-1', now)
        if t == INTERACTIVE_HOLD_S + 5 and not reject_interactive:
            pool.release('dbg-2', now)
    return pool.log


if __name__ == '__main__':
    mode = os.environ.get('DRILL_MODE', 'reject')
    log = run_drill(reject_interactive=(mode == 'reject'))
    print(json.dumps(log, indent=2))

Run both admission modes and compare queue age.
Save the logs so you can diff hold time later.

python3 lease_hold_drill.py > /tmp/reject.json
DRILL_MODE=admit python3 lease_hold_drill.py > /tmp/admit.json

python3 - <<'PY'
import json
for name in ('reject', 'admit'):
    rows = json.load(open('/tmp/%s.json' % name))
    ages = [row['age'] for row in rows if row.get('event') == 'dequeue_batch']
    rejects = sum(1 for row in rows if row['event'].startswith('reject'))
    print(name, 'dequeued', len(ages), 'max_age', max(ages, default=0),
          'rejects', rejects)
PY

Expected output, clearly labeled unexecuted

On a quiet laptop the shape should look like this.
Treat the block as a target shape, not a benchmark claim.

reject dequeued 6 max_age 25.0 rejects 1
admit  dequeued 6 max_age 180.0 rejects 0

Your exact numbers will move with scheduler timing.
The shape should not move if the leak is real.
A 180-second hold on two slots starves batch slack.

Slack-based admission keeps max age near compute time.
The admit mode spends hold time, retries, and tokens together.
That is the cost story hiding under a calm CPU chart.

Decision table: keep free capacity or reject

Use this table during the 02:14 argument.
Pick one primary signal before you debate replicas.

Signal	Threshold	Action	Rationale
Interactive lease while slack is under 90s	90s slack	Reject debug	Deadline work owns the slots
CPU under 0.3 and queue age over 60s	60s age	Do not scale	This is a hold leak
Interactive retry count over 2	2 retries	Shed the client	Retries clone hold time
Tokens grow with no batch progress	Any growth	Kill the lease	Free tokens are still finite
Slack over 180s and a slot free	180s slack	Admit debug	Spare slack is the only window

Ask which threshold you would defend at 02:14.
Choose among queue age, utilization, and deadline slack.
Pick deadline slack as the primary reject signal.

Utilization lags and flatters a quietly held worker.
Queue age without remaining compute still lies to you.
Slack folds remaining compute into the same reject line.

Where a free isolated box still helps

You still need a place to run the drill.
Production serving is the wrong blast radius tonight.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode provides free model access and a free server option.
Use that isolation for the lease drill described above.

Do not park deadline-bound production batch on that pool.
A free shared pool is a lab, not spare production capacity.
Treat it as a blast-radius box you can throw away.

Inject holds, watch slack, then destroy the box.
If you already isolate drills on a free server, keep both signals.
Place deadline_slack_seconds beside lease_hold_seconds on that board.

Failure handling

Inject these faults on the laptop only.
Stop if any step escapes the local process namespace.

Hold one streaming client open for 180 seconds.
Retry the same debug prompt three times on timeout.
Start six batch jobs during that interactive hold.
Watch deadline slack cross the 90-second line.
Confirm new interactive leases return 429 with reason.

Use this proposed gateway body for the reject path.
Keep the reason field stable so clients can fail closed.

{
  "error": "interactive_lease_rejected",
  "reason": "deadline_slack_seconds below 90",
  "queue_age_seconds": 410,
  "retry_after_seconds": 120
}

Do not convert that 429 into a client retry loop.
Debug clients must fail closed on admission rejects.
Batch clients may retry with jitter and a budget.

Rollback and cleanup

Tear the drill down before you walk away.
Leftover leases are how tomorrow's 02:14 page is born.

kill %1 2>/dev/null || true
curl -sf -X POST localhost:8080/admin/drain || true
rm -f /tmp/reject.json /tmp/admit.json

Rollback if admission shipped wider than you intended.
Keep metrics hot even while the reject flag is off.

Flip the reject_interactive feature flag to off.
Keep emitting slack metrics so you are not blind.
Cap interactive leases at one until you re-enable.
Page on slack, never on CPU busy ratio alone.

Leave the pool empty after the last batch drain.
Do not leave the notebook holding a production-like slot.

Who should not use this approach

Skip this pattern if you have dedicated debug hardware.
Skip it when every job already has a hard lease timeout.
Skip it if you do not hold a batch deadline at all.

Do not mix paid production traffic onto a free shared server.
Noisy neighbors and preemptible capacity will break the story.
Free capacity is the wrong bet under unbounded hold time.

Limitations

The script is a discrete-time toy, not a load test.
It does not model tokenizer stalls or network jitter.
It does not name models, quotas, or hardware SKUs.

Your real threshold will differ from ninety seconds.
Treat 90 seconds as a starting argument, not law.
Measure expected_compute_seconds from traces, never from hope.

Token counters still matter when the invoice reads zero.
Retries spend both tokens and hold time together.
Count both before you call the pool cheap.

What you do at 02:14

You do not scale the quiet pool under this alert.
You kill the debug lease that stole the slot.
You reject new interactive work until slack recovers.

File hold time as the cause, not low CPU.
Rerun the local drill after the reject flag stays on.
Admit debug sessions only when slack is honestly large.

Which page threshold will you defend tomorrow morning?
Will it be slack, age, or utilization under lease contention?

Reject Tool-Call Fanout Before Token Burn Beats Slack

Odd_Background_328 — Tue, 22 Sep 2026 14:28:33 +0000

You get paged at 02:14 UTC tonight.
Queue age already sits at 47 seconds.
Interactive CPU still looks almost idle.

You open the token meter next.
Spend is climbing at 3x the planned rate.
Deadline slack on the batch window is 90 seconds.

Which operational action follows that evidence?
Scale the free worker, or reject the fanout class?
This note argues for reject, then prove it locally.

The contradiction you should not ignore

Queue age is high. Utilization is low.
That pairing is not a capacity shortage.
It is usually a work-shape problem under metering.

Tool-calling jobs hide nested HTTP work.
Each nested call can enqueue sibling calls.
Retries then multiply tokens without moving the head job.

Free capacity makes that shape look cheap.
Wall-clock slack still expires on the original deadline.
Tokens still leave the shared pool either way.

Topology for the local drill

Keep the lab topology tiny and explicit.

admitd: admission proxy with class rules
workq: FIFO queue with age gauges
worker: one process, one in-flight job
meter: token and retry counters
deadline: remaining slack per job class

Do not share this worker with interactive traffic.
Label every number below as a lab expectation.
Do not treat it as a production observation.

Proposed config, saved as admitd.yaml:

classes:
  tool_fanout:
    max_depth: 1
    max_siblings: 2
    reject_when:
      queue_age_ms: 15000
      slack_ms_below: 60000
      tokens_per_success_est: 8000
  interactive:
    max_depth: 0
    reserved: true
queue:
  visibility_timeout_s: 30
  max_inflight: 1
meter:
  window_s: 60
rollback:
  drain_timeout_s: 45
  fail_closed: true

Declared lab workload:

20 parent jobs, each with two tool calls.
Each tool call may spawn one retry on 429.
Parent deadline is 120 seconds after enqueue.
Worker concurrency stays at one.
No other publishers touch workq.

Start from the alert, not the model card

You should bind alerts to evidence fields.
Do not page on “tokens used” alone.
Page on slack versus age versus burn together.

Proposed telemetry fields:

queue_age_ms
deadline_slack_ms
tokens_in_window
retries_in_window
tool_depth
class
admit_decision

Expected lab log line, clearly labeled expected:

# expected lab output, not a live incident
ts=02:14:07 class=tool_fanout queue_age_ms=47012
deadline_slack_ms=90000 tokens_in_window=24110
retries_in_window=14 tool_depth=2 admit_decision=reject
reason=token_burn_vs_slack

If CPU is idle while that line prints, you do not scale.
You reject the class that fans out.
Then you drain what already entered.

Artifact: a fail-closed admission check

The checker below is a local, unexecuted example.
Run it against recorded metrics, not guesswork.
Keep it fail-closed when slack is unknown.

from dataclasses import dataclass

@dataclass
class Sample:
    queue_age_ms: int
    deadline_slack_ms: int
    tokens_in_window: int
    successes_in_window: int
    tool_depth: int
    retries_in_window: int

class Reject(Exception):
    pass


def tokens_per_success(sample: Sample) -> float:
    if sample.successes_in_window <= 0:
        return float("inf")
    return sample.tokens_in_window / sample.successes_in_window


def admit_tool_fanout(sample: Sample, cfg: dict) -> str:
    rules = cfg["classes"]["tool_fanout"]["reject_when"]
    if sample.deadline_slack_ms < rules["slack_ms_below"]:
        raise Reject("slack_below_floor")
    if sample.queue_age_ms > rules["queue_age_ms"]:
        raise Reject("queue_age_over_limit")
    if sample.tool_depth > cfg["classes"]["tool_fanout"]["max_depth"]:
        raise Reject("tool_depth_over_limit")
    est = tokens_per_success(sample)
    if est > rules["tokens_per_success_est"]:
        raise Reject("token_burn_vs_success")
    if sample.retries_in_window >= 10 and sample.successes_in_window == 0:
        raise Reject("retry_amplification")
    return "admit"

Wire a one-shot probe. Do not loop this in production yet.

python3 - <<'PY'
from admit_check import Sample, admit_tool_fanout, Reject
import yaml
cfg = yaml.safe_load(open("admitd.yaml"))
sample = Sample(
    queue_age_ms=47012,
    deadline_slack_ms=90000,
    tokens_in_window=24110,
    successes_in_window=1,
    tool_depth=2,
    retries_in_window=14,
)
try:
    print(admit_tool_fanout(sample, cfg))
except Reject as e:
    print("reject", e)
PY

Expected lab output:

reject tool_depth_over_limit

You now have a binary operational action.
Reject beats adding another free worker here.
The extra worker would only clone the fanout.

When free capacity is the wrong bet

Free model access can still be useful.
A free server option can isolate the drill.
Neither one repairs a bad work shape.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
If you need an isolated box for the same admission drill, MonkeyCode's free model access and free server option can host the worker without mixing it into paid interactive traffic.

Use this decision table before you bet on free capacity.

Evidence	Free capacity bet	Operational action
Queue age high, CPU idle, depth=1	Maybe	Keep one worker, watch slack
Depth >= 2, retries rising	Wrong	Reject class, fail closed
Tokens/success unknown	Wrong	Reject until meter exists
Slack < 60s, age > 15s	Wrong	Drain, do not enqueue more
Interactive class shares the pool	Wrong	Split queues before any scale
Successes move, retries stay flat	Acceptable	Keep free worker, cap depth

Ask for one threshold and one rationale.
Prefer deadline slack over utilization.
Utilization lies when nested calls wait on I/O.

Rationale you can defend in a review:

Queue age over 15s with one inflight job means head-of-line delay.
Slack under 60s leaves no drain window.
Tokens per success over 8k, in this lab, means nested waste.
Retry count without successes means amplification, not load.

Change those numbers only after a declared rerun.
Do not copy them into production blindly.

Failure injection you can run locally

Inject one fault at a time.
Record age, slack, tokens, and retries after each.
Stop if slack crosses the floor.

# 1) nested fanout
python3 inject_fanout.py --parents 20 --depth 2 --siblings 2

# 2) retry storm on 429
python3 inject_fanout.py --parents 20 --depth 1 --force-429 8

# 3) slow tool, no extra tokens
python3 inject_fanout.py --parents 5 --depth 1 --tool-sleep-ms 4000

# 4) meter blackout
python3 inject_fanout.py --parents 5 --drop-meter

Expected lab pattern, labeled expected:

Case 1: reject on tool_depth_over_limit.
Case 2: reject on retry_amplification.
Case 3: admit if age stays under 15s.
Case 4: reject because tokens per success is infinite.

Case 3 is the trap.
Slow tools raise age without burning tokens.
You still reject if age beats slack, not if tokens look fine.

Drain, rollback, cleanup

Rejection without drain still burns the in-flight parent.
Practice the rollback on the lab queue.

# stop publishers for tool_fanout only
curl -X POST localhost:8080/classes/tool_fanout/reject

# wait for inflight, then snapshot meters
sleep 45
curl -s localhost:8080/metrics > /tmp/meter.after-reject

# fail closed if drain exceeds timeout
if [ "$(curl -s localhost:8080/queue/age_ms)" -gt 15000 ]; then
  curl -X POST localhost:8080/queue/purge?class=tool_fanout
fi

# restore only after slack recovers
curl -X POST localhost:8080/classes/tool_fanout/enable \
  --data 'max_depth=1&max_siblings=2'

Cleanup the lab files after the drill.

rm -f /tmp/meter.after-reject
rm -rf ./workq-data ./meter-wal

If you pointed the worker at a free server, tear that worker down too.
Do not leave a preemptible box holding visibility locks.
Stale locks look like queue age the next morning.

What this drill does not claim

This is an admission workflow, not a model benchmark.
It does not name models, quotas, or hardware SKUs.
It does not claim free capacity is faster or slower.

It also does not replace tracing of each tool HTTP call.
You still need span ids on nested calls.
Without spans, depth is only an inference.

Separate observed fields from architectural guesses.
queue_age_ms is observed.
“The model is slow” is not observed here.

Who should not use this approach

Skip this reject path if you lack a token meter.
Skip it if parents have unknown tool graphs.
Skip it for user-facing streaming with sub-second SLOs.

Do not use a shared free worker for mixed classes.
Interactive work will inherit batch fanout delays.
Paid isolation is cheaper than a missed deadline then.

Do not tune thresholds during an active incident.
Run the injectors on a laptop first.
Promote the rule only after two clean lab passes.

The action that follows the 02:14 evidence

You had high age, idle CPU, and rising tokens.
That is reject-and-drain, not scale-on-free.
Free capacity is the wrong bet once fanout starts.

Keep depth at one on cheap workers.
Keep a slack floor of 60 seconds in this lab.
Keep retries from counting as progress.

Then rerun the four injectors after every parser change.
Tool schemas change more often than queue code.
Your reject rule must track the schema, not the slogan.