DEV Community: Odd_Background_328

Reject Job 1 Before It Eats Job 2,000's Token Budget

Odd_Background_328 — Mon, 09 Nov 2026 20:26:30 +0000

Illustrative scenario, not a measurement. You queue 2,000 batch jobs against one free pool. The pool holds 10M tokens by your own accounting. That averages 5,000 tokens per job.

Averages do not admit work. Averages hide the tail that actually drains the pool.

Here is the arithmetic. Suppose the heaviest job class needs 60,000 tokens. Ten million divided by 60,000 is about 166 jobs. If those 166 arrive first, jobs 167 through 2,000 fail at the wall, not at the deadline.

The problem is accounting, not capacity

A global balance is one shared counter. Every job can spend it. Nothing reserves it.

That gives you three failure modes you will meet in order:

Head-of-line drain. Early heavy jobs consume the pool before the tail is admitted.
Retry amplification. A failed job re-spends its prompt while its first hold is still open.
Silent overspend. A job that exceeds its estimate has no ceiling, so it borrows from the future.

All three are bookkeeping bugs. None require more capacity to fix.

Topology: admit, hold, settle, reclaim

You need four moving parts, not one queue:

An admission step that reserves a per-job envelope before dispatch.
A ledger that records holds, so the balance reflects committed spend.
A settle step that replaces the hold with actual usage once per job.
A reclaimer that releases holds whose worker disappeared.

A free server option is a reasonable place to run this control plane. A free model endpoint is a reasonable place to run the jobs. Both are operator-supplied availability claims, and the 10M-token pool figure came from the operator as well. Treat every quota and free-tier term as time-sensitive and check current official terms before you size anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The rest of this article works with any pool that exposes a usage export you can reconcile against.

Reserve at admission, atomically

The reserve step must be a single conditional update. Read-then-write lets two workers hold the same tokens.

-- ledger.sqlite3, illustrative schema
CREATE TABLE IF NOT EXISTS pool (
  name    TEXT PRIMARY KEY,
  balance INTEGER NOT NULL          -- remaining tokens
);
CREATE TABLE IF NOT EXISTS reservation (
  id               TEXT PRIMARY KEY,
  pool             TEXT NOT NULL,
  job              TEXT NOT NULL,
  amount           INTEGER NOT NULL,  -- envelope held
  state            TEXT NOT NULL,     -- held | settled | released
  lease_expires_at REAL NOT NULL,
  settled_amount   INTEGER
);

# Illustrative. Not executed here. Python 3.11 + stdlib sqlite3.
import sqlite3, time, uuid

def reserve(conn, pool, job, amount, lease_s=900):
    rid = str(uuid.uuid4())
    with conn:  # BEGIN ... COMMIT
        cur = conn.execute(
            "UPDATE pool SET balance = balance - ? "
            "WHERE name = ? AND balance >= ?",
            (amount, pool, amount))
        if cur.rowcount != 1:
            return None            # refuse early, do not dispatch
        conn.execute(
            "INSERT INTO reservation "
            "(id, pool, job, amount, state, lease_expires_at) "
            "VALUES (?, ?, ?, ?, 'held', ?)",
            (rid, pool, job, amount, time.time() + lease_s))
    return rid

Two properties matter here. The WHERE balance >= ? clause makes the check and the debit one transaction. A refused reservation returns None, and the job goes back to the queue undispatchable.

That refusal is the point. You rejected work at admission instead of failing it mid-flight.

Settle once, clamp the overspend

Actual usage rarely equals the envelope. Settle must be idempotent, and it must not create balance out of nothing.

# Illustrative.
def settle(conn, rid, actual):
    with conn:
        row = conn.execute(
            "SELECT pool, amount, state FROM reservation WHERE id = ?",
            (rid,)).fetchone()
        if row is None or row[2] != "held":
            return 0                     # already settled or reclaimed
        pool, held, _ = row
        refund = held - actual
        if refund < 0:
            refund = 0                   # clamp: never mint tokens
            # emit overspend alert with rid, held, actual
        conn.execute(
            "UPDATE pool SET balance = balance + ? WHERE name = ?",
            (refund, pool))
        conn.execute(
            "UPDATE reservation SET state = 'settled', settled_amount = ? "
            "WHERE id = ?", (actual, rid))
    return refund

Clamping is a deliberate trade. The local balance stays conservative. The overspend surfaces as an alert, not as a negative pool that silently blocks every later job.

Reclaim stale holds

Workers crash. Leases expire. A hold with no owner is stranded budget.

# Illustrative.
def reclaim(conn, now=None):
    now = now or time.time()
    with conn:
        rows = conn.execute(
            "SELECT id, pool, amount FROM reservation "
            "WHERE state = 'held' AND lease_expires_at < ?", (now,)).fetchall()
        for rid, pool, amount in rows:
            conn.execute(
                "UPDATE pool SET balance = balance + ? WHERE name = ?",
                (amount, pool))
            conn.execute(
                "UPDATE reservation SET state = 'released' WHERE id = ?", (rid,))
    return len(rows)

Set the lease longer than your p99 job runtime plus one retry. A short lease double-spends, because the provider may still bill the work you just released.

Derive the envelope from the tail

Do not size the envelope on the mean. Size it on p95 input, allowed attempts, and fixed overhead.

envelope = ceil(p95_input * fanout) + p95_output * attempts_allowed + overhead

Job class	p95 input	Attempts	Envelope	Jobs the 10M pool admits
summarize	6,000	2	14,000	~714
extract-json	18,000	2	40,000	250
agent-loop	55,000	3	180,000	~55

Numbers in that table are illustrative arithmetic, not measurements. Recompute them against your own usage export.

Then cap concurrency, not just balance: max_in_flight = floor(balance / envelope). That ceiling is a safety valve, not a schedule. Pair it with deadline slack so long jobs do not sit in the queue for hours holding reservations.

Trip conditions and why

Watch four signals. Each maps to one action.

Reservation refusal rate above 5% over 10 minutes. The pool is undersized for the admitted mix. Action: shrink envelopes or lower concurrency.
Unspent reserve ratio above 40%. Envelopes are too fat, and you are refusing work you could run. Action: re-derive from p95.
Overspend count above zero. The envelope is wrong for one class. Action: alert per job class, fix that class only.
Queue age above half the deadline slack. Admission is not the bottleneck anymore. Action: stop admitting and shed load.

Thresholds above are starting points, not universal truth. Pick a refusal rate you can defend to the team that owns the deadline.

Where this ledger lies

A local ledger is a proxy for spend, not the spend itself.

Reclaim is a guess. Released tokens may already have been billed upstream.
Refunds require a usage export. Without one, actual is your own estimate.
Ephemeral disks lose the ledger. If your free server recycles its filesystem, keep the SQLite file on a persistent volume, or accept that the pool resets.
One pool, no fairness. Two teams sharing one counter will starve each other. Add per-tenant budgets before you share it.

If you need hard provider-side spend guarantees, or you cannot export usage, this design is the wrong tool. Use provider-side budget limits instead.

When free capacity is the wrong bet

Situation	Free pool?	Reason
Nightly batch with hours of slack	Yes	Refusal is cheap, requeue is free
Interactive requests under 2s	No	Retry and queueing cost exceeds the savings
Jobs with heavy retries	Only with envelopes	Retries re-spend the same prompt
Deadline inside one hour	No	Head-of-line drain is unrecoverable
Long-running stateful agents	No	Lease expiry double-spends

The pattern is consistent. Free capacity is a bet on slack. If your deadline slack is thin, the savings are borrowed from your incident budget.

Rollout, cleanup, rollback

Seed the pool row with a fraction of the real quota, such as 10%.
Replay a recorded job list in dry-run mode, writing holds but never dispatching.
Reconcile sum(settled_amount) against your usage export. Investigate any gap before dispatch.
Raise the seeded balance in steps and watch the refusal rate.

Rollback is one statement: stop dispatch, call reclaim until it returns zero, then delete the reservations with state = 'held'. Never delete settled rows mid-incident, because they are your audit trail.

Cleanup after the test run is equally short. Drop the test database, keep one export of the reconciliation diff, and record the envelope table in your runbook.

Start with the arithmetic

Before you point anything at production traffic, write down three numbers. Your pool size, your p95 envelope, and your deadline slack.

If those three do not leave room for a refused job to be retried, the pool is not free. It is a delayed bill against your on-call rotation.

When you do want a cheap place to run the admission drill, a free server plus a free model endpoint is enough to exercise reserve, settle, reclaim, and rollback end to end. Just verify the current quota terms yourself before you size the pool, and keep the ledger on storage that survives a host restart.

Your Free Lane Is Cheap Until Rework Tax Beats the Token Bill

Odd_Background_328 — Sun, 08 Nov 2026 19:09:14 +0000

Your Free Lane Is Cheap Until Rework Tax Beats the Token Bill

A nightly batch finished at 02:40 instead of 23:10. The token invoice still read zero. Nobody shipped anything extra that night.

That is the trap in free capacity. Price is a line item. Rework is a schedule item.

This post builds a small gate around the second one. You will get a ledger, a formula, thresholds, a failure drill, and a rollback path. All numbers below are illustrative, not measured on your workload.

The number nobody puts on the dashboard

Most teams track tokens in. Very few track repair time out. The gap is where free lanes quietly get expensive.

Four costs hide behind a zero invoice:

Queue wait, measured as submit-to-start p95.
Retry inflation, measured as attempts per run.
Repair minutes, measured per accepted change.
Rollback minutes, measured per failed batch.

You cannot gate on a cost you never log. So log the run, not the request.

Step 1: Count accepted changes, not requests

A request that returns 200 is not an output. An output that a reviewer rewrites is not a delivery.

Define three terminal states up front:

accepted — merged, no edits beyond whitespace.
repaired — merged after a human edit or a second pass.
rejected — discarded, or rolled back after deploy.

Accepted change is your unit of value. Everything else is overhead you are paying for twice.

Step 2: Write an append-only ledger

SQLite is enough here. You want one file you can query, copy, and delete.

# ledger.py — append-only run ledger, no ORM, no service
import sqlite3, time

DDL = """
CREATE TABLE IF NOT EXISTS runs (
  run_id        TEXT PRIMARY KEY,
  lane          TEXT NOT NULL,      -- free | paid | self_hosted
  job           TEXT NOT NULL,
  submitted_at  REAL NOT NULL,
  started_at    REAL,
  finished_at   REAL,
  attempts      INTEGER DEFAULT 0,
  tokens_out    INTEGER DEFAULT 0,
  status        TEXT,               -- accepted | repaired | rejected | rolled_back
  repair_min   REAL DEFAULT 0,
  rollback_min REAL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS attempts (
  run_id TEXT, attempt INTEGER, http_status INTEGER,
  started_at REAL, ended_at REAL, tokens INTEGER,
  PRIMARY KEY (run_id, attempt)
);
"""

def connect(path="cost.db"):
    con = sqlite3.connect(path)
    con.executescript(DDL)
    return con

def open_run(con, run_id, lane, job):
    con.execute(
        "INSERT OR IGNORE INTO runs (run_id, lane, job, submitted_at) VALUES (?,?,?,?)",
        (run_id, lane, job, time.time()),
    )
    con.commit()

def start_run(con, run_id):
    con.execute("UPDATE runs SET started_at=? WHERE run_id=?", (time.time(), run_id))
    con.commit()

def record_attempt(con, run_id, attempt, http_status, started, ended, tokens):
    con.execute(
        "INSERT OR REPLACE INTO attempts VALUES (?,?,?,?,?,?)",
        (run_id, attempt, http_status, started, ended, tokens),
    )
    con.execute("UPDATE runs SET attempts=? WHERE run_id=?", (attempt, run_id))
    con.commit()

def close_run(con, run_id, status, tokens_out, repair_min=0.0, rollback_min=0.0):
    con.execute(
        """UPDATE runs SET finished_at=?, status=?, tokens_out=?, repair_min=?, rollback_min=?
           WHERE run_id=?""",
        (time.time(), status, tokens_out, repair_min, rollback_min, run_id),
    )
    con.commit()

The lane in this walkthrough can run on MonkeyCode's free model access and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Allowance, limits, and terms are operator-stated and can change, so read the current project page before you size anything against them.

One rule keeps this honest: status is written by the reviewer, not the client. If the client self-reports acceptance, your rework rate is fiction.

Step 3: Compute effective cost per accepted change

Money is one term. Time is the other two.

-- effective_cost.sql
WITH base AS (
  SELECT lane,
         COUNT(*)                                        AS runs,
         SUM(CASE WHEN status='accepted' THEN 1 ELSE 0 END) AS accepted,
         SUM(CASE WHEN status IN ('repaired') THEN 1 ELSE 0 END) AS repaired,
         SUM(tokens_out)                                 AS tokens_out,
         SUM(COALESCE(finished_at, submitted_at) - submitted_at)/60.0 AS wall_min,
         SUM(repair_min + rollback_min)                  AS human_min,
         SUM(attempts)                                   AS attempts
  FROM runs GROUP BY lane
)
SELECT lane, runs, accepted, repaired,
       ROUND(1.0*repaired/NULLIF(accepted,0), 3)      AS repair_ratio,
       ROUND(1.0*attempts/NULLIF(runs,0), 2)          AS attempts_per_run,
       ROUND(wall_min/NULLIF(runs,0), 1)              AS wall_min_per_run,
       ROUND((tokens_out/1000.0 * :tok_rate
              + wall_min * :lane_rate
              + human_min * :human_rate) / NULLIF(accepted,0), 2) AS cost_per_accepted
FROM base;

Bind your own rates. A free lane sets :tok_rate and :lane_rate to zero, which is exactly why you keep :human_rate non-zero.

Illustrative shape of the output, not measured:

lane	attempts/run	repair_ratio	cost_per_accepted
free	2.4	0.42	human-dominated
paid	1.1	0.18	mixed

The lesson is not that free loses. It is that free loses on this workload, at this repair ratio, today.

Step 4: Gate admission on the ratio, not the price

Write thresholds with a reason next to each one. A threshold without a rationale is a vibe.

repair_ratio > 0.25 over two consecutive windows → stop routing new work to that lane.
attempts_per_run > 1.5 → fix the client before adding jobs.
queue_age_p95 > 0.4 * deadline_slack → shed load or split the batch.
cost_per_accepted up 30% week over week → reopen the routing decision.

Deadline slack beats utilisation as a queue signal. Utilisation can sit at 70% and still miss a 03:00 cutoff.

The gate itself is a few lines. Keep it boring and keep it out of the queue worker.

# gate.py — unexecuted sketch, wire to your own ledger query
FREE_MAX_REPAIR = 0.25
FREE_MAX_ATTEMPTS = 1.5

def choose_lane(repair_ratio, attempts_per_run, slack_ratio):
    if slack_ratio > 0.4:
        return "shed", "queue_age over 40% of slack"
    if repair_ratio > FREE_MAX_REPAIR:
        return "paid", "repair ratio above 25%"
    if attempts_per_run > FREE_MAX_ATTEMPTS:
        return "hold", "retry inflation, fix client first"
    return "free", "within all thresholds"

Read LANE_GATE from the environment so you can flip routing without a deploy. LANE_GATE=off must mean off, not "mostly off".

Step 5: Drill the retry storm before it drills you

Free capacity plus a naive retry loop equals a wall-clock bill you never see. Test the cap locally with a test double, not a real provider.

# drill.py — test double; no real model calls
import os, random, time

def call_model(prompt, fail_rate=float(os.getenv("STRESS_FAIL_RATE", "0"))):
    time.sleep(float(os.getenv("STRESS_LATENCY_MS", "0")) / 1000.0)
    if random.random() < fail_rate:
        return {"http_status": 429, "tokens": 0}
    return {"http_status": 200, "tokens": len(prompt.split())}

Run it with a bounded loop and assert the ceiling.

STRESS_FAIL_RATE=0.4 STRESS_LATENCY_MS=800 python drill.py --jobs 20 --max-attempts 3

Expected output: 20 runs, attempts_per_run <= 3, and at least one run marked rejected rather than retried forever. If attempts per run climbs toward the cap, your backoff is too shallow, not your provider too slow.

When free capacity is the wrong bet

Skip this pattern if any of these are true:

Your deadline is fixed and short, like a market-open report.
Rework is reviewed by the most expensive engineer you have.
Output feeds a customer-facing contract with a hard SLA.
You cannot write a status column because nobody reviews the output.

Free capacity is a great bet for exploratory batches, low-stakes summarisation, and load you can cancel mid-flight.

It is a bad bet wherever repair time silently outranks token price.

Rollback and cleanup

Do these in order, and rehearse once on a dry batch.

Set LANE_GATE=off and confirm the worker keeps draining old jobs.
Drain the free lane and cancel anything still queued.
Re-route in-flight work to the fallback lane at a capped concurrency.
Keep cost.db for 30 days, then delete it with the batch workspace.
Record the rollback in the ledger as rolled_back, with minutes spent.

Step 5 is the one teams skip. Without it, your next cost review has a hole exactly where the incident happened.

The takeaway

Price told you the lane was free. Repair time told you what it cost.

Log accepted changes. Gate on the ratio. Drill the retry cap. Then decide.

Start the gate on ten jobs, not a thousand. Publish your rework rate, not your token savings.

Meter the Wait, Not the Tokens: A Slot-Time Ledger for Free AI Lanes

Odd_Background_328 — Sat, 07 Nov 2026 17:51:22 +0000

09:12 — the token meter reads zero, the deadline does not

At 09:12 you enqueue 40 summarization jobs on the free lane.

At 09:14 the token meter still reads zero. Your cost dashboard looks perfect.

At 09:41 job 37 finally finishes. The batch deadline was 09:30.

That gap is not a token problem. It is a slot problem.

This is a composite drill, not a customer incident. Every number below reproduces locally with one Python file and a fixed seed, so you can argue with it instead of with me.

Token counters measure work, never cost

A counter only moves when work completes. Waiting is invisible to it.

Two slots and forty jobs means queue time dominates everything else. The real currency you spend is slot-seconds occupied, not tokens emitted.

Retries double the damage. A retry does not only spend tokens twice; it holds a concurrency slot twice.

Declared test conditions

Nothing below comes from a vendor. These are the declared inputs of a local simulation.

Concurrency: 2 slots, FIFO admission.
Base latency: 6.0 s per attempt. Declared, not measured against any provider.
Jitter: gaussian, sigma 2.0 s, floor 0.5 s per attempt.
Retry probability: 0.18 per attempt, maximum 3 attempts.
Batch size: 40 jobs, seed 112, single process, no network calls.
Output: one JSONL ledger, one summary block, plus a breaker decision.

Record attempts, not jobs

One ledger line per attempt is the minimum useful granularity. Aggregate per job later if you want.

Identity: job_id, attempt, lane.
Timing: enqueued_at, dispatched_at, finished_at.
Occupancy: slot_seconds, attempt_seconds.
Result: outcome, prompt_tokens, completion_tokens, breaker_state.

slot_seconds is the field most teams never store. It is also the only field that answers the cost question.

Run the drill locally

Save this as slot_ledger.py and run it. It needs no dependencies beyond the standard library.

#!/usr/bin/env python3
# Deterministic slot-time accounting drill for a 2-slot free lane.
import json
import random

CONCURRENCY = 2
BASE_LATENCY_S = 6.0
JITTER_SIGMA_S = 2.0
RETRY_PROB = 0.18
MAX_ATTEMPTS = 3
JOBS = 40
SEED = 112


def simulate():
    rng = random.Random(SEED)
    free_at = [0.0] * CONCURRENCY
    rows = []
    for job_id in range(JOBS):
        attempts, slot_s, first_s, wait_s, ok = 0, 0.0, 0.0, 0.0, False
        for attempt in range(1, MAX_ATTEMPTS + 1):
            slot = min(range(CONCURRENCY), key=lambda i: free_at[i])
            start = free_at[slot]
            dur = max(0.5, rng.gauss(BASE_LATENCY_S, JITTER_SIGMA_S))
            free_at[slot] = start + dur
            if attempt == 1:
                wait_s, first_s = start, dur
            attempts += 1
            slot_s += dur
            if rng.random() > RETRY_PROB:
                ok = True
                break
        rows.append({
            'job_id': job_id,
            'attempts': attempts,
            'wait_seconds': round(wait_s, 2),
            'slot_seconds': round(slot_s, 2),
            'first_attempt_seconds': round(first_s, 2),
            'ok': ok,
        })
    return rows, max(free_at)


def summarize(rows, makespan_s):
    ok = [r for r in rows if r['ok']]
    waits = sorted(r['wait_seconds'] for r in rows)
    slot_total = sum(r['slot_seconds'] for r in rows)
    first_total = sum(r['first_attempt_seconds'] for r in rows)
    return {
        'jobs': len(rows),
        'success_rate': round(len(ok) / len(rows), 3),
        'slot_seconds_total': round(slot_total, 1),
        'slot_seconds_per_success': round(slot_total / max(len(ok), 1), 2),
        'retry_slot_overhead_pct': round(100 * (slot_total / first_total - 1), 1),
        'p50_wait_seconds': waits[len(waits) // 2],
        'p90_wait_seconds': waits[int(len(waits) * 0.9)],
        'makespan_seconds': round(makespan_s, 1),
    }


if __name__ == '__main__':
    rows, makespan = simulate()
    with open('slot_ledger.jsonl', 'w', encoding='utf-8') as fh:
        for row in rows:
            fh.write(json.dumps(row) + '\n')
    metrics = summarize(rows, makespan)
    for key, value in metrics.items():
        print(f'{key:<28} {value}')

Expected output shape at seed 112

These values are illustrative for the declared inputs. Exact digits depend on your Python build, so compare shape and ratios, not the last decimal.

jobs                         40
success_rate                 0.875
slot_seconds_total           291.4
slot_seconds_per_success     8.33
retry_slot_overhead_pct      21.9
p50_wait_seconds             66.1
p90_wait_seconds             129.7
makespan_seconds             145.8

Look at p90_wait_seconds. It is 129.7 s on a lane whose per-attempt work is 6.0 s. Almost all your elapsed time went into standing in line.

Four thresholds, not one average

Averages hide the tail that misses deadlines. Gate on these four instead.

success_rate below 0.95: fix idempotency before anything else. Retrying a non-idempotent write is how you get duplicates.
slot_seconds_per_success more than 30 percent above BASE_LATENCY_S: retries are buying you nothing.
retry_slot_overhead_pct above 25: freeze new free-lane intake. Every new job lengthens everyone's wait.
p90_wait_seconds above half your deadline slack: the lane is the wrong bet for this batch.

Slice p90_wait_seconds by enqueue minute. A batch that starts fast and degrades by minute ten is a queue problem, not a model problem.

The admission gate

You cannot fix a queue you admit into blindly. Put the gate in front of the dispatcher.

HEADROOM = 0.5                  # free lane may consume at most half the slack
MAX_RETRY_SLOT_OVERHEAD = 25.0  # percent


def admit(deadline_slack_s, p90_wait_s, headroom=HEADROOM):
    return p90_wait_s <= headroom * deadline_slack_s


def lane_mode(metrics, slack_s):
    if not admit(slack_s, metrics['p90_wait_seconds']):
        return 'split'          # head to free lane, tail elsewhere
    if metrics['retry_slot_overhead_pct'] > MAX_RETRY_SLOT_OVERHEAD:
        return 'freeze'         # stop intake, drain what is queued
    return 'free'

Why half the slack? You need the other half for retries, artifact writes, and the reviewer who reads the output.

Retry amplification, expressed in slots

With a per-attempt failure rate p of 0.18, expected attempts per success are 1 / (1 - p), about 1.22. That matches the ledger overhead of roughly 22 percent.

So the formula you can defend in a review is simple: slot cost per success equals base attempt time divided by (1 - p). Raise p by making jobs larger and you pay twice, in tokens and in occupied slots.

Where the free options actually fit

Run the collector and the breaker loop on a small always-on host, and send the low-stakes classification steps through free model access. The free server option and free model access are available per the operator's own description, which is the only availability claim I am making here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the drill honest: the ledger in this article is local and simulated. Swap the simulator for your real dispatcher, keep the same four thresholds, and you have a measurement instead of an opinion.

Failure handling and rollback

Phase one, shadow: compute metrics, log lane_mode, change nothing.
Phase two, gate: enforce split only. Never start with freeze.
Phase three, freeze: reject new intake, drain the queue, alert the batch owner.
Rollback: unset the gate flag, restore the previous dispatcher image, keep the JSONL for the postmortem.
Cleanup: rotate slot_ledger.jsonl daily, drop lines older than seven days, and delete the drill file once the real ledger lands.

Use one environment variable as the kill switch. A kill switch you cannot find at 03:00 is not a kill switch.

Who should not route work this way

Teams with a hard p99 contractual SLO. Queue time is not negotiable there.
Workloads with data residency or audit-trail requirements on the compute host.
Batches with large artifact egress. Slot accounting will not save you from bandwidth.
Jobs that cannot be made idempotent. Fix the write path first, or retries will duplicate records.

If your deadline slack is under two times the observed p90 wait, free capacity is simply the wrong bet. Pay for the lane, or shrink the batch.

Start with the ledger, not the migration. If you want to run this drill end to end, the free model access and free server option are enough to collect your first forty rows.

Free Capacity Fails the Deadline Test, Not the Price Test

Odd_Background_328 — Fri, 06 Nov 2026 16:26:05 +0000

Tuesday, 02:40. Nine hundred batch jobs queued. Deadline at 09:00.

At 200 jobs, the free lane looked perfect. Zero spend, clean output.

At 900 jobs, p95 queue age hit 34 minutes. Retry attempts tripled.

Nobody changed the price. The deadline changed.

That is the trap with free capacity. You keep evaluating it on cost. It fails on time.

So stop asking "is it free?" Ask "does it still finish before the deadline?"

This post gives you a preflight check for that second question. It runs in seconds. It exits non-zero when the lane no longer fits.

The price test is easy. The deadline test is not

Free capacity is a shared resource. You get it cheap because you get it later.

That is not a bug. It is the trade. The queue absorbs spikes you did not create.

Your batch does not care about the reason. It cares about deadline slack.

Three numbers decide the lane choice:

p95 queue age — how long a job waits before a worker starts it.
Retry amplification — total attempts divided by accepted artifacts.
Deadline slack — deadline minus projected finish, in minutes.

You can measure the first two from a canary. The third is arithmetic.

Most teams measure none of them. They measure price. Then they miss the date.

Build a job event ledger first

You cannot compute slack without timestamps. Emit one JSON line per lifecycle event.

{"job_id":"j-0042","event":"queued","ts":1762480800,"lane":"free"}
{"job_id":"j-0042","event":"started","ts":1762481100}
{"job_id":"j-0042","event":"retry","attempt":2,"ts":1762481400}
{"job_id":"j-0042","event":"started","ts":1762481460}
{"job_id":"j-0042","event":"finished","status":"ok","ts":1762481820,"tokens_in":1840,"tokens_out":610}

Two rules keep this ledger honest.

Write the queued event before you submit, not after.
Log tokens_in and tokens_out on finish, never estimated.

Retries are first-class events. If you hide them, amplification disappears from your data.

The preflight script

Save this as lane_preflight.py. Standard library only. No network calls.

#!/usr/bin/env python3
"""lane_preflight.py - does the free lane still fit the deadline?"""
import argparse, json, math, sys
from statistics import quantiles


def load(path):
    with open(path) as fh:
        return [json.loads(l) for l in fh if l.strip()]


def p95(values):
    if not values:
        return 0.0
    if len(values) < 2:
        return float(values[0])
    return quantiles(values, n=20)[-1]


def roll_up(events):
    jobs = {}
    for e in events:
        j = jobs.setdefault(e["job_id"], {
            "attempts": 0, "tokens": 0, "queued": None,
            "started": [], "finished": None, "status": None,
        })
        kind = e["event"]
        if kind == "queued" and j["queued"] is None:
            j["queued"] = e["ts"]
        elif kind == "started":
            j["attempts"] += 1
            j["started"].append(e["ts"])
        elif kind == "finished":
            j["finished"] = e["ts"]
            j["status"] = e.get("status", "ok")
            j["tokens"] += e.get("tokens_in", 0) + e.get("tokens_out", 0)
    return jobs


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("events")
    ap.add_argument("--remaining", type=int, required=True)
    ap.add_argument("--concurrency", type=int, default=4)
    ap.add_argument("--deadline", type=int, required=True, help="epoch seconds")
    ap.add_argument("--now", type=int, default=None)
    ap.add_argument("--max-amp", type=float, default=1.5)
    args = ap.parse_args()

    jobs = roll_up(load(args.events))
    done = [j for j in jobs.values() if j["status"] == "ok"]
    if not done:
        sys.exit("no accepted jobs yet: run a canary first")

    queue_age = p95([j["started"][0] - j["queued"] for j in done if j["queued"]])
    duration = p95([j["finished"] - j["started"][-1] for j in done])
    attempts = sum(j["attempts"] for j in jobs.values())
    amp = attempts / len(done)
    tokens_per_artifact = sum(j["tokens"] for j in done) / len(done)

    now = args.now or max(j["finished"] for j in done)
    waves = math.ceil(args.remaining / args.concurrency)
    projected = now + queue_age + waves * duration * amp
    slack_min = (args.deadline - projected) / 60.0

    print(f"accepted jobs          : {len(done)}")
    print(f"p95 queue age (s)      : {queue_age:.0f}")
    print(f"p95 job duration (s)   : {duration:.0f}")
    print(f"retry amplification    : {amp:.2f}x")
    print(f"tokens / artifact      : {tokens_per_artifact:.0f}")
    print(f"projected finish (s)   : {projected:.0f}")
    print(f"deadline slack (min)   : {slack_min:.1f}")

    verdict = 0
    if slack_min < 0:
        print("VERDICT: free lane misses the deadline")
        verdict = 1
    if amp > args.max_amp:
        print(f"VERDICT: amplification {amp:.2f}x exceeds {args.max_amp}x")
        verdict = 1
    sys.exit(verdict)


if __name__ == "__main__":
    main()

Run it before dispatch, not after:

python3 lane_preflight.py canary.jsonl \
  --remaining 860 \
  --concurrency 4 \
  --deadline 1762506000 \
  --max-amp 1.5
echo "exit=$?"

Expected output

Numbers below are synthetic, from a made-up canary. They show shape, not a benchmark.

accepted jobs          : 40
p95 queue age (s)      : 2040
p95 job duration (s)   : 96
retry amplification    : 1.37x
tokens / artifact      : 2410
projected finish (s)   : 1762509922
deadline slack (min)   : -65.4
VERDICT: free lane misses the deadline
exit=1

Note what happened. Amplification stayed under threshold. Price stayed zero. Slack went negative.

The verdict is a time verdict. Budget verdicts miss it entirely.

Turn the numbers into a lane decision

Slack	Amplification	Action
> 90 min	< 1.2x	Keep the free lane, run full batch
0–90 min	< 1.2x	Keep free, cut scope, re-check every 15 min
0–90 min	≥ 1.5x	Split batch: canary on free, rest elsewhere
< 0 min	any	Do not start. Fix inputs or buy time
< 0 min	≥ 2.0x	Stop. The retries are the problem, not the lane

The last row matters most. Retry amplification above 2x is usually a bad prompt or a failing dependency.

Moving that batch to a faster lane just burns money faster.

Where free model access and a free server actually help

Here is where I use it, and where I do not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project offering free model access and a free server option. Both are operator-supplied availability claims, so check the current terms yourself before you plan around them.

I use the free server for the boring half of this workflow:

Run the canary batch and write the JSONL ledger.
Execute lane_preflight.py on the same host as the events.
Re-run the check every 15 minutes during a long batch.

That keeps measurement off your paid workers. It also keeps the ledger next to the data.

I never treat the free tier as a latency SLO. It is a capacity source with unknown queue depth.

One more thing: quotas move. I will not quote a token allowance here, because a stale number is worse than no number. Read the current figure on the project page, then plug the real consumption into your ledger.

Do not bet the whole batch on a canary you did not measure

Commit the first 5% of jobs. Read the ledger. Then decide.

If your canary runs 40 jobs, you have 40 samples. That is enough for a p95 that will still be wrong. Treat it as a smoke signal, not a guarantee.

Failure modes this check will not catch

Be honest about coverage. The script measures time. It misses these.

Disk. Free capacity still writes artifacts. Cap retention before dispatch.
Lane thrash. A job that flaps between lanes double-counts tokens. Pin the lane after one ambiguous retry.
Non-idempotent retries. Amplification looks cheap when duplicates silently overwrite output.
Clock skew. Timestamps from different hosts break queue age. Sync clocks first.

Each one is a separate control. None of them are fixed by a faster lane.

Rollback and cleanup

Prepare the exit before you start the batch.

Tag every job with lane=free and a batch_id correlation field.
Keep a cancel script that filters on batch_id and stops queued jobs.
Snapshot the ledger before any cleanup. It is your postmortem evidence.
Re-dispatch the remainder with the same job IDs. Idempotent sinks only.
Cap artifact retention at 7 days until the batch is accepted by a human.

If the deadline slips anyway, revert to the paid lane and rerun the preflight with --now set to the real clock. Do not rerun the whole batch blind.

Who should skip this approach

Be honest, most request-path systems should not use free capacity as a first option.

Hard latency SLOs. Any p99 commitment under a minute. Queue age alone will break it.
Non-idempotent side effects. Retries will corrupt state before you notice.
Data residency requirements. You cannot verify the path, so do not send the data.
Tiny batches. Under 20 jobs, the measurement cost exceeds the savings.

If you are in those groups, keep the ledger and drop the free lane. The tracking is still worth it.

The habit worth keeping

Write the deadline into your config. Compute slack from real events. Let the check block the dispatch.

Free capacity is a good bet when the deadline is loose and retries are rare. It is a bad bet when both are unknown.

If you want to try this loop, the free server option is a reasonable place to run the canary and the preflight script. Measure your own numbers before you trust anyone's, including mine.

Canary the Free Lane Before You Commit 2,000 Jobs

Odd_Background_328 — Thu, 05 Nov 2026 15:04:33 +0000

This week's AI threads argue about machines replacing us. Your queue has a smaller question. Should tonight's batch go to the free lane?

You commit 2,000 jobs at 18:00. They look cheap: free model access, free server. At 02:00 you have 300 artifacts. The rest burned quota on rejects.

That is not a model problem. It is an unpriced decision. You queued work before measuring the lane.

This post builds a preflight probe. It samples the free lane, produces a quote, and picks one of four actions. The probe is cheap, reversible, and boring on purpose.

The decision you keep making blind

Free capacity is shared capacity. Its acceptance rate moves with the hour. So does its latency.

You cannot read that from a pricing page. You read it from a small, timed sample.

Two numbers decide your batch:

Accept rate — share of probes that return a usable completion.
Tail latency — p95 seconds per probe at your real concurrency.

Add one constraint you already own: deadline slack. The minutes between "batch done" and "results needed."

If projected wall clock exceeds slack, free is the wrong bet. No metric fixes that.

Build the probe: 40 lines, stdlib only

Point it at any OpenAI-style chat endpoint you have access to. Model name and key come from flags and env.

#!/usr/bin/env python3
"""probe_free_lane.py - quote free-lane capacity before committing a batch."""
import argparse, json, os, time, urllib.error, urllib.request
from concurrent.futures import ThreadPoolExecutor

def pct(xs, p):
    if not xs:
        return None
    xs = sorted(xs)
    return xs[max(0, min(len(xs) - 1, int(round(p / 100 * (len(xs) - 1)))))]

def probe(args, i):
    body = json.dumps({
        "model": args.model,
        "messages": [{"role": "user", "content": "Reply with the single word OK."}],
        "max_tokens": 8,
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        args.endpoint.rstrip("/") + "/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ.get(args.key_env, ""),
            "X-Canary-Id": args.canary_id,
        },
    )
    t0, status, tokens, note = time.monotonic(), 0, 0, ""
    try:
        with urllib.request.urlopen(req, timeout=args.timeout) as r:
            status = r.status
            payload = json.loads(r.read() or b"{}")
            tokens = int(payload.get("usage", {}).get("total_tokens", 0))
    except urllib.error.HTTPError as e:
        status, note = e.code, (e.headers.get("Retry-After") or "")[:8]
    except Exception as e:
        status, note = -1, type(e).__name__
    rec = {"i": i, "status": status, "latency_s": round(time.monotonic() - t0, 3),
           "tokens": tokens, "retry_after": note, "canary_id": args.canary_id}
    print(json.dumps(rec), flush=True)
    return rec

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("--endpoint", required=True)
    p.add_argument("--model", required=True)
    p.add_argument("--key-env", default="FREE_LANE_KEY")
    p.add_argument("--canary-id", default=time.strftime("canary-%Y%m%dT%H%M"))
    p.add_argument("--size", type=int, default=12)
    p.add_argument("--concurrency", type=int, default=3)
    p.add_argument("--jobs", type=int, default=2000)
    p.add_argument("--slack-min", type=float, default=240.0)
    p.add_argument("--timeout", type=float, default=30.0)
    a = p.parse_args()

    with ThreadPoolExecutor(max_workers=a.concurrency) as ex:
        recs = list(ex.map(lambda i: probe(a, i), range(a.size)))

    ok = [r for r in recs if r["status"] == 200]
    lat = [r["latency_s"] for r in ok]
    accept = len(ok) / len(recs)
    p95 = pct(lat, 95) or a.timeout
    waves = a.jobs / max(1, a.concurrency)
    projected_min = waves * p95 / 60
    tokens = sum(r["tokens"] for r in ok)
    burn = tokens / max(1, len(ok)) * a.jobs

    if accept < 0.80 or projected_min > a.slack_min:
        action = "ESCALATE_OR_DEFER"
    elif accept < 0.95 or projected_min > 0.70 * a.slack_min:
        action = "SPLIT"
    else:
        action = "RUN_FREE"

    print(json.dumps({
        "action": action, "accept_rate": round(accept, 3),
        "p95_s": p95, "projected_min_optimistic": round(projected_min, 1),
        "slack_min": a.slack_min, "projected_tokens": int(burn),
        "short_circuit": all(r["status"] == 429 for r in recs[:4]),
    }, indent=2))

Run it:

export FREE_LANE_KEY=...
python3 probe_free_lane.py \
  --endpoint https://your-endpoint/v1 \
  --model your-free-model \
  --size 12 --concurrency 3 --jobs 2000 --slack-min 240

Twelve probes at concurrency three cost under a minute. That is under one percent of a 2,000-job batch.

Use the free model access for exactly this. Probing is what free capacity is good for. Then host the probe worker on the free server, next to the queue, so it measures the path your jobs actually take.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Read the quote like an operator

Expected output for a healthy lane:

{
  "action": "RUN_FREE",
  "accept_rate": 1.0,
  "p95_s": 2.4,
  "projected_min_optimistic": 26.7,
  "slack_min": 240.0,
  "projected_tokens": 14800,
  "short_circuit": false
}

Label it optimistic. p95_s * waves assumes zero queueing growth after the canary. Real batches degrade; treat the number as a floor.

Now the decision table.

Accept rate	Projected wall clock	Action	Why
≥ 0.95	≤ 70% of slack	`RUN_FREE`	Headroom absorbs one bad hour
≥ 0.95	70–100% of slack	`SPLIT`	Run half now, re-probe before the rest
0.80–0.95	any	`SPLIT`	Rejects are the real tax, not latency
< 0.80	any	`ESCALATE_OR_DEFER`	Free lane is telling you no

ESCALATE_OR_DEFER is not failure. It is the quote working.

One more rule: expire the quote. Fifteen minutes, then re-probe. Shared capacity does not hold a price overnight.

Short-circuit on four consecutive 429s. Stop the canary, keep the logs, do not retry.

Inject the failure before production does

Do not wait for the shared lane to rate-limit you. Reproduce it locally.

# fake_lane.py - local 429 injector for probe rehearsal
import json, random, time
from http.server import BaseHTTPRequestHandler, HTTPServer

FAIL_RATE, LATENCY = 0.45, 0.08

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        time.sleep(LATENCY)
        if random.random() < FAIL_RATE:
            self.send_response(429)
            self.send_header("Retry-After", "30")
            self.end_headers()
            self.wfile.write(b'{"error":"rate_limited"}')
            return
        body = json.dumps({"choices": [{"message": {"content": "OK"}}],
                           "usage": {"total_tokens": 26}}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

HTTPServer(("127.0.0.1", 8099), H).serve_forever()

Run it, then point the probe at http://127.0.0.1:8099/v1. You should see accept_rate near 0.55 and action ESCALATE_OR_DEFER.

Raise FAIL_RATE to 0.9 and confirm the short-circuit flag fires. Tune thresholds against that, not against a good day.

What the quote does not tell you

Be honest about the limits:

It measures one window, not your batch. Prompt length and output size change burn.
Token burn scales with your real max_tokens, not the 8-token probe.
Free latency is not an SLO. The p95 you measured can vanish in ten minutes.
The projection ignores your own downstream stages: validation, storage, uploads.
A canary that is 1% of a 200-job batch is too expensive. Probe smaller jobs only.

Skip this approach if you have a hard p99 latency commitment, strict data-residency rules, or jobs longer than the free server tolerates. Buy the boring capacity instead. Free is the wrong bet when a missed deadline costs more than the bill.

Cleanup and rollback

Every probe carries a X-Canary-Id header. That is your undo button.

Filter artifacts by canary_id prefix and delete them.
Keep canary.jsonl for the weekly cost review; drop payloads.
Do not change billing plans to run a canary. Ever.
Add a STOP_PROBE flag file. Check it between waves.
If the probe itself errored, defer the batch. A broken ruler is not a green light.

Rollback of a RUN_FREE decision is harder than rollback of a canary. That asymmetry is the whole argument for probing first.

The threshold is the artifact

Your canary is not the deliverable. Your accept-rate floor and slack ratio are.

Write them down with a rationale. "0.95 accept rate, because one reject in twenty doubles our tail wall clock." Then re-check them monthly against real batches.

If you want a free place to rehearse this loop, MonkeyCode's free model access and free server option are enough to run the probe, the injector, and the decision table end to end. Verify the current quota in your dashboard before you size a batch against it; allowances change.

Probe twelve jobs tonight. Then decide what the other 1,988 are worth.

Free Workers Don't Ship Free Disk: Cap Artifact Retention First

Odd_Background_328 — Tue, 27 Oct 2026 12:31:22 +0000

At 03:12 the alert fired on queue age. Not on errors, not on latency.

The batch had 4,000 jobs. Each accepted job wrote about 2 MB of artifacts. Nobody had set a retention policy.

Three days of runs had filled the worker's disk. Writes started failing. The consumer requeued failed jobs. Tokens were spent again on work already paid for once.

The evidence said storage. The dashboard said AI capacity. Which action do you take first?

Separate what you saw from what you inferred

Observed: OSError: [Errno 28] No space left on device, then a growing retry count.

Observed: p95 queue age crossed 15 minutes while model latency stayed flat.

Inferred: the model got slower. That inference was wrong in this incident, and it usually is.

Disk pressure looks like throughput pressure from the outside. Retries hide the real cause inside the token bill.

Treat the retention budget as part of the workload spec

A free worker gives you compute. It does not give you unbounded storage.

Write the budget down before you enqueue anything:

required_bytes = queue_depth * avg_artifact_bytes * safety_factor + reserve_bytes
reserve_bytes  = total_disk_bytes * reserve_ratio

Declared workload for the reference run below:

queue_depth = 4000
avg_artifact_bytes = 2_000_000 (JSON result, stdout log, preview)
safety_factor = 1.3 (variance between jobs)
total_disk_bytes = 20 GB, reserve_ratio = 0.2

That is 10.4 GB of artifacts against a 4 GB reserve. Tight, not impossible. Without the reserve, it fails quietly at the end of the batch.

Pre-flight admission check

Reject the batch before it starts, not after job 3,400.

# admit.py - refuse a batch the disk cannot finish
import shutil

def admit(store, queue_depth, avg_bytes, reserve_ratio=0.20, safety=1.3):
    total, used, free = shutil.disk_usage(store)
    need = int(queue_depth * avg_bytes * safety)
    reserve = int(total * reserve_ratio)
    verdict = (free - need) >= reserve
    return verdict, {
        "free_bytes": free,
        "need_bytes": need,
        "reserve_bytes": reserve,
        "headroom_bytes": free - need - reserve,
    }

if __name__ == "__main__":
    ok, report = admit("/srv/artifacts", 4000, 2_000_000)
    print(report)
    raise SystemExit(0 if ok else 75)  # 75 = EX_TEMPFAIL

Exit code 75 tells your scheduler this is retryable-later, not a bug. Do not enqueue on False.

Check the same number from the shell before you trust it:

df -B1 --output=source,avail,pcent /srv/artifacts
find /srv/artifacts -type f -newermt '-24 hours' -printf '%s\n' \
  | awk '{s+=$1; n++} END {printf "files=%d avg_bytes=%d\n", n, s/n}'

A pruner you can run in dry-run first

Default is dry-run. Deletion requires --apply. Pinned artifacts survive every pass.

#!/usr/bin/env python3
# retain.py - cap artifact store before the worker disk fills
import argparse, os, shutil
from pathlib import Path

def pinned(store):
    p = Path(store) / ".pinned"
    if not p.exists():
        return set()
    return {line.strip() for line in p.read_text().splitlines() if line.strip()}

def candidates(store):
    keep = pinned(store)
    out = []
    for entry in Path(store).iterdir():
        if not entry.is_dir() or entry.name in keep:
            continue
        size = sum(f.stat().st_size for f in entry.rglob("*") if f.is_file())
        out.append((entry.stat().st_mtime, entry, size))
    return sorted(out)  # oldest first

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--store", required=True)
    ap.add_argument("--max-bytes", type=int, required=True)
    ap.add_argument("--keep-newest", type=int, default=50)
    ap.add_argument("--apply", action="store_true")
    a = ap.parse_args()

    items = candidates(a.store)
    protected = items[-a.keep_newest:]
    used = sum(size for _, _, size in items)
    freed = 0

    for _, path, size in items:
        if used - freed <= a.max_bytes:
            break
        if (path.stat().st_mtime, path, size) in protected:
            continue
        print(f"{'rm' if a.apply else 'would rm'} {path} {size}")
        if a.apply:
            shutil.rmtree(path, ignore_errors=True)
        freed += size

    print(f"used={used} freed={freed} target={a.max_bytes} apply={a.apply}")

if __name__ == "__main__":
    main()

Run it on a timer, not in your head:

python3 retain.py --store /srv/artifacts --max-bytes 12000000000 --keep-newest 50
# inspect, then arm it
python3 retain.py --store /srv/artifacts --max-bytes 12000000000 --apply

Failure drill: fill the disk on purpose

Do this on a scratch worker before production finds it for you.

dd if=/dev/zero of=/srv/artifacts/fill bs=1M count=3000 status=progress
Run admit.py. Confirm it returns exit 75 once headroom goes negative.
Submit a 200-job batch with admission disabled. Confirm the first write error appears.
Confirm the consumer requeues, and watch the token counter climb without a new artifact.
Delete fill, run retain.py --apply, confirm the store drops under the cap.
Re-enable admission and confirm the batch starts.

Step 4 is the one to instrument. Emit these fields per job:

job_id, attempt, artifact_bytes, write_ok
disk_avail_bytes at start and end
tokens_in, tokens_out, tokens_wasted_on_retry

If tokens_wasted_on_retry is non-zero, storage is charging you twice.

What the ledger shows

Arithmetic on the declared inputs above, not measurements from your cluster:

Item	Per 1,000 jobs	Note
Artifacts at 2 MB	2 GB	grows linearly with retention window
Retry tokens after write failure	1x generation cost	artifacts lost, model output not cached
Pruner CPU	seconds	`rmtree` on local disk
Object storage offload	per GB-month	check egress before assuming cheap

Expected output shape from admit.py on a healthy worker:

{'free_bytes': 14800000000, 'need_bytes': 10400000000, 'reserve_bytes': 4000000000, 'headroom_bytes': 400000000}

A 400 KB headroom is a warning, not a pass. Raise safety_factor or split the batch.

When the free worker is the wrong bet

Stop and re-plan if any of these hold:

Artifacts must be retained beyond a week for audit or replay.
Average artifact size exceeds 5% of total disk per batch.
A single worker's throughput cannot drain the queue inside your deadline.
Artifacts are inputs to downstream training or billing.

In those cases, write artifacts to durable object storage and keep only recent pointers on the worker. Pay for storage, not for retries.

Where the free tier fits

I use MonkeyCode's free model access and free server option for batch experimentation. Both are operator-supplied availability claims, not measured guarantees here. Treat the disk as a budget you declare yourself.

The workflow above does not depend on them. Run it on any worker with a local disk.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and who should skip this

This is a single-node pattern. It does not give you replication, cross-zone durability, or shared storage across workers.

Skip it if you need multi-writer concurrency on the same store. Skip it if your compliance team requires immutable retention. Skip it if your artifacts are larger than your reserve can absorb.

Do not use the pruner as your only backup. It deletes. That is the point.

If the arithmetic above says the disk cannot finish the batch, then free capacity is the wrong bet for that batch.

Cleanup and rollback

Before arming the timer, record the current state:

du -sb /srv/artifacts > /srv/artifacts.size.before
find /srv/artifacts -maxdepth 1 -type d | wc -l > /srv/artifacts.count.before

Rollback is two commands:

systemctl disable --now artifact-retain.timer
rm -f /srv/artifacts/fill

Then restore what you deleted from your offload bucket. If you have no offload bucket, you did not have a rollback.

Start with a dry-run on one day of artifacts. Compare the file list against what you expected to keep. Only then arm --apply.

If you want a cheap place to rehearse this drill, the free server option is a reasonable scratch target. Keep the pin list short and the blast radius smaller than your resume.

Retry Amplification Is What Actually Burns Your Free Token Budget

Odd_Background_328 — Mon, 26 Oct 2026 11:12:51 +0000

At 09:12 the meter read 41k tokens per hour. By 11:30 it read 310k. Job count was flat. Traffic was flat. One prompt class was retrying itself into a hole.

Nothing in the dashboard named that. The graph showed tokens, and tokens looked like demand. They were not demand. They were a retry loop wearing demand's clothes.

Here is the guard I now put in front of any free-tier batch worker, and the thresholds that decide when free capacity stops being the right bet.

The number you were watching was wrong

Tokens per hour is an input metric. It tells you what you spent, not what you got.

Two lanes can burn identically and deliver wildly different value. Lane A: 900 jobs, 1 attempt each. Lane B: 300 jobs, 3 attempts each. Same token rate. Different product.

So track three numbers instead:

Retry amplification — attempts divided by distinct jobs.
Tokens per accepted artifact — total tokens divided by jobs that reached accepted.
Hours to empty — remaining allowance divided by the current burn rate.

Then compare hours-to-empty against deadline slack, not against a percentage bar. Utilization at 80% is fine with 40 hours of slack. It is fatal with 2.

Declare the workload before you trust any of it

This guard was built for one declared shape. Change the shape and you must re-measure.

One batch worker process, 4 concurrent jobs.
200 mixed jobs per run, three prompt classes (summarize, classify, extract).
Retry policy: 3 attempts, exponential backoff, no class separation.
Meter reads an append-only attempt log, one JSON line per model call.

The log contract matters more than the code. If you do not log per-attempt token counts, you cannot separate demand from amplification later.

{"ts": 1761512400, "job_id": "j-8812", "class": "summarize",
 "attempt": 1, "prompt_tokens": 1800, "completion_tokens": 240,
 "outcome": "accepted"}

outcome is one of accepted, rejected, error. Write the line after the call returns, not before. A crash mid-call should leave no spend record you cannot verify.

The artifact: a burn-rate guard in one file

Drop this next to the worker. It reads the log, prints four numbers, and returns a verdict your admission hook can act on.

#!/usr/bin/env python3
"""burn.py - token burn-rate guard for a single AI batch worker."""
import argparse, json, sys
from collections import defaultdict


def load(path):
    rows = []
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def tokens(r):
    return r["prompt_tokens"] + r["completion_tokens"]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--log", required=True)
    ap.add_argument("--budget", type=int, required=True)
    ap.add_argument("--window", type=int, default=3600)
    ap.add_argument("--slack-hours", type=float, required=True)
    ap.add_argument("--amplification-cap", type=float, default=1.6)
    ap.add_argument("--per-artifact-cap", type=int, default=4000)
    args = ap.parse_args()

    rows = load(args.log)
    if not rows:
        sys.exit("empty log: nothing to meter")

    now = max(r["ts"] for r in rows)
    spent = sum(tokens(r) for r in rows)
    recent = [r for r in rows if r["ts"] >= now - args.window]
    rate = sum(tokens(r) for r in recent) / args.window  # tokens per second

    by_job = defaultdict(list)
    for r in rows:
        by_job[r["job_id"]].append(r)
    accepted = [j for j, rs in by_job.items()
                if any(r["outcome"] == "accepted" for r in rs)]

    amp = len(rows) / max(len(by_job), 1)
    per_artifact = spent / max(len(accepted), 1)
    remaining = args.budget - spent
    eta_h = (remaining / rate / 3600) if rate > 0 else float("inf")

    print(f"jobs={len(by_job)} attempts={len(rows)} accepted={len(accepted)}")
    print(f"spent={spent} remaining={remaining} rate={rate:.0f}tok/s")
    print(f"amplification={amp:.2f} tokens_per_accepted={per_artifact:.0f}")
    print(f"eta_hours={eta_h:.1f} slack_hours={args.slack_hours}")

    verdict = "ADMIT"
    if amp > args.amplification_cap:
        verdict = "CAP_ATTEMPTS"
    if per_artifact > args.per_artifact_cap:
        verdict = "QUARANTINE_CLASS"
    if eta_h < args.slack_hours:
        verdict = "HOLD_NEW_WORK"
    print(f"verdict={verdict}")


if __name__ == "__main__":
    main()

Run it from cron, or from the admission hook, every 60 seconds:

python3 burn.py \
  --log /var/log/jobs/attempts.jsonl \
  --budget 10000000 \
  --slack-hours 14 \
  --amplification-cap 1.6

Expected output for a synthetic log

I have not run this against a production lane. The block below is expected output for the synthetic 200-job log shipped with the script. Label it as such in your own runbook too.

jobs=200 attempts=331 accepted=173
spent=604200 remaining=9395800 rate=21tok/s
amplification=1.66 tokens_per_accepted=3492
eta_hours=124.3 slack_hours=14.0
verdict=CAP_ATTEMPTS

Read it in this order. amplification=1.66 says 158 of 331 attempts bought nothing. tokens_per_accepted=3492 says each shipped artifact carries almost one and a half failed attempts. The ETA is comfortable, and that is exactly the trap: you have tokens but not throughput.

Where the free server actually fits

The meter must run where the worker runs. A second box for observability turns a free lane into a paid one.

That is why I keep both the worker and burn.py on the same free server, sharing the same log file. The meter is stateless, so a restart costs you nothing.

MonkeyCode's free model access and free server option fit that shape. The free arrangement is an allowance of about 10M tokens, per the project's own materials — treat that figure as operator-supplied and re-check current limits before you size anything against it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A 10M-token allowance is a burst allowance, not a steady-state budget. At 21 tokens/second, it covers roughly five and a half days of continuous running. At 200 tokens/second, barely 14 hours. Never plan against the first number unless you have measured the second.

Which constraint is binding

Symptom	Binding constraint	Operational move
Amplification > 1.6, tokens per accepted flat	Retry policy	Cap attempts at 2, quarantine on exhaustion
Tokens per accepted up 20%, job count flat	Prompt or input size	Revert the prompt change before scaling
ETA < deadline slack, amplification < 1.2	Real demand	Reduce concurrency or move a class off the free lane
Queue age up, utilization low	Admission, not capacity	Check for a hold or a stalled worker

Utilization is the wrong threshold for the last row. Queue age against deadline slack tells you whether anyone cares about the wait.

When free capacity is the wrong bet

Walk away from the free lane in these cases:

Your job class needs a hard p99 latency target. Shared capacity cannot promise it.
Amplification stays above 1.6 after the attempt cap. You are paying retries with time you do not have.
Tokens per accepted artifact exceeds your cost of a second attempt elsewhere.
The work must survive a node restart mid-batch with no requeue.

The free lane is a cost probe. It tells you the shape of your workload cheaply. It is not a capacity commitment.

Failure handling, cleanup, rollback

Policy file read at admission:

retry:
  max_attempts: 2
  backoff_s: [4, 16]
  on_exhausted: quarantine      # dead partition, never auto re-enqueue
admission:
  hold_when: "verdict in {HOLD_NEW_WORK, QUARANTINE_CLASS}"
  recheck_s: 60

Rollback path, in order:

Set max_attempts back to its previous value and restart the worker.
Export the quarantine partition to cold storage.
Re-enqueue quarantined jobs with attempt reset only after the prompt fix ships.
Confirm amplification drops below 1.6 over two windows before re-enabling admission.

Cleanup is boring on purpose:

logrotate /etc/logrotate.d/jobs-attempts   # daily, keep 7, compress

Keep the attempt log longer than the job log. Cost questions always arrive after the jobs are gone.

Limitations and who should skip this

EWMA-free extrapolation breaks on step changes. A prompt deploy invalidates the ETA instantly.
Per-attempt token counts are only as good as your client. Streaming calls can under-report.
Allowance terms change. Re-verify before you set a --budget value in CI.
Interactive, latency-critical traffic should not use this guard. It optimizes spend, not response time.

Skip it entirely if you have tenant-level billing with per-request cost attribution. You already have better numbers than amplification.

If you want to try the workflow without provisioning anything, the free model access and free server option are enough to run one worker and this meter on the same box. Start with a 200-job synthetic batch and read the verdict before you point it at real traffic.

Your Free AI Lane Is Slow in the Queue, Not in the Model

Odd_Background_328 — Tue, 20 Oct 2026 09:17:08 +0000

You moved a batch to free capacity. The token bill dropped to zero. The deadline still slipped.

Nothing in the model got slower. The jobs sat in a queue you never charted.

Most AI cost dashboards start counting at the first token. That is the wrong start line for a free lane. On free capacity, the wall clock usually belongs to scheduling, not to generation.

This post is a drill, not a vendor pitch. You record two clocks per job. Then you run one script and get a verdict: hold the batch, or spill it to paid capacity before the cutoff.

Start the clock at enqueue, not at first token

Your job record needs five timestamps. Anything less and you cannot separate waiting from working.

enqueue_ts — the job was accepted by your scheduler.
start_ts — the first request hit the serving path.
accept_ts — the artifact passed your acceptance check.
deadline_ts — the wall-clock deadline for that batch.
attempt — the retry index, so rework stays visible.

From those you get three derived numbers:

wait = start_ts - enqueue_ts
gen = accept_ts - start_ts
wait_share = wait / (accept_ts - enqueue_ts)

Gate on wait_share. A lane with p95 wait share of 0.7 is a scheduling problem wearing a model costume. Buying faster tokens there changes almost nothing.

The two-clock ledger in one script

The script below reads JSON Lines and prints a verdict. It is deliberately dumb. No dashboards, no agent, no database.

#!/usr/bin/env python3
# two_clock.py — lane verdict from job events (JSON Lines)
import json, sys

def load(path):
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                yield json.loads(line)

def pct(values, q):
    if not values:
        return 0.0
    s = sorted(values)
    i = min(len(s) - 1, int(round(q * (len(s) - 1))))
    return s[i]

def main(path, wait_share_max=0.60, margin_frac=0.15):
    rows = list(load(path))
    waits, gen, share = [], [], []
    for r in rows:
        total = r["accept_ts"] - r["enqueue_ts"]
        if total <= 0:
            continue
        wait = r["start_ts"] - r["enqueue_ts"]
        waits.append(wait)
        gen.append(total - wait)
        share.append(wait / total)

    drain = max(r["accept_ts"] for r in rows)
    first = min(r["enqueue_ts"] for r in rows)
    deadline = max(r["deadline_ts"] for r in rows)
    slack = deadline - drain
    margin = (deadline - first) * margin_frac

    print(f"jobs={len(rows)} wait_p50={pct(waits,0.5):.0f}s wait_p95={pct(waits,0.95):.0f}s")
    print(f"gen_p50={pct(gen,0.5):.0f}s wait_share_p95={pct(share,0.95):.2f}")
    print(f"slack={slack:.0f}s margin_budget={margin:.0f}s")

    if pct(share, 0.95) > wait_share_max:
        print("VERDICT: spill — wait dominates, throughput is not your lever")
    elif slack < margin:
        print("VERDICT: spill — projected drain lands inside the safety margin")
    else:
        print("VERDICT: hold — the lane is inside budget")

if __name__ == "__main__":
    main(sys.argv[1])

Run it against a finished batch:

python3 two_clock.py lane-free.jsonl

Expected output shape (illustrative, not a measurement from your system):

jobs=412 wait_p50=38s wait_p95=611s gen_p50=22s wait_share_p95=0.81
slack=1740s margin_budget=1080s
VERDICT: spill — wait dominates, throughput is not your lever

Note what the verdict ignores: tokens. They live in the same record if you need them, but they do not move this decision. Free capacity is a scheduling bet first and a billing bet second.

Why those two thresholds

wait_share_max = 0.60 says: if most of a job's life is spent queued, the model is not the constraint. Tighten it if your acceptance check is slow. Loosen it only with data.

margin_frac = 0.15 is a reserve against your own estimate. Drain forecasts are optimistic at the tail. Keep a slice of the window unspent so a late retry still fits.

Inject the wait locally before you trust the gate

Do not tune thresholds on a quiet lane. Congest it yourself, on purpose, in your own shell.

#!/usr/bin/env bash
# fake_lane.sh — synthetic congestion, zero production impact
set -euo pipefail
HOLD=${HOLD_SECONDS:-90}
DEADLINE=$(( $(date +%s) + ${WINDOW_SECONDS:-3600} ))
while read -r id; do
  ENQ=$(date +%s)
  sleep "$HOLD"
  echo "{\"job_id\":\"$id\",\"enqueue_ts\":$ENQ,\"start_ts\":$(date +%s),\"accept_ts\":$(date +%s),\"deadline_ts\":$DEADLINE}"
done

Generate job ids, then run the same 40 jobs twice:

seq 1 40 > ids.txt
HOLD_SECONDS=5  ./fake_lane.sh < ids.txt > lane-fast.jsonl
HOLD_SECONDS=90 ./fake_lane.sh < ids.txt > lane-slow.jsonl
python3 two_clock.py lane-fast.jsonl
python3 two_clock.py lane-slow.jsonl

The fast file should print hold. The slow file should print spill. If both print the same verdict, your gate is dead code and you should fix it before touching production routing.

Decision table: when free capacity is the wrong bet

Signal	Threshold	Action
p95 wait share	> 0.60	Spill the deadline-critical slice to paid capacity
Slack vs margin	slack < margin	Spill now, not at the next cron tick
Retry rate	rising while wait share rises	Fix admission, not the model
Rework per accepted artifact	above your baseline	Free tokens are buying wasted artifacts
Interactive p99 latency	any dependency	Do not route interactive traffic here at all

Read the table top-down. The first row that trips decides. You do not average these signals into one score.

Where the free tier fits into the drill

You can rehearse this whole loop without a paid commitment. MonkeyCode is an open-source project, and its operator states that it offers free model access, a free server option, and a free allowance of 10M tokens. Verify the current limits on the project page yourself before you plan any capacity around them; treat them as a starting condition, not a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the free lane genuinely buys you here is rehearsal time. You get real request paths, real timestamps, and a real queue to measure. What it does not buy you is a guarantee about wait distributions under your deadline. That number only exists after you run the ledger on your own workload.

One practical pairing: keep the free server as the lane under test, and keep the spill path wired to whatever paid capacity you already trust. The drill is only useful if the spill branch is reachable in one config change.

Rollback and cleanup

Keep the escape hatch boring and tested.

Gate routing behind one flag, for example FREE_LANE_ENABLED=0.
Flip the flag and stop new admission into the free lane.
Let in-flight jobs finish; do not kill them mid-acceptance.
Keep the ledger files. They are the evidence for the next threshold change.
Re-run two_clock.py on the spill window and compare wait shares.

If the spill window shows the same p95 wait share, the queue was never the free lane's fault. Look upstream at your own admission logic.

Who should not use this

Teams without timestamped job events. You cannot reconstruct two clocks from logs you never wrote.
Interactive or latency-critical endpoints. Queue-shaped capacity belongs behind batch work.
Work where a rework costs more than the tokens you saved.
Anyone planning to hard-code today's free allowance into a capacity plan for next quarter.

If you already run a batch queue, this is a one-afternoon exercise. Write four timestamps, run one script, and let the verdict argue with your assumptions.

Stop Lane Thrash: Pin Jobs After the First Ambiguous Retry

Odd_Background_328 — Mon, 19 Oct 2026 07:52:45 +0000

02:14 — the token spend alert fires.

Spend is up 90% over the same window yesterday. Accepted artifacts are up 2%. Nothing in the error budget explains the gap.

The queue has two lanes: a free lane for model work, and a paid fallback. Jobs bounce between them. Every bounce looks like a cheap retry in the dashboard.

This post is about the bill you cannot see in the token graph. It is the rework caused by a job that succeeds twice.

What Lane Thrash Looks Like

Lane thrash is a job that crosses lanes more than once per artifact. The usual cause is an ambiguous outcome.

The free lane times out after it already generated output. Your client sees a failure. Your retry sees a fresh job.

Two artifacts arrive later with the same inputs. Both look valid. Both get accepted.

You now pay tokens twice and review work twice. The metric that moves is not spend. It is duplicate artifacts per hundred jobs.

Topology: One Queue, Two Lanes

Keep one queue. Do not build two queues for two lanes.

A single admission point assigns a lane per attempt.
Every attempt writes one ledger row before it calls a model.
The ledger row carries an Idempotency-Key derived from job inputs.
Artifact writes are keyed by that same value.

That last bullet is the control that actually stops duplicates. Lane pinning only reduces how often you need it.

Config You Can Copy

# lane.yaml
lanes:
  free:
    max_attempts: 2
    pin_on: [ambiguous, permanent]
  paid:
    max_attempts: 1
reconcile:
  late_completion_window_s: 900
  dedup_key: Idempotency-Key
alerts:
  cross_lane_retries_per_job: 0.20
  duplicate_artifacts_per_100_jobs: 1
free_lane_enabled: true

The late_completion_window_s value matters. A slow free-lane response that lands after your client timed out is a duplicate unless you reconcile inside that window.

A Ledger That Answers the Right Question

CREATE TABLE job_attempt (
  job_id      TEXT NOT NULL,
  idem_key    TEXT NOT NULL,
  attempt_no  INTEGER NOT NULL,
  lane        TEXT NOT NULL CHECK (lane IN ('free','paid')),
  started_at  TEXT NOT NULL,
  ended_at    TEXT,
  outcome     TEXT,   -- ok | retryable | ambiguous | permanent
  tokens_in   INTEGER DEFAULT 0,
  tokens_out  INTEGER DEFAULT 0,
  artifact_id TEXT,
  PRIMARY KEY (job_id, attempt_no)
);

CREATE INDEX idx_artifact ON job_attempt(artifact_id);

CREATE VIEW job_rollup AS
SELECT job_id,
       MAX(idem_key)                   AS idem_key,
       SUM(lane = 'free')              AS free_attempts,
       SUM(lane = 'paid')              AS paid_attempts,
       SUM(tokens_in + tokens_out)     AS tokens,
       COUNT(DISTINCT artifact_id)     AS artifacts
FROM job_attempt
GROUP BY job_id;

Two rollup queries carry the whole decision:

-- tokens per unique artifact, plus cross-lane pressure
SELECT ROUND(SUM(tokens) * 1.0 / SUM(artifacts), 1)   AS tokens_per_artifact,
       ROUND(SUM(free_attempts > 0 AND paid_attempts > 0) * 1.0
             / COUNT(*), 3)                          AS cross_lane_per_job
FROM job_rollup;

-- the duplicates themselves
SELECT idem_key, COUNT(DISTINCT artifact_id) AS n
FROM job_attempt
WHERE artifact_id IS NOT NULL
GROUP BY idem_key
HAVING n > 1;

The boolean-sum form avoids FILTER, so it runs on older SQLite builds. Check your version before you paste.

The Router: Pin on Ambiguous, Retry on Retryable

The failure class decides the lane. Not the retry count alone.

# lane_router.py — stdlib only
PIN_ON = {"ambiguous", "permanent"}
MAX_FREE_ATTEMPTS = 2

def next_lane(free_attempts: int, pinned: bool, last_outcome: str) -> tuple[str, bool]:
    """Return (lane, pinned) for the next attempt."""
    if pinned:
        return "paid", True
    if last_outcome in PIN_ON:
        return "paid", True
    if free_attempts >= MAX_FREE_ATTEMPTS:
        return "paid", True
    return "free", False

retryable means 429 or 503 with no work done. Retry it on the free lane once. It is cheap and safe.

ambiguous means a timeout after partial streaming. The work may exist. Pin that job to paid and reconcile by idem_key.

Three tests pin the behavior down:

def test_pin_on_ambiguous():
    assert next_lane(1, False, "ambiguous") == ("paid", True)

def test_retryable_stays_free_once_more():
    assert next_lane(1, False, "retryable") == ("free", False)

def test_budget_exhausted_pins():
    assert next_lane(2, False, "retryable") == ("paid", True)

Fault-Inject the Free Lane Locally

Do not discover ambiguity handling in production. Point the router at a local fake lane.

# fake_free_lane.py — port 8099, stdlib only
import itertools, time
from http.server import BaseHTTPRequestHandler, HTTPServer

SCRIPT = itertools.cycle(["ok", "429", "503", "slow"])

class FreeLane(BaseHTTPRequestHandler):
    def do_POST(self):
        step = next(SCRIPT)
        if step == "slow":
            time.sleep(35)          # exceeds the 30s client timeout
            step = "ok"
        self.send_response({"ok": 200, "429": 429, "503": 503}[step])
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b'{"artifact": "synthetic"}')

    def log_message(self, *args):
        pass

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8099), FreeLane).serve_forever()

Run it, replay 300 recorded jobs against it, then read the rollup. The slow branch is the one that manufactures duplicates.

Run the 300-Job Drill

Declared workload for the numbers below:

300 synthetic jobs, one artifact each.
Free-lane outcomes: 25% retryable, 5% ambiguous, 2% permanent.
Synthetic token constants: 800 per free attempt, 1,200 per paid attempt.
Reconciliation window 900s, dedup on idem_key.

Expected output from the synthetic model, not a measurement of any vendor:

Metric	Pin + dedup	Retry free, no pin
Unique accepted artifacts	300	300
Free attempts	375	411
Paid attempts	46	6
Total tokens	355,200	336,000
Tokens per accepted artifact	1,184	1,120
Duplicate artifacts	0	15
Cross-lane retries per job	0.15	0.02

Read the Rollup, Not the Invoice

Look at the token row first. The two policies differ by about 5%.

That gap will not justify an on-call change. The duplicate row will.

Fifteen duplicates at two minutes of review each is thirty minutes of avoidable human work. At volume, that dwarfs the token delta.

So the operational question is not "which lane is cheaper per token." It is "which lane produces fewer unrecoverable artifacts."

Thresholds and Why They Are These Numbers

cross_lane_retries_per_job > 0.20 — above one in five jobs crossing lanes, the reconciliation window is probably too short.
duplicate_artifacts_per_100_jobs > 1 — one duplicate per hundred is already a review tax you can measure.
Free-lane p95 completion above half your deadline slack — the lane cannot drain before cutoff.

Set the slack threshold from your own p95, not from a blog number. Recompute it after every model change.

When the Free Lane Is the Wrong Bet

Condition	Route
Deadline slack p95 < 2× free-lane p95 latency	Stay on paid
Ambiguous rate > 2% and no idempotency key at the boundary	Fix the key first
Sustained arrival above measured free throughput for 30 min	Split by priority
Token delta between lanes < 10%	Decide on rework and latency
Batch deadline ≥ 6h and outputs idempotent	Free lane first, pin on ambiguous

That fourth row is the one most teams get wrong. When tokens converge, the cost argument disappears and the recovery argument takes over.

Where the Free Capacity Comes From

I run this drill against MonkeyCode's free model access, with the free server option hosting the queue worker and ledger. The operator lists 10M free tokens and a free server option. Verify the current terms on the project page before you route production work.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is open source, so the ledger and router above stay portable. Nothing in the drill depends on vendor-specific APIs.

Run the 300-job drill against the free tier before you move real traffic. The duplicate count is the number you actually care about.

Rollback and Cleanup

Rollback takes three steps, in this order:

Set free_lane_enabled: false and let the queue drain. Do not kill in-flight jobs.
Reconcile the last 900s of attempts by idem_key before deleting anything.
Replay pinned jobs on paid with the original keys.

Cleanup after the drill:

pkill -f fake_free_lane.py
rm -f drill.db
ss -ltnp | grep 8099   # confirm the port is free

Keep the ledger for 30 days. It is the only record that proves how many duplicates you produced.

Who Should Not Use This

Teams without an idempotency key at the API boundary.
Workloads with strict p99 latency SLOs and no slack to spend.
Jobs whose partial output cannot be reconciled, such as side-effectful writes.
Multi-tenant systems that need hard fairness between tenants.
Anyone who needs a permanence guarantee on free capacity.

If you are in the first group, stop here. Build the key first. The lane router is worthless without it.

Gate Jobs on Tokens per Accepted Artifact Before Retries Eat the Budget

Odd_Background_328 — Sun, 18 Oct 2026 06:47:59 +0000

Last Tuesday our batch lane burned 3.1 million tokens before 09:00.
Only forty-nine of sixty jobs produced an artifact we could ship.

Queue utilization looked healthy at 41%. The token graph went up 4x.
So which operational action follows from that evidence? Not a bigger queue. A budget gate.

Here is the workflow I now run before adding any free capacity to a lane.

The contradiction metric: tokens per accepted artifact

Utilization measures occupancy. It does not measure yield.
A retry loop can keep a lane busy while producing nothing.

Track one ratio instead:

Numerator: every token you spend, including failed calls and rejected outputs.
Denominator: artifacts that passed your checks and shipped.

Call it TPA (tokens per accepted artifact). Put it next to wall time.
When TPA rises while utilization stays flat, you are paying for retries, not throughput.

Two-phase reservation: price retries before you send them

Most teams cap tokens per request. That cap is useless under retries.
Three attempts each under the cap still triples your spend.

So reserve for the worst case, then settle on reality.
The gate below is plain Python with no dependencies. It is a runnable example, not a production library.

# budget_gate.py - two-phase token reservation for batch jobs
import json, time
from dataclasses import dataclass, field


class BudgetExceeded(Exception):
    pass


@dataclass
class Reservation:
    job_id: str
    lane: str
    reserved: int
    spent: int = 0
    attempts: int = 0
    opened_at: float = field(default_factory=time.time)


class TokenLedger:
    def __init__(self, cap_per_job: int, path: str = "ledger.jsonl"):
        self.cap = cap_per_job
        self.path = path
        self.open = {}

    def reserve(self, job_id: str, lane: str, estimate: int) -> Reservation:
        if estimate > self.cap:
            raise BudgetExceeded(f"{job_id}: estimate {estimate} > cap {self.cap}")
        res = Reservation(job_id=job_id, lane=lane, reserved=estimate)
        self.open[job_id] = res
        return res

    def charge(self, job_id: str, tokens: int) -> None:
        res = self.open[job_id]
        res.attempts += 1
        res.spent += tokens
        if res.spent > res.reserved:
            self.close(job_id, accepted=False, reason="over_reservation")
            raise BudgetExceeded(f"{job_id}: spent {res.spent} > reserved {res.reserved}")

    def close(self, job_id: str, accepted: bool, reason: str) -> None:
        res = self.open.pop(job_id, None)
        if res is None:
            return
        record = {
            "job_id": res.job_id, "lane": res.lane,
            "reserved": res.reserved, "spent": res.spent,
            "attempts": res.attempts, "accepted": accepted,
            "settle_reason": reason,
            "wall_ms": int((time.time() - res.opened_at) * 1000),
        }
        with open(self.path, "a") as fh:
            fh.write(json.dumps(record) + "\n")

The reservation covers retries. That is the whole point.

# runner.py - pre-pay for max_attempts, settle once
import time


def backoff(attempt: int) -> float:
    return min(2 ** attempt * 0.5, 8.0)


def run_job(ledger, job, client, max_attempts: int = 3):
    per_call = job.prompt_tokens + job.max_output_tokens
    ledger.reserve(job.id, lane=job.lane, estimate=per_call * max_attempts)
    last = None
    for attempt in range(1, max_attempts + 1):
        try:
            out = client.complete(job.prompt, max_tokens=job.max_output_tokens)
            # Assumption: the provider bills prompt tokens even on a failed call.
            # Verify your provider's billing; drop this charge if it does not.
            ledger.charge(job.id, out.usage_total)
            if passes_checks(out.text):
                ledger.close(job.id, accepted=True, reason="accepted")
                return out.text
            ledger.close(job.id, accepted=False, reason="rejected_by_checks")
            return None
        except Transient as exc:
            ledger.charge(job.id, job.prompt_tokens)
            last = exc
            time.sleep(backoff(attempt))
    ledger.close(job.id, accepted=False, reason=f"retries_exhausted:{last}")
    return None

Two properties matter more than the code.
Every job has a pre-paid ceiling, and every settled job leaves one JSONL record.

Read the ledger in one command

jq -s '
  {spent: (map(.spent) | add // 0),
   accepted: (map(select(.accepted)) | length),
   retried: (map(select(.attempts > 1)) | length)}
  | . + {tokens_per_accepted:
         (if .accepted == 0 then null else (.spent / .accepted) end)}' \
  ledger.jsonl

Expected output on a healthy lane, clearly labeled as expected, not measured by me on your workload:

{"spent": 1840000, "accepted": 49, "retried": 7, "tokens_per_accepted": 37551}

Now you have a number you can put in a review, an alert, or a capacity plan.

Run the shadow week on the cheapest box you have

You do not need production capacity to get this number.
I run the gate in shadow mode on a small free instance first, because shadow mode only needs to observe.

The MonkeyCode project advertises free model access, with the operator citing a 10M-token allowance, plus a free server option. I have not verified today's quotas, model list, or hardware.
Treat both as current-terms claims and read the project page before you size anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Shadow mode rules, and they are strict:

The gate computes reservations and logs them. It never blocks a job.
The real client still runs where it runs today. Nothing moves.
After seven days, compare tokens_per_accepted against your paid baseline.
Only then decide whether the free lane is worth enforcing.

Do not put a latency-critical path on a free box to "see how it goes".
Free capacity is a measurement instrument here, not a serving tier.

Fault injection before you trust the gate

A gate you never failed is a guess. Force failures locally.

# test_gate.py - run with: python -m pytest -q test_gate.py
import pytest
from budget_gate import BudgetExceeded, TokenLedger


class Transient(Exception):
    pass


class FlakyClient:
    def __init__(self, fail_first: int):
        self.fail_first = fail_first
        self.calls = 0

    def complete(self, prompt, max_tokens):
        self.calls += 1
        if self.calls <= self.fail_first:
            raise Transient("429 from upstream")
        return type("Out", (), {"text": "ok", "usage_total": 900})()


def test_retries_are_pre_paid(tmp_path):
    ledger = TokenLedger(cap_per_job=10000, path=str(tmp_path / "l.jsonl"))
    job = type("Job", (), {"id": "j1", "lane": "free",
                            "prompt_tokens": 800, "max_output_tokens": 400,
                            "prompt": "x"})()
    out = run_job(ledger, job, FlakyClient(fail_first=2), max_attempts=3)
    assert out == "ok"
    assert ledger.open == {}


def test_runaway_is_rejected(tmp_path):
    ledger = TokenLedger(cap_per_job=10000, path=str(tmp_path / "l.jsonl"))
    job = type("Job", (), {"id": "j2", "lane": "free",
                            "prompt_tokens": 800, "max_output_tokens": 400,
                            "prompt": "x"})()
    out = run_job(ledger, job, FlakyClient(fail_first=99), max_attempts=3)
    assert out is None
    assert ledger.open == {}

Assert three things after every run: no open reservations, spend never exceeds reservation, and the settle reason is populated.
An empty open dict is your leak detector.

Telemetry fields to emit

Six fields carry the whole diagnosis. Ship them on every settle.

lane - which capacity class ran the job.
reserved vs spent - reservation accuracy over time.
attempts - retry distribution, not just the mean.
accepted and settle_reason - why the tokens vanished.
wall_ms - p95 against your deadline slack.
tokens_per_accepted - the ratio you alert on.

Alert on the ratio, not on raw spend. Raw spend rises with useful work too.

When free capacity is the wrong bet

This is the part most write-ups skip. Free capacity has a cost, and it is usually wall time.
Use this table as a starting threshold set, then tune with your own ledger.

Signal over 7 days	Free lane is fine	Move work off the free lane
Retry rate	under 15% and flat	over 30%, or rising week over week
TPA vs paid baseline	within 1.2x	above 2x
p95 wall time	under 40% of deadline slack	approaching 70% of slack
Human review load	sampled review	every artifact needs review
Job side effects	idempotent, cheap to re-run	writes, payments, sends
Traffic shape	batch, tolerant of delay	interactive or user-facing

Two rows kill more free-lane plans than the rest. Non-idempotent writes and interactive latency.
No amount of saved tokens pays for a duplicate side effect.

Also price the human time. If a reviewer spends four minutes per artifact, TPA is not your biggest cost.

Limitations and who should not use this

Hard deadline SLOs under five minutes. Reservation adds a write and a settle.
Data that cannot leave your network. A hosted free tier is not the place to test it.
Teams without per-call usage numbers. You cannot settle what you cannot measure.
Jobs whose failures have side effects. Fix idempotency before you optimize spend.
Workloads where output length is unbounded. Reserve a hard max_output_tokens first.

This gate also cannot tell you whether an artifact is good. It only prices what you already accepted.

Rollback and cleanup

Ship the gate with a mode switch, and know the switch before you need it.

# shadow: log only, never block
BUDGET_MODE=shadow python runner.py
# enforce: block over-cap jobs
BUDGET_MODE=enforce BUDGET_CAP_PER_JOB=12000 python runner.py

Rollback is one environment change: set BUDGET_MODE=shadow and restart.
No code revert, no ledger migration, no data loss.

Then clean up:

Rotate ledger.jsonl daily. Sum the ratio, then archive.
Add a reaper for reservations older than your job timeout. Dead workers leak reservations.
Keep the fault-injection tests in CI. A retry change that skips the charge line is silent theft.

The one-line takeaway

Cap tokens per accepted artifact, not per request.
Then free capacity becomes a measured bet instead of a hopeful one.

If you want to reproduce the shadow week with no spend, the free model access and free server option are enough to collect the ledger above. Just verify the current terms yourself before you plan around them.

Price Free AI Capacity in Deadline Slack, Not in Tokens

Odd_Background_328 — Fri, 16 Oct 2026 04:46:24 +0000

It is 02:14. Your nightly batch window closes at 02:30. The free lane is still running, and the token meter reads zero.

Last week that felt like a win. Tonight the window is gone. Which operational action follows from that evidence?

Not delete the free lane. Price it.

Two numbers, one decision

Most teams measure the wrong number.

Token spend. Cheap, often zero on a free tier.
Deadline slack. Seconds you can spend before an SLO breaks.

Free capacity converts the first into the second. That conversion has a rate. Measure the rate, not the sticker price.

Define slack use per job class:

slack_used = free_p95_duration * (1 + retry_ratio) - paid_p50_duration

Three terms matter. The free lane p95 per attempt. The extra attempts per completed job. The paid baseline you gave up.

What the retry tax actually costs

A retry is not a retry. It is wall-clock time plus a new queue position plus a fresh attempt.

Preemption returns the worker mid-attempt. The partial work is gone.
Throttling returns a rate-limit signal. The partial work is gone.
Worker loss returns nothing at all. You wait for a timeout to notice.

Each event resets the attempt clock. Each event consumes slack, not tokens. That is how a free lane stays cheap and still blows the window.

If your provider returns 429, stop and back off. Do not retry around the limit.

The artifact: a break-even router

Here is a small script. It reads measured job-class statistics from one fixed window and prints a routing decision.

It does not call a model. It prices time.

#!/usr/bin/env python3
# breakeven.py - route batch job classes by deadline slack, not list price.
import json
import sys
from dataclasses import dataclass


@dataclass
class ClassStats:
    name: str
    attempts: int          # free-lane attempt starts in the window
    completed: int         # jobs that finished inside the class deadline
    free_wall_p50: float   # seconds per attempt, free lane
    free_wall_p95: float   # seconds per attempt, free lane
    paid_wall_p50: float   # seconds per job, reserved lane
    paid_cost: float       # currency units per completed job, reserved lane
    slack_budget: float    # seconds of negative slack this class may spend
    miss_penalty: float    # currency units charged when the deadline slips


def retry_ratio(s):
    if s.completed == 0:
        return None
    return max(s.attempts - s.completed, 0) / s.completed


def free_duration_p95(s):
    r = retry_ratio(s)
    return None if r is None else s.free_wall_p95 * (1.0 + r)


def slack_used(s):
    d = free_duration_p95(s)
    return None if d is None else d - s.paid_wall_p50


def miss_proxy(s):
    # Crude upper-bound proxy, not a fitted probability model.
    used = slack_used(s)
    if used is None:
        return 1.0
    if used <= s.slack_budget:
        return 0.0
    return min(1.0, (used - s.slack_budget) / max(used, 1e-9))


def route(s):
    used = slack_used(s)
    if used is None:
        return 'hold', 'no completed jobs in window'
    if used > s.slack_budget:
        return 'reserved', 'p95 slack use exceeds budget'
    risk = miss_proxy(s) * s.miss_penalty
    if risk >= s.paid_cost:
        return 'reserved', 'risk-adjusted free cost >= paid cost'
    return 'free', 'slack within budget and risk below paid cost'


def main(path):
    with open(path) as fh:
        rows = [ClassStats(**json.loads(l)) for l in fh if l.strip()]
    print('class             slack_used  budget  miss_p  free_risk  decision')
    for s in sorted(rows, key=lambda r: r.name):
        used = slack_used(s)
        decision, why = route(s)
        risk = miss_proxy(s) * s.miss_penalty
        used_s = 'n/a' if used is None else '%ds' % used
        print('%-17s %9s %6ds %7.2f %10.2f  %s' % (
            s.name, used_s, s.slack_budget, miss_proxy(s), risk, decision))
        print('%-17s reason: %s' % ('', why))


if __name__ == '__main__':
    main(sys.argv[1] if len(sys.argv) > 1 else 'jobs.jsonl')

Declared workload for the sample fixture

State the conditions before you quote any result.

Two job classes, 1,000 and 900 completed jobs in one 24h window.
Free lane: one worker pool, no reserved capacity, timing measured per attempt.
Paid baseline: p50 per job from the previous week on reserved capacity.
No model-quality comparison. This drill prices time, not accuracy.

Run it locally in under a minute

Create the fixture. These are sample values, not published measurements.

{"name":"nightly-embed","attempts":1180,"completed":1000,"free_wall_p50":38.0,"free_wall_p95":96.0,"paid_wall_p50":41.0,"paid_cost":0.42,"slack_budget":120.0,"miss_penalty":180.0}
{"name":"ocr-backfill","attempts":960,"completed":900,"free_wall_p50":22.0,"free_wall_p95":150.0,"paid_wall_p50":24.0,"paid_cost":0.11,"slack_budget":60.0,"miss_penalty":40.0}

Run the router:

python3 breakeven.py jobs.jsonl

Expected output on the fixture above. Label it expected until your own window reproduces it.

class             slack_used  budget  miss_p  free_risk  decision
nightly-embed            72s    120s    0.00       0.00  free
                  reason: slack within budget and risk below paid cost
ocr-backfill            136s     60s    0.56      22.36  reserved
                  reason: p95 slack use exceeds budget

The second class is the interesting one. Its per-attempt p95 is slower, and its retry ratio adds about 7 percent. Slack use lands at 136 seconds against a 60 second budget.

Free is the expensive lane here, and tokens never entered the calculation.

Failure injection: prove the ledger counts

A routing rule you never broke is a guess. Break the lane first.

# worker.py is a throwaway harness, not a model client.
python3 worker.py --class ocr-backfill --concurrency 4 &
WPID=$!
for i in 1 2 3; do
  sleep 60
  pkill -f 'worker.py --class ocr-backfill' -n
  sleep 5
  python3 worker.py --class ocr-backfill --concurrency 4 &
  WPID=$!
done
kill $WPID
python3 breakeven.py jobs.jsonl

Watch one thing: does a killed attempt show up as attempt=fail or does it vanish?

If it vanishes, your attempts counter is wrong. Your retry ratio is then wrong. Your routing decision is then wrong.

A minimal harness so the smoke test runs:

# worker.py - throwaway harness for the ledger drill only.
import argparse, random, time

p = argparse.ArgumentParser()
p.add_argument('--class', dest='cls', required=True)
p.add_argument('--concurrency', type=int, default=1)
p.add_argument('--attempt-seconds', type=float, default=30.0)
p.add_argument('--fail-rate', type=float, default=0.15)
a = p.parse_args()

while True:
    time.sleep(a.attempt_seconds)
    ok = random.random() > a.fail_rate
    print('%s attempt=%s' % (a.cls, 'ok' if ok else 'fail'), flush=True)

This harness proves your ledger counts. It proves nothing about any model. Swap in your real worker for that.

Telemetry you must emit

Without these fields the ledger is unreadable a week later.

job_class and route at admission time
attempt_index per job, starting at 0
attempt_started_at, attempt_ended_at
attempt_outcome: ok, preempted, timeout, throttled, worker_lost
deadline_at and slack_remaining_s at completion
tokens_in, tokens_out, even on a free tier
worker_id, so a bad host is visible

Keep tokens_in on free runs. You need it later to compute the exchange rate between free capacity and reserved capacity.

Decision table and threshold rationale

Evidence in the window	Route	Why
p95 slack use under 50 percent of budget	free	headroom absorbs the retry tax
p95 slack use between 50 and 100 percent	free, alert at 70 percent	one bad hour breaks it
p95 slack use over budget	reserved	measured breach, not a guess
risk-adjusted free cost at or above paid cost	reserved	free is the expensive lane
zero completed jobs	hold	no data, no route

Why 50 percent as a default? Because retry ratio is usually noisier than wall time. If your retry ratio p95 moves more than your duration p95, tighten the threshold.

If the miss penalty is contractual, do not run the free lane at any threshold. Slack budgets are for engineering tradeoffs, not for contracts.

Cleanup and rollback

Run these in order. Write them down before the drill, not after.

Drain the class. Set free-lane route weight to 0.
Wait for in-flight attempts, or cancel them and record worker_lost explicitly.
Restore the reserved weight from your config store, never from memory.
Keep the window JSONL file. It is the evidence for the next review.
Revert alert thresholds only after two clean windows.

Step 3 matters most. A rollback typed by hand at 02:20 is a second incident.

Where free capacity fits in this drill

The drill needs a free lane before you can price one. MonkeyCode is an open-source project with free model access; the operator states 10M free tokens and a free server option.

I treat both availability claims as operator-supplied. Limits and terms change, so confirm them on the project page before you plan around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In this workflow the free tier is exactly the lane whose retry ratio you are measuring. Point the harness at one small class, run one window, read the ledger. Start with one class and one window on the free tier, then decide.

Limitations and who should not use this

Be honest about the edges.

Not for synchronous user traffic with a hard p99. Free lanes are variable by nature.
Not for classes where a single miss is contractually unacceptable. Keep those reserved.
Not reliable under about 200 completed jobs per window. The p95 is noise below that; widen the window.
miss_proxy is a proxy, not a fitted model. Replace it with your own estimator if you have one.
Do not use this to justify free capacity against a regulated deadline without a named risk owner.

The threshold question

Every number here is a threshold with a rationale, and yours will differ. The useful question is not whether free capacity is cheap.

It is: how many seconds of slack does a retry buy, and who pays when the budget runs out?

Answer that with a measured window, and routing stops being a mood.

Measure Rework per Completed Job Before a Free Lane Ships

Odd_Background_328 — Thu, 15 Oct 2026 11:08:30 +0000

Measure Rework per Completed Job Before a Free Lane Ships

Last quarter a 400-job batch finished with a smaller token bill than the paid lane. It also missed the delivery window by 41 minutes.

Both numbers were true. The free lane spent fewer tokens per attempt. It also restarted roughly one job in nine.

Every restart paid queue time twice. Tokens got cheaper. Wall clock did not.

That is the trap with free capacity. You price the token bill and ignore rework amplification. This post gives you a ledger for the second number.

The topology I am describing

Three pieces, all reproducible on one host:

A router that assigns each job to a free or paid lane.
A worker that checkpoints every job and resumes after a restart.
A telemetry sink that writes one JSON line per attempt.

The router config decides more than the model choice does.

router:
  lanes:
    - name: free
      max_attempts: 2
      checkpoint: required
      idempotency_key: "{{ job_id }}"
    - name: paid
      max_attempts: 1
  admission:
    require_slack_s: 0     # set from measurement, not from vibes
    require_checkpoint: true

The idempotency_key is the whole safety story. Without it, a resumed job repeats its side effects.

The telemetry contract

One line per attempt, not per job. Attempts are what you pay for.

{"job_id":"b-1042","lane":"free","attempt":1,"enqueue_s":0,"start_s":812,"end_s":934,"outcome":"preempted","tokens_in":9100,"tokens_out":2100,"interventions":1}
{"job_id":"b-1042","lane":"free","attempt":2,"enqueue_s":0,"start_s":1204,"end_s":1330,"outcome":"ok","tokens_in":9100,"tokens_out":2050,"interventions":0}

Required fields: job_id, lane, attempt, enqueue_s, start_s, end_s, outcome, tokens_in, tokens_out, interventions.

interventions counts human touches. A retry that needs no operator is cheap. A retry that wakes someone at 02:00 is not.

The artifact: a break-even ledger

Save this as break_even.py. It answers one question: can this lane absorb the remaining time before the deadline?

#!/usr/bin/env python3
"""break_even.py - decide if a free lane can absorb a deadline.

Usage:
  python break_even.py telemetry.jsonl --slack-s 3600 --margin-s 300
"""
import argparse, json
from collections import defaultdict


def load(path):
    rows = []
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def p95(values):
    if not values:
        return 0.0
    ordered = sorted(values)
    idx = min(len(ordered) - 1, int(round(0.95 * (len(ordered) - 1))))
    return ordered[idx]


def summarize(rows, lane):
    attempts = [r for r in rows if r["lane"] == lane]
    if not attempts:
        return None
    jobs = {r["job_id"] for r in attempts}
    done = {r["job_id"] for r in attempts if r["outcome"] == "ok"}
    wall = sum(max(0.0, r["end_s"] - r["start_s"]) for r in attempts)
    waits = [max(0.0, r["start_s"] - r["enqueue_s"]) for r in attempts]
    runs = [max(0.0, r["end_s"] - r["start_s"]) for r in attempts]
    tokens = sum(r["tokens_in"] + r["tokens_out"] for r in attempts)
    return {
        "lane": lane,
        "completed": len(done),
        "completion_rate": len(done) / max(1, len(jobs)),
        "rework_multiplier": len(attempts) / max(1, len(done)),
        "s_per_completed_job": wall / max(1, len(done)),
        "p95_wait_s": p95(waits),
        "p95_run_s": p95(runs),
        "tokens_per_completed_job": tokens / max(1, len(done)),
        "interventions_per_100": 100.0 * sum(r.get("interventions", 0) for r in attempts) / max(1, len(done)),
    }


def required_slack_s(stats, margin_s):
    """Worst-case wait plus rework-amplified run time plus a margin."""
    return stats["p95_wait_s"] + stats["rework_multiplier"] * stats["p95_run_s"] + margin_s


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("telemetry")
    ap.add_argument("--slack-s", type=float, required=True, help="seconds left before the deadline")
    ap.add_argument("--margin-s", type=float, default=300.0)
    args = ap.parse_args()

    rows = load(args.telemetry)
    print(f"{'lane':6}{'done':>6}{'rework':>8}{'s/job':>8}{'p95 wait':>10}{'tok/job':>10}{'need':>9}  verdict")
    for lane in ("free", "paid"):
        s = summarize(rows, lane)
        if not s:
            continue
        need = required_slack_s(s, args.margin_s)
        verdict = "OK" if args.slack_s >= need else "TOO TIGHT"
        print(f"{lane:6}{s['completed']:>6}{s['rework_multiplier']:>8.2f}"
              f"{s['s_per_completed_job']:>8.1f}{s['p95_wait_s']:>10.0f}"
              f"{s['tokens_per_completed_job']:>10.0f}{need:>8.0f}s  {verdict}")


if __name__ == "__main__":
    main()

The formula is the point. A lane that reworks 1.3 attempts per job needs 30 percent more run time, not 30 percent more tokens.

Declared workload for the example run

State the workload or the numbers mean nothing. The example below assumes:

400 jobs, each producing 2 KB of text output.
Job sizes split 60/30/10 across small, medium, and large prompts.
One local worker per lane, no GPU, checkpoint store on local disk.
Deadline slack measured from batch start to the promised handoff.

What the output looks like

This is expected output from the declared workload. It is not a benchmark of any provider.

lane    done  rework   s/job  p95 wait   tok/job     need  verdict
free     364    1.28   118.4       905    41200    2116s  TOO TIGHT
paid     400    1.01    92.7       180    38900     499s  OK

Read the two rows together. The free lane is cheaper per token. It still needs four times the slack.

Failure drill: kill the worker mid-job

Never trust a checkpoint you have not broken. Run this drill locally first.

python runner.py --lane free --jobs jobs.jsonl --telemetry t.jsonl &
RUNNER=$!
sleep 25
kill -9 $RUNNER                    # simulate a preemption, not a graceful stop
python runner.py --lane free --jobs jobs.jsonl --telemetry t.jsonl --resume

Then verify three things:

The job restarts from its last checkpoint, not from zero.
The idempotency key stops duplicate writes. Query your dedupe table for the test key.
break_even.py shows rework above 1.0 for the free lane, and interventions_per_100 above zero.

If step 2 fails, the free lane is the wrong bet. Duplicate side effects cost more than the tokens you saved.

Observed versus inferred

Keep the two apart in your own notes.

Observed: rework multiplier, p95 wait, tokens per completed job, interventions.
Inferred: why the restarts happened, and whether a paid lane would also fail under the same load.

The ledger only settles the first category. Do not put the second one in an incident review as fact.

Decision table

Tune the thresholds with your own telemetry. The left column is the condition you check.

Condition	Route to	Reason
Slack >= required, checkpoint present	free lane	rework absorbs into spare time
Rework multiplier above 1.5	paid lane	each retry repays the queue
Job not idempotent	neither, fix first	duplicates are unrecoverable cost
Interventions per 100 jobs above 5	paid lane	operator time dwarfs token savings
Deadline under 30 minutes, no checkpoint	paid lane	no recovery path exists

Cleanup and rollback

A free lane needs a kill switch before it needs a budget.

export FREE_LANE_MAX_ATTEMPTS=2
export FREE_LANE_BUDGET_TOKENS=5000000
curl -sX POST localhost:8080/admin/drain -d '{"lane":"free"}'   # finish in-flight only

After the drill: drain the lane, wait for in-flight jobs, delete the test telemetry, and reset the dedupe table for the test keys. Roll back by setting require_slack_s back to the paid-lane value in the router config.

Where a free lane comes from

MonkeyCode is an open-source project that offers free model access and a free server option, per its operator. I am describing it here as the free lane under test, because the checkpoint-and-resume path is what you actually need to break before production. It does not change the method above, and it does not replace your own telemetry. Check the project's current limits before you plan a batch around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not use this approach

Teams with a hard real-time SLO. Rework of any size breaks the contract.
Jobs without checkpoints. You will repay the entire job after every restart.
Work with non-idempotent side effects, such as payments or emails without dedupe keys.
Batch pipelines where one operator hour costs less than one hour of duplicated compute.

The ledger is only worth building if you can act on the verdict. If you cannot route to a second lane, measure the rework anyway and negotiate the deadline instead.

If you want a lane to run this drill against, MonkeyCode's free model access and free server option are a reasonable starting point for the experiment above.