Measure Rework per Completed Job Before a Free Lane Ships

Last quarter a 400-job batch finished with a smaller token bill than the paid lane. It also missed the delivery window by 41 minutes.

Both numbers were true. The free lane spent fewer tokens per attempt. It also restarted roughly one job in nine.

Every restart paid queue time twice. Tokens got cheaper. Wall clock did not.

That is the trap with free capacity. You price the token bill and ignore rework amplification. This post gives you a ledger for the second number.

The topology I am describing

Three pieces, all reproducible on one host:

1. A router that assigns each job to a free or paid lane.
2. A worker that checkpoints every job and resumes after a restart.
3. A telemetry sink that writes one JSON line per attempt.

The router config decides more than the model choice does.

router:
  lanes:
    - name: free
      max_attempts: 2
      checkpoint: required
      idempotency_key: "{{ job_id }}"
    - name: paid
      max_attempts: 1
  admission:
    require_slack_s: 0     # set from measurement, not from vibes
    require_checkpoint: true

The idempotency_key is the whole safety story. Without it, a resumed job repeats its side effects.

The telemetry contract

One line per attempt, not per job. Attempts are what you pay for.

{"job_id":"b-1042","lane":"free","attempt":1,"enqueue_s":0,"start_s":812,"end_s":934,"outcome":"preempted","tokens_in":9100,"tokens_out":2100,"interventions":1}
{"job_id":"b-1042","lane":"free","attempt":2,"enqueue_s":0,"start_s":1204,"end_s":1330,"outcome":"ok","tokens_in":9100,"tokens_out":2050,"interventions":0}

Required fields: job_id, lane, attempt, enqueue_s, start_s, end_s, outcome, tokens_in, tokens_out, interventions.

interventions counts human touches. A retry that needs no operator is cheap. A retry that wakes someone at 02:00 is not.

The artifact: a break-even ledger

Save this as break_even.py. It answers one question: can this lane absorb the remaining time before the deadline?

#!/usr/bin/env python3
"""break_even.py - decide if a free lane can absorb a deadline.

Usage:
  python break_even.py telemetry.jsonl --slack-s 3600 --margin-s 300
"""
import argparse, json
from collections import defaultdict


def load(path):
    rows = []
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def p95(values):
    if not values:
        return 0.0
    ordered = sorted(values)
    idx = min(len(ordered) - 1, int(round(0.95 * (len(ordered) - 1))))
    return ordered[idx]


def summarize(rows, lane):
    attempts = [r for r in rows if r["lane"] == lane]
    if not attempts:
        return None
    jobs = {r["job_id"] for r in attempts}
    done = {r["job_id"] for r in attempts if r["outcome"] == "ok"}
    wall = sum(max(0.0, r["end_s"] - r["start_s"]) for r in attempts)
    waits = [max(0.0, r["start_s"] - r["enqueue_s"]) for r in attempts]
    runs = [max(0.0, r["end_s"] - r["start_s"]) for r in attempts]
    tokens = sum(r["tokens_in"] + r["tokens_out"] for r in attempts)
    return {
        "lane": lane,
        "completed": len(done),
        "completion_rate": len(done) / max(1, len(jobs)),
        "rework_multiplier": len(attempts) / max(1, len(done)),
        "s_per_completed_job": wall / max(1, len(done)),
        "p95_wait_s": p95(waits),
        "p95_run_s": p95(runs),
        "tokens_per_completed_job": tokens / max(1, len(done)),
        "interventions_per_100": 100.0 * sum(r.get("interventions", 0) for r in attempts) / max(1, len(done)),
    }


def required_slack_s(stats, margin_s):
    """Worst-case wait plus rework-amplified run time plus a margin."""
    return stats["p95_wait_s"] + stats["rework_multiplier"] * stats["p95_run_s"] + margin_s


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("telemetry")
    ap.add_argument("--slack-s", type=float, required=True, help="seconds left before the deadline")
    ap.add_argument("--margin-s", type=float, default=300.0)
    args = ap.parse_args()

    rows = load(args.telemetry)
    print(f"{'lane':6}{'done':>6}{'rework':>8}{'s/job':>8}{'p95 wait':>10}{'tok/job':>10}{'need':>9}  verdict")
    for lane in ("free", "paid"):
        s = summarize(rows, lane)
        if not s:
            continue
        need = required_slack_s(s, args.margin_s)
        verdict = "OK" if args.slack_s >= need else "TOO TIGHT"
        print(f"{lane:6}{s['completed']:>6}{s['rework_multiplier']:>8.2f}"
              f"{s['s_per_completed_job']:>8.1f}{s['p95_wait_s']:>10.0f}"
              f"{s['tokens_per_completed_job']:>10.0f}{need:>8.0f}s  {verdict}")


if __name__ == "__main__":
    main()

The formula is the point. A lane that reworks 1.3 attempts per job needs 30 percent more run time, not 30 percent more tokens.

Declared workload for the example run

State the workload or the numbers mean nothing. The example below assumes:

• 400 jobs, each producing 2 KB of text output.
• Job sizes split 60/30/10 across small, medium, and large prompts.
• One local worker per lane, no GPU, checkpoint store on local disk.
• Deadline slack measured from batch start to the promised handoff.

What the output looks like

This is expected output from the declared workload. It is not a benchmark of any provider.

lane    done  rework   s/job  p95 wait   tok/job     need  verdict
free     364    1.28   118.4       905    41200    2116s  TOO TIGHT
paid     400    1.01    92.7       180    38900     499s  OK

Read the two rows together. The free lane is cheaper per token. It still needs four times the slack.

Failure drill: kill the worker mid-job

Never trust a checkpoint you have not broken. Run this drill locally first.

python runner.py --lane free --jobs jobs.jsonl --telemetry t.jsonl &
RUNNER=$!
sleep 25
kill -9 $RUNNER                    # simulate a preemption, not a graceful stop
python runner.py --lane free --jobs jobs.jsonl --telemetry t.jsonl --resume

Then verify three things:

1. The job restarts from its last checkpoint, not from zero.
2. The idempotency key stops duplicate writes. Query your dedupe table for the test key.
3. break_even.py shows rework above 1.0 for the free lane, and interventions_per_100 above zero.

If step 2 fails, the free lane is the wrong bet. Duplicate side effects cost more than the tokens you saved.

Observed versus inferred

Keep the two apart in your own notes.

• Observed: rework multiplier, p95 wait, tokens per completed job, interventions.
• Inferred: why the restarts happened, and whether a paid lane would also fail under the same load.

The ledger only settles the first category. Do not put the second one in an incident review as fact.

Decision table

Tune the thresholds with your own telemetry. The left column is the condition you check.

Condition	Route to	Reason
Slack >= required, checkpoint present	free lane	rework absorbs into spare time
Rework multiplier above 1.5	paid lane	each retry repays the queue
Job not idempotent	neither, fix first	duplicates are unrecoverable cost
Interventions per 100 jobs above 5	paid lane	operator time dwarfs token savings
Deadline under 30 minutes, no checkpoint	paid lane	no recovery path exists

Cleanup and rollback

A free lane needs a kill switch before it needs a budget.

export FREE_LANE_MAX_ATTEMPTS=2
export FREE_LANE_BUDGET_TOKENS=5000000
curl -sX POST localhost:8080/admin/drain -d '{"lane":"free"}'   # finish in-flight only

After the drill: drain the lane, wait for in-flight jobs, delete the test telemetry, and reset the dedupe table for the test keys. Roll back by setting require_slack_s back to the paid-lane value in the router config.

Where a free lane comes from

MonkeyCode is an open-source project that offers free model access and a free server option, per its operator. I am describing it here as the free lane under test, because the checkpoint-and-resume path is what you actually need to break before production. It does not change the method above, and it does not replace your own telemetry. Check the project's current limits before you plan a batch around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not use this approach

• Teams with a hard real-time SLO. Rework of any size breaks the contract.
• Jobs without checkpoints. You will repay the entire job after every restart.
• Work with non-idempotent side effects, such as payments or emails without dedupe keys.
• Batch pipelines where one operator hour costs less than one hour of duplicated compute.

The ledger is only worth building if you can act on the verdict. If you cannot route to a second lane, measure the rework anyway and negotiate the deadline instead.

If you want a lane to run this drill against, MonkeyCode's free model access and free server option are a reasonable starting point for the experiment above.