Stop Lane Thrash: Pin Jobs After the First Ambiguous Retry

02:14 — the token spend alert fires.

Spend is up 90% over the same window yesterday. Accepted artifacts are up 2%. Nothing in the error budget explains the gap.

The queue has two lanes: a free lane for model work, and a paid fallback. Jobs bounce between them. Every bounce looks like a cheap retry in the dashboard.

This post is about the bill you cannot see in the token graph. It is the rework caused by a job that succeeds twice.

What Lane Thrash Looks Like

Lane thrash is a job that crosses lanes more than once per artifact. The usual cause is an ambiguous outcome.

The free lane times out after it already generated output. Your client sees a failure. Your retry sees a fresh job.

Two artifacts arrive later with the same inputs. Both look valid. Both get accepted.

You now pay tokens twice and review work twice. The metric that moves is not spend. It is duplicate artifacts per hundred jobs.

Topology: One Queue, Two Lanes

Keep one queue. Do not build two queues for two lanes.

• A single admission point assigns a lane per attempt.
• Every attempt writes one ledger row before it calls a model.
• The ledger row carries an Idempotency-Key derived from job inputs.
• Artifact writes are keyed by that same value.

That last bullet is the control that actually stops duplicates. Lane pinning only reduces how often you need it.

Config You Can Copy

# lane.yaml
lanes:
  free:
    max_attempts: 2
    pin_on: [ambiguous, permanent]
  paid:
    max_attempts: 1
reconcile:
  late_completion_window_s: 900
  dedup_key: Idempotency-Key
alerts:
  cross_lane_retries_per_job: 0.20
  duplicate_artifacts_per_100_jobs: 1
free_lane_enabled: true

The late_completion_window_s value matters. A slow free-lane response that lands after your client timed out is a duplicate unless you reconcile inside that window.

A Ledger That Answers the Right Question

CREATE TABLE job_attempt (
  job_id      TEXT NOT NULL,
  idem_key    TEXT NOT NULL,
  attempt_no  INTEGER NOT NULL,
  lane        TEXT NOT NULL CHECK (lane IN ('free','paid')),
  started_at  TEXT NOT NULL,
  ended_at    TEXT,
  outcome     TEXT,   -- ok | retryable | ambiguous | permanent
  tokens_in   INTEGER DEFAULT 0,
  tokens_out  INTEGER DEFAULT 0,
  artifact_id TEXT,
  PRIMARY KEY (job_id, attempt_no)
);

CREATE INDEX idx_artifact ON job_attempt(artifact_id);

CREATE VIEW job_rollup AS
SELECT job_id,
       MAX(idem_key)                   AS idem_key,
       SUM(lane = 'free')              AS free_attempts,
       SUM(lane = 'paid')              AS paid_attempts,
       SUM(tokens_in + tokens_out)     AS tokens,
       COUNT(DISTINCT artifact_id)     AS artifacts
FROM job_attempt
GROUP BY job_id;

Two rollup queries carry the whole decision:

-- tokens per unique artifact, plus cross-lane pressure
SELECT ROUND(SUM(tokens) * 1.0 / SUM(artifacts), 1)   AS tokens_per_artifact,
       ROUND(SUM(free_attempts > 0 AND paid_attempts > 0) * 1.0
             / COUNT(*), 3)                          AS cross_lane_per_job
FROM job_rollup;

-- the duplicates themselves
SELECT idem_key, COUNT(DISTINCT artifact_id) AS n
FROM job_attempt
WHERE artifact_id IS NOT NULL
GROUP BY idem_key
HAVING n > 1;

The boolean-sum form avoids FILTER, so it runs on older SQLite builds. Check your version before you paste.

The Router: Pin on Ambiguous, Retry on Retryable

The failure class decides the lane. Not the retry count alone.

# lane_router.py — stdlib only
PIN_ON = {"ambiguous", "permanent"}
MAX_FREE_ATTEMPTS = 2

def next_lane(free_attempts: int, pinned: bool, last_outcome: str) -> tuple[str, bool]:
    """Return (lane, pinned) for the next attempt."""
    if pinned:
        return "paid", True
    if last_outcome in PIN_ON:
        return "paid", True
    if free_attempts >= MAX_FREE_ATTEMPTS:
        return "paid", True
    return "free", False

retryable means 429 or 503 with no work done. Retry it on the free lane once. It is cheap and safe.

ambiguous means a timeout after partial streaming. The work may exist. Pin that job to paid and reconcile by idem_key.

Three tests pin the behavior down:

def test_pin_on_ambiguous():
    assert next_lane(1, False, "ambiguous") == ("paid", True)

def test_retryable_stays_free_once_more():
    assert next_lane(1, False, "retryable") == ("free", False)

def test_budget_exhausted_pins():
    assert next_lane(2, False, "retryable") == ("paid", True)

Fault-Inject the Free Lane Locally

Do not discover ambiguity handling in production. Point the router at a local fake lane.

# fake_free_lane.py — port 8099, stdlib only
import itertools, time
from http.server import BaseHTTPRequestHandler, HTTPServer

SCRIPT = itertools.cycle(["ok", "429", "503", "slow"])

class FreeLane(BaseHTTPRequestHandler):
    def do_POST(self):
        step = next(SCRIPT)
        if step == "slow":
            time.sleep(35)          # exceeds the 30s client timeout
            step = "ok"
        self.send_response({"ok": 200, "429": 429, "503": 503}[step])
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b'{"artifact": "synthetic"}')

    def log_message(self, *args):
        pass

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8099), FreeLane).serve_forever()

Run it, replay 300 recorded jobs against it, then read the rollup. The slow branch is the one that manufactures duplicates.

Run the 300-Job Drill

Declared workload for the numbers below:

• 300 synthetic jobs, one artifact each.
• Free-lane outcomes: 25% retryable, 5% ambiguous, 2% permanent.
• Synthetic token constants: 800 per free attempt, 1,200 per paid attempt.
• Reconciliation window 900s, dedup on idem_key.

Expected output from the synthetic model, not a measurement of any vendor:

Metric	Pin + dedup	Retry free, no pin
Unique accepted artifacts	300	300
Free attempts	375	411
Paid attempts	46	6
Total tokens	355,200	336,000
Tokens per accepted artifact	1,184	1,120
Duplicate artifacts	0	15
Cross-lane retries per job	0.15	0.02

Read the Rollup, Not the Invoice

Look at the token row first. The two policies differ by about 5%.

That gap will not justify an on-call change. The duplicate row will.

Fifteen duplicates at two minutes of review each is thirty minutes of avoidable human work. At volume, that dwarfs the token delta.

So the operational question is not "which lane is cheaper per token." It is "which lane produces fewer unrecoverable artifacts."

Thresholds and Why They Are These Numbers

• cross_lane_retries_per_job > 0.20 — above one in five jobs crossing lanes, the reconciliation window is probably too short.
• duplicate_artifacts_per_100_jobs > 1 — one duplicate per hundred is already a review tax you can measure.
• Free-lane p95 completion above half your deadline slack — the lane cannot drain before cutoff.

Set the slack threshold from your own p95, not from a blog number. Recompute it after every model change.

When the Free Lane Is the Wrong Bet

Condition	Route
Deadline slack p95 < 2× free-lane p95 latency	Stay on paid
Ambiguous rate > 2% and no idempotency key at the boundary	Fix the key first
Sustained arrival above measured free throughput for 30 min	Split by priority
Token delta between lanes < 10%	Decide on rework and latency
Batch deadline ≥ 6h and outputs idempotent	Free lane first, pin on ambiguous

That fourth row is the one most teams get wrong. When tokens converge, the cost argument disappears and the recovery argument takes over.

Where the Free Capacity Comes From

I run this drill against MonkeyCode's free model access, with the free server option hosting the queue worker and ledger. The operator lists 10M free tokens and a free server option. Verify the current terms on the project page before you route production work.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is open source, so the ledger and router above stay portable. Nothing in the drill depends on vendor-specific APIs.

Run the 300-job drill against the free tier before you move real traffic. The duplicate count is the number you actually care about.

Rollback and Cleanup

Rollback takes three steps, in this order:

1. Set free_lane_enabled: false and let the queue drain. Do not kill in-flight jobs.
2. Reconcile the last 900s of attempts by idem_key before deleting anything.
3. Replay pinned jobs on paid with the original keys.

Cleanup after the drill:

pkill -f fake_free_lane.py
rm -f drill.db
ss -ltnp | grep 8099   # confirm the port is free

Keep the ledger for 30 days. It is the only record that proves how many duplicates you produced.

Who Should Not Use This

• Teams without an idempotency key at the API boundary.
• Workloads with strict p99 latency SLOs and no slack to spend.
• Jobs whose partial output cannot be reconciled, such as side-effectful writes.
• Multi-tenant systems that need hard fairness between tenants.
• Anyone who needs a permanence guarantee on free capacity.

If you are in the first group, stop here. Build the key first. The lane router is worthless without it.