Meter the Wait, Not the Tokens: A Slot-Time Ledger for Free AI Lanes

09:12 — the token meter reads zero, the deadline does not

At 09:12 you enqueue 40 summarization jobs on the free lane.

At 09:14 the token meter still reads zero. Your cost dashboard looks perfect.

At 09:41 job 37 finally finishes. The batch deadline was 09:30.

That gap is not a token problem. It is a slot problem.

This is a composite drill, not a customer incident. Every number below reproduces locally with one Python file and a fixed seed, so you can argue with it instead of with me.

Token counters measure work, never cost

A counter only moves when work completes. Waiting is invisible to it.

Two slots and forty jobs means queue time dominates everything else. The real currency you spend is slot-seconds occupied, not tokens emitted.

Retries double the damage. A retry does not only spend tokens twice; it holds a concurrency slot twice.

Declared test conditions

Nothing below comes from a vendor. These are the declared inputs of a local simulation.

• Concurrency: 2 slots, FIFO admission.
• Base latency: 6.0 s per attempt. Declared, not measured against any provider.
• Jitter: gaussian, sigma 2.0 s, floor 0.5 s per attempt.
• Retry probability: 0.18 per attempt, maximum 3 attempts.
• Batch size: 40 jobs, seed 112, single process, no network calls.
• Output: one JSONL ledger, one summary block, plus a breaker decision.

Record attempts, not jobs

One ledger line per attempt is the minimum useful granularity. Aggregate per job later if you want.

• Identity: job_id, attempt, lane.
• Timing: enqueued_at, dispatched_at, finished_at.
• Occupancy: slot_seconds, attempt_seconds.
• Result: outcome, prompt_tokens, completion_tokens, breaker_state.

slot_seconds is the field most teams never store. It is also the only field that answers the cost question.

Run the drill locally

Save this as slot_ledger.py and run it. It needs no dependencies beyond the standard library.

#!/usr/bin/env python3
# Deterministic slot-time accounting drill for a 2-slot free lane.
import json
import random

CONCURRENCY = 2
BASE_LATENCY_S = 6.0
JITTER_SIGMA_S = 2.0
RETRY_PROB = 0.18
MAX_ATTEMPTS = 3
JOBS = 40
SEED = 112


def simulate():
    rng = random.Random(SEED)
    free_at = [0.0] * CONCURRENCY
    rows = []
    for job_id in range(JOBS):
        attempts, slot_s, first_s, wait_s, ok = 0, 0.0, 0.0, 0.0, False
        for attempt in range(1, MAX_ATTEMPTS + 1):
            slot = min(range(CONCURRENCY), key=lambda i: free_at[i])
            start = free_at[slot]
            dur = max(0.5, rng.gauss(BASE_LATENCY_S, JITTER_SIGMA_S))
            free_at[slot] = start + dur
            if attempt == 1:
                wait_s, first_s = start, dur
            attempts += 1
            slot_s += dur
            if rng.random() > RETRY_PROB:
                ok = True
                break
        rows.append({
            'job_id': job_id,
            'attempts': attempts,
            'wait_seconds': round(wait_s, 2),
            'slot_seconds': round(slot_s, 2),
            'first_attempt_seconds': round(first_s, 2),
            'ok': ok,
        })
    return rows, max(free_at)


def summarize(rows, makespan_s):
    ok = [r for r in rows if r['ok']]
    waits = sorted(r['wait_seconds'] for r in rows)
    slot_total = sum(r['slot_seconds'] for r in rows)
    first_total = sum(r['first_attempt_seconds'] for r in rows)
    return {
        'jobs': len(rows),
        'success_rate': round(len(ok) / len(rows), 3),
        'slot_seconds_total': round(slot_total, 1),
        'slot_seconds_per_success': round(slot_total / max(len(ok), 1), 2),
        'retry_slot_overhead_pct': round(100 * (slot_total / first_total - 1), 1),
        'p50_wait_seconds': waits[len(waits) // 2],
        'p90_wait_seconds': waits[int(len(waits) * 0.9)],
        'makespan_seconds': round(makespan_s, 1),
    }


if __name__ == '__main__':
    rows, makespan = simulate()
    with open('slot_ledger.jsonl', 'w', encoding='utf-8') as fh:
        for row in rows:
            fh.write(json.dumps(row) + '\n')
    metrics = summarize(rows, makespan)
    for key, value in metrics.items():
        print(f'{key:<28} {value}')

Expected output shape at seed 112

These values are illustrative for the declared inputs. Exact digits depend on your Python build, so compare shape and ratios, not the last decimal.

jobs                         40
success_rate                 0.875
slot_seconds_total           291.4
slot_seconds_per_success     8.33
retry_slot_overhead_pct      21.9
p50_wait_seconds             66.1
p90_wait_seconds             129.7
makespan_seconds             145.8

Look at p90_wait_seconds. It is 129.7 s on a lane whose per-attempt work is 6.0 s. Almost all your elapsed time went into standing in line.

Four thresholds, not one average

Averages hide the tail that misses deadlines. Gate on these four instead.

1. success_rate below 0.95: fix idempotency before anything else. Retrying a non-idempotent write is how you get duplicates.
2. slot_seconds_per_success more than 30 percent above BASE_LATENCY_S: retries are buying you nothing.
3. retry_slot_overhead_pct above 25: freeze new free-lane intake. Every new job lengthens everyone's wait.
4. p90_wait_seconds above half your deadline slack: the lane is the wrong bet for this batch.

Slice p90_wait_seconds by enqueue minute. A batch that starts fast and degrades by minute ten is a queue problem, not a model problem.

The admission gate

You cannot fix a queue you admit into blindly. Put the gate in front of the dispatcher.

HEADROOM = 0.5                  # free lane may consume at most half the slack
MAX_RETRY_SLOT_OVERHEAD = 25.0  # percent


def admit(deadline_slack_s, p90_wait_s, headroom=HEADROOM):
    return p90_wait_s <= headroom * deadline_slack_s


def lane_mode(metrics, slack_s):
    if not admit(slack_s, metrics['p90_wait_seconds']):
        return 'split'          # head to free lane, tail elsewhere
    if metrics['retry_slot_overhead_pct'] > MAX_RETRY_SLOT_OVERHEAD:
        return 'freeze'         # stop intake, drain what is queued
    return 'free'

Why half the slack? You need the other half for retries, artifact writes, and the reviewer who reads the output.

Retry amplification, expressed in slots

With a per-attempt failure rate p of 0.18, expected attempts per success are 1 / (1 - p), about 1.22. That matches the ledger overhead of roughly 22 percent.

So the formula you can defend in a review is simple: slot cost per success equals base attempt time divided by (1 - p). Raise p by making jobs larger and you pay twice, in tokens and in occupied slots.

Where the free options actually fit

Run the collector and the breaker loop on a small always-on host, and send the low-stakes classification steps through free model access. The free server option and free model access are available per the operator's own description, which is the only availability claim I am making here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the drill honest: the ledger in this article is local and simulated. Swap the simulator for your real dispatcher, keep the same four thresholds, and you have a measurement instead of an opinion.

Failure handling and rollback

1. Phase one, shadow: compute metrics, log lane_mode, change nothing.
2. Phase two, gate: enforce split only. Never start with freeze.
3. Phase three, freeze: reject new intake, drain the queue, alert the batch owner.
4. Rollback: unset the gate flag, restore the previous dispatcher image, keep the JSONL for the postmortem.
5. Cleanup: rotate slot_ledger.jsonl daily, drop lines older than seven days, and delete the drill file once the real ledger lands.

Use one environment variable as the kill switch. A kill switch you cannot find at 03:00 is not a kill switch.

Who should not route work this way

• Teams with a hard p99 contractual SLO. Queue time is not negotiable there.
• Workloads with data residency or audit-trail requirements on the compute host.
• Batches with large artifact egress. Slot accounting will not save you from bandwidth.
• Jobs that cannot be made idempotent. Fix the write path first, or retries will duplicate records.

If your deadline slack is under two times the observed p90 wait, free capacity is simply the wrong bet. Pay for the lane, or shrink the batch.

Start with the ledger, not the migration. If you want to run this drill end to end, the free model access and free server option are enough to collect your first forty rows.