Queue Time Is a Line Item

You already meter what your jobs cost. Almost nobody meters how long each job sat still before anything happened — and in most AI pipelines that wait is the largest cost a single afternoon of instrumentation can find. This is a measurement recipe and a routing rule, not another retry story.

Token meters answer "what did the work cost". They do not answer "what did waiting cost", and for queued workloads the second number is usually bigger. A job that waits 40 seconds and runs for 3 spends 93% of its life in a queue. You will not see that in a token dashboard; you will see it in a timestamps file nobody is writing. Think of a restaurant kitchen: ingredient cost per plate is a fine metric, but if you never count how long each table sat empty, you can optimise the wrong number all the way to closing time.

Record four timestamps, nothing else

You need enqueued_at, started_at, finished_at, deadline_at, plus a route label and a token count if you have one. That is it. If you use a queue, three of those already exist somewhere; enqueued_at is the one people forget to propagate, and it is the only one that reveals the bill you are not looking at.

Write one JSONL record per job:

{"id":"j1","route":"free","enqueued_at":0.0,"started_at":12.5,"finished_at":40.1,"deadline_at":60.0,"tokens":8200}

Then a script that turns timestamps into a bill. This one is about forty lines, has no dependencies, and does one thing.

#!/usr/bin/env python3
"""wait_bill.py - turn job timestamps into a queue-time bill.

Reads JSONL, one record per job, all times in epoch seconds.
  {"id":"j1","route":"free","enqueued_at":0.0,"started_at":12.5,
   "finished_at":40.1,"deadline_at":60.0,"tokens":8200}

Usage: python wait_bill.py jobs.jsonl
"""
import json
import sys
from collections import defaultdict


def quantile(values, p):
    if not values:
        return float("nan")
    xs = sorted(values)
    idx = min(len(xs) - 1, max(0, int(round(p / 100 * (len(xs) - 1)))))
    return xs[idx]


def rows(path):
    with open(path, encoding="utf-8") as fh:
        for line in fh:
            line = line.strip()
            if line:
                yield json.loads(line)


def bill(jobs):
    lanes = defaultdict(list)
    for j in jobs:
        wait = j["started_at"] - j["enqueued_at"]
        service = j["finished_at"] - j["started_at"]
        lanes[j["route"]].append(
            {
                "wait": wait,
                "wall": wait + service,
                "on_time": j["finished_at"] <= j["deadline_at"],
                "tokens": j.get("tokens", 0),
            }
        )
    return lanes


def report(lanes):
    cols = ("lane", "n", "wait p50", "wait p90", "wait%", "late%", "tok/ok")
    header = "{:<12}{:>6}{:>10}{:>10}{:>8}{:>8}{:>10}".format(*cols)
    print(header)
    print("-" * len(header))
    for name, rs in sorted(lanes.items()):
        waits = [r["wait"] for r in rs]
        wall = sum(r["wall"] for r in rs) or 1.0
        on_time = [r for r in rs if r["on_time"]]
        tok_ok = sum(r["tokens"] for r in on_time) / len(on_time) if on_time else float("nan")
        late = 100 * (1 - len(on_time) / len(rs))
        print(
            "{:<12}{:>6}{:>10.1f}{:>10.1f}{:>7.0f}%{:>7.0f}%{:>10.0f}".format(
                name, len(rs), quantile(waits, 50), quantile(waits, 90),
                100 * sum(waits) / wall, late, tok_ok,
            )
        )


if __name__ == "__main__":
    report(bill(rows(sys.argv[1])))

If you have no production data yet, generate a synthetic file so you can prove the script runs end to end. The numbers it produces are fabricated on purpose and say nothing about any real system:

import json, random


def synth(path, n=2000, seed=7):
    rng = random.Random(seed)
    with open(path, "w", encoding="utf-8") as fh:
        for i in range(n):
            free = rng.random() < 0.7
            enq = i * 0.8
            wait = rng.expovariate(1 / (18 if free else 2))
            service = rng.expovariate(1 / (6 if free else 5))
            fh.write(json.dumps({
                "id": f"j{i}",
                "route": "free" if free else "committed",
                "enqueued_at": enq,
                "started_at": enq + wait,
                "finished_at": enq + wait + service,
                "deadline_at": enq + (45 if free else 30),
                "tokens": rng.randint(2000, 20000),
            }) + "\n")


synth("jobs.jsonl")

Read the bill before you optimise anything

wait% is the share of wall-clock time your jobs spend queued on that lane. late% is how often they miss the deadline. tok/ok is tokens spent per job that still landed on time, which is the only token ratio worth arguing about, because tokens burned on a job that missed its deadline are tokens you will spend again.

Two patterns deserve names. If wait% is high on a lane and late% is near zero, the lane is queue-bound but healthy and the work is genuinely deferrable. If wait% is high and late% is also high, you are not buying throughput, you are buying surprises, and no amount of retry logic repairs that. Measure first, because the fix differs.

Route cheapest-feasible, and let reject be a route

Once you have measured p90 wait per lane, routing becomes a cheap predicate instead of a vibe:

def route(job, lanes, now, safety=1.5):
    """Cheapest lane that can still make the deadline. Reject is a lane."""
    for name in ("free", "committed"):
        p90_wait = lanes[name]["p90_wait"]      # from wait_bill.py output
        p90_service = lanes[name]["p90_service"]
        if now + safety * p90_wait + p90_service <= job["deadline_at"]:
            return name
    return "reject"

The interesting branch is the last one. A job that cannot make its deadline on any affordable lane should fail fast and be re-planned by a scheduler or a human, not quietly occupy capacity another job could have used. reject is a cost control, not an error path, and treating it as one is the difference between a budget and a wish.

Where a free lane actually earns its place

Free model access and a free server option — both operator-supplied availability claims for MonkeyCode — fit the free slot in the predicate above: batch summarisation, overnight re-indexing, evaluation sweeps, anything whose deadline is soft and whose wait is genuinely cheap. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What free capacity does not do is shorten a queue tail. If your p90 wait is dominated by burst concurrency, a free lane absorbs the burst and hands you a worse tail than a capped queue would, so cap concurrency and measure again. Current limits, quotas and server details belong in the project's own documentation rather than in a blog post. I have therefore quoted no number here that I could not verify today, and you should treat any quota you read elsewhere the same way.

When free capacity is the wrong bet

Job class	Can it wait?	Cheapest sane lane	Why
Nightly batch summarisation	hours	free	soft deadline; wait costs almost nothing
PR review bot, 10-minute SLA	minutes	free first, escalate on measured p90	you need the escalation path, not a faster default
Interactive assistant	no	committed	the user experiences every queued second
5k-job fan-out burst	yes, but	free with a concurrency cap	uncapped, the tail eats the burst
Hard external deadline	no	committed plus reject	a missed deadline costs more than the tokens

Limitations, and who should not do this

Timestamps from different hosts drift; sync clocks or accept a few hundred milliseconds of noise. A p90 computed from thirty jobs is not a p90. This method tells you nothing about output quality, and nothing useful about capacity you already pay for up front, where wait is a sunk cost rather than a marginal one. If your workload is a single interactive request with a sub-second expectation, the queue is not your problem and this whole exercise is overhead. Same if you cannot yet emit enqueued_at — fix that first, because it is one line and it unlocks everything above.

Start by counting the wait you already have. Then decide which jobs deserve committed capacity and which can sit in a free lane until the queue reaches them. If you want somewhere to point the script while you calibrate, MonkeyCode's free model access and free server option are a reasonable place to run the batch-shaped half of a workload — verify the current details on the project's own pages, and let the measured p90 decide the rest.