Charge Wait Time to the Same Job

You can close a job under the token budget and still miss the ship window. Shared free capacity does not fail like an empty wallet. It fails like a standby line that never called your name.

Tokens remaining answer a different question than the one your deadline asks. The grant says you may speak. The queue says when. Those two clocks drift the moment anyone else on the same pool starts a burst.

Treat that drift as a cost of the job, not as weather. If you only log prompt size and completion tokens, you will keep green dashboards for work that arrived late. Late work is not free. It spent the deploy slot, the review hour, or the nightly window you cannot buy back.

A standby boarding pass is the right picture. Nobody charged you the fare. Nobody promised a seat on this flight either. An agent loop on spare capacity is the same ticket: permission without a reservation. Headroom in the grant is not a gate assignment.

Agent pipelines make the mismatch louder. Each tool call is a fresh arrival in a queue you do not own. A cheap prompt that retries twice is not merely twice the tokens. It is twice as exposed to someone else's burst sitting in front of you. Head-of-line delay never appears on a token invoice. It appears on the wall clock you already promised to a human.

Serial tool use is the amplifier. One lookup, one patch, one test command, one summary: four queue admissions for a single user-visible result. If any hop waits, the whole chain waits. Parallel fan-out looks faster until the slowest hop holds the merge. You did not overspend tokens. You overspent minutes you were not measuring.

So stop arguing about average token price before you have a wait sample. Price without delay is a brochure. Delay without a budget line is how “we were under quota” becomes “the job did not ship.”

Here is a small, labeled example you can run locally. It invents no vendor quotas, no model names, and no hardware. You bring your own observed waits. The script charges idle seconds to the same job that spent the tokens, then tells you whether the remaining deadline still fits the p95 wait you actually saw.

#!/usr/bin/env python3
"""dual_clock.py — charge wait time to the same LLM job.

Example only. Feed it waits you measured. It does not call a model.
"""
from __future__ import annotations

import argparse
import json
import math
import sys
from pathlib import Path


def percentile(xs: list[float], p: float) -> float:
    if not xs:
        raise ValueError("need at least one wait sample")
    ys = sorted(xs)
    idx = min(len(ys) - 1, max(0, math.ceil(p / 100.0 * len(ys)) - 1))
    return ys[idx]


def load_samples(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text().splitlines():
        line = line.strip()
        if not line:
            continue
        rows.append(json.loads(line))
    return rows


def charge(rows: list[dict], deadline_s: float, work_s: float) -> dict:
    waits = [float(r["queue_wait_s"]) for r in rows]
    latencies = [float(r.get("model_latency_s", 0.0)) for r in rows]
    tokens = [int(r.get("tokens_in", 0)) + int(r.get("tokens_out", 0)) for r in rows]
    hops = max(1, int(rows[-1].get("hop", 1))) if rows else 1

    p50 = percentile(waits, 50)
    p95 = percentile(waits, 95)
    token_total = sum(tokens)
    wait_total = sum(waits)
    # Remaining path still has to admit `hops` times on a similar queue.
    projected_s = hops * (p95 + (sum(latencies) / len(latencies) if latencies else 0.0))
    projected_s += work_s
    late = projected_s > deadline_s
    return {
        "attempts": len(rows),
        "hops": hops,
        "tokens_total": token_total,
        "wait_s_total": round(wait_total, 3),
        "wait_p50_s": round(p50, 3),
        "wait_p95_s": round(p95, 3),
        "deadline_s": deadline_s,
        "projected_s": round(projected_s, 3),
        "under_token_count": True,  # this file does not know your grant
        "late_if_you_stay": late,
        "decision": "divert" if late else "stay",
    }


def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--samples", type=Path, required=True)
    p.add_argument("--deadline-s", type=float, required=True)
    p.add_argument("--work-s", type=float, default=15.0,
                   help="non-model work left: tests, patch, upload")
    args = p.parse_args()
    rows = load_samples(args.samples)
    result = charge(rows, args.deadline_s, args.work_s)
    json.dump(result, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 2 if result["late_if_you_stay"] else 0


if __name__ == "__main__":
    raise SystemExit(main())

Pair it with a JSONL file you actually wrote down. Do not invent the waits. A throwaway session that records queue time next to token counts is enough. The shape looks like this.

{"hop": 1, "queue_wait_s": 4.1, "model_latency_s": 2.2, "tokens_in": 812, "tokens_out": 140}
{"hop": 2, "queue_wait_s": 19.8, "model_latency_s": 3.0, "tokens_in": 901, "tokens_out": 88}
{"hop": 3, "queue_wait_s": 41.2, "model_latency_s": 2.7, "tokens_in": 940, "tokens_out": 210}

Those three lines are a labeled illustration, not a benchmark. Notice hop three did not explode tokens. It exploded wait. If your nightly window has 90 seconds left and local tests still need 15, the projector will call that path late even though the token column looks polite.

python3 dual_clock.py --samples waits.jsonl --deadline-s 90 --work-s 15; echo exit:$?

Exit 2 means stay-on-spare is the wrong bet for this deadline. Exit 0 means the wait samples you have still fit. Either way you charged idle seconds to the job instead of hiding them under “queueing, ignore.”

You still need a way to capture queue_wait_s. Wrap the call so the wait is visible even when the vendor SDK collapses it into one latency number. A crude shell pattern is enough to start. Time the blocked open separately from the bytes-on-the-wire if your client exposes both. If it does not, log the gap between “request object built” and “first token received.” That gap is mostly queue plus cold start. It belongs on the job.

python3 - <<'PY'
import json, time, urllib.request
# Labeled stub: replace the URL with your own client.
# Measure build -> first-byte. Do not pretend this is model quality.
t0 = time.monotonic()
req = urllib.request.Request("https://example.invalid/v1/not-a-real-call", method="POST")
try:
    urllib.request.urlopen(req, timeout=2)
except Exception:
    pass
print(json.dumps({"queue_wait_s": round(time.monotonic() - t0, 3)}))
PY

Put that logger next to retries, not inside the happy-path dashboard. A retry is a new admission. If you average it away, you will keep riding a pool that is fine at p50 and fatal at p95. Deadlines live in the tail. So should the divert rule.

The divert rule should be boring. If projected wait plus remaining local work exceeds the window, you leave the spare pool. You do not “try one more hop” because the grant still has tokens. Tokens do not open a slot. A slot is a reservation, a paid route, a machine you started, or a decision to cut scope. Spare capacity is a lab and a backlog absorber. It is a bad production scheduler.

If you need a scratch box to collect those samples without standing up your own host, MonkeyCode’s free model access and free server option can run the logger and the projector. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use that box to time the wait. Do not park a customer cron there and call the grant a capacity plan.

Write the two clocks into the same record. One field for tokens in and out. One field for seconds blocked before work began. One field for hops already admitted. When someone asks “were we over budget,” you can answer with both currencies. Under on tokens and over on minutes is still over.

Keep the decision local and testable. The function above does not need a network. Feed it three lines and assert the divert. That is the whole unit test: a late tail must not hide behind a small token total.

def test_tail_wait_diverts():
    rows = [
        {"hop": 1, "queue_wait_s": 5, "model_latency_s": 2, "tokens_in": 100, "tokens_out": 20},
        {"hop": 2, "queue_wait_s": 40, "model_latency_s": 2, "tokens_in": 120, "tokens_out": 20},
        {"hop": 3, "queue_wait_s": 48, "model_latency_s": 2, "tokens_in": 120, "tokens_out": 20},
    ]
    out = charge(rows, deadline_s=60, work_s=15)
    assert out["tokens_total"] < 1000
    assert out["decision"] == "divert"

Limitations are sharp. This projector assumes the next hops look like the last ones. A quiet Sunday sample will lie on Monday morning. It also treats p95 of a tiny file as if it were a distribution. Five rows are a smell, not a study. It cannot see preemption after you are already running. It cannot see a queue that reorders you because your prompt is long. And it cannot convert wait into money unless you already know what an hour of that deadline is worth.

Do not use this approach for on-call pages, user-facing chat, regulated data, or anything that cannot tolerate jitter. Those jobs need a reservation, isolation, and a kill switch that does not consult a grant balance. Do not put secrets on a shared free server while you “just measure.” Measurement is not an excuse to mix traffic classes.

Also do not use it as a reason to disable retries. Retries are still how you survive a single blip. You budget them on both clocks. A retry that cannot finish before the window is not resilience. It is a second idle wait charged to a job that already lost.

The operational habit is small. Before you argue about model choice, dump the JSONL. If wait dominates, changing temperature will not save the release. Shorten the hop count, cut the serial chain, or leave the spare pool. If tokens dominate and wait is tiny, then you have a prompt problem, not a scheduler problem. Most teams mix the two and tune the wrong knob.

Free capacity is still useful. It is useful for probes, for scoring a prompt, for proving the logger, for work that may slip. It becomes expensive the moment you pretend a grant is a reservation. Charge the idle seconds. Then decide with both clocks in the same sentence.