Do the Deadline Math Before You Enqueue

Free capacity does not fail on price. It fails on the tail. The median request comes back fast enough that you stop worrying, and then one job in twenty sits in a slot for six minutes while your downstream step holds a lock and a human waits.

So stop pricing free capacity in tokens. Price it in the only currency the queue actually charges you: wall-clock hours of pipeline occupancy per accepted result. Once you have that number, admission control becomes arithmetic instead of vibes.

The rule I use is short: a job may enter free capacity only if its measured p90 completion time, inflated by its retry rate, still fits inside the deadline with a margin. Everything else gets routed somewhere with a contract. Here is how to get those three numbers without guessing.

Log attempts, not averages

Averages hide the tail, and the tail is the whole argument. You need one log line per attempt, with monotonic timestamps taken at submit, first byte, and completion, plus whether the result was actually accepted by your validator.

# measure.py - one JSONL line per attempt, no dependencies beyond stdlib
import json, os, time, urllib.request, uuid

ENDPOINT = os.environ["MC_ENDPOINT"]
KEY = os.environ["MC_KEY"]
LOG = "attempts.jsonl"

def call(prompt, timeout=180):
    body = json.dumps({"prompt": prompt}).encode("utf-8")
    req = urllib.request.Request(
        ENDPOINT,
        data=body,
        headers={"content-type": "application/json", "authorization": f"Bearer {KEY}"},
    )
    t_submit = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        t_first = time.monotonic()
        payload = json.loads(resp.read().decode("utf-8"))
    return t_submit, t_first, time.monotonic(), payload

def run(job_id, prompt, attempt, deadline_s):
    rec = {"job_id": job_id, "attempt": attempt, "deadline_s": deadline_s, "ok": False}
    try:
        t0, t1, t2, payload = call(prompt)
        rec.update(ttft_s=round(t1 - t0, 3), total_s=round(t2 - t0, 3),
                   tokens=payload.get("usage", {}).get("total_tokens"), ok=True)
    except Exception as exc:
        rec.update(error=type(exc).__name__, total_s=None)
    with open(LOG, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec) + "\n")
    return rec

Two details matter more than they look. deadline_s is written at submit time, not reconstructed later, so you can count missed deadlines honestly. And ok means your validator accepted the output, not that HTTP returned 200. A 200 with a truncated answer is a failure that still burned a slot.

Run the harness against the prompts you actually send, not a synthetic list. Fifty to a hundred samples is enough to see the shape; a handful is enough to fool yourself.

Turn the log into three numbers

You want p90 end-to-end time, retry amplification, and tokens per accepted result. Retry amplification is just total attempts divided by accepted results. If a fifth of your jobs need a second try, every slot you book is 20 percent larger than the job.

import json, statistics

rows = [json.loads(l) for l in open("attempts.jsonl")]
done = sorted(r["total_s"] for r in rows if r.get("total_s") is not None)
ak = sum(1 for r in rows if r["ok"])

def pct(xs, p):
    if not xs:
        return float("inf")
    k = max(0, min(len(xs) - 1, int(round((p / 100) * len(xs) + 0.5)) - 1))
    return xs[k]

print("attempts      ", len(rows))
print("accepted      ", ak)
print("p50_s         ", round(pct(done, 50), 2))
print("p90_s         ", round(pct(done, 90), 2))
print("amplification ", round(len(rows) / ak, 2) if ak else "n/a")
print("tok/accepted  ", round(sum(r.get("tokens") or 0 for r in rows) / ak, 1) if ak else "n/a")

That last line is the honest free-tier cost. Free tokens are not zero cost if you spend three attempts to get one accepted answer; the grant is consumed by the retries, and the clock is consumed by all three attempts whether they succeed or not. This is the same reason a retry is rent rather than insurance: you pay it every time the job runs, including the runs that work.

Where the free tier earns its place

This is the part where MonkeyCode is genuinely relevant to the method rather than bolted onto it. The harness above is worthless if sampling it costs more than the pipeline it measures, and the project advertises a free model tier plus a free server option, currently promoted with a large token grant (the exact figure moves, so read the current terms before you budget against it). That makes it a reasonable place to run the first hundred samples and to host the collector that writes attempts.jsonl.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Host the collector there, keep the log on a volume you control, and re-run the sampler whenever the free tier changes. If you would rather not run a second machine, the free server is enough for a cron job that posts fifty prompts a day and appends to a file.

One caution: a shared free server measures the vendor's contention as well as your code. Compare its p90 against a small paid box before you conclude anything about your own application.

The gate itself

With p90 and amplification in hand, admission is four lines. Ask for the job's deadline, compute the inflated p90, and refuse anything that does not fit inside 80 percent of the remaining budget. The missing 20 percent is your margin for the day the tail is worse than the sample.

def admit(deadline_s, p90_s, amplification, margin=0.20, in_flight=0, max_slots=4):
    if in_flight >= max_slots:
        return False, "no slot"
    expected = p90_s * amplification
    budget = deadline_s * (1 - margin)
    return expected <= budget, round(expected, 1)

Refusals are not failures. They are the queue telling you the truth before you have paid for it. Route the refused jobs to committed capacity, or shrink the job until it fits, or move the deadline. What you must not do is enqueue optimistically and discover the overrun at the downstream lock.

A useful side effect: the ratio of refused to admitted jobs is a leading indicator. When refusals climb, either your prompts grew, or the free tier's tail widened, or both. Either way you learn it from a log line instead of an incident.

When free capacity is the wrong bet

It is the wrong bet when the deadline is hard and the consequence of missing it is not yours to absorb: payment capture windows, anything a human is actively waiting on, and jobs whose side effects are not idempotent so a retry is not a retry but a duplicate. It is also wrong when the tail is invisible to you, because then no admission rule can fire.

Skip this whole approach if your traffic is steady and small enough that a fixed committed plan is cheaper than the engineering time to build the gate. A ledger nobody reads is more expensive than the overrun it was meant to catch.

For everyone else, the sequence is unglamorous and it works: log attempts with deadlines attached, compute p90 and amplification, admit only what fits, and treat the free grant as a place to learn the shape of the tail rather than a place to park production work. If you want to start measuring without paying for the first batch of samples, the free model access and free server are a fair place to begin.