Route by Slack, Not by Price

Free capacity is only cheap when your deadline is longer than its queue. That single comparison decides more of your bill than any per-token price. Everything below is a way to measure the queue and then choose a pool per request instead of per month.

Agent loops make this bite. One user question fans out into a planner call, three tool calls, a summarizer, and a repair pass when JSON comes back malformed. On a token sheet each call looks affordable. What actually arrives at the other end is wall-clock time, and someone is sitting there waiting for it.

Separate the queue from the work

Most dashboards collapse a request into one latency number. That number is useless for routing, because it hides whether you are paying in queue time or in generation time. A free pool with a long queue and fast generation and a committed pool with no queue and slow generation look identical once you average them.

So log four timestamps per call: submitted, accepted by the pool, first token, last token. The gap between submitted and accepted is your queue. The gap after that is the work.

# timestamps.py -- illustrative, not executed
import time

def timed_call(submit, deadline_monotonic):
    t0 = time.monotonic()
    handle = submit()                    # returns once the pool accepts the job
    t1 = time.monotonic()               # accept / dequeue
    stream = handle.stream()
    first = next(stream)
    t2 = time.monotonic()               # first token
    rest = first + "".join(stream)
    t3 = time.monotonic()               # last token
    return rest, {
        "queue_s": t1 - t0,
        "first_token_s": t2 - t1,
        "stream_s": t3 - t2,
        "slack_at_entry_s": deadline_monotonic - t0,
    }

Convert every deadline to the same monotonic clock at request entry, or the slack column will drift with clock adjustments and quietly lie to you.

Turn the numbers into a lane

A lane is a pool plus the numbers you measured for it. You do not need a scheduler. You need a comparison.

# lanes.py -- illustrative, not executed
from dataclasses import dataclass

@dataclass(frozen=True)
class Lane:
    name: str
    queue_p95_s: float
    service_p95_s: float
    retries_per_call: float      # measured, not guessed

    def expected_s(self, safety: float = 0.25) -> float:
        base = self.queue_p95_s + self.service_p95_s
        return base * (1.0 + self.retries_per_call) * (1.0 + safety)


def admission(slack_s, free, committed):
    if slack_s >= free.expected_s():
        return free
    if slack_s >= committed.expected_s():
        return committed
    return None   # shed or degrade: queueing here just spends the caller's deadline

The third branch is the one people skip. If neither lane fits, waiting is not a strategy, it is a slow failure that also costs tokens. Shedding, returning a cheaper partial result, or telling the caller to poll later is honest. Silently adding four seconds is not.

Note the retry term inside the expectation. A retry inherits the remains of the original deadline, not a fresh one, so a call that failed once is a worse candidate for the slower pool than the same call was before it failed. That asymmetry is easy to miss when retries live in a wrapper nobody reads.

Check it before you believe it

The routing rule is only as good as the percentiles under it, so run the same prompt set twice: once sending everything to the slow lane, once routed by slack. Compare deadline miss rate per lane and total calls, not just average latency. Averages hide exactly the tail that breaks user trust.

# harness.py -- illustrative, not executed
import asyncio, time

async def one(prompt, deadline_s, route, lanes, submit):
    t0 = time.monotonic()
    lane = route(deadline_s, lanes["free"], lanes["committed"])
    if lane is None:
        return {"lane": None, "miss": True, "calls": 0}
    text, usage = await submit(lane.name, prompt)
    return {"lane": lane.name,
            "miss": (time.monotonic() - t0) > deadline_s,
            "calls": usage.get("attempts", 1)}

async def sweep(prompts, deadline_s, route, lanes, submit, n=200):
    results = await asyncio.gather(
        *(one(prompts[i % len(prompts)], deadline_s, route, lanes, submit) for i in range(n))
    )
    return results

Run it with two flags and the same seed, and read the miss rate per lane rather than a combined score.

# illustrative invocations; substitute your own submit() implementation
python harness.py --route all-free      --n 200 --deadline 4
python harness.py --route by-slack      --n 200 --deadline 4

If the routed run has the same miss rate as the committed-only run while pushing a real share of calls to the slow lane, you have earned the right to route. If the miss rate climbs, your percentiles were optimistic, and no amount of prompt tuning will fix a queue you did not measure.

The decision, written down

Workload	Slack at entry	Route	Reason
Interactive turn, human typing	shorter than the fast lane's p95	committed	the slow lane's queue alone eats the budget
Agent tool call, human still waiting	roughly twice the slow lane's p95	slow lane	slack absorbs one tail spike
Nightly re-summarize of a backlog	hours	slow lane	the deadline is a maintenance window, not a person
Retry after a failed attempt	already halved by the first try	committed	the retry inherited a shorter deadline
Post-incident backfill	hours, very high volume	slow lane, throttled	volume is the risk, not latency

Read the table as a template, not a recommendation. Your numbers belong in the middle column.

Where the free tier fits, and where it does not

This is the point in the workflow where an offer like the one around MonkeyCode becomes testable rather than aspirational. The operator-supplied terms are free model access and a free server option, with 10 million free tokens quoted for the free tier; the project is presented as open source. Treat those as claims to check on the project's official page, including the license, which models are covered, and whether anything expires.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The reason the free server option matters for a cost note is subtler than the token count. A free pool changes what you spend: you trade money for queue position. That trade is excellent for a batch job whose deadline is a maintenance window, and bad for a request a person is watching. Free capacity is a loan against your latency, and something has to repay it.

A one-paragraph version of when free is the wrong bet: whenever the measured slow-lane wait plus retry inflation exceeds the slack you actually have. Not the slack you wish you had. Not the average slack. The slack on the request in front of you.

Who should skip this

If your total spend is small enough that routing logic costs more engineering time than it saves, skip it. If you have no per-request telemetry, skip it until you do, because a router fed guessed percentiles is worse than no router. And if you need hard guarantees about availability, retention, or long-term free tiers, do not build a deadline class on top of something you have not read the terms of. Verify first, then route.

Limitations worth stating plainly: the snippets above are illustrative and were not executed for this article. Percentiles go stale when traffic shape changes, so re-measure on a schedule. A shared free pool can move under you in ways your own logs lag behind. And the strictest deadline class always wins the fast lane, which means the fast lane never gets cheaper no matter how much you route around it.

If you want your own slack numbers instead of mine, the free model access and free server option are a reasonable place to collect real queue-wait data cheaply. Measure first. Migrate second.