Routing Agent Jobs on p95 Time-to-First-Token, Not Average Latency

A missed deadline the averages never showed

A two-person team runs three agent jobs from one laptop: inbox triage that must land inside forty seconds, a log summariser that can take three minutes, and a nightly report with a nine-minute window. On most days all three finish early. On the day the laptop suspended mid-run, triage resumed into a stale socket, retried twice, and delivered at 4 minutes 12 seconds. The dashboard reported mean latency of 3.1 seconds for the same period. This scene is a composite rather than a measured customer case, but the shape is familiar: the mean stayed green because the tail was rare, and the tail was the only part with a deadline.

The fix is not a faster model. It is to treat routing as admission control over jobs with deadlines, and to rank candidate lanes by a tail statistic instead of an average.

Why p95 time-to-first-token is the useful unit

Time-to-first-token (TTFT) covers everything a caller waits through before any useful output exists: queueing, cold start, connection setup, and prompt prefill. Total completion time still matters, but for streaming agents the perceived stall is TTFT, and the tail is where deadlines break. A lane advertising a 900 ms mean can hide a 6 s p95, which means a job with a 5 s budget fails roughly once in twenty runs while every average-based chart stays flat.

Measure the distribution per lane and per prompt-size class, and keep the raw samples on disk. Two lanes that look identical on averages routinely differ by a factor of three at p95 once cold starts are included.

Step 1: Probe each lane and store a distribution

The probe below records TTFT samples for streaming chat completions against any OpenAI-compatible endpoint. It uses only the standard library, so it runs on the same restricted box as the agent itself. Collect at least fifty samples per lane before trusting the quantiles.

# probe_ttft.py - measure TTFT samples for one lane
import json, os, time, urllib.request

BASE   = os.environ['LANE_BASE_URL']        # e.g. https://host/v1
KEY    = os.environ.get('LANE_API_KEY', '')
MODEL  = os.environ['LANE_MODEL']
PROMPT = 'Return the word ready and nothing else.'

def one_probe(timeout=30.0):
    body = json.dumps({'model': MODEL, 'stream': True,
                       'messages': [{'role': 'user', 'content': PROMPT}]}).encode()
    req = urllib.request.Request(
        f'{BASE}/chat/completions', data=body,
        headers={'Content-Type': 'application/json',
                 'Authorization': f'Bearer {KEY}'})
    t0 = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        for raw in resp:
            line = raw.decode('utf-8', 'replace').strip()
            if line.startswith('data:') and '[DONE]' not in line:
                return time.monotonic() - t0
    return time.monotonic() - t0

def quantile(xs, q):
    xs = sorted(xs)
    if not xs:
        return None
    return xs[min(len(xs) - 1, int(round(q * (len(xs) - 1))))]

if __name__ == '__main__':
    samples, errors = [], 0
    for _ in range(int(os.environ.get('LANE_SAMPLES', '50'))):
        try:
            samples.append(one_probe())
        except Exception:
            errors += 1
        time.sleep(0.2)
    print(json.dumps({'lane': os.environ['LANE_NAME'], 'n': len(samples),
                      'errors': errors, 'p50': quantile(samples, 0.50),
                      'p95': quantile(samples, 0.95),
                      'max': max(samples, default=None)}))

Run it once per candidate lane and keep the error count alongside the quantiles. A lane that times out on ten percent of probes is not a fallback; it is a second outage arriving on schedule.

Step 2: Turn measurements into a lane scorecard

The scorecard adds the fields a probe cannot measure: whether the lane can hold a secret, and whether it survives with no network. The values below are illustrative placeholders, not benchmarks.

# lanes.py - replace every number with your own probe output
LANES = [
    {'name': 'local',       'p95_ttft_s': 1.4, 'retry_rate': 0.05, 'handoff_s': 0.2,
     'offline_capable': True,  'can_hold_secret': True,  'rank': 0},
    {'name': 'free_server', 'p95_ttft_s': 2.6, 'retry_rate': 0.20, 'handoff_s': 0.8,
     'offline_capable': False, 'can_hold_secret': False, 'rank': 1},
]

This is where a hosted lane earns its place. The operator describes MonkeyCode as an open-source project offering free model access and a free server option, along with a 10M-token free allowance; I have not independently audited those terms, so check the project's current pages before depending on them. What matters for routing is narrower: a free server lane adds a third column to the scorecard that costs nothing when idle, and that idle capacity is exactly what a deadline-aware controller needs. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 3: Encode the policy as an admission controller

The controller answers one question per job: which lane can meet this deadline with the least expected cost. Expected cost multiplies p95 by the observed retry rate and adds the handoff penalty, which penalises flaky lanes far more honestly than a mean does.

# admit.py - choose a lane for a job with a deadline
SAFETY_MARGIN_S = 1.0

def eligible(lane, job):
    if job['secret_class'] == 'restricted' and not lane['can_hold_secret']:
        return False
    if job['requires_offline'] and not lane['offline_capable']:
        return False
    return True

def expected_cost(lane):
    return lane['p95_ttft_s'] * (1 + lane['retry_rate']) + lane['handoff_s']

def choose(job, lanes):
    budget = job['deadline_s'] - SAFETY_MARGIN_S
    ranked = sorted((l for l in lanes if eligible(l, job)),
                    key=lambda l: (expected_cost(l), l['rank']))
    for lane in ranked:
        if expected_cost(lane) <= budget:
            return lane['name'], expected_cost(lane)
    return None, None   # refuse early instead of missing silently

Two behaviours matter more than the ranking rule. A job carrying a restricted credential can never be admitted to a lane with can_hold_secret = False, regardless of how much deadline headroom exists. And when no lane fits, the controller returns None immediately, so the caller degrades its output instead of discovering the miss four minutes later.

Step 4: Log every lane decision

An admission decision that is not logged cannot be audited. Emit one line per job with the chosen lane, the expected cost, the budget, and the eventual outcome.

python probe_ttft.py >> ttft_samples.jsonl
python -c 'from admit import choose; from lanes import LANES; \
print(choose({"deadline_s": 40, "secret_class": "restricted", \
"requires_offline": False}, LANES))'

After a week, compare the logged expected_cost against the real completion time. A persistent gap in one direction means the scorecard is stale, and re-probing that lane is cheaper than debugging a mystery deadline miss later.

The tradeoff, stated plainly

Job property	Local-first wins when	Free server lane wins when
Latency	p95 stays under budget once the model is warm	local p95 exceeds the deadline on cold start
Secrets	credentials must never leave the machine	the job carries no restricted material
Offline	work must continue with no network	connectivity is assumed and monitored
Capacity	job volume fits one device at night	bursts outrun the device, and idle capacity is free

The table is not a recommendation to move everything. It is a filter: most jobs are local-first, a narrow band of stateless, network-tolerant, secret-free jobs benefits from an external lane, and the controller should make that band explicit rather than accidental.

Where this breaks, and who should skip it

Single-sample probing produces noise, so anyone unwilling to schedule repeated probes will get a scorecard that misroutes confidently. Lanes with wildly bimodal latency need separate distributions per prompt-size class, which doubles the bookkeeping. Teams already running a real queue with priority and preemption should use that scheduler instead, because a sixty-line controller duplicates its weakest parts.

Skip this approach entirely if your jobs have no deadlines, if every job carries restricted credentials, or if you cannot store and review per-lane samples. The controller is a deadline tool, not a general load balancer.

The script and the scorecard format run unchanged against any OpenAI-compatible endpoint, including the free tier the operator provides, so a first comparison costs one afternoon and fifty probes per lane.