Every Extra Hop Buys Another Queue Ticket

Free tokens do not make a multi-hop agent cheap. They make the invoice quiet while the wall clock gets loud. If your job fans out into tool calls, retrievals, or “one more check” loops, each hop is a fresh ticket in whoever is scheduling the model. The money line can stay at zero. The deadline still moves.

You already know how to count completion tokens. That habit fails the moment the work is a chain. A chain is not one request with a larger prompt. It is N separate admissions into a queue, N separate cold decisions, and N chances for backoff to stack. The happy-path token math never sees that.

Think of a deli that does not charge for the sandwich. You still take a number. Eight people in front of you is not a pricing problem. It is a calendar problem. An agent with eight tool hops on shared free capacity is that line, except the agent cannot see the other tickets and will cheerfully pull another one.

The hop is the unit, not the job

A single chat completion is one hop. A tool call that comes back and asks the model what to do next is two. A planner that fans out three searches, then a synthesizer, then a verifier is five if they run serially—and still five admissions if you thought “parallel” but the free pool serializes you anyway.

You should budget hops the way ops budgets packets. Loss and delay live on the hop. So does retry. A 429, a truncated tool JSON, a model that “helpfully” assumes a file exists: each of those is not a footnote. It is another ticket.

Write the identity down before you argue about vendors.

wall_clock ≈ hops × (queue_wait + inference + tool_io) + retry_tax
money      ≈ hops × token_price × tokens_per_hop
retry_tax  ≈ extra_hops × (queue_wait + inference)

When token_price is zero, people stop reading the first line. That is the mistake. Zero price deletes money. It does not delete queue_wait, and queue_wait is multiplied by hops you have not measured yet.

A small simulator you can actually run

The numbers below are labeled examples, not product benchmarks. Plug in waits you measured. If you have not measured, you do not have a cost model. You have a wish.

#!/usr/bin/env python3
"""Hop-budget a serial agent loop. Example inputs only — replace waits."""

from dataclasses import dataclass

@dataclass(frozen=True)
class HopBudget:
    name: str
    hops: int
    queue_wait_s: float   # measured p50/p95 per admission
    inference_s: float    # measured, not the brochure
    tool_io_s: float      # retrieval, HTTP, sandbox
    usd_per_hop: float    # 0.0 on a free path
    retry_rate: float     # 0.0–1.0 extra hops / hop

    def effective_hops(self) -> float:
        return self.hops * (1.0 + self.retry_rate)

    def wall_clock_s(self) -> float:
        per = self.queue_wait_s + self.inference_s + self.tool_io_s
        return self.effective_hops() * per

    def money_usd(self) -> float:
        return self.effective_hops() * self.usd_per_hop


def report(b: HopBudget, deadline_s: float) -> str:
    wc = b.wall_clock_s()
    slack = deadline_s - wc
    verdict = "fits" if slack >= 0 else "misses"
    return (
        f"{b.name:18} hops~{b.effective_hops():5.2f}  "
        f"wall={wc:7.1f}s  usd={b.money_usd():6.3f}  "
        f"slack={slack:7.1f}s  {verdict}"
    )


if __name__ == "__main__":
    deadline = 90.0  # your SLO, not a default
    free = HopBudget(
        name="free-serial",
        hops=6,
        queue_wait_s=8.0,   # EXAMPLE: replace
        inference_s=2.5,    # EXAMPLE: replace
        tool_io_s=0.8,
        usd_per_hop=0.0,
        retry_rate=0.25,
    )
    paid = HopBudget(
        name="paid-serial",
        hops=6,
        queue_wait_s=0.3,   # EXAMPLE: replace
        inference_s=1.8,
        tool_io_s=0.8,
        usd_per_hop=0.012,  # EXAMPLE: replace
        retry_rate=0.10,
    )
    hybrid = HopBudget(
        name="paid-critical",
        hops=3,             # pin planner+verifier
        queue_wait_s=0.3,
        inference_s=1.8,
        tool_io_s=0.8,
        usd_per_hop=0.012,
        retry_rate=0.10,
    )
    explore = HopBudget(
        name="free-explore",
        hops=3,             # park searches off the SLO path
        queue_wait_s=8.0,
        inference_s=2.5,
        tool_io_s=0.8,
        usd_per_hop=0.0,
        retry_rate=0.25,
    )
    print(f"deadline={deadline}s\n")
    for b in (free, paid):
        print(report(b, deadline))
    # Hybrid wall clock is the max of the pinned path and the parked path
    # only if you truly overlap them. Serial composition is the safe default.
    combo = hybrid.wall_clock_s() + explore.wall_clock_s()
    print(
        f"{'hybrid-serial':18} hops~{hybrid.effective_hops()+explore.effective_hops():5.2f}  "
        f"wall={combo:7.1f}s  usd={hybrid.money_usd()+explore.money_usd():6.3f}  "
        f"slack={deadline-combo:7.1f}s  "
        f"{'fits' if combo <= deadline else 'misses'}"
    )

Run it as a dry lamp test before you wire it to a provider.

python3 hop_budget.py

You should see the free path win on dollars and lose on slack whenever queue_wait_s is not tiny. That is the whole note. Change hops from 2 to 8 and watch the miss appear long before the invoice does.

If you want p95 instead of a point estimate, do not average waits in your head. Sample.

import random

def sample_wait(p50: float, p95: float) -> float:
    # crude mixture so a tail event can show up in a local rehearsal
    return p95 if random.random() < 0.05 else p50

random.seed(7)
waits = [sample_wait(4.0, 22.0) for _ in range(6)]
print("example hop waits_s", [round(w, 1) for w in waits], "sum", round(sum(waits), 1))

A tail wait on hop three is not “noise.” In a serial agent it delays hops four through six. Free capacity with a fat tail is a convoy. You do not average a convoy. You miss the boat or you don’t.

Measure the ticket, not the blog post

Wrap every model admission. Log four fields and refuse to debate without them: enqueue time, first-byte time, finish time, hop index. Hop index matters because hop 1 and hop 5 are different jobs even when they share a prompt template. Hop 1 is usually short context. Hop 5 is tools, residue, and a model that wants to “just fix it.”

import time
from contextlib import contextmanager

@contextmanager
def hop_timer(hop_index: int, sink: list):
    t0 = time.monotonic()
    marked = {"hop": hop_index, "enqueued_s": t0}
    try:
        yield marked
    finally:
        marked["elapsed_s"] = time.monotonic() - t0
        sink.append(marked)

You can fill queue_wait_s later if the API does not expose it. Elapsed minus inference is a blunt instrument, but it is better than a slogan. If elapsed grows while output tokens stay flat, you are paying the queue, not the model.

Command-line sanity check on your own logs, once you have JSON lines:

python3 - <<'PY'
import json, statistics, sys
rows = [json.loads(l) for l in sys.stdin if l.strip()]
by = {}
for r in rows:
    by.setdefault(r["hop"], []).append(r["elapsed_s"])
for hop in sorted(by):
    xs = by[hop]
    print(f"hop {hop} n={len(xs)} p50={statistics.median(xs):.2f}s max={max(xs):.2f}s")
PY

If hop 4’s p50 is several times hop 1, stop calling it “the same free model.” It is a different wait profile attached to a heavier prompt. Treating them as one bucket hides the tax.

When free capacity is the right bet anyway

Overnight evals, prompt sweeps, offline red-team dumps, and anything that can sit behind a deadline measured in hours still belong on a free path. The queue is a scheduler. Schedulers are good at work that does not have a customer in the room.

Interactive agents are the opposite. A user-facing copilot that does three retrievals and a patch proposal is a fan-out machine. So is a CI bot that must comment before the next push. Spare, free, bursty capacity is a poor fit there even when the token meter reads zero. You are buying variance. Variance is the product.

A hybrid is often the adult move: pin the hops that sit on the SLO—plan, final patch, verifier—on a path with boring waits. Park the hops that invent options—extra searches, style rewrites, “try another test file”—on free capacity, and only merge them if they arrive before the pin path finishes. If you cannot overlap, do not pretend. Serial hybrid is just two queues in a trench coat.

This is where a sandbox with free model access and a free server option is useful as a measurement fixture, not as a promise. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option; use them to capture hop timers on non-critical sweeps, then copy the same instrumentation onto the path you actually ship. Do not treat an unmetered queue as production SLO. If you try that workflow, steal the script above first and fill it with your waits, not anyone else’s.

Limitations, and who should skip this

The simulator assumes a serial loop. If you truly run tools in parallel against a pool that does not serialize, wall clock collapses toward max(hop) plus join time. Prove that with timestamps. Many “parallel” agent graphs still enqueue one completion at a time behind a global rate limit. The graph was parallel. The scheduler was not.

The script also assumes retries are a constant fraction. Real retries cluster: a bad tool schema fails three hops in a row. Clustered retries turn retry_rate=0.25 into a cliff. If you cannot bound extra hops with a kill switch, the model is incomplete and you should not use it to green-light a launch.

Do not use this approach if your product needs a hard p95 under a few seconds and you have no paid, reserved, or otherwise boring path. Free admission is the wrong control loop for that job. Do not use it if you cannot log hop index. Blended latency will lie to you. Do not use it to justify zero spend to a finance partner without showing slack; you will win the token argument and lose the outage argument.

Skip it for one-shot completions with no tools. One hop is a sandwich, not a line. The queue still exists, but the multiplier that makes this note worth reading is gone.

The core conclusion does not change with the week’s agent glossary. Count hops. Meter wait. Keep free capacity on the work that can wait, and stop calling a silent invoice a cheap system. A cheap system is one that still has slack after the last hop comes back.