A Burst-Timing Test: Does Free Inference Hold Up at Your Busiest Hour?

Free model access rarely collapses on average latency. It collapses inside your busiest ninety minutes, when everyone on the shared path wants the same GPU at the same time. So the only comparison worth running is a burst test scheduled at the hours you actually work, not a single quiet-afternoon benchmark. Below is a decision guide I use to choose between a free hosted lane, a free server running your own runtime, a paid hosted lane, and hardware you own.

The core claim is simple: pick your inference lane by how it behaves in your peak window, then by ops load, and only then by sticker price. Price per thousand tokens is a tiebreaker, not a starting point. If a lane cannot clear your floor at 16:00 on a Tuesday, its cheapness is a rounding error on the cost of a missed deadline.

1. Define the floor before you measure anything

A floor is the slowest acceptable round trip for one unit of work in your peak window, and it must be a number you can defend. For an inline coding assistant that a developer waits on, my default floor is 25 seconds to first complete answer for a small, fixed task. For batch jobs, the floor is throughput per hour instead of per-request latency. Write the number and the window into the harness config so it cannot drift.

Why does the window matter more than the number? Because free and shared capacity is a queue, and queues are shaped by other people's schedules, not yours. A p50 measured at 09:00 tells you almost nothing about 16:30. If you only have time for one measurement, measure the busy hour.

2. Build one harness, run it against every lane

The harness below sends the same prompt in fixed-size bursts at four local times, records wall-clock latency and retries per attempt, and appends to CSV. I keep the prompt small and deterministic so differences come from the lane rather than the task. The script is a template: I have not executed it against any provider for this article, and the numbers you get will be your own.

# burst_timing.py -- template, unexecuted here
import csv, json, os, time, urllib.request

BASE_URL = os.environ["LLM_BASE_URL"].rstrip("/")   # e.g. https://host/v1
API_KEY  = os.environ.get("LLM_API_KEY", "")
MODEL    = os.environ["LLM_MODEL"]                  # whatever the lane lists today
PROMPT   = "Write a Python function that reverses a linked list, plus one test."
BURSTS   = ["09:00", "13:00", "16:00", "22:00"]
REQS     = 8          # attempts per burst
FLOOR_S  = 25.0       # your floor from step 1
TIMEOUT  = 180
OUT      = "bursts.csv"

def call_once():
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT}],
        "temperature": 0,
    }).encode()
    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    req = urllib.request.Request(f"{BASE_URL}/chat/completions",
                                 data=body, headers=headers)
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
            json.loads(r.read())
        return round(time.monotonic() - t0, 4), True, ""
    except Exception as e:
        return round(time.monotonic() - t0, 4), False, type(e).__name__

with open(OUT, "a", newline="") as f:
    w = csv.writer(f)
    if f.tell() == 0:
        w.writerow(["ts", "burst", "attempt", "latency_s", "ok", "err"])
    for burst in BURSTS:
        for i in range(REQS):          # sequential on purpose: no self-inflicted queue
            lat, ok, err = call_once()
            w.writerow([time.strftime("%Y-%m-%dT%H:%M:%S"), burst, i, lat, ok, err])
            f.flush()
        time.sleep(5)

print("done")

Run it for three working days per lane. Swap lanes only through environment variables so the measurement code never changes:

export LLM_BASE_URL="https://lane-a/v1" LLM_MODEL="<listed-model>" LLM_API_KEY="..."
python burst_timing.py
# repeat with lane-b, lane-c, lane-d; one CSV per lane

3. Reduce the CSV to three numbers per burst

Latency alone is misleading, so reduce each burst to a pass rate against your floor, a p90 latency, and a retry rate. Those three numbers capture waiting, tail behavior, and flakiness in one line each.

# summarize.py -- template, unexecuted here
import csv, statistics, sys

floor = float(sys.argv[2])
rows = list(csv.DictReader(open(sys.argv[1])))
for burst in sorted({r["burst"] for r in rows}):
    b = [r for r in rows if r["burst"] == burst]
    ok = [float(r["latency_s"]) for r in b if r["ok"] == "true"]
    if not ok:
        print(burst, "all failed", len(b)); continue
    ok.sort()
    p90 = ok[min(len(ok) - 1, int(round(0.9 * (len(ok) - 1))))]
    passes = sum(1 for x in ok if x <= floor) / len(b)
    print(f"{burst}  pass={passes:.0%}  p90={p90:.1f}s  "
          f"retries={1 - len(ok)/len(b):.0%}  n={len(b)}")

The output line I care about most is not p90; it is the pass rate inside the busy burst. A lane that passes 95 percent at 09:00 and 50 percent at 16:00 is not a 72 percent lane, it is a lane with a hard scheduling constraint, and you should treat it that way.

4. Place each lane on the tradeoff table

Once you have three days of numbers, the decision usually stops being about speed and becomes about the shape of the failure. This table is the framework I apply after the measurements land.

Signal at your peak window	Free hosted model access	Free server + your runtime	Paid hosted	Self-hosted on owned hardware
Tail latency	Moves with shared load	Predictable once warm, but cold start is yours	Usually smoothed by capacity you pay for	Stable, bounded by your box
Cost shape	Zero spend, unknown queue	Zero spend, machine-limited	Linear with volume	Fixed, plus your maintenance hours
Ops load	Near zero	You own updates, restarts, disk	Low	Highest
Failure mode	Slow or rate-limited during peaks	Dies with the box	Bill grows quietly	Dies with the box, no support path
Data path	Leaves your network	Stays on the server you control	Leaves your network	Stays local
Exit cost	Low; switch endpoints	Medium; rebuild runtime	Low to medium	High; hardware and setup

Fit criteria, stated as rules rather than vibes. Choose free hosted access when your pass rate at your busiest burst stays at or above 90 percent and retries stay under 5 percent for three days. Choose the free server option when the same harness on that machine clears the floor and you have a named person who will patch it. Go paid when the busy-hour pass rate is fine but you need a contractual latency or higher concurrency. Go self-hosted when the data path, not the latency, decides the question.

MonkeyCode is relevant here because it advertises both of those free lanes, model access and a server option, which means one harness can compare them without a procurement step. I treat the free allowance as operator-stated rather than permanent, so I check the current terms and limits on the day I run the test instead of trusting a number from a screenshot.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

5. Set a graduation trigger instead of arguing later

Decision guides fail when they are one-time. Write the trigger down before you start: if the busy-burst pass rate drops below 80 percent on two of three working days, or retries exceed 10 percent, move the affected task class one lane over. That trigger makes the choice reversible and keeps the debate about data rather than preference.

Then re-run the counter-test. Change one thing, wait two days, and rerun the same bursts at the same hours. If the pass rate does not move, your original lane was never the problem, and you just saved yourself a migration.

Where this test is the wrong tool

Small samples lie, and this design uses eight attempts per burst, which is enough to catch a gross scheduling problem and not enough to prove a p99. If you have a hard latency SLO, regulated data that cannot leave your network, or a workload of one forty-minute generation per day, burst statistics will not help you and a contract or a local box will.

The free server lane also assumes someone owns maintenance; if nobody does, that machine becomes an unpatched mystery within a quarter. And no free tier is a commitment, so treat every measurement as a snapshot with an expiry date, not a capacity plan.

If you want to run the harness before you spend anything, the two free lanes are a reasonable place to start, and the useful thing you can bring back is your own burst table rather than a verdict.