Does a Free Tier Survive a 90-Minute Batch? Pre-Register the Thresholds, Then Spike

Free tiers are easy to try and hard to trust. A demo proves a request works. It does not prove the tier holds under a real batch.

My rule is simple. Time-box the question to 90 minutes. Write the pass/kill thresholds before you run anything. Then let the logs decide.

This post is that protocol. It targets one narrow question: can free model access plus a free server option carry a small batch job? The harness is a template you run yourself.

The hypothesis must be falsifiable in one sentence

Bad hypothesis: "the free tier seems fast enough." That cannot fail, so it teaches nothing.

Good hypothesis: "A 200-item batch at concurrency 4 completes in under 20 minutes, with error rate below 2% and p95 latency under twice the median."

One claim, three numbers, one deadline. If the run misses, the answer is kill or extend, not "it depends."

Why a spike beats a benchmark post

Evaluation fatigue is real. Long model comparisons age badly and rarely match your traffic. A spike has a different goal.

It answers a decision you actually face this week. Ship on the free tier, or pay for something with an SLA? That is a budget question, not a leaderboard question.

A spike also expires honestly. Run it again next month, and the numbers may change. That is a feature, not a flaw.

Pre-register these thresholds first

Write them down before the first request. Otherwise you will rationalize whatever you see.

Signal	Pass	Extend	Kill
Error rate	< 2%	2–8%	> 8%
p95 latency	< 2x median	2–4x median	> 4x median
Throughput at 2x concurrency	flat or better	drops < 30%	drops > 30%
Failure shape	isolated timeouts	retryable errors	hard rate-limit storm
Operator effort	zero babysitting	one fix	manual restarts

Five signals, one table, no debate after the run. Copy it into the repo before you start.

Stage the 90 minutes into three blocks

Time-boxing only works if the stages are separate.

1. Smoke, 5 minutes. Ten sequential requests. Confirm auth, response shape, and log writing.
2. Ladder, 40 minutes. Concurrency 1, 2, 4. Sixty items each. Stop early on a kill signal.
3. Soak, 45 minutes. Hold concurrency 4 and stream the remaining items. Watch for drift.

If stage 1 fails, stop. Debugging an auth error at concurrency 8 is wasted time.

The harness is stdlib only

The script below uses no SDK and no retries. Retries hide the failure rate you are trying to measure.

# spike_harness.py — unexecuted template; set env vars before running
import json, os, time, statistics, urllib.request, urllib.error
from concurrent.futures import ThreadPoolExecutor, as_completed

BASE = os.environ["BASE_URL"]        # e.g. OpenAI-compatible endpoint
MODEL = os.environ["MODEL"]          # fill in from the provider's docs
KEY = os.environ["API_KEY"]
OUT = os.environ.get("LOG_PATH", "spike.jsonl")

def one(item_id, prompt, timeout=60):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 128,
    }).encode()
    req = urllib.request.Request(
        BASE, data=body, method="POST",
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    t0 = time.perf_counter()
    code, err = 200, None
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            raw = r.read()
            code = r.status
    except urllib.error.HTTPError as e:
        code, err, raw = e.code, f"http_{e.code}", b""
    except Exception as e:
        code, err, raw = 0, type(e).__name__, b""
    ms = (time.perf_counter() - t0) * 1000
    return {"item": item_id, "status": code, "error": err,
            "latency_ms": round(ms, 1), "bytes": len(raw),
            "ts": round(time.time(), 3)}

def run(items, concurrency, prompt="Reply with the word ok."):
    rows = []
    with open(OUT, "a") as f, ThreadPoolExecutor(concurrency) as pool:
        futures = [pool.submit(one, i, prompt) for i in items]
        for fut in as_completed(futures):
            row = fut.result()
            rows.append(row)
            f.write(json.dumps(row) + "\n")
            if row["status"] == 429:
                raise SystemExit("rate limited: stop, do not retry around the limit")
    return rows

def summarize(rows):
    ok = [r["latency_ms"] for r in rows if r["status"] == 200]
    ok.sort()
    p = lambda q: ok[min(len(ok) - 1, int(len(ok) * q))] if ok else None
    return {"n": len(rows), "ok": len(ok),
            "err_rate": round(1 - len(ok) / len(rows), 3),
            "median_ms": round(statistics.median(ok), 1) if ok else None,
            "p95_ms": p(0.95), "max_ms": ok[-1] if ok else None}

if __name__ == "__main__":
    items = list(range(int(os.environ.get("N", "60"))))
    for c in (1, 2, 4):
        s = summarize(run(items, c))
        print(c, json.dumps(s))
        if s["err_rate"] > 0.08:
            print("kill signal reached, stopping ladder")
            break

Run it in three commands, one per stage:

N=10  python spike_harness.py   # smoke
N=60  python spike_harness.py   # ladder
N=200 python spike_harness.py   # soak, watch LOG_PATH grow

Read the log, not the dashboard

The one-line JSON log is the artifact. It survives the spike and answers later questions.

• status and error give you the failure taxonomy for free.
• latency_ms per item shows drift across the soak window.
• bytes catches truncated responses that still returned 200.

Bucket latencies by minute. A flat median with a rising p95 is a queue problem, not a model problem.

Tag every non-200 with one of four codes: timeout, http_429, http_5xx, bad_body. Mixed causes need different fixes.

Where the free options fit

This spike exists because provisioning is the expensive part. A free server option removes the setup step. Free model access removes the billing step.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That is the whole role: a low-friction place to run the first 90 minutes. I am not quoting token quotas or hardware here. Those numbers change, and a stale number is worse than no number. Check the project page for current limits before you plan a budget around them.

If the spike passes, keep the same harness pointed at a paid endpoint. The logs stay comparable, so the upgrade decision is arithmetic.

What this spike cannot tell you

• Single machine, single network. Your laptop is not production egress.
• One prompt shape. Real prompts vary in length and cost per call.
• No retry policy. A pass here means the raw path is healthy, not that it is resilient.
• No SLA. Free tiers can change capacity without notice, and that is legitimate.
• No quality signal. Latency and error rate say nothing about output correctness.

Add a separate quality check before you trust outputs. This harness only measures whether the pipe holds.

Do not use this approach if

• You need contractual uptime or audit trails.
• You process regulated data that cannot leave your boundary.
• Your workload needs tight rate-limit guarantees at high concurrency.
• You expect the spike to replace load testing against real traffic shapes.

In those cases, pay for an SLA and write a proper load test. A 90-minute spike is a decision tool, not a warranty.

The 90-minute checklist

1. Write the hypothesis with three numbers.
2. Copy the threshold table into the repo.
3. Record the start time and stop at 90 minutes.
4. Run smoke, ladder, soak in that order.
5. On a 429, stop. Do not retry around the limit.
6. Summarize per stage, then apply the table mechanically.
7. Ship, extend by 45 focused minutes, or kill and move on.

The value is not the number you get. It is that you decide once, with evidence, and then go build.

If you want to try the protocol without wiring a GPU box, the free server option is the cheapest entry point. Read the current limits first, then start the clock.