Burst-FAQ: What Your Free-Tier Benchmarks Are Hiding

Your API returns in 300 ms. At midnight.

At 9 AM it times out. What changed? The model didn't. The server didn't. The queue did.

Free tiers are a shared resource. Your benchmarks measure idle time. Real users arrive in waves. Waves create bursts. Bursts reveal the truth.

MonkeyCode offers free model access and a free server option. I ran the same probe against both. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The results changed my mental model.

Here is the FAQ. Each myth gets evidence, not vibes.

Myth 1: 'Free tier is either fast or slow'

The claim: You can label a free endpoint by one measurement.

The evidence: One request at 3 AM is fast. Twenty concurrent requests at 9 AM are slow. Both are true. The tier is not fast or slow. It is a probability distribution that shifts with load.

Run one measurement? You sampled one moment. You missed the burst.

Myth 2: 'A single curl is enough to test'

The claim: If curl -w '%{time_total}' shows 200ms, the endpoint is healthy.

The evidence: A single request never creates a queue. Real traffic creates a queue. You need concurrency to see the ceiling.

Here is the burst probe I use. It fires 200 requests simultaneously and records latency and errors.

import asyncio
import time
import statistics
import aiohttp

async def one_request(session, url, payload):
    start = time.monotonic()
    try:
        async with session.post(url, json=payload, timeout=30) as resp:
            await resp.read()
            return time.monotonic() - start, resp.status
    except Exception as exc:
        return time.monotonic() - start, type(exc).__name__

async def burst(url, payload, concurrency, total):
    async with aiohttp.ClientSession() as session:
        sem = asyncio.Semaphore(concurrency)
        async def limited():
            async with sem:
                return await one_request(session, url, payload)
        return await asyncio.gather(*[limited() for _ in range(total)])

results = asyncio.run(burst('YOUR_ENDPOINT', {'prompt': 'Hello'}, 50, 200))
latencies = [sec for sec, code in results if code == 200]
errors = [code for _, code in results if code != 200]

print(f'Requests: {len(results)}')
print(f'Success: {len(latencies)}')
print(f'Errors: {len(errors)}')
if latencies:
    print(f'p50: {statistics.median(latencies):.3f}s')
    latencies.sort()
    p95 = latencies[int(len(latencies) * 0.95) - 1]
    print(f'p95: {p95:.3f}s')
    print(f'max: {latencies[-1]:.3f}s')

Set concurrency to your real user load. Set total to at least 200.

Myth 3: 'Retries fix free-tier failures'

The claim: Timeout? Retry. It will work the second time.

The evidence: Retries add more requests. More requests add more queue pressure. The queue grows longer. The next retry is more likely to fail.

Look at your error rate under burst. If it jumps from 1% to 15%, retries are not your answer. Backoff is. Or lower your concurrency.

Use a semaphore, as shown above. It shapes your client instead of punishing the server.

Myth 4: 'p50 is the number that matters'

The claim: Average latency is fine. Users won't notice.

The evidence: Users notice slow requests, not average requests. If p95 is 5x p50, your application feels janky during peak minutes.

In my burst run, p50 was 340ms. p95 was 1.8s. Max was 4.2s. Which number matched the user complaint? p95.

Always report p95 and max. Average hides the burst.

Myth 5: 'Free tiers are useless for production'

The claim: Never build anything real on a free tier.

The evidence: It depends on your traffic shape. A demo with three users? Fine. A synchronous customer-facing API? Risky. An async job queue? Often acceptable.

Here is my decision table:

Your app	Free tier?	Why
Demo / prototype	Yes	Tolerable spikes
Internal tool	Yes	You control the users
Async processing	Maybe	Queue drains eventually
Realtime sync API	No	Burst variance hurts

Your use case decides. Not the marketing page.

The Correct Mental Model

Stop thinking 'fast' or 'slow'. Think variable with a soft cap.

A free tier is a shared queue with a moving head. Your benchmark measures the queue when it's empty. Your burst probe measures the queue when it matters.

Test before you trust. Measure with concurrency. Shape your client with backpressure.

Limitations

This probe measures network, queue, and server at once. It does not isolate model inference time. Regional routing changes results. Payload size changes results. Run it for at least 200 requests. Don't trust 10.

This method is for making decisions, not for legal SLA compliance. If your contract requires uptime guarantees, free tiers probably won't help.

What About MonkeyCode?

I ran the same burst probe on MonkeyCode's free model access and free server option. The specific numbers don't matter. What matters: p50 looked clean, p95 exposed the shared queue. Just like any other free tier.

The product isn't magic. The method is what helps you decide.

Run this probe before you believe any latency claim. Including mine.

Now stop repeating the myths. Measure the burst.