Your Free Tier Is Not a Capacity Plan: Five Throughput Myths and a Concurrency Ramp

A demo handles one request at a time. A launch sends ten at once. The endpoint falls over.

Was the model worse today? Or did nobody measure the shape of the curve?

I keep seeing the same five claims about free models and free servers. Each one sounds reasonable. Each one falls apart under a ten-minute test.

I run these probes against MonkeyCode's free model access and its free server option while prototyping.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Nothing below depends on that relationship. Swap in any endpoint you control. The method is the point.

Myth 1: "Latency grows linearly with concurrency"

If one call takes two seconds, ten calls take twenty. Right?

No. That is a queue, not a multiplier. Little's Law says concurrency equals arrival rate multiplied by time in system. When arrivals outrun service capacity, waiting time stops being bounded.

The curve has a knee. Left of it, latency is flat. Right of it, latency goes vertical and errors appear.

You cannot read the knee in a doc page. You find it by ramping.

Myth 2: "Retries add capacity"

Retries feel like resilience. Under saturation they are a load multiplier.

Say a third of calls fail and you retry twice. Offered load can nearly double. At the knee, that turns a brownout into an outage.

A corrected model: retry only classified transient failures. Use exponential backoff with jitter. Keep a retry budget, something like ten percent of successful calls.

Primary sources worth reading: AWS on backoff and jitter and the SRE book chapter on handling overload.

Myth 3: "It is free, so the numbers do not matter"

Free tiers make people skip measurement. "It is just a prototype."

But the shape transfers. The knee moves. The existence of a knee does not.

A prototype that dies at ten users teaches you nothing. A prototype with a written envelope teaches you where the queue belongs.

Myth 4: "A 429 or a timeout means the service is broken"

A 429 is admission control talking. It says your offered load exceeded what the server will accept.

A timeout says your deadline was shorter than time to first byte.

Neither one is a bug report. Both are signals to slow down, queue, or shed.

Make 429 a first-class state in the client. Do not swallow it as an exception.

Myth 5: "The model is the slow part"

Can you prove that? Most people cannot.

Split each request into a time budget:

1. Client prep: serialize the prompt, sign the request.
2. Connection: DNS, TCP, TLS.
3. Time to first byte: queue wait plus prefill.
4. Generation: first token to last token.
5. Client parse: decode, validate, store.

Without all five numbers, "the model is slow" is a guess with good manners.

The artifact: a concurrency ramp

Here is the probe. It is a template, not a run. I am not publishing numbers from it.

Stop rule first. Stop a level when either condition fires:

• p95 exceeds three times the level-1 baseline.
• Error rate passes five percent.

# ramp.py - template only. Point it at an endpoint you control.
import asyncio, time
import httpx

URL = 'https://your-endpoint.example/v1/chat'
LEVELS = [1, 2, 4, 8, 16, 32]
PER_LEVEL = 24
P95_STOP = 3.0
ERR_STOP = 0.05

async def call(client, sem, rows):
    async with sem:
        t0, ttfb = time.perf_counter(), None
        try:
            async with client.stream('POST', URL, json={'input': 'ping'}) as r:
                async for _ in r.aiter_bytes():
                    if ttfb is None:
                        ttfb = time.perf_counter()
                rows.append({'ok': r.status_code < 400, 'code': r.status_code,
                             't0': t0, 'ttfb': ttfb, 'end': time.perf_counter()})
        except Exception as e:
            rows.append({'ok': False, 'code': type(e).__name__,
                         't0': t0, 'ttfb': None, 'end': time.perf_counter()})

def summarize(rows, wall):
    lat = sorted((r['end'] - r['t0']) * 1000 for r in rows)
    q = lambda p: lat[min(len(lat) - 1, int(p * len(lat)))]
    errs = sum(1 for r in rows if not r['ok'])
    return {'rps': round(len(rows) / wall, 2),
            'p50_ms': round(q(0.50)), 'p95_ms': round(q(0.95)),
            'err': round(errs / len(rows), 3)}

async def main():
    async with httpx.AsyncClient(timeout=60) as client:
        base = None
        for level in LEVELS:
            sem, rows = asyncio.Semaphore(level), []
            start = time.perf_counter()
            await asyncio.gather(*[call(client, sem, rows) for _ in range(PER_LEVEL)])
            s = summarize(rows, time.perf_counter() - start)
            base = base or s['p95_ms']
            print(level, s)
            if s['p95_ms'] > P95_STOP * base or s['err'] > ERR_STOP:
                print('stop: knee near level', level)
                break

asyncio.run(main())

Reading the output

Three fields matter more than the rest:

• rps tells you whether more concurrency bought throughput.
• p95_ms tells you where waiting started.
• err tells you where the server pushed back.

If rps flattens while p95 climbs, you are past the knee. More concurrency is now just more waiting.

The decision table

Zone	What you see	What to do
Left of the knee	p95 flat, errors near zero	Raise the cap slowly, keep the ramp in CI
At the knee	p95 jumps 2-3x	Set your cap at half the knee
Right of the knee	p95 vertical, 429s or timeouts	Queue, shed, or return a fast 503
No ramp at all	no baseline	You have no capacity envelope, only hope

Half the knee, not the knee. Headroom is for the day your prompt gets longer.

The gate you ship

Now encode what you measured. Do not leave it in a notebook.

# gate.py - cap below the knee you measured.
import asyncio, random

class Retryable(Exception):
    pass

class Envelope:
    def __init__(self, knee, headroom=0.5):
        self.cap = max(1, int(knee * headroom))
        self.sem = asyncio.Semaphore(self.cap)

    async def run(self, fn, *args, **kwargs):
        async with self.sem:
            for attempt in range(3):
                try:
                    return await fn(*args, **kwargs)
                except Retryable:
                    delay = (2 ** attempt) * 0.2 + random.random() * 0.2
                    await asyncio.sleep(delay)
            raise

Retryable is yours to define. Map 429, 503, and connect timeouts to it. Map everything else to a hard failure.

What this does not tell you

• The generator shares a network with the target. Measure from a second host when you can.
• TTFB mixes queue wait with prefill. The client cannot split them apart.
• A ramp is a snapshot. Provider-side changes move the knee without warning.
• One prompt shape is not all prompt shapes. Long prompts bend the curve down.
• Free tiers can change. That is a reason to re-run the ramp, not a reason to skip it.

Who should not use this

• One-shot scripts and nightly batch jobs. Nobody is waiting on them.
• Teams with real load tests and SLO dashboards. Reuse those numbers instead.
• Anyone who cannot label synthetic traffic as synthetic. Ask first.

So why ship a cap you never measured? You cannot defend a number you never took.

Run one ramp. Write the knee in your README. If you are prototyping on a budget, run the same script against MonkeyCode's free model access and free server option before you commit to a paid plan. The script does not care whose endpoint answers.