Before You Add Workers to a Free Model API, Sweep Its Effective Concurrency

#api #ai #python #performance

A 200 status from a free model API only proves that the server accepted one request. It does not prove that your requested concurrency is real.

If you add workers, retries, or queues before measuring effective concurrency, you are tuning against an imagined number.

Why HTTP status hides the limit

A 200 is per request, not a throughput signal.
A gateway can accept many sockets and execute one at a time.
A p50 latency can stay low while p95 explodes under queueing.
Free tiers may not publish a concurrency or queue limit.

Many failures look like model errors but are actually concurrency failures. A request times out only because it waited behind other requests. The model never saw a bad prompt. Measuring effective concurrency separates model failures from queue failures.

The probe

Run a concurrency sweep with a fixed, small prompt. Record completed requests, wall time, p50, p95, and max latency. Compare the first run with each larger run.

Use an environment variable for the endpoint so the script works with any OpenAI-style model API.

import asyncio
import os
import time

import httpx

BASE_URL = os.getenv("MODEL_URL", "https://free-model.example/v1/chat/completions")
TOKEN = os.getenv("MODEL_TOKEN", "")
MODEL = os.getenv("MODEL_NAME", "model")
CONCURRENCIES = [1, 2, 4, 8]
PROMPT = "Return the single word ok."

async def one_request(client, _):
    started = time.perf_counter()
    try:
        response = await client.post(
            BASE_URL,
            headers={"Authorization": f"Bearer {TOKEN}"},
            json={
                "model": MODEL,
                "messages": [{"role": "user", "content": PROMPT}],
                "max_tokens": 8,
            },
        )
        return {
            "ok": response.status_code == 200,
            "status": response.status_code,
            "latency": time.perf_counter() - started,
        }
    except Exception as exc:
        return {
            "ok": False,
            "status": type(exc).__name__,
            "latency": time.perf_counter() - started,
        }

def percentile(values, p):
    ordered = sorted(values)
    index = int((len(ordered) - 1) * p / 100)
    return ordered[index]

async def run_sweep(concurrency):
    async with httpx.AsyncClient(timeout=60) as client:
        started = time.perf_counter()
        results = await asyncio.gather(
            *(one_request(client, i) for i in range(concurrency))
        )
        wall = time.perf_counter() - started
    return results, wall

async def main():
    baseline = None
    for concurrency in CONCURRENCIES:
        results, wall = await run_sweep(concurrency)
        latencies = sorted(result["latency"] for result in results)
        ok = sum(1 for result in results if result["ok"])
        if baseline is None:
            baseline = latencies[len(latencies) // 2] or wall
        completed = len(results)
        effective = round(baseline * completed / wall, 2) if wall else completed
        print(
            f"c={concurrency} ok={ok}/{completed} wall={wall:.2f}s "
            f"p50={percentile(latencies, 50):.2f}s "
            f"p95={percentile(latencies, 95):.2f}s "
            f"max={latencies[-1]:.2f}s effective={effective}"
        )
        await asyncio.sleep(2)

asyncio.run(main())

Read the table

Use the c=1 wall time as the baseline T.

Effective parallelism: T * completed / wall.
- Close to 1 means requests are serialized.
- Close to concurrency means the endpoint truly runs them in parallel.
When effective parallelism stops growing, adding workers only adds queue pressure.
A falling ok count or a 429 means you crossed the limit.
A p95 far above p50 means queueing started even when p50 still looks fine.

Example reading, not a benchmark:

c	ok	wall	p50	p95	effective
1	1/1	2.10s	2.10s	2.10s	1.0
2	2/2	2.15s	2.12s	2.82s	1.95
4	4/4	4.31s	2.40s	7.90s	1.95
8	6/8	8.02s	3.10s	14.2s	2.09

The endpoint stops giving real parallelism after two workers. c=8 looks worse because two requests fail and p95 climbs past seven seconds. The answer is not more workers; it is a semaphore set to two.

Apply the measurement

Pin the worker pool to the largest concurrency where effective stays near concurrency.
Add a queue in front of the worker pool instead of raising the pool size.
Keep retries out of the unbounded path. Retry only requests that failed under a measured, bounded worker count.
Re-run the sweep after any quota change, region change, or model change.

A small semaphore does the limiting:

sem = asyncio.Semaphore(2)

async def bounded_request(client, i):
    async with sem:
        return await one_request(client, i)

Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Operator-supplied availability: MonkeyCode offers free model access and a free server option. If you point this harness at a MonkeyCode free model endpoint, use the current docs for the exact URL, token format, and model name. Do not assume those details from a generic OpenAI example.

The probe's value is the same either way: it turns "how many workers should I use?" from a guess into a measured number.

Limitations

This is a capacity probe, not a quality benchmark.
Free endpoints can change quotas, regions, and queue behavior. Results are a snapshot.
Prompt length, token limits, and output length change latency. Use a fixed representative payload.
One client location cannot observe rate limits applied elsewhere.
The script makes real requests. Keep the sweep small and respect published limits.

Who should skip this

You already have server-side metrics for queue depth, concurrency, and p95.
The endpoint publishes exact concurrency or rate limits.
The endpoint is paid or expensive, so extra probe traffic costs money.
You are evaluating output quality, not capacity.

Start with one worker, run the sweep, and pin the worker count to measured effective concurrency before you build the retry layer.

DEV Community