48-Hour Field Notes: I Ramped Worker Threads From 1 to 12 and Watched Throughput Fall Off a Cliff at 8

I had roughly 400 short model calls to make and a free server that I did not pay for, so the obvious move was to crank up the thread count and walk away. Twenty minutes in, my script was faster on paper and slower on wall clock, and I had no measurement that could tell me which of those two facts was real. Have you ever tuned a number because it felt faster?

These are my 48-hour field notes from that week: what I tried, what broke, and the one small harness I would build again before writing any job queue. Everything below is from a single box, a single run, and a free model endpoint — treat the numbers as an observation, not a benchmark of anything.

What I was actually optimizing

The job was boring on purpose: for each of ~400 inputs, ask a model for a short structured summary, then append the result to a file. The per-call latency was the whole cost — my Python did almost no work. So the only lever I thought I had was concurrency.

• Serial run: roughly 8 minutes, which felt wasteful.
• 12 threads: finished, but slower than serial on one of the three runs.
• Retries: silently multiplied the request count, so my "throughput" was fiction.

That last bullet is the trap. If you count successes per second without counting attempts, concurrency looks free right up until it does not.

The product side, briefly

I was running this on MonkeyCode's free server option and calling its free model access, which is why per-call latency was worth measuring at all rather than just paying for headroom. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not get internal queue metrics, so every claim below is something I observed from the client side with a stopwatch.

Attempt 1: more threads, less thinking

My first version was the classic "just raise the pool" edit:

from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=12) as pool:
    results = list(pool.map(call_model, inputs))

It is four lines and it hides everything I needed. There is no per-call deadline, no record of attempts, no separation between "the endpoint was slow" and "my box was busy", and the results only exist after the last call returns. When the host recycling happened at hour 31 and the process died, I had nothing to analyze.

So the fix was not a better pool size. The fix was instrumentation first.

The harness I should have written on day one

Ninety lines, standard library only, no dependencies. It sends N calls at a given concurrency, records every attempt as one JSON line, and prints a small summary. You can paste this and change the two constants.

import json, time, urllib.request, urllib.error
from concurrent.futures import ThreadPoolExecutor

URL = "http://127.0.0.1:8000/v1/chat/completions"  # your local/forwarded endpoint
PAYLOAD = {"messages": [{"role": "user", "content": "Reply with the single word: ok"}]}

def one_call(deadline_s=30):
    req = urllib.request.Request(
        URL,
        data=json.dumps(PAYLOAD).encode(),
        headers={"Content-Type": "application/json"},
    )
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=deadline_s) as r:
            r.read()
            status = r.status
        return {"ok": True, "ms": round((time.monotonic() - t0) * 1000), "status": status}
    except urllib.error.HTTPError as e:
        return {"ok": False, "ms": round((time.monotonic() - t0) * 1000), "status": e.code}
    except Exception as e:
        return {"ok": False, "ms": round((time.monotonic() - t0) * 1000), "status": type(e).__name__}

def pct(xs, p):
    xs = sorted(xs)
    if not xs:
        return None
    k = max(0, min(len(xs) - 1, int(round((p / 100) * (len(xs) - 1)))))
    return xs[k]

def sweep(concurrency, total=40):
    t0 = time.monotonic()
    with ThreadPoolExecutor(max_workers=concurrency) as pool:
        rows = list(pool.map(lambda _: one_call(), range(total)))
    wall = time.monotonic() - t0
    lat = [r["ms"] for r in rows]
    oks = [r for r in rows if r["ok"]]
    print(json.dumps({
        "conc": concurrency,
        "rps": round(len(oks) / wall, 2),
        "p50_ms": pct(lat, 50), "p95_ms": pct(lat, 95),
        "errors": len(rows) - len(oks),
    }))

for c in (1, 2, 4, 8, 12):
    sweep(c)

Two details matter more than the code. First, rps is computed from successful calls only, so a failing endpoint cannot look fast. Second, every attempt is timed individually, so a p95 that triples while p50 barely moves tells you something is queueing — and it is probably not your CPU.

What one sweep showed

One run, one free box, 40 calls per step, all replies identical. This is a single observation, not a product benchmark, and I would not quote it as capacity.

Workers	ok req/s	p50 (ms)	p95 (ms)	errors
1	0.8	1,180	1,900	0
2	1.5	1,240	2,300	0
4	2.6	1,410	3,100	0
8	3.0	2,050	6,800	1
12	2.7	3,600	11,400	6

Read it as two curves, not one. Throughput flattens between 4 and 8; tail latency keeps climbing past 8 and errors appear. The knee is where p95 grows faster than req/s, and for this box on that afternoon it was somewhere around 4 to 6.

Is that number stable? Almost certainly not. It depends on time of day, on whatever else the free host is doing, and on how long my prompts are. That is exactly why the harness matters more than the number.

What broke, and what the failure was not

My first theory was CPU saturation on the free server. Wrong. When I logged the box's own load while the sweep ran, it stayed low — the waiting was happening on the network hop and at the endpoint, not in my process. The retries were the real damage: every timeout re-sent a request that the endpoint was probably still working on, which made the next sweep look worse and pushed me toward even more threads. Does that loop sound familiar?

Three concrete changes came out of it.

1. Fixed pool, explicit queue. Four workers, a bounded queue, and a deadline_s short enough that a stuck call dies before the next batch starts.
2. Attempt-level logging. One JSON line per attempt with conc, ms, ok, status, and a run id. Re-analysis never requires re-running.
3. Warm-up before measuring. The first two calls after an idle period were consistently the slowest, so I throw them away instead of letting them pollute p50.

After that, the 400-call run landed near 3 minutes and, more importantly, it was predictable. I cared less about the ceiling than about not being surprised by it.

Decision table

Your situation	Do this	Skip
One-off script, <100 calls	Serial, log everything	Concurrency tuning
Recurring batch on a free box	Sweep 1→12 once per day, pick the knee	Copying someone else's worker count
Latency-sensitive user requests	Measure p95 per concurrency step first	Retrying aggressively on timeout
Hard quota or per-call cost	Count attempts, not successes	Optimizing req/s in isolation

Who should not use this approach

The honest limits: my sweep is 40 calls per step on a shared free box, with no control over neighboring load, so it cannot tell you what capacity you will get tomorrow. If you need a guaranteed p95, contractual throughput, or spend accounting at the request level, a free tier and a home-grown harness are the wrong foundation — measure, then move to a paid plan with published limits. And do not run any of this against data you would not send to a third party.

What I would repeat, and what I wasted time on

I would absolutely repeat the sweep and the per-attempt JSONL log; together they turned a vague "it feels slow" into a table I could argue with. I would skip the day I spent reading about thread pool tuning before I had a single measurement of my own, because the bottleneck was never where the blog posts said it would be.

If you take one thing: make the endpoint prove its speed to you at the concurrency you actually plan to run, and write that proof down before you trust it. If you try the sweep, change URL and total and post your table — I would like to know whether your knee sits where mine did.