The 7.6x Speedup Hiding in Your HTTP Client

A 100-request batch of AI-assisted diff reviews completed in seven minutes and twenty seconds on the first attempt. The same batch finished in fifty-eight seconds after two modest changes: a shared connection and a four-worker cap. No prompt edits. No model swap. The fix lived entirely in the HTTP client.

For this experiment I used MonkeyCode's free models and the free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The scenario

The workload was straightforward. One hundred diffs, each around 60 lines of changed code, queued for a review endpoint. Every request sent a diff and received a structured critique. A runnable benchmark, not a production pipeline. The endpoint runs on the free server. The models are free-tier models, adequate for prototyping.

The first client implementation was the most obvious one. A new httpx.Client for every request.

import httpx

results = []
for diff in diffs:
    client = httpx.Client(timeout=30)
    res = client.post(REVIEW_URL, json={"diff": diff})
    results.append(res.json())
    client.close()

This code is correct. It is also the slowest reliable option available.

What the timing said

Before changing anything, I measured the request lifecycle with curl. The -w flag prints the connection time, time to first byte, and total time. That is enough to separate client cost from server cost.

for i in $(seq 1 10); do
  curl -o /dev/null -s \
    -w "connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
    -X POST "$REVIEW_URL" \
    -H "content-type: application/json" \
    -d '{"diff": "sample diff one two three"}'
done

The sequence repeated every few requests. Slow, fast, fast, slow, fast, slow.

request   connect   ttfb    total
   1       1.84s    3.41s   5.52s
   2       0.04s    0.88s   1.21s
   3       0.03s    0.91s   1.33s
   4       1.79s    3.22s   5.29s
   5       0.05s    0.86s   1.18s
   6       1.88s    3.30s   5.47s

The connect time told the whole story. Every second request paid a fresh TLS handshake. The server side did not spend two seconds thinking on those requests. The transport did.

Fix one: reuse the connection

The remedy is a single shared client.

import httpx

with httpx.Client(timeout=30) as client:
    for diff in diffs:
        res = client.post(REVIEW_URL, json={"diff": diff})
        results.append(res.json())

One connection is established at the start of the batch. Keep-alive takes over for the remaining 99 requests. The first request still pays the cold-start tax. The rest do not.

The numbers changed immediately. These are from one run on one day, so treat them as a reproducible example rather than a promise.

variant            median total   wall time (100 requests)
fresh client       4.41s          7m 20s
shared client      1.17s          2m 10s

That is a 3.4x improvement from the first fix alone.

Fix two: bounded concurrency

The shared client solved the handshake problem. A new problem surfaced: serialization. Each request waited for the previous response before sending the next. Round-trip time dominated the batch wall clock.

The endpoint does not require strict ordering. Request boundaries are independent. That makes the workload a candidate for bounded parallelism. Four concurrent workers kept the server busy without hammering it. The bound was a hard cap of four in-flight requests.

from concurrent.futures import ThreadPoolExecutor

with httpx.Client(timeout=30) as client:
    with ThreadPoolExecutor(max_workers=4) as pool:
        results = list(pool.map(
            lambda d: client.post(REVIEW_URL, json={"diff": d}).json(),
            diffs
        ))

The wall time dropped from two minutes ten seconds to fifty-eight seconds. Individual request latency stayed roughly the same. The batch just stopped waiting on its own idle time. The full path from the naive client to the bounded pool was a 7.6x reduction in wall time.

Decision table

The table below is a rough map of what to use when. The times come from a single run on the free server. They will differ elsewhere; the relationship between the rows usually holds anyway.

Strategy	Wall time for 100 requests	Best when	Risk
New client per request	~7m 20s	Rare, one-off calls	Cold connections dominate
Shared client, serial	~2m 10s	Batches under 50	Latency grows linearly
Shared client, 4 workers	~1m	Batches of 50–500	Rate limits and backpressure
Shared client, 8+ workers	faster, then plateau	Latency-bound tuning	Thread-safety and timeouts

The plateau matters. At some worker count the bottleneck stops being the client. It becomes the server's queue, the rate limiter, or the network path. Push past it and the error rate climbs while throughput stalls.

Timeouts and retries

A batch without retry logic loses to a single transient error. A batch with naive retries creates a thundering herd. Both failure modes are avoidable. Use a short connection timeout and a generous read timeout. Retry only on idempotent failures.

import random
import time

def retry_request(client, diff, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            res = client.post(REVIEW_URL, json={"diff": diff}, timeout=30)
            res.raise_for_status()
            return res.json()
        except Exception:
            if attempt == max_attempts - 1:
                raise
            time.sleep((2 ** attempt) + random.uniform(0, 1))

Backoff with jitter keeps retries from stacking into synchronized bursts.

Limits and who should skip this

The free models and free server have real constraints. They are an experiment surface, not a production promise. Expect variable latency, shared capacity, and a quota model that is not an SLA. Read the current terms before building on top of them.

Teams that should not use this pattern at all: anyone handling regulated customer data on shared infrastructure, anyone who needs a committed response-time guarantee, and anyone whose traffic volume is high enough to require formal capacity planning. The workflow described here is for prototypes, evaluation harnesses, and load testing.

The takeaway

The model was not the slow part of this batch. The transport was. A new connection per request is a hidden tax on every batch workload. Three curl timing fields reveal the truth in minutes. Apply keep-alive, add bounded concurrency, then measure again. The graph tells you who owes you the missing milliseconds.

MonkeyCode's open-source codebase is a short walk from the free models and free server. A curl smoke test is faster than reading the README twice.