A Load-Testing Playbook for Free AI Servers (and Why I Wrote One)

Free AI servers are dangerously easy to trust. The docs promise zero cost, the setup takes minutes, and your first curl returns a perfect 200. Then your second user shows up, and the server starts answering with 503s or, worse, hangs forever. I've been burned by this pattern enough times that I finally wrote a 30-line load tester to check a free tier before I build anything on it. Here's the script, how to run it, and how to read the numbers without fooling yourself.

The 30-line load tester

The idea is simple: fire a fixed number of concurrent HTTP requests at a health endpoint, record status codes and latencies, and let the results speak. I used asyncio and aiohttp because they handle concurrency without spawning a thread per request, which would skew the test on a small machine.

import asyncio
import time
import aiohttp

async def hit(session, url, results):
    start = time.perf_counter()
    try:
        async with session.get(url) as resp:
            await resp.read()
            status = resp.status
    except Exception as exc:
        status = type(exc).__name__
    results.append((status, time.perf_counter() - start))

async def main(url, concurrency, total):
    results = []
    async with aiohttp.ClientSession() as session:
        tasks = []
        for _ in range(total):
            tasks.append(hit(session, url, results))
            if len(tasks) >= concurrency:
                await asyncio.gather(*tasks)
                tasks = []
        if tasks:
            await asyncio.gather(*tasks)
    return results

if __name__ == "__main__":
    import sys
    url = sys.argv[1] if len(sys.argv) > 1 else "https://example.com/health"
    concurrency = int(sys.argv[2]) if len(sys.argv) > 2 else 20
    total = int(sys.argv[3]) if len(sys.argv) > 3 else 200

    results = asyncio.run(main(url, concurrency, total))
    ok = [r for r in results if r[0] == 200]
    errors = [r for r in results if r[0] != 200]
    latencies = [r[1] for r in ok]

    print(f"Total: {len(results)}")
    print(f"OK: {len(ok)}")
    print(f"Errors: {len(errors)}")
    if latencies:
        print(f"Avg latency: {sum(latencies) / len(latencies):.3f}s")
        print(f"Max latency: {max(latencies):.3f}s")
    if errors:
        print(f"First error type: {errors[0][0]}")

Save it as loadtest.py, install aiohttp with pip install aiohttp, and point it at your endpoint. The script sends requests in batches, so it won't flood the server with 200 simultaneous sockets on a tiny free instance.

How to run it and what to look for

Start with a gentle baseline: one request, then ten, then fifty. I usually run this sequence:

python loadtest.py https://your-free-server.example.com/health 1 10
python loadtest.py https://your-free-server.example.com/health 10 100
python loadtest.py https://your-free-server.example.com/health 50 500

The first run tells you the raw latency. The second reveals connection limits. The third shows how the server behaves under sustained pressure. Three numbers matter most:

• Error rate: any non-200 response is a red flag, but 429s are different from 503s. 429 means rate limiting, which you can often work around. 503 means the server is overloaded or the free tier is throttled.
• Latency growth: if average latency doubles when concurrency goes from 1 to 10, the server has a queueing problem. If it stays flat until 50, you're probably fine for a small hobby project.
• Max latency: a single 30-second response will kill a user-facing app even if the average is 200ms. Watch the tail, not just the mean.

How I use this with MonkeyCode

MonkeyCode is an open-source project that caught my attention because it bundles free model access with a free server option, which is exactly the combination that needs this kind of testing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I ran the load tester against a small service deployed on their free server, and the process was genuinely useful: it gave me concrete numbers before I wrote any application code, and it exposed a cold-start delay on the first request that I would have missed otherwise.

The free tier is not a production SLA, and I don't treat it as one. But for a prototype or a weekend project, knowing that the server can handle 20 concurrent requests with a 400ms average latency is enough to move forward. The tester doesn't validate model quality or token limits, so I check the current docs for those numbers before committing.

Limitations and who should skip this

This script only measures HTTP behavior, not model correctness, token throughput, or data privacy. If you're building something that handles sensitive information, a shared free server is the wrong choice regardless of load test results. Also, a single endpoint test won't catch every failure mode: database connections, background workers, and cold starts all hide behind the health check. Run the tester against the actual routes your users will hit, not just /health.

If you're already on a paid platform with a guaranteed SLA, you don't need this playbook. But if you're evaluating a free tier for a side project, or you're tired of discovering limits after your users do, spend ten minutes with this script first. The numbers will tell you more than any README promise.