The Free Server Doesn't Leak. It Drifts.

I stopped asking whether a free server crashes. Wrong question, and it cost me a demo.

The things rarely crash. They drift — a few kilobytes of resident memory per request, a few extra prompt tokens per turn — and then the demo dies at minute forty with no stack trace and no obvious culprit. A crash hands you a line number. Drift hands you a shrug. Which one would you rather debug at 11pm?

So I measure the slope instead. One fixed workload, one sample of resident memory per request, one least-squares line through the second half of the run. That slope is what decides whether a free box can hold a real session, and it costs about fifteen minutes to get.

A single reading lies

Memory graphs breathe. Allocators hold pages they intend to reuse, garbage collectors sawtooth, and a snapshot taken at a lucky moment looks like a plateau. Take two snapshots an hour apart and you have learned almost nothing about the trend between them.

The second half is where the truth lives, because the first half is warmup — import caches, JIT, connection pools, the one-time cost of parsing your own prompt file. Fit a line through the whole run and warmup drags the intercept up while flattening the slope you are hunting for. Fit only the tail and you are finally measuring growth.

Do you need a heap profiler for this? Eventually, yes. First you need to know whether there is anything worth profiling.

The harness

Three small files. The service does one boring thing, so the only remaining variables are the box and the model endpoint.

# soak_service.py — one endpoint, one upstream call, one CSV row per request
import json, os, threading, time, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

BASE  = os.environ["MODEL_BASE_URL"].rstrip("/")  # confirm path + auth for your provider
KEY   = os.environ["MODEL_API_KEY"]
MODEL = os.environ["MODEL_NAME"]
OUT   = open(os.environ.get("SOAK_LOG", "soak.csv"), "a", buffering=1)
LOCK, SEQ = threading.Lock(), 0

def rss_kib():
    try:
        with open("/proc/self/status") as fh:
            for line in fh:
                if line.startswith("VmRSS:"):
                    return int(line.split()[1])
    except FileNotFoundError:
        pass
    return -1  # macOS: resource.getrusage gives a peak, not a sample — use it knowingly

def ask(prompt):
    payload = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 64, "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        BASE + "/chat/completions", data=payload,
        headers={"Content-Type": "application/json", "Authorization": "Bearer " + KEY})
    with urllib.request.urlopen(req, timeout=120) as resp:
        return json.loads(resp.read())

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        global SEQ
        raw = self.rfile.read(int(self.headers.get("Content-Length", 0)))
        prompt = json.loads(raw).get("prompt", "ping")
        t0, pt, ct, code, text = time.monotonic(), -1, -1, 200, ""
        try:
            body = ask(prompt)
            usage = body.get("usage") or {}
            pt = usage.get("prompt_tokens", -1)
            ct = usage.get("completion_tokens", -1)
            text = body["choices"][0]["message"]["content"]
        except Exception as exc:
            code, text = 500, str(exc)
        ms = int((time.monotonic() - t0) * 1000)
        with LOCK:
            SEQ += 1
            OUT.write(f"{SEQ},{time.time():.3f},{rss_kib()},{pt},{ct},{ms},{code}\n")
        blob = text.encode()
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(blob)))
        self.end_headers()
        self.wfile.write(blob)

    def log_message(self, *a): pass

if __name__ == "__main__":
    if OUT.tell() == 0:
        OUT.write("seq,ts,rss_kib,prompt_tokens,completion_tokens,ms,status\n")
    ThreadingHTTPServer(("0.0.0.0", 8080), H).serve_forever()

Notice the deliberate constraints. max_tokens is capped so completion length cannot masquerade as a memory trend, temperature is zero so two runs are comparable, and RSS is sampled after the upstream call returns — that is the moment your handler has held every buffer at once. The CSV has one row per request, which means the analysis needs no daemon and no metrics stack.

The driver cycles a fixed prompt list with a fixed seed, so a slow request is slow because of the box, not because of a random harder question.

# soak_driver.py — usage: python soak_driver.py 600 prompts.txt
import json, random, sys, time, urllib.request

URL = "http://127.0.0.1:8080/"
PROMPTS = [p.strip() for p in open(sys.argv[2]) if p.strip()]
N = int(sys.argv[1])
random.seed(7)   # same list, same order, every run

lat = []
for i in range(N):
    body = json.dumps({"prompt": PROMPTS[i % len(PROMPTS)]}).encode()
    req = urllib.request.Request(URL, data=body,
                                 headers={"Content-Type": "application/json"})
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=180) as r:
            r.read()
        lat.append(time.monotonic() - t0)
    except Exception:
        lat.append(float("nan"))
    if (i + 1) % 50 == 0:
        print(f"{i+1}/{N}", flush=True)

lat = sorted(x for x in lat if x == x)
print(f"ok {len(lat)}/{N}  p50 {lat[len(lat)//2]:.2f}s  p95 {lat[int(len(lat)*0.95)]:.2f}s")

Then the fit. Pure stdlib, because a soak harness that needs numpy is a soak harness nobody reruns.

# soak_fit.py — the only interesting number is the slope of the tail
import csv, sys

def slope(xs, ys):
    n = len(xs)
    mx, my = sum(xs) / n, sum(ys) / n
    den = sum((x - mx) ** 2 for x in xs)
    return 0.0 if den == 0 else sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / den

rows = list(csv.DictReader(open(sys.argv[1])))
ok   = [r for r in rows if r["status"] == "200"]
tail = ok[len(ok) // 2:]
xs   = range(len(tail))
print(f"kept {len(ok)}/{len(rows)} requests, fitting the last {len(tail)}")
print(f"rss slope          : {slope(xs, [int(r['rss_kib']) for r in tail]) * 1024:+9.1f} B/request")
print(f"prompt-token slope : {slope(xs, [int(r['prompt_tokens']) for r in tail]):+9.3f} tok/request")
print(f"completion slope   : {slope(xs, [int(r['completion_tokens']) for r in tail]):+9.3f} tok/request")

Reading the number

Here is the part I will not fake for you. I am deliberately not pasting slopes from a run, because a slope from a box you do not own tells you nothing about yours. Run the harness, then read the output against thresholds you can defend.

RSS slope (tail)	Prompt-token slope	Read	Action
under ±0.5 KiB/req	~0	noise floor	ship it
0.5–4 KiB/req	~0	real drift, slow	restart on a timer, cap history
4–20 KiB/req	~0	growth, findable	trim per-request caches and unbounded lists
over 20 KiB/req	any	leak, not drift	do not put demo traffic on it
any	positive	prompt assembly accumulates	fix retrieval or history trimming first

The last row is the one people miss. If prompt tokens climb while memory is flat, your context assembler is quietly appending and you will blame the model for getting slower. It is not slower. You are feeding it more.

And once you have a slope, the restart timer stops being a vibe. Take the memory you actually have, keep 30% as headroom, and divide:

max_requests ≈ 0.3 * free_RAM_bytes / slope_bytes_per_request

If that number lands below one demo session, you have learned something a crash would have taught you in front of an audience instead.

Where a free tier actually helps

This harness burns volume by design — a few hundred to a few thousand calls of deliberate, identical traffic. That is exactly the workload a free model allowance is good for, and MonkeyCode offers free model access plus a free server option, both operator-supplied claims worth verifying on their own page along with the current allowance, since those numbers move. The server half matters more than it looks: if you soak on the same class of box you would demo from, the RSS slope transfers.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The project is open source, so you can read what runs in the box rather than taking a dashboard's word for it. But the value here is not the product, it is the discipline of producing your own slope before you trust anyone's uptime story.

What this cannot tell you

One soak is one workload. Change your prompt length distribution and the token slope moves; add concurrency and the memory slope moves again, sometimes downward, because four workers each holding less can beat one worker hoarding. Casualties of shared free hardware — noisy neighbors, throttled CPU — show up as latency spread, not memory, so a flat RSS ladder with terrible p95 means you measured the wrong thing today.

The harness also assumes an OpenAI-compatible POST /chat/completions shape. Verify the path and auth for your provider; if usage is missing from responses, the token columns read -1 and you are down to memory only. On macOS the sampler returns -1 and the fit is meaningless — there is no substitute for sampling the real process RSS.

Who should skip this? If you already run Prometheus plus a real allocator profiler, this is a step backwards in fidelity. And if your actual traffic is three requests per user session, drift is not your failure mode — variance is, so go measure p99 instead. This technique is for long-lived processes on hardware you do not control, which is precisely the shape of a free server.

Run the soak before the demo. If the slope comes out ugly, that is a useful result — cheaper now than at minute forty.