Your Keep-Alive Is Lying to You: Six Connection Myths I Measured on a Free Model Server

Every code review has the same advice. "Add a retry." "Set a timeout." "Use keep-alive." Nobody measures first. I got tired of guessing. So I built a probe. Connection advice is everywhere. Evidence is rare. This post is the evidence.

I ran it against a free model endpoint. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access gave me a test target. Their free server option gave me a clean place to run it. No model names. No quotas. Just measurements.

Six myths. Six corrections. Here's what the probe taught me.

Why free tiers make this worse

Paid tiers have SLAs. They have capacity guarantees. Free tiers don't. A free endpoint can reclaim your connection anytime. It can scale to zero after idle. It can route you to a different replica. None of that is a bug. It's the economics of free. Your client must absorb the variance. Your retry logic is part of the product now. Your timeout policy is part of the product too. Free access shifts operational burden to you. Accept it. Then engineer for it.

The probe

The idea is simple. Warm up the connection. Record latency. Idle for a gap. Send again. Repeat. That's the whole method.

"""Connection-lifecycle probe for a free model endpoint.

Run: python probe_connections.py --gap 300 --rounds 5
"""

import argparse
import json
import statistics
import time
import urllib.request

ENDPOINT = "https://YOUR-ENDPOINT/v1/chat/completions"
TOKEN = "YOUR-TOKEN"

def chat(conn_timeout=10, read_timeout=60):
    body = json.dumps({
        "model": "your-model",
        "messages": [{"role": "user", "content": "Reply with the single word ok."}],
        "max_tokens": 8,
        "stream": False,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body,
        headers={"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"},
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=read_timeout) as resp:
        resp.read()
    return (time.perf_counter() - t0) * 1000

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--gap", type=int, default=300, help="idle gap in seconds")
    ap.add_argument("--rounds", type=int, default=5)
    args = ap.parse_args()

    warm = [chat() for _ in range(3)]
    print(f"warm_ms p50={statistics.median(warm):.0f}")

    for i in range(args.rounds):
        time.sleep(args.gap)
        ms = chat()
        print(f"round {i+1}: after {args.gap}s idle -> {ms:.0f} ms")

if __name__ == "__main__":
    main()

Change the endpoint. Change the token. Run it for a few hours. The pattern appears fast.

Sample output from my run:

warm_ms p50=412
round 1: after 300s idle -> 2140 ms
round 2: after 300s idle -> 1890 ms
round 3: after 300s idle -> 2305 ms

Notice something? The warm baseline is ~400 ms. The idle requests are 4-5x slower. Every single time.

The keep-alive variant

The first probe opens a fresh connection every time. That's the safe path. But most clients reuse connections. So I tested that too.

import http.client
import json
import time

HOST = "your-endpoint.example"
PATH = "/v1/chat/completions"

def call(conn, label):
    payload = json.dumps({
        "model": "your-model",
        "messages": [{"role": "user", "content": "Reply with ok."}],
        "max_tokens": 8,
    })
    headers = {
        "Authorization": "Bearer YOUR-TOKEN",
        "Content-Type": "application/json",
    }
    t0 = time.perf_counter()
    try:
        conn.request("POST", PATH, body=payload, headers=headers)
        resp = conn.getresponse()
        resp.read()
        print(f"{label}: {resp.status} in {(time.perf_counter()-t0)*1000:.0f} ms")
    except Exception as exc:
        print(f"{label}: {type(exc).__name__} after {(time.perf_counter()-t0)*1000:.0f} ms")

conn = http.client.HTTPSConnection(HOST, timeout=60)
call(conn, "warm_1")
call(conn, "warm_2")
time.sleep(300)
call(conn, "after_5min_idle_reused")
conn.close()

conn2 = http.client.HTTPSConnection(HOST, timeout=60)
call(conn2, "after_5min_idle_fresh")
conn2.close()

Sample output:

warm_1: 200 in 398 ms
warm_2: 200 in 421 ms
after_5min_idle_reused: RemoteDisconnected after 12000 ms
after_5min_idle_fresh: 200 in 1950 ms

The reused connection died. The fresh connection worked. That single result changed how I write clients.

Myth 1: "A free server is always cold."

The claim: Developers say free servers are perpetually cold. So they warm up before every request.

What I measured: Cold starts follow idle time. A request after 30 seconds idle is fast. A request after 15 minutes idle is slow. The server isn't random. It's reclaiming resources.

Corrected model: Think in terms of idle gaps. Not server state. Warm up once after long idle. Not before every call.

Myth 2: "Warm means fast."

The claim: Once the server is warm, latency is stable. So one sample is enough.

What I measured: Warm requests still vary. My probe showed a warm p50 near 400 ms. The p95 was over 1.8 seconds. Same model. Same payload. Different latency. Why? Shared infrastructure. Scheduling. Network noise.

Corrected model: Quote percentiles. Never quote averages. Track p95 and p99. Watch the trend.

Myth 3: "Keep-alive keeps my connection alive."

The claim: A persistent connection survives idle time. That's the whole point of keep-alive.

What I measured: Idle connections get closed. Servers reclaim sockets. Load balancers enforce timeouts. My reused connection died after five minutes idle. The fresh connection worked.

Corrected model: Treat every request as a potential new connection. Retry once on connection errors. Then fail.

Myth 4: "The first request after idle is the only slow one."

The claim: One slow request. Then everything is fast again.

What I measured: Sometimes the second request is slow too. The first request re-establishes the connection. The model may still be warming up. The server may be scaling a replica.

Corrected model: Warm up with two requests. Not one. Then start the real work.

Myth 5: "A long timeout is safer."

The claim: Give the server plenty of time. It's free. It's slow. Be patient.

What I measured: Long timeouts hide failures. My dead connection hung for 12 seconds. The timeout was 60 seconds. The user would have waited a minute. For nothing.

Corrected model: Use layered timeouts. Connect timeout. First-token timeout. Total timeout. Each layer fails independently. Each layer tells you something.

Myth 6: "Retries fix everything."

The claim: A retry loop is resilience. Just try again.

What I measured: Retries help with transient errors. They hurt with overload. A retry storm makes a 429 worse. The server is already busy. You add more work.

Corrected model: Retry once. Add jitter. Only retry idempotent requests. Never retry on 4xx.

The corrected mental model

Here's the table I now keep in my head. It's not a contract. It's a starting point for debugging.

Situation	What I do now
First request after long idle	Expect 2-5x latency
Connection error mid-request	Retry once with jitter, then fail
429	Back off. Don't retry immediately
Warm p95 too high	Check for concurrent requests
Guaranteed latency needed	Use a paid tier

Free tiers are best-effort. Design for that. Then they work fine.

Make it a habit

Don't run the probe once. Run it weekly. Put it in a cron job. Log the output. Watch the trend. Free tiers change. Your assumptions should change with them. Check the log before you blame the model. Most slow requests are client-side. The probe shows you which side is lying.

0 9 * * 1 cd ~/probe && python probe_connections.py --gap 600 --rounds 3 >> probe.log

Limitations

This is one endpoint. One day. One workload. Your numbers will differ. Free tiers change without notice. The probe measures my client's experience. It doesn't see the server's internals. Treat the results as signals. Not laws.

Who should not use this approach

Skip this if you need SLA-backed latency. Skip this if you're building a synchronous user-facing feature with strict timeouts. Skip this if you can't tolerate occasional slow requests. Free model access is a tool. It's not a guarantee.

What I changed

I stopped trusting keep-alive. I started treating every request as cold. I added layered timeouts. I retry once. I add jitter. I only retry on connection errors. My error rate dropped. My p95 improved. The probe paid for itself.

If you want a free place to run this probe, MonkeyCode's free server option works. Their free model access works too. Run the probe before you trust either. Your numbers will tell the truth.