A Free Server Is a Failure-Domain Map

Free compute is rarely your bottleneck. Unowned failure is. Map the failure domains before you deploy on a free server.

That sounds abstract, so here is the concrete version. You get a machine you did not buy, running a model you did not train. Your job is not to trust it. Your job is to decide what breaks when it disappears.

Start with ownership, not features

Draw four boxes and label the owner of each.

client ──▶ gateway (you) ──▶ free server (borrowed) ──▶ model endpoint (borrowed)
              │                    │                          │
        deadlines, retries    process lifetime           model version
        logs, cache keys      disk, env vars             rate limits
        fallback path         restart policy             response shape

Only the second column is yours. Everything to the right can change without a changelog. You will not get a migration guide when that machine reboots.

So the review question is not "does the free server work?" It is "what does my gateway do while it does not?"

Borrowed disk and environment variables are state you never designed. Any file written there can vanish between requests. Any secret placed there can outlive your intent. Keep both out of the critical path.

Turn the review into drills

Opinions about failure are cheap. Reproductions are not. Write three drills and run them on a schedule.

# drills/run.sh — template; point BASE at your own gateway
set -u
BASE="${BASE:-http://localhost:8080}"
for drill in cold-start rate-limit truncation; do
  python drills/drill.py --target "$BASE" --drill "$drill" --runs 5
done

The driver stays small. It only injects one fault at a time.

# drills/drill.py — template, not executed against a live endpoint for this article
import random, time, urllib.request

def call(target, prompt, timeout):
    body = json.dumps({"prompt": prompt}).encode()
    req = urllib.request.Request(
        f"{target}/generate", data=body,
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return resp.read().decode()

def cold_start(target, runs):
    for i in range(runs):
        time.sleep(random.uniform(0, 45))   # aim at an idle window
        t0 = time.monotonic()
        try:
            call(target, "warm up", timeout=30)
            print("cold-start", i, round(time.monotonic() - t0, 2), "s", "ok")
        except Exception as exc:
            print("cold-start", i, "fail", type(exc).__name__)

def rate_limit(target, runs):
    for i in range(runs):
        status = "ok"
        try:
            call(target, "burst", timeout=10)
        except Exception as exc:
            status = type(exc).__name__
        print("rate-limit", i, status)

def truncation(target, runs):
    for i in range(runs):
        try:
            out = call(target, "long answer please", timeout=20)
        except Exception as exc:
            print("truncation", i, "error", type(exc).__name__)
            continue
        ok = out.rstrip().endswith((".", "}", "!"))
        print("truncation", i, len(out), "terminated" if ok else "cut")

Run the rate-limit drill with concurrency, not a serial loop. A serial loop will pass and teach you nothing. That is the most common way this review produces false comfort.

Read the results as architecture

Cold start tells you about your first-paint budget. If an idle machine answers in tens of seconds, your synchronous endpoint is the bug. Move the call behind a queue, or keep a short warm path with a cheap probe.

Rate limit tells you who owns retries. If both the client and the gateway retry, you built a multiplier. Pick one owner, and give that owner a single deadline that all attempts share.

Truncation tells you whether you validate output. A response that stops mid-sentence is not an error code. Your parser must reject it explicitly, or your users become the validator.

Write the outcome down. A drill you ran once is an anecdote. A drill wired into CI is a constraint you can defend in review.

A decision table you can defend

Use plain rules, written before the incident, not after it.

Workload	Runs where	Reason
Interactive request	gateway, hard timeout, fail fast	someone is waiting
Batch summarization	queue, generous retry	nobody is waiting
Secrets or customer records	local only	boundary you do not own
Deterministic transforms	local, no model at all	model adds risk, not value

The last row saves the most money and the most incident reviews. Not every step deserves a model call.

What I would change next

Three changes, in order of payoff.

First, one owner for retries. The gateway retries. The client does not. Every attempt spends from the same budget.

# gateway.py — deadline-carrying wrapper (template)
import time

class Deadline(Exception):
    pass

def call_with_budget(fn, budget_s, attempts=3):
    end = time.monotonic() + budget_s
    for _ in range(attempts):
        left = end - time.monotonic()
        if left <= 0:
            raise Deadline("budget exhausted")
        try:
            return fn(timeout=left)
        except TimeoutError:
            continue          # same deadline, smaller remaining budget
    raise Deadline("attempts exhausted")

Second, a health probe that records the remote build. Log the version string you saw with each rollout. When response shape changes, you want a timestamp, not a guess.

Third, a fallback that is boring. Serve a cached answer or a clear error. Never silently swap in another model and pretend nothing happened. Quiet substitution turns a small outage into a data quality incident.

Each change is small. Together they turn a borrowed machine into a bounded dependency.

Limits and who should skip this

Drills measure your gateway, not the provider. They cannot promise availability, and they cannot detect a limit you never hit.

Free capacity can change shape without notice. Treat every number you measure as a snapshot with a date attached. Re-run the drills after any change on the far side.

Skip this approach entirely if you have data residency rules, sub-second latency targets, or contractual uptime. Rent capacity you can audit instead, and keep the drills anyway.

For a low-stakes sandbox, MonkeyCode provides free model access and a free server option. That is enough room to run these three drills before you point real traffic at anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Point BASE at your own gateway. Keep the drills in CI. Then you can replace the machine without losing sleep over it.