Your First Request After Idle Is a Lie: Measuring Cold Start on Free-Tier Agent Hosts

You ship a small agent to a free host, wire a nightly batch to it, and close your laptop feeling clever. The next morning the run has failed twice with a timeout and succeeded on the third attempt, and nobody changed a line of code. The logs show a healthy service, the health check was green the whole time, and the only unusual thing is that the failures all happened right after a long idle window. That gap between "the process answers" and "the process can actually do the work" is what this playbook measures.

The scenario above is a composite, not a logged incident from one company, so treat the numbers in your own run as the only evidence that counts. What follows is a reproducible way to separate three clocks that most teams accidentally average together, plus a one-page run you can paste into a wiki. You will end up with a cold penalty figure you can defend in a review and a readiness definition that stops lying to you.

Why the first request after idle is not a sample of your latency

When you time a request to a free-tier agent host, you are timing at least three unrelated things at once. The platform has to start or resume your container, your interpreter has to import modules and initialise the model client, and the model provider has to return a first token. Only the first two belong to your server, and the third can dominate both when free model access is under load.

Averaging them produces a single p50 that looks acceptable and hides the exact interval where your batch job breaks. Your scheduled run does not sample the warm distribution at all, because it always fires after the longest idle gap of the day. That is why a dashboard can look calm while a cron job fails on its first attempt every single night.

There is a second trap: most health endpoints answer before the service is useful. A /healthz route that returns {"ok": true} from a process that has imported FastAPI but never opened a connection to the model backend tells your orchestrator to send real work too early. You need a readiness signal that can only be produced by a completed model call.

Step 1 — Instrument the phases inside the service

Start on the inside, because an external timer can only see the total. Add a tiny context manager that records named phases, and expose them from a probe route that performs one real, tiny model call. The goal is a number per phase in milliseconds, attached to the same response your harness times from outside.

# app/timing.py
import time
from contextlib import contextmanager

PHASES = {}

@contextmanager
def phase(name):
    t0 = time.perf_counter()
    try:
        yield
    finally:
        PHASES[name] = round((time.perf_counter() - t0) * 1000, 1)

# app/serve.py (excerpt)
import time
from fastapi import FastAPI
from app.timing import phase, PHASES
from app.model import ask  # thin wrapper over your model client

app = FastAPI()
BOOT = time.perf_counter()

@app.get("/healthz")
def healthz():
    return {"ok": True, "uptime_s": round(time.perf_counter() - BOOT, 3)}

@app.get("/probe")
def probe():
    PHASES.clear()
    with phase("model_first_token"):
        answer = ask("Reply with the single word: ok", max_tokens=8)
    return {"answer": answer, "phases_ms": dict(PHASES),
            "uptime_s": round(time.perf_counter() - BOOT, 3)}

This is where a free-tier setup is actually convenient: you can run the same probe against a local process and against the hosted one without touching your production billing. MonkeyCode's free model access and free server option are the two operator-supplied ingredients I used for this shape of test, since a cheap target lets you repeat the restart cycle many times in one sitting.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honesty notes before you plan capacity on it. First, treat availability, quotas, model names, and server specs as claims that change, and confirm the current ones on the project's own pages rather than trusting a blog post from any week, including this one. Second, the project is presented as open source, so read the repository and its license yourself if self-hosting or auditing matters to you.

Step 2 — Probe from outside with a restart loop

The harness below measures two intervals: spawn-to-listening and listening-to-answer. It writes one JSON line per run so you can aggregate later instead of eyeballing a terminal. Run it in local mode to restart a process repeatedly, or in remote mode against a host you restart through your platform's UI or API.

#!/usr/bin/env python3
"""coldstart_probe.py - measure cold vs warm start phases.

Local:
  python coldstart_probe.py --mode local --cmd "python -m app.serve" \
    --health http://127.0.0.1:8000/healthz --probe http://127.0.0.1:8000/probe
Remote:
  python coldstart_probe.py --mode remote --runs 5 \
    --health https://your-host/healthz --probe https://your-host/probe
"""
import argparse, json, subprocess, time, urllib.request


def now():
    return time.perf_counter()


def hit(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as r:
        r.read()
        return r.status


def wait_listening(url, deadline):
    t0 = now()
    while now() - t0 < deadline:
        try:
            if hit(url, timeout=2.0) == 200:
                return now() - t0
        except Exception:
            time.sleep(0.1)
    raise TimeoutError(f"no 200 from {url} within {deadline}s")


def one_run(args, label):
    proc, t0 = None, now()
    if args.mode == "local":
        proc = subprocess.Popen(args.cmd.split(), stdout=subprocess.DEVNULL)
    try:
        t_listen = wait_listening(args.health, args.deadline)
        hit(args.probe, timeout=args.deadline)  # forces one real model call
        row = {"run": label, "mode": args.mode,
               "spawn_to_listen_s": round(t_listen, 3),
               "listen_to_answer_s": round(now() - t0 - t_listen, 3),
               "spawn_to_answer_s": round(now() - t0, 3)}
        with open("coldstart.jsonl", "a") as f:
            f.write(json.dumps(row) + "\n")
        return row
    finally:
        if proc is not None:
            proc.terminate()
            proc.wait(timeout=30)


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--mode", choices=["local", "remote"], default="local")
    p.add_argument("--cmd", default="python -m app.serve")
    p.add_argument("--health", required=True)
    p.add_argument("--probe", required=True)
    p.add_argument("--runs", type=int, default=3)
    p.add_argument("--deadline", type=float, default=180.0)
    args = p.parse_args()
    for i in range(args.runs):
        row = one_run(args, f"cold-{i}")
        print(json.dumps(row))
        for j in range(1, 4):  # same process, no restart: warm samples
            t0 = now()
            hit(args.probe, timeout=args.deadline)
            with open("coldstart.jsonl", "a") as f:
                f.write(json.dumps({"run": f"warm-{i}-{j}", "mode": args.mode,
                                    "spawn_to_listen_s": 0.0,
                                    "listen_to_answer_s": round(now() - t0, 3),
                                    "spawn_to_answer_s": round(now() - t0, 3)}) + "\n")


if __name__ == "__main__":
    main()

Label discipline matters more than sample size here. A run is only cold if it starts within a short window after a genuine restart, so record the restart time in your notes and discard anything ambiguous. Warm samples must come from the same process, otherwise you are comparing two cold populations and the ratio below will look like noise.

Step 3 — Compute the cold penalty and set a gate

Aggregate the JSONL file instead of reading individual lines, because the tail is the part that breaks your batch. The snippet below reports p95 for cold runs against p50 for warm runs, then divides them into a single ratio you can put in a review comment.

import json


def pct(vals, q):
    vals = sorted(vals)
    return vals[min(len(vals) - 1, int(round(q * (len(vals) - 1))))]


rows = [json.loads(line) for line in open("coldstart.jsonl")]
key = "spawn_to_answer_s"
cold = [r[key] for r in rows if r["run"].startswith("cold")]
warm = [r[key] for r in rows if r["run"].startswith("warm")]

cold_p95, warm_p50 = pct(cold, 0.95), pct(warm, 0.50)
print(f"n_cold={len(cold)} n_warm={len(warm)}")
print(f"cold p95={cold_p95:.2f}s warm p50={warm_p50:.2f}s "
      f"penalty={cold_p95 / warm_p50:.1f}x")

Now translate the ratio into an action instead of an opinion. The table below is the decision rule I would defend, and you should adjust the thresholds to your own deadline rather than copying mine.

Cold penalty	Cold p95	Decision
under 2x	under 8s	Ship it; keep the probe in CI as a regression check
2x to 5x	8s to 20s	Keep batch work, pre-warm before every scheduled run
over 5x	over 20s	Move interactive paths off the free host, batch only

Write the decision down with the date and the run count next to it. A number without its sample size and its test date is a rumor, and the next person to touch the deployment will treat it as permanent when it is not.

Step 4 — Make readiness mean warm, not listening

Your orchestrator should never route user traffic to a process that answers health checks but has not completed a model call. Add a readiness route that returns 503 until the warm probe has succeeded once in that process, and point your platform's health check at it.

from fastapi.responses import JSONResponse

STATE = {"warmed": False, "warm_ms": None}

@app.get("/readyz")
def readyz():
    if not STATE["warmed"]:
        return JSONResponse({"ready": False, "reason": "no warm probe yet"},
                            status_code=503)
    return {"ready": True, "warm_ms": STATE["warm_ms"]}

Then gate your scheduled job on that route rather than on a sleep, so the wait is evidence-based and self-documenting.

# block until the instance is warm, then start the batch
until curl -fsS "$BASE/readyz" >/dev/null; do sleep 2; done
python -m jobs.nightly_batch

If your platform health check cannot use a 503, keep /healthz for liveness and add /readyz to the job script only. Splitting the two signals is usually enough, and it costs you one route and one line of shell.

Step 5 — The one-page run for your wiki

Paste this block into the team wiki and fill in names, because a measurement with no owner becomes folklore within a month. The handoffs are deliberate: the person who builds the service should not be the only person who interprets its cold numbers.

Role	Owns	Handoff artifact
Builder	/probe, /readyz, phase timing	merged routes plus probe command
Prober	restart cycle, 3 cold runs, 12 warm samples	coldstart.jsonl plus penalty line
Reviewer	threshold decision and expiry date	one paragraph in the runbook

Run the cycle whenever the model client, the base image, or the hosting tier changes, and always before a launch that depends on scheduled work. Keep the JSONL file in the repo for the current quarter so the next reviewer can compare like with like. The whole loop takes about thirty minutes on a cheap host, which is the main argument for doing it at all.

Limitations and who should not use this

This harness measures wall-clock phases, not correctness, so a model that returns a wrong answer in 400 ms will pass every check here. Network variance between your laptop and the host can also inflate remote numbers, so run remote probes from the same region as your batch job. Free-tier capacity changes without notice, which means a penalty measured this week may not hold next week.

Do not use this approach if you need a sub-second first token under a contractual SLO, because a free host cannot give you that guarantee and the measurement will not create one. Skip it as well if automated probing would violate the provider's rate limits or acceptable-use terms, and check those before you loop anything. Finally, if your agent has no human-facing latency requirement at all, the cold penalty is trivia and your time is better spent on output quality.

If you maintain a small agent on a free host, the cheapest win this week is to add /readyz, run three cold cycles, and write one sentence with the ratio and the date. If you try the harness against MonkeyCode's free model access and free server, the phase split you get back is the part worth sharing with your team.