Is Your Free Model Server Cold or Just Slow? A Probe That Tells the Difference

Your free model server feels slow. Is it sleeping? Or is it just slow?

Those are different problems. They need different fixes.

Free model endpoints are everywhere. Free servers are everywhere. Almost nobody measures their sleep cycles.

I used to guess. I blamed the network. I blamed the model. I blamed the free tier.

Then I wrote a probe. It measures one thing: how a server sleeps and wakes.

This article busts five cold-start myths. The probe is included. Run it before you trust your first request.

What a cold start is

A free server goes idle. The platform reclaims its memory. The next request wakes it up.

That wake-up costs seconds. Warm requests cost milliseconds.

Cold starts are not failures. They are slow successes. The difference matters.

The probe

Here is the script. It needs Node 18+ and nothing else.

// coldstart-probe.mjs
// Measures how a free model server sleeps and wakes.
//
// Usage:
//   node coldstart-probe.mjs find <url> [max-idle-min]
//   node coldstart-probe.mjs burst <url> <idle-min> <requests>

// --- adjust for your endpoint ---
const METHOD = 'POST';
const HEADERS = { 'content-type': 'application/json' };
const BODY = JSON.stringify({
  messages: [{ role: 'user', content: 'ping' }],
  max_tokens: 1,
});
const TIMEOUT_MS = 120_000;
// ---------------------------------

const [mode, url, a, b] = process.argv.slice(2);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

if (!url) {
  console.error('Missing URL.');
  process.exit(1);
}

async function hit(label) {
  const start = performance.now();
  let status = 0;
  try {
    const res = await fetch(url, {
      method: METHOD,
      headers: HEADERS,
      body: METHOD === 'GET' ? undefined : BODY,
      signal: AbortSignal.timeout(TIMEOUT_MS),
    });
    status = res.status;
    await res.text();
  } catch (err) {
    console.log(`${label},error,${err.name},${(performance.now() - start).toFixed(0)}`);
    return;
  }
  console.log(`${label},${status},ok,${(performance.now() - start).toFixed(0)}`);
}

async function findThreshold(maxIdleMin) {
  const intervals = [1, 2, 5, 10, 15, 30, 45, 60].filter((m) => m <= maxIdleMin);
  console.log('label,status,result,latency_ms');
  for (const min of intervals) {
    await hit(`warmup_${min}`);
    console.error(`# sleeping ${min} min`);
    await sleep(min * 60_000);
    await hit(`after_${min}min`);
  }
}

async function burst(idleMin, requests) {
  console.log('label,status,result,latency_ms');
  await hit('warmup');
  console.error(`# sleeping ${idleMin} min to force a cold start`);
  await sleep(idleMin * 60_000);
  for (let i = 1; i <= requests; i++) {
    await hit(`request_${i}`);
  }
}

if (mode === 'find') await findThreshold(Number(a || 60));
else if (mode === 'burst') await burst(Number(a || 10), Number(b || 5));
else console.error('Unknown mode. Use find or burst.');

Two modes. find maps the sleep threshold. burst shows the warm-up curve.

Progress notes go to stderr. Stdout stays clean CSV.

I ran both against the free server option in MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The same probe works for free model endpoints. Point it at any HTTP API you do not fully trust.

How to run it

1. Save the script as coldstart-probe.mjs.
2. Adjust METHOD, HEADERS, and BODY for your endpoint.
3. Run find first. Start with max-idle-min at 30.
4. Find the first interval where latency jumps.
5. Run burst with that interval. Confirm the warm-up curve.
6. Save the CSV. Re-run it after any plan change.

Example output

Your numbers will differ. The shape will not.

$ node coldstart-probe.mjs find https://your-endpoint.example/v1/chat 30

label,status,result,latency_ms
warmup_1,200,ok,812
after_1min,200,ok,734
warmup_2,200,ok,701
after_2min,200,ok,688
warmup_5,200,ok,655
after_5min,200,ok,742
warmup_10,200,ok,690
after_10min,200,ok,9801

See the jump? after_5min is fast. after_10min is not.

The threshold sits between five and ten minutes. That is your server's real sleep cycle.

Now the burst:

$ node coldstart-probe.mjs burst https://your-endpoint.example/v1/chat 10 5

label,status,result,latency_ms
warmup,200,ok,698
request_1,200,ok,11203
request_2,200,ok,689
request_3,200,ok,671
request_4,200,ok,702
request_5,200,ok,655

Request one is cold. Requests two through five are warm. That is the cold-start signature.

Compare the two outputs. Cold is a spike after a long sleep. Slow is a flat line on every request.

The probe tells them apart.

Myth 1: Free servers sleep after exactly five minutes

Someone measured one platform. Everyone repeated the number.

Free tiers rarely document idle behavior. Even when they do, the number changes.

The probe settles it. Run find.

Your threshold might be five minutes. It might be thirty. It might be 'never sleeps, but throttles the first request.'

Corrected mental model: idle timeout is an empirical property. Measure it. Re-measure after every plan change.

Myth 2: Ping it every 30 seconds to keep it warm

A ping is a request. Thirty seconds means 2,880 pings per day.

On a request-quota free tier, that is real consumption. Some platforms also ignore pings when resetting the idle timer.

The probe shows a cheaper path. One warmup request is enough. The server stays warm for your whole burst.

Corrected mental model: warmth is a resource. Spend one request to warm up. Do not spend 2,880.

Myth 3: A cold start means the request failed

A cold start is a slow success. The status is 200. The latency is ten seconds.

The failure comes from your client. It gave up at three seconds.

The probe records both. Status tells the truth. Latency tells the story.

Corrected mental model: set timeouts from the cold-start tail. Add one retry for real failures. Do not confuse slow with down.

Myth 4: Cold-start latency is one number

It is a distribution. It depends on sleep duration, host load, and model size.

The first request after idle is the slow one. The second is usually ten times faster.

The burst output proves it. request_1 is the outlier. The rest cluster together.

Corrected mental model: report p50, p95, and p99. Design for the tail. Never design for the average.

Myth 5: A free server can't handle real traffic

It cannot handle synchronous, user-facing traffic with a tight latency budget. It handles batch jobs, CI runs, preview environments, and background tasks fine. Those workloads do not care about a ten-second wake-up.

The probe shows the split. Warm, the server answers in hundreds of milliseconds. Cold, it answers in seconds.

The workload decides which one you can afford.

Corrected mental model: match the workload to the sleep behavior. Queue the work. Warm the server. Or accept the cold start.

The decision table

Workload	Cold start observed	Verdict
CI job, cron, batch	10s	Fine. Add retries.
Interactive dev tool	2-3s	Fine. Show a spinner.
User-facing API	10s	Add warm-up or a queue.
Realtime, sub-second	any	Wrong tool.

The thresholds are examples. Your users set the real ones.

Limitations

The probe measures your network path too. Run it from the same region as your real clients.

One run is a sample, not a guarantee. Run it at different times. Run it on different days.

Free tiers change. Re-run the probe after any plan or pricing change.

The probe consumes quota. Budget for it.

It does not measure throughput, rate limits, or quota caps. Those need separate probes.

Who should not use this

Teams with strict sub-second p99 requirements. Free servers are the wrong tool. This probe will not fix that.

Anyone who needs a long-running warm container for stateful work. Free servers sleep. State does not survive the nap.

Teams with complex auth flows. Adapt the script before running it.

The takeaway

Cold starts are measurable. Sleep thresholds are measurable. Warm-up curves are measurable.

The only thing you should not measure is your patience with guessing.

Run the probe. Then decide. Guessing is the only thing that is actually free.

Comments

[Luis Cruz]

The insight about differentiating between cold starts and regular latency is quite valuable, especially for developers relying on free model servers where performance can be unpredictable. Your probe script is a fantastic tool for pinpointing the sleep thresholds and understanding server behavior more accurately. One improvement idea could be to add a logging feature that tracks and visualizes the latency over time, providing clearer trends for developers. If you’re looking to expand this tool or address other related performance metrics, I’d love to discuss potential collaboration on that!