Time the Cold Start Before You Trust a Free Server With Your Review Bot

The question nobody in the room could answer

A pairing hour in mid-October, two engineers, one prototype. The bot takes a unified diff, asks a model for structured findings, and posts them on a pull request. It ran on a free server instance and called a free model endpoint.

Nobody could say what the first request after ten minutes of idle actually cost in wall-clock time.

Three things were unknown, and all three were being papered over:

• the latency of the first request after an idle period;
• the timeout the caller had assumed (somewhere between 10 seconds and never);
• whether the person who found the number could reproduce it a week later.

Five questions the senior wrote on the whiteboard

1. What is the first-request latency after ten minutes of idle, measured on this deployment, today?
2. Which component owns the timeout budget?
3. Can someone who was not in this session reproduce the measurement in one command?
4. What happens to a request that arrives during a cold start?
5. If the number is bad, what changes — the code, the deploy, or the promise made to the reviewer?

Question five is the one that mattered later. It forces the decision to be written down before the measurement, so the measurement cannot be reinterpreted to fit a preference.

Two dead ends before the first line of code

Dead end one: read the logs. The request log on the free tier listed status codes and nothing else. No timestamps, no upstream duration. There was nothing to compute from.

Dead end two: read the documentation. Sleep behaviour for that plan was not documented. Three blog posts produced two contradictory assumptions, which is worse than zero assumptions because each side now had a citation.

The rule the senior set after that: an external observation is only trustworthy if the observed process also reports its own timestamps, so two clocks can be compared instead of one being guessed.

Where the free model access and the free server fit

MonkeyCode provides free model access and a free server option, and that combination is what made this prototype cheap enough to exist at all. Without it, the pairing hour would have been a budget conversation instead of a measurement.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest notes before any code. The current limits of the free access and the free server — quotas, sleep policy, hardware, model list — are not inputs to the method below and are not asserted here; read them from the project's own documentation on the day you run the probe, because they can change and this article would not know. And every number in the sample output further down is a placeholder that shows column shape, not a benchmark.

Artifact one: a service that describes its own state

The first kept decision was small and unglamorous: the service must report its own clock. Every response carries a server-side duration, and /health reports how long ago the model was last called. That single field separates "the network is slow" from "nothing has run in eleven minutes."

// server.mjs — Node 20+, zero dependencies
import http from 'node:http';
import crypto from 'node:crypto';
import { performance } from 'node:perf_hooks';

const PORT = Number(process.env.PORT ?? 8080);
const BASE_URL = process.env.MODEL_BASE_URL; // OpenAI-compatible endpoint
const MODEL_ID = process.env.MODEL_ID;
const API_KEY = process.env.MODEL_API_KEY;
const BUDGET_MS = Number(process.env.REVIEW_BUDGET_MS ?? 45000);

let lastModelCallAt = null;

function json(res, code, body) {
  const payload = JSON.stringify(body);
  res.writeHead(code, {
    'content-type': 'application/json',
    'content-length': Buffer.byteLength(payload),
  });
  res.end(payload);
}

async function review(diff, requestId) {
  const t0 = performance.now();
  try {
    const upstream = await fetch(`${BASE_URL}/chat/completions`, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        authorization: `Bearer ${API_KEY}`,
      },
      body: JSON.stringify({
        model: MODEL_ID,
        temperature: 0,
        response_format: { type: 'json_object' },
        messages: [
          { role: 'system', content: 'Return JSON only: {"findings":[{"file":string,"line":number,"claim":string}]}. No prose.' },
          { role: 'user', content: diff },
        ],
      }),
      signal: AbortSignal.timeout(BUDGET_MS),
    });
    const serverMs = Math.round(performance.now() - t0);
    lastModelCallAt = Date.now();
    if (!upstream.ok) return { requestId, ok: false, stage: 'upstream', upstreamStatus: upstream.status, serverMs };
    const body = await upstream.json();
    return { requestId, ok: true, serverMs, raw: body.choices?.[0]?.message?.content ?? null };
  } catch (err) {
    const serverMs = Math.round(performance.now() - t0);
    return { requestId, ok: false, stage: err.name === 'TimeoutError' ? 'budget' : 'transport', serverMs };
  }
}

const server = http.createServer(async (req, res) => {
  const requestId = crypto.randomUUID();

  if (req.url === '/health') {
    return json(res, 200, {
      status: BASE_URL && MODEL_ID ? 'configured' : 'missing_env',
      budgetMs: BUDGET_MS,
      lastModelCallAgoMs: lastModelCallAt ? Date.now() - lastModelCallAt : null,
    });
  }

  if (req.url === '/review' && req.method === 'POST') {
    const chunks = [];
    for await (const chunk of req) chunks.push(chunk);
    const diff = Buffer.concat(chunks).toString('utf8');
    if (diff.length > 40000) return json(res, 413, { requestId, error: 'diff_too_large' });
    return json(res, 200, await review(diff, requestId));
  }

  json(res, 404, { requestId, error: 'not_found' });
});

server.listen(PORT, () => console.log(`listening on ${PORT}`));

Two details are deliberate. The timeout is a named budget (REVIEW_BUDGET_MS) rather than a scattered literal, so question three has one answer. And a timeout returns a structured failure with the stage that failed, instead of an exception that the caller has to classify.

Artifact two: idle, then measure, then print CSV

Guessing was the whole problem, so the probe does the boring thing. It sleeps, fires one request, fires a second one immediately, and records both. Attempt 1 is the cold request; attempt 2 is the warm one. The difference between them is the entire finding.

// probe-idle.mjs — node probe-idle.mjs <baseUrl> <idleMinutes> <cycles>
const base = process.argv[2] ?? 'http://127.0.0.1:8080';
const idleMs = Number(process.argv[3] ?? 10) * 60000;
const cycles = Number(process.argv[4] ?? 3);
const diff = '--- a/x.js\n+++ b/x.js\n@@ -1,2 +1,2 @@\n-const a = 1\n+const a = 1;\n';

console.log('cycle,endpoint,attempt,status,ms');

for (let cycle = 1; cycle <= cycles; cycle++) {
  await new Promise((r) => setTimeout(r, idleMs));
  for (const endpoint of ['health', 'review']) {
    for (let attempt = 1; attempt <= 2; attempt++) {
      const t0 = performance.now();
      const res = endpoint === 'health'
        ? await fetch(`${base}/health`)
        : await fetch(`${base}/review`, { method: 'POST', body: diff });
      const ms = Math.round(performance.now() - t0);
      await res.arrayBuffer();
      console.log([cycle, endpoint, attempt, res.status, ms].join(','));
    }
  }
}

Run it and keep the CSV in the repo, because a committed CSV is a reproducible claim and a screenshot is not:

node probe-idle.mjs https://review-bot.example 10 3 > cold-start.csv

awk -F, 'NR>1 && $2=="review" && $3==1 {print $5}' cold-start.csv \
  | sort -n \
  | awk '{v[NR]=$1} END {printf "n=%d p50=%dms max=%dms\n", NR, v[int(NR/2)], v[NR]}'

Sample shape of the output — placeholders, not measurements:

cycle	endpoint	attempt	status	ms
1	health	1	200	NN
1	health	2	200	NN
1	review	1	200	NNNN
1	review	2	200	NNN

If attempt 1 and attempt 2 differ by an order of magnitude, the deployment sleeps and the reviewer will eventually see a timeout that has nothing to do with the model's answer quality.

The decision table, written before the run

Pattern	Fits when	Cost	How to verify
On-demand only	cold-start latency is inside the caller's budget	cheapest, but one slow first reviewer	probe attempt 1 under the caller's real timeout
Keep warm with a timer	cold start exceeds the budget and the free tier permits periodic requests	a timer that must not be mistaken for real traffic	read the tier's terms first, then confirm lastModelCallAgoMs stays bounded
Queue and batch reviews	reviewers tolerate minutes, not seconds	a queue you now operate	end-to-end timestamp on the queued job, not the HTTP call
Retry inside the gateway	never, for cold starts	hides the failure and doubles cost	it should fail review; a second request measures, it does not fix

The senior crossed out the fourth row personally. A retry that masks a cold start turns a measurable problem into an intermittent one.

What was kept, and what was dropped

Kept: /health with lastModelCallAgoMs, the named timeout budget, the structured failure stage, and probe-idle.mjs with its CSV.

Dropped: the keep-alive cron job, at least until someone reads the terms of the free tier rather than assuming periodic pings are welcome. A five-minute systemd timer is trivial to write and trivial to get wrong:

# /etc/systemd/system/review-health.timer
[Unit]
Description=Ping the review bot health endpoint

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

If that timer ever ships, it ships with a note explaining why it exists and what it costs, because the next reader will otherwise delete it as noise.

Limitations, and who should skip this

This method measures one deployment on one day. Sleep policy, quotas, and free-tier behaviour can change without a changelog, so a CSV from last month is evidence about last month.

The probe also says nothing about answer quality — a fast wrong review is still wrong, and latency numbers should never be used to rank models. And a single cold observation is not an SLO; if you need a p95 promise, you need paid capacity or your own host, not a free tier and a hopeful timer.

Skip this approach if the diffs contain anything that cannot leave your network, if reviewers already expect sub-second feedback, or if nobody on the team will own the CSV. A measurement nobody re-runs decays into folklore within a quarter.

The probe is the part worth keeping. If you want a free model endpoint and a free server to point it at while you learn your own numbers, MonkeyCode is a reasonable place to start — run probe-idle.mjs before you promise anyone a latency figure.