Mean Latency Is Not a Pass: A Timeout-Aware Load-Check Take-Home

Hiring screens that accept a green mean-latency number routinely miss bimodal services that timeout under a stated budget. This proposed take-home packet treats measurement design as the deliverable, not as an optional afterthought. The fixture is a small Node.js HTTP service with a hidden timeout cliff, plus a checker that must fail closed. The listings below are a proposed reference for local reproduction and are not production benchmark results.

The packet exists because generated clients often loop one request, print an arithmetic mean, and declare the API healthy. That workflow resembles a vanity dashboard more than an SLO, and it now appears frequently in agent-authored take-homes. A service can answer most calls in a few milliseconds and still drop a tail that exceeds the product budget. Mean latency then looks calm while callers retry, pile on, and turn a small stall into an outage.

The failure this packet isolates

The grader should score three claims together, because a fast happy path is not the same as a passing check.

1. The service stays correct under overlapping clients, including invalid JSON and missing fields.
2. Each request has a deadline, and a late HTTP 200 counts as a failure against the budget.
3. Reported numbers include percentiles, timeout rate, and error rate, not only a mean.

A mean-only checker can bless a handler that is fast on ninety-five percent of calls and over budget on the rest. Closed-loop tests make the lie worse by waiting for each response before issuing the next call. That waiting hides lost capacity, which load testers have long described as coordinated omission. This packet therefore grades the measurement method as strictly as the service implementation.

Why the mean hides a cliff

Consider a bimodal handler under a five-hundred-millisecond budget. Ninety-five percent of requests return in about ten milliseconds, while five percent sleep long enough to miss the budget. The arithmetic mean can still land near one hundred fifty milliseconds, which looks acceptable on a summary slide. Percentile views and timeout counts tell a different story, because the slow class already violated the budget. Treating those late two-hundred responses as successes then trains both humans and agents to ship a cliff.

The second-phase trick is to run the candidate checker against two servers that share a contract. A clean server must produce pass: true. A stalled fixture must produce pass: false even when its mean still looks modest. Checkers that never abort, never compute percentiles, or never count timeouts will pass the wrong server.

Take-home prompt for the candidate or agent

Time box: 75 minutes. Stack: Node.js 18 or later, standard library only. Constraint: no extra packages, no cloud accounts, bind only to 127.0.0.1.

Prompt text to paste:

Build two programs in one directory.

1. server.js listens on 127.0.0.1:8080 and handles POST /checkout with body { "cartId": string, "items": integer }. Valid carts return 200 and { "ok": true, "totalItems": <n> }. Invalid JSON or missing fields return 400 with { "ok": false }. Unknown paths return 404. The process must remain up under concurrent requests.
2. loadcheck.js issues one hundred twenty timed requests to that endpoint with concurrency eight, after a twenty-request warmup that is discarded. Each request must abort at 500ms. Print one JSON object and then exit 0 on pass or 2 on fail.

Required JSON shape:

{
  "meanMs": 0,
  "p50Ms": 0,
  "p95Ms": 0,
  "p99Ms": 0,
  "timeoutRate": 0,
  "errorRate": 0,
  "pass": true
}

pass may be true only when timeoutRate is 0, errorRate is 0, and p95Ms is at most 500. A low mean cannot override a high p95 or a nonzero timeout rate. Timed-out calls must be counted in timeoutRate and included in the percentile sample at 500ms, so a tail cannot vanish from p95.

Hidden grader note, not shown to the candidate: keep a fixture server that stalls every twentieth request by 800ms. Run the candidate checker against that fixture and against a clean server. The checker must fail the fixture and pass the clean process.

Rubric

Score out of 12. A packet that only prints a mean cannot pass, even if server.js is tidy.

Score	Signal	What the grader actually runs
0–2	Server contract	Valid POST returns 200; bad JSON returns 400; process survives overlap
0–3	Deadlines	Each call aborts at 500ms; hung sockets do not stall the checker
0–3	Statistics	Warmup discarded; p50/p95/p99 from sorted samples; timeouts included at 500ms
0–2	Fail-closed pass flag	Clean server passes; stalled fixture fails despite a calm mean
0–2	Concurrency	Eight in-flight requests, not a sequential for/await loop

Decision table for the checker, which is the real artifact:

Observation	Mean-only checker	Timeout-aware checker
Fast service, no stall	pass	pass
Five-percent 800ms stall, 500ms budget	often pass	fail
HTTP 200 after two seconds	pass	fail, counted as timeout or error
Connection hang	may hang forever	abort, timeoutRate rises
Sequential single-socket loop	green and slow	concurrency score is lost

Proposed fixture server

The stall is an environment switch so the same file can act as the clean server or the cliff. This is proposed reference code, not a measured lab report.

'use strict';

const http = require('http');

const PORT = Number(process.env.PORT || 8080);
const STALL_MS = Number(process.env.STALL_MS || 0);
const STALL_EVERY = Number(process.env.STALL_EVERY || 0);

function send(res, status, body) {
  const json = JSON.stringify(body);
  res.writeHead(status, {
    'content-type': 'application/json',
    'content-length': Buffer.byteLength(json),
  });
  res.end(json);
}

let n = 0;
const server = http.createServer((req, res) => {
  if (req.method !== 'POST' || req.url.split('?')[0] !== '/checkout') {
    send(res, 404, { ok: false, error: 'not_found' });
    return;
  }

  const chunks = [];
  req.on('data', (c) => chunks.push(c));
  req.on('end', () => {
    let payload;
    try {
      payload = JSON.parse(Buffer.concat(chunks).toString('utf8') || '{}');
    } catch {
      send(res, 400, { ok: false, error: 'invalid_json' });
      return;
    }

    const cartId = payload && payload.cartId;
    const items = payload && payload.items;
    if (typeof cartId !== 'string' || cartId.length < 1 || !Number.isInteger(items) || items < 1) {
      send(res, 400, { ok: false, error: 'invalid_cart' });
      return;
    }

    n += 1;
    const stall = STALL_EVERY > 0 && n % STALL_EVERY === 0;
    const delay = stall ? STALL_MS : 8;
    setTimeout(() => {
      send(res, 200, { ok: true, totalItems: items });
    }, delay);
  });
});

server.listen(PORT, '127.0.0.1', () => {
  process.stdout.write(`listening on 127.0.0.1:${PORT}\n`);
});

Proposed reference checker

The important details are abort, warmup discard, concurrency, and the fail-closed pass flag. Percentiles use a simple nearest-rank on a sorted list, which is enough for this packet.

'use strict';

const BASE = process.env.BASE_URL || 'http://127.0.0.1:8080';
const TOTAL = 120;
const WARMUP = 20;
const CONCURRENCY = 8;
const BUDGET_MS = 500;

function percentile(sorted, p) {
  if (sorted.length === 0) return null;
  const idx = Math.min(
    sorted.length - 1,
    Math.max(0, Math.ceil((p / 100) * sorted.length) - 1)
  );
  return sorted[idx];
}

async function oneRequest() {
  const started = Date.now();
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), BUDGET_MS);
  try {
    const res = await fetch(`${BASE}/checkout`, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ cartId: 'cart-1', items: 3 }),
      signal: ac.signal,
    });
    const ms = Date.now() - started;
    if (!res.ok) return { kind: 'error', ms };
    await res.json();
    return { kind: 'ok', ms };
  } catch (err) {
    const ms = Date.now() - started;
    if (err && err.name === 'AbortError') {
      return { kind: 'timeout', ms: BUDGET_MS };
    }
    return { kind: 'error', ms };
  } finally {
    clearTimeout(timer);
  }
}

async function pool(count, concurrency, fn) {
  const out = [];
  let i = 0;
  async function worker() {
    while (i < count) {
      const idx = i;
      i += 1;
      out[idx] = await fn();
    }
  }
  await Promise.all(Array.from({ length: concurrency }, () => worker()));
  return out;
}

(async () => {
  await pool(WARMUP, CONCURRENCY, oneRequest);
  const results = await pool(TOTAL, CONCURRENCY, oneRequest);
  const samples = results.map((r) => r.ms).sort((a, b) => a - b);
  const timeouts = results.filter((r) => r.kind === 'timeout').length;
  const errors = results.filter((r) => r.kind === 'error').length;
  const meanMs = samples.reduce((a, b) => a + b, 0) / samples.length;
  const report = {
    meanMs: Math.round(meanMs * 100) / 100,
    p50Ms: percentile(samples, 50),
    p95Ms: percentile(samples, 95),
    p99Ms: percentile(samples, 99),
    timeoutRate: timeouts / results.length,
    errorRate: errors / results.length,
  };
  report.pass =
    report.timeoutRate === 0 &&
    report.errorRate === 0 &&
    report.p95Ms !== null &&
    report.p95Ms <= BUDGET_MS;
  process.stdout.write(JSON.stringify(report) + '\n');
  process.exitCode = report.pass ? 0 : 2;
})();

Commands the grader actually types

Use two terminals on the same loopback interface. Do not point this fixture at a shared staging host.

# Terminal A: clean server, expect loadcheck pass
STALL_MS=0 STALL_EVERY=0 node server.js

# Terminal B
node loadcheck.js
# expect pass: true and exit 0

# Terminal A: timeout cliff, expect loadcheck fail
STALL_MS=800 STALL_EVERY=20 node server.js

# Terminal B
node loadcheck.js
# expect pass: false, timeoutRate > 0, p95Ms at least 500, exit 2

A mean that still looks modest on the stalled run is the teaching moment. The checker must not treat that mean as a pass.

Common failure modes

Agents and candidates collapse on the same handful of shortcuts. Graders should keep this list beside the rubric rather than debating style.

• Sequential awaits. A for loop with await fetch never reaches concurrency eight, so the cliff is sampled too politely.
• Mean as the only number. The stalled fixture still looks fine, which is exactly the hiring lie this packet exists to catch.
• Percentile arithmetic. Computing p95 as 0.95 * max or as the mean times a constant does not inspect a sorted tail.
• Timeouts omitted from samples. Dropping aborted calls makes p95 look fast while users were already waiting on the budget.
• Deadline as setTimeout logging only. Logging a warning without AbortController or req.destroy() still waits for the stall.
• Warmup mixed into the score. The first requests can include module load and socket setup, which inflates or hides the cliff.
• HTTP 200 worship. Accepting a two-second 200 against a 500ms budget trains the same vanity dashboard the packet rejects.
• Process death on the first 400. A checker that sends only happy bodies never proves the server stays up on invalid JSON.

Running the packet in a shared workspace

The checker needs a real port, real timers, and overlapping sockets, so a paste-only chat window is a weak harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which can hold server.js and loadcheck.js in one process environment without adding cloud credentials to the exercise. Those two availability claims are the only product facts used here; this article does not name models, quotas, hardware, or promised duration.

A useful workflow is to let the agent draft both files, then have the grader run the two command blocks above on that server. The rubric stays unchanged. The free model pass is for producing a first draft, not for replacing the fail-closed checks.

Limitations and who should skip this packet

This approach is a local interview filter, not a capacity plan. One hundred twenty requests on loopback will not expose garbage collection, disk, TLS, or multi-region behavior. Nearest-rank percentiles on a short sample are coarse, and Date.now() millisecond buckets are not a histogram. Coordinated omission is only partly addressed by aborting and by counting timeouts at the budget.

Teams should not use this packet as production load testing against customer systems. It is also a poor first screen when the role does not own SLOs, timeouts, or concurrency. Skip it for exercises that only need a correct JSON parser, and skip it when the candidate has no Node.js runtime. Do not treat a green mean from any vendor dashboard as a substitute for the two-server grader run.

The core conclusion stays narrow. Average latency is a weak pass condition, and a take-home should fail a checker that cannot see a timeout cliff. Readers who want a shared process for the fixture can use that free server option and keep the rubric, the stall, and the fail-closed flag exactly as written.