Real Traffic Is the Only Benchmark: Replay Requests Before You Swap Models

A demo passes. Production fails. That is the normal model swap story. Replay real traffic before you switch. A replay harness captures live requests. It sends them to a candidate model. Then it diffs the answers against production. This tutorial builds that harness in five verified stages.

A repo test tells you if a model can code. It does not tell you if a swap is safe. Your users' traffic is the test set you already have. Use it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The candidate endpoint below is a MonkeyCode free model access endpoint. The harness is designed to run on a free server. You can substitute any OpenAI-compatible endpoint. Model names and quotas are not part of this method.

What You Need

• A server with Node.js 18 or newer
• A current production endpoint (the model you run today)
• A candidate endpoint (MonkeyCode free model access)
• One hour and a small JSONL file

Stage 1: Capture Real Requests

Run a logging proxy in front of your current endpoint. It records every request body. It forwards the request upstream. Then it appends one JSON line per call.

// capture-proxy.mjs
import { createServer } from "node:http";
import { appendFile } from "node:fs/promises";

const TARGET = process.env.TARGET_URL;
const LOG = process.env.LOG_FILE || "./traffic.jsonl";
const PORT = process.env.PORT || 3000;

const server = createServer(async (req, res) => {
  const chunks = [];
  for await (const chunk of req) chunks.push(chunk);
  const raw = Buffer.concat(chunks).toString("utf8");

  const started = Date.now();
  const upstream = await fetch(TARGET, {
    method: req.method,
    headers: { "content-type": "application/json" },
    body: raw || undefined,
  });
  const upstreamBody = await upstream.text();
  const latency = Date.now() - started;

  if (req.method === "POST" && req.url.includes("/chat/completions")) {
    const record = {
      ts: new Date().toISOString(),
      path: req.url,
      status: upstream.status,
      latency_ms: latency,
      request: JSON.parse(raw),
      response: JSON.parse(upstreamBody),
    };
    await appendFile(LOG, JSON.stringify(record) + "\n", "utf8");
  }

  res.writeHead(upstream.status, { "content-type": "application/json" });
  res.end(upstreamBody);
});

server.listen(PORT, () => console.log(`capture proxy on :${PORT}`));

Save the file as capture-proxy.mjs. Node treats .mjs files as ES modules. Top-level await needs that.

Verify Stage 1:

node capture-proxy.mjs &
curl -s -X POST localhost:3000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}]}'
wc -l traffic.jsonl   # expect 1

Run the proxy for one day. Or run it for one hour. The value grows with volume. Redact secrets before you start. Request bodies can contain user data.

Stage 2: Replay Against the Candidate

The replay script reads traffic.jsonl. It sends each request to the candidate endpoint. It buffers the full response before parsing. A streaming chunk is not a payload.

// replay.mjs
import { readFile, appendFile } from "node:fs/promises";

const [logFile, target, apiKey] = process.argv.slice(2);
const lines = (await readFile(logFile, "utf8")).trim().split("\n");

for (const line of lines) {
  const record = JSON.parse(line);
  const started = Date.now();
  const res = await fetch(target, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${apiKey}`,
    },
    body: JSON.stringify(record.request),
  });
  const latency = Date.now() - started;
  const text = await res.text();

  let response = null;
  try { response = JSON.parse(text); } catch { /* keep null */ }

  await appendFile("results.jsonl", JSON.stringify({
    ts: record.ts,
    status: res.status,
    latency_ms: latency,
    response,
  }) + "\n", "utf8");
}

Verify Stage 2:

node replay.mjs traffic.jsonl "$CANDIDATE_URL" "$API_KEY"
wc -l results.jsonl   # same count as traffic.jsonl

Keep the order stable. The diff joins both files by line number. A failed parse stays null. A null response is a signal, not a crash.

Stage 3: Diff Production Against Candidate

The diff script joins traffic.jsonl and results.jsonl. It compares status, latency, content, and token usage.

// diff.mjs
import { readFile } from "node:fs/promises";

const traffic = (await readFile("traffic.jsonl", "utf8")).trim().split("\n").map(JSON.parse);
const results = (await readFile("results.jsonl", "utf8")).trim().split("\n").map(JSON.parse);

const rows = traffic.map((t, i) => {
  const r = results[i] ?? {};
  const prod = t.response?.choices?.[0]?.message?.content ?? null;
  const cand = r.response?.choices?.[0]?.message?.content ?? null;
  return {
    request: i,
    prod_status: t.status,
    cand_status: r.status,
    prod_ms: t.latency_ms,
    cand_ms: r.latency_ms,
    exact_match: prod === cand,
    both_valid: prod !== null && cand !== null,
    prod_tokens: t.response?.usage?.total_tokens ?? null,
    cand_tokens: r.response?.usage?.total_tokens ?? null,
  };
});

console.table(rows);

const valid = rows.filter((x) => x.cand_status === 200 && x.both_valid).length;
const exact = rows.filter((x) => x.exact_match).length;
console.log(`valid: ${valid}/${rows.length}`);
console.log(`exact match: ${exact}/${rows.length}`);

Verify Stage 3:

node diff.mjs
# valid: 47/50
# exact match: 41/50

A count mismatch means the replay failed partway. Investigate before trusting the diff. Read the mismatches. Exact match is a weak signal for creative tasks. It is a strong signal for extraction, classification, and formatting jobs. Look at three mismatches by hand. Then decide.

Stage 4: Apply a Decision Table

Numbers need thresholds. Start with these. Tune them to your traffic.

Signal	Switch threshold	Reject threshold
Valid response rate	≥ 99%	< 95%
Exact content match	≥ 80%	< 50%
Median latency	≤ 1.5× production	> 3× production
Token usage per call	≤ 1.2× production	> 2× production

Compute medians, not means. One slow outlier inflates an average. It does not move a median. The table uses medians for that reason. Two rules keep the table honest. First, a single failing signal can block a swap. Second, all four signals must pass to switch. A fast model with bad content is still bad.

Stage 5: Schedule the Replay

Free servers kill long-lived processes. A replay job is short-lived. Run it as a cron job. One line covers the pattern.

0 */6 * * * cd /opt/replay && node replay.mjs traffic.jsonl "$CANDIDATE_URL" "$API_KEY" >> replay.log 2>&1

Then run the diff after every replay. Keep the last seven reports. A model can drift. Your replay set is the early warning.

Limitations

Replay is offline. It does not test concurrency, rate limits, or real-time behavior. Captured traffic reflects your users. It does not reflect the general public. Token deltas are approximate. Tokenizers differ across models. The harness stores full payloads. Delete the capture file after the swap decision. Add a retention policy if you run replay weekly.

Who Should Not Use This

Teams without production traffic should skip replay. There is nothing to capture. Teams testing safety policy need human review. A structural diff will not catch subtle refusals. Teams needing live shadowing should build an online shadow proxy instead. Replay answers one question: does the candidate behave like the current model on your real traffic?

The One-Hour Swap Check

Replay is not a benchmark. It is a before-and-after measurement. It turns your existing traffic into a regression test. Five stages take about an hour. Run it before every swap. Your users already wrote the test cases.

MonkeyCode's free model access and free server option are enough to run this harness end to end. Start with one day of captured traffic. Then decide with data.