Real API Traffic Is Your Missing Test Data: A Free-Server Capture Workflow

Real API traffic is the hardest test data to get and the most valuable one to have. Unit tests manufactured inside a test suite are clean. Production requests are not. They contain empty strings, huge payloads, duplicated keys, unexpected encodings, and timing-sensitive fields. You know this. The problem is you cannot just install a tcpdump next to your production API and start recording.

You need a capture point outside production. It should be always-on, separate from your own machine, and cheap. A free server is exactly that. This article shows a workflow that records real requests to a staging or secondary API endpoint, then uses a free model to mine that capture for edge cases worth turning into tests. No production changes, no sensitive data hoarding, and no invented fixtures.

The Capture Proxy

The proxy is the gateway in front of the service you want to observe. Every request passes through it, is forwarded unchanged, and only a tiny canonical description of the request/response is kept. You run the proxy on a remote host that has network access to the target service.

The core is a few lines of Node.js using the http module. It parses the request, forwards it, and captures a filtered trace.

// capture-proxy.js
const http = require("node:http");

const TARGET = new URL(process.env.TARGET_URL); // e.g. http://staging-api:3000

const seenHashes = new Set();

const server = http.createServer((req, res) => {
  let raw = "";
  req.on("data", (c) => (raw += c));
  req.on("end", () => {
    const forwardReq = http.request({
      hostname: TARGET.hostname,
      port: TARGET.port,
      path: req.url,
      method: req.method,
      headers: req.headers,
    }, (forwardRes) => {
      let responseBody = "";
      forwardRes.on("data", (c) => (responseBody += c));
      forwardRes.on("end", () => {
        const trace = makeTrace(req, raw, forwardRes, responseBody);
        const key = hash(trace);
        if (!seenHashes.has(key)) {
          seenHashes.add(key);
          appendTrace(trace);
        }
        res.writeHead(forwardRes.statusCode, forwardRes.headers);
        res.end(responseBody);
      });
    });
    forwardReq.write(raw);
    forwardReq.end();
  });
});

server.listen(8080);

The trace contains only the shape of the request, not the payload itself. You want enough to reproduce the call path but not enough to leak secrets. makeTrace implements that mask.

function makeTrace(req, body, response) {
  const url = new URL(req.url, "http://localhost");
  return {
    method: req.method,
    path: url.pathname,
    query: url.search,
    status: response.statusCode,
    bodyHash: sha1(body).slice(0, 16),
    bodySize: body.length,
    headerKeys: Object.keys(req.headers)
      .filter((k) => !k.startsWith("cookie") && k !== "authorization")
      .join(","),
  };
}

We store a hash of the body, the size, and the header names. The actual body is never written to disk. Later, when you see an interesting trace, you can replay it against the staging service manually to get the real body. That is the trick: the proxy does not need to retain sensitive data for you to build a useful corpus.

Running It on a Free Server

One practical way to get an always-on capture host without opening your wallet is to use MonkeyCode's free server option. That server can call your staging API from outside your local network. You keep the proxy on that box, let it run for a week, and collect a corpus of real requests.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free model access MonkeyCode offers then helps you mine that corpus, so you do not have to read every trace by hand.

Mining Traces With a Free Model

After a few days you have a JSONL file with hundreds of trace entries. The structure looks like this:

{"method":"POST","path":"/orders","query":"?plan=pro","status":400,"bodyHash":"a3f9...","bodySize":342,"headerKeys":"content-type,user-agent,x-request-id"}

Now the model comes in. You do not need to write a parser that guesses what is interesting. You feed a small sample of traces to a free model with a strict prompt that asks it to identify which traces would be worth replaying and turning into tests.

You are given API trace summaries. For each trace, classify it as:

- HIGH value: anomalous status, unusual body size, uncommon header set
- MEDIUM value: normal-looking call that might still hit an edge case
- LOW value: routine success

Output JSON array of objects with trace index and label. Do not explain. No code output.

MonkeyCode's free model access lets you run this classification over batches again and again without counting it against a paid quota. You can paste ten traces at a time and get labels back quickly. The goal is to reset your intuition: which calls are the strange ones? The model will usually flag a trace with a 400 response and a bodySize of 2 bytes, or a GET to /health with a query string that is never used. Those are exactly the kind of input that a test suite would never generate because they look like mistakes. They are not mistakes; they are clients being sloppy, and your API needs to handle them.

Turning Interesting Traces Into Tests

Once you have a shortlist, replay each trace against the staging service manually and record the exact request body. Then write a test that pins the server's behavior for that weird input. This is not characterization testing in the "lock everything" sense; it is targeted regression protection for the edges that matter.

test("POST /orders with empty plan query returns 400", async () => {
  const res = await fetch(
    `${API_BASE}/orders?plan=pro&plan=`,
    {
      method: "POST",
      body: "{}",
      headers: { "content-type": "application/json" },
    }
  );
  assert.equal(res.status, 400);
});

You are not testing for a specific output value; you are testing that the system does not crash and returns a controlled error. That is the real insight: production traffic often exposes "does it survive" more than "does it compute correctly".

Deciding What to Keep or Discard

Use this simple table when reviewing captured traces.

Trace property	Keep?	Reason
status >= 500	Always	Failure needs a repro
status 4xx with nonstandard body	Always	Client misuse reveals missing validation
body size >> P95	Often	Large payloads can stress buffer limits
same bodyHash but different status	Often	Same body but different outcomes suggests flakiness
Routine 200s with stable size	Skip	Already covered by happy-path tests
Contains authorization header	Never	Do not store credential metadata

The proxy already strips auth headers, but your replay step must also use safe test credentials.

Limitations and Who Should Not Use This

First, a capture point works only when you have traffic flowing to the target. If your staging API gets only one call per week, this workflow gives you almost nothing. You need a real consumer base, even if it is a handful of developers.

Second, the proxy as shown is synchronous and holds the socket open while forwarding. Under high throughput, it will start dropping or timing out. This is meant for a low-traffic staging or QA environment, not for production traffic. If you need production-scale capture, use a proper message queue and a dedicated ingestion pipeline.

Third, trace summaries hide the actual content. If you want to use them for differential testing, you will need a separate mechanism to fetch the original bodies. This method biases toward presence of oddities, not content of those oddities. For content-level analysis, you need a sanitized replay service.

Fourth, and this is important: a free model may hallucinate a reason for an edge case. Use its classifications as a signal, not as a verified truth. Replay the trace yourself before writing a test.

Who should skip this? If your project has a full contract test suite with schema validation and property-based testing, you may already cover most edge cases. This workflow is for teams that rely on examples and happy paths, and for those who want a no-cost way to find out what their real clients are doing. It is also not for security-sensitive APIs where even hashes of bodies could leak information via length.

The Net Effect

Your tests are only as good as your inputs. A free remote server costs you nothing to run the capture, and a free model shrinks the reading time from three hours to thirty minutes. The artifact is small: one Node.js script, a JSONL file, and a handful of regression tests born from real-world weirdness.

Stop inventing test cases. Start recording them.