The hardest regressions from AI-suggested changes are the ones every unit test green-lighted. A patch can pass compile, pass the function suite, and still break on the sixth request in a burst because it assumes the items array is never empty. Static review catches style and obvious logic errors. It does not see the request pattern your service actually receives on a Wednesday morning.
A common next step is to ask another model to review the diff. That gives you an opinion, usually with confident wording. What it doesn't give you is evidence about runtime behavior. This article describes a different step: deploy the candidate change to a shadow service, replay a day of real traffic against it, and compare what comes back.
Why replay gives you evidence, not a verdict
When two implementations answer the same request, you can compare status codes and bodies directly. That comparison is reproducible. Two runs over the same log should produce the same mismatch list, except for non-deterministic features you deliberately normalize. A model-generated review is harder to reproduce because temperature, prompt wording, and recent context change the output.
A shadow service fits a disposable server well. You do not need it to be production-grade; you need it to run the candidate code with enough of the same dependencies to respond realistically. The cheaper that server is to throw away, the more often you can run the comparison. That is where a free server option, if you have one, changes the frequency of verification rather than the final verdict.
A replay harness you can run on a disposable server
The harness below reads a JSONL request log, sends each request to both a baseline service and a candidate service, then prints mismatches. It skips write methods by default because replaying writes safely requires a shadow database and careful cleanup.
import json
import sys
from urllib.request import Request, urlopen
from urllib.error import HTTPError
WRITE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}
SENSITIVE_HEADERS = {"authorization", "cookie", "x-api-key"}
def load_requests(path):
with open(path, encoding="utf-8") as f:
for line in f:
if line.strip():
yield json.loads(line)
def safe_headers(headers):
return {
k: v for k, v in headers.items()
if k.lower() not in SENSITIVE_HEADERS
}
def call(base_url, req, timeout=5):
url = base_url + req["path"]
body = req.get("body")
data = None if body is None else json.dumps(body).encode()
request = Request(
url,
data=data,
headers=safe_headers(req.get("headers", {})),
method=req.get("method", "GET"),
)
try:
with urlopen(request, timeout=timeout) as resp:
payload = resp.read().decode()
try:
payload = json.loads(payload)
except json.JSONDecodeError:
pass
return resp.status, payload
except HTTPError as e:
payload = e.read().decode()
try:
payload = json.loads(payload)
except json.JSONDecodeError:
pass
return e.code, payload
def normalize(value, unstable_keys):
"""Replace values at given dotted paths with a sentinel."""
# A production version should use JSONPath or a recursive walker.
if isinstance(value, dict):
return {k: normalize(v, unstable_keys) for k, v in value.items()}
if isinstance(value, list):
return [normalize(v, unstable_keys) for v in value]
return value
def main():
baseline = sys.argv[1]
candidate = sys.argv[2]
request_log = sys.argv[3]
mismatches = []
for req in load_requests(request_log):
if req.get("method") in WRITE_METHODS:
continue
status_a, body_a = call(baseline, req)
status_b, body_b = call(candidate, req)
# Shallow compare for the article; add canonicalization in practice.
if status_a != status_b or body_a != body_b:
mismatches.append({
"path": req["path"],
"method": req.get("method", "GET"),
"baseline_status": status_a,
"candidate_status": status_b,
"baseline_body": body_a,
"candidate_body": body_b,
})
print(f"{len(mismatches)} mismatches")
for item in mismatches[:20]:
print(json.dumps(item, indent=2))
if __name__ == "__main__":
main()
This is intentionally simple. Before using it, replace the shallow body comparison with a canonicalizer that strips volatile fields such as IDs, timestamps, trace IDs, and generated URLs. Otherwise, the mismatch list will be noisy enough to ignore.
Decide what counts as a mismatch
Not every difference deserves the same response. A useful decision table looks like this:
| Difference type | Action |
|---|---|
| Status code differs | Investigate immediately |
| Body differs only in normalized volatile fields | Treat as a match |
| Body differs in a known date field format | Fix the candidate serializer |
| Same status, unexpected body difference | Review the specific request and response |
| Timeout or connection error on candidate | Profile the changed code path |
After the deterministic compare runs, a model can help group the mismatches into a shorter report—for example, "12 mismatches are timestamp-only, 3 are empty-array handling, and 1 is a 500 on missing locale." That reduces reading time without making the model the approver. If MonkeyCode's free model access and free server option are available to you, they map to those two resource slots: the server hosts the shadow instance, and the model summarizes the deterministic mismatch list. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
What replay will not catch
Replay only covers the requests in your log. It misses cold-start latency, enormous payloads you have never seen, and the first request after a cache flush. If your candidate introduces a new dependency, the shadow server may fail differently from production because the dependency environment is not identical. That does not make the test useless; it means you should treat a clean replay as a reduction in risk, not proof of correctness.
Write operations are the biggest gap. Running a candidate against the same database as production is dangerous. If you need to verify writes, start with a restored read replica or a dedicated shadow database, and make sure the request log cannot contain credentials or personal data. Redact before storing the log, not after.
Start with one read-heavy endpoint
Pick an endpoint that receives real traffic and has a stable response shape. Capture one day of requests, strip sensitive headers, and run the harness on the smallest disposable server you can find. If the mismatch list is zero after canonicalization, you have evidence the candidate behaves like today's service for that traffic. If it is not zero, you have a list of concrete requests to debug—far better than arguing with a model about whether a diff is safe.
Do not skip the baseline. Comparing candidate against what you think should happen is weaker than comparing it against what currently happens. Real traffic is messy, and the baseline already encodes that mess. Use it.
Top comments (0)