Replay Fixtures Are Still Live Traffic

The core conclusion is simple enough to keep on a note beside your editor while you debug. When you attach a recorded HTTP cassette to a remote coding model, you are not sending dummy data. The same is true for a HAR export or a snapshot of a failing request. You are sending a compact copy of production-shaped traffic, complete with cookies, bearer tokens, and internal hostnames.

Most teams treat fixtures as theatrical props because they live under test directories and look boring in review. That folder is closer to a flight recorder than a costume trunk, and the remote model cannot tell the difference. You already learned not to paste database passwords, and you already keep dotenv files off the prompt. Replay files are the remaining envelope that still leaves the building with the original letter inside.

A useful analogy is the security-camera recorder sitting in a closed shop after the staff have gone home. You would not email last night's tape to a stranger just to ask why the door sensor false-triggered. That tape also shows customer faces, badge numbers, and the person who typed at the keypad. A VCR cassette, a Playwright trace, or a saved HAR file is that tape for your API.

The boundary sits at the socket

Draw the trust boundary at the network socket that carries the prompt, not at gitignore and not at the chat UI. Everything assembled into the request body has already left your process and entered another administrative domain. That includes the snippet you highlighted, the buffer an agent indexed, and the fixture a test runner dumped after a red build.

You should name three observers before anything leaves the machine that you actually control today. The first is your laptop, which you mostly trust because you can pull the power cord. The second is the inference host, which you should treat as a curious intern with a perfect memory for this session. The third is anyone who later obtains logs, support dumps, or cached prompts from that host.

Free remote endpoints do not redraw that picture; they only make the second and third observers easier to ignore during a late debugging session. Secrets in this model are not limited to cloud access keys that sit in environment files. A session cookie, a pagination cursor that embeds a user id, and a signed upload URL are secrets of different grades. Logs make the problem worse, because CI prints headers when a test fails and you paste the job output to explain a 401.

One ordinary bad paste

Imagine a checkout test that failed on staging after you merged a routine dependency bump. You copy the GitHub Actions log, a YAML cassette under fixtures, and a Playwright trace you unpacked because the screenshot looked wrong. The log contains a bearer token because someone left HTTP debug enabled on the client library. The cassette repeats that same header on every recorded call toward an internal payments hostname.

None of those files looks like a dotenv file, so scanners that only hunt environment variables stay quiet. The remote model then proposes a retry helper and, trying to be useful, inlines the bearer token inside a comment. You have now copied a live credential into a transcript whose retention you do not set. That is the failure this walkthrough is meant to interrupt before it becomes a rotation ticket.

Inventory the working set

Treat every outbound prompt as a document with attachments, then inventory those attachments on disk before the socket opens. The script below is a local preflight, not a vendor product and not a promise of complete coverage. It walks a directory, reads common replay formats as text, and prints why a given file should stay off the wire.

#!/usr/bin/env python3
"""Local preflight: find replay-shaped files that look unsafe to send to a model."""
from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

HEADER_RE = re.compile(
    r"(?i)(authorization|cookie|set-cookie|x-api-key|x-amz-security-token)\s*[:=]\s*\S+"
)
JWT_RE = re.compile(
    r"eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}"
)
EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
SUFFIXES = {".har", ".yml", ".yaml", ".json", ".snap", ".txt", ".log"}


def looks_like_replay(path: Path, text: str) -> str | None:
    name = path.name.lower()
    head = text[:2000]
    if name.endswith(".har") or ('"entries"' in head and '"request"' in head):
        return "har-log"
    if "interactions:" in text or "recorded_with:" in text:
        return "vcr-cassette"
    if "nock(" in text or "scope.filteringPath" in text:
        return "nock-fixture"
    if name.endswith(".snap") and ("Authorization" in text or "Set-Cookie" in text):
        return "snapshot-headers"
    if "playwright" in head.lower() and "cookies" in text.lower():
        return "playwright-trace-text"
    return None


def findings(text: str) -> list[str]:
    hits: list[str] = []
    if HEADER_RE.search(text):
        hits.append("auth-or-cookie-header")
    if JWT_RE.search(text):
        hits.append("jwt-shaped-token")
    if EMAIL_RE.search(text):
        hits.append("email-address")
    if "internal." in text or ".svc." in text:
        hits.append("internal-hostname")
    return hits


def main() -> int:
    parser = argparse.ArgumentParser(description="Inventory replay files before they leave the machine.")
    parser.add_argument("root", type=Path, nargs="?", default=Path("."))
    parser.add_argument("--max-bytes", type=int, default=2_000_000)
    args = parser.parse_args()
    rows = []
    for path in args.root.rglob("*"):
        if not path.is_file():
            continue
        cassette_name = "cassette" in path.name.lower()
        if path.suffix.lower() not in SUFFIXES and not cassette_name:
            continue
        try:
            text = path.read_bytes()[: args.max_bytes].decode("utf-8", errors="replace")
        except OSError as exc:
            print(f"# skip {path}: {exc}", file=sys.stderr)
            continue
        kind = looks_like_replay(path, text)
        if not kind:
            continue
        hits = findings(text)
        if hits:
            rows.append(
                {
                    "path": str(path),
                    "kind": kind,
                    "hits": hits,
                    "bytes": path.stat().st_size,
                }
            )
    json.dump(rows, sys.stdout, indent=2)
    print()
    return 1 if rows else 0


if __name__ == "__main__":
    raise SystemExit(main())

Save that file as preflight_replays.py and point it at the directory you were about to drop into the assistant. The JSON document it prints is the threat model in miniature: path, kind, hit classes, and size. You want those names in front of your eyes before a remote model ever sees the bytes.

python3 preflight_replays.py ./tests | tee /tmp/replay-inventory.json
if [ -s /tmp/replay-inventory.json ] && grep -q '"path"' /tmp/replay-inventory.json; then
  echo "strip or withhold the paths in /tmp/replay-inventory.json"
else
  echo "no text replay hits; still withhold binary traces"
fi

Labeled example of a hit you should take seriously, not an observation from a private repo:

[
  {
    "path": "tests/fixtures/checkout.yml",
    "kind": "vcr-cassette",
    "hits": ["auth-or-cookie-header", "jwt-shaped-token"],
    "bytes": 18422
  }
]

The command writes an inventory instead of rewriting fixtures in place, because silent mutation will break tests and hide the lesson you need. Look at the paths, then decide whether a sanitized excerpt is enough for the model to reason about status codes. If the inventory is empty, you still may have binary traces the decoder never opened, so do not treat silence as a blessing.

Here is a blunt sanitizer for a YAML cassette after you have copied a backup out of the repository tree. It is not cryptography, and it will not notice tokens hiding in base64 response bodies. Use it to produce an attachment, not to create a false sense that the original file became safe.

#!/usr/bin/env python3
from pathlib import Path
import re
import sys

REPLACERS = [
    (re.compile(r"(?i)(authorization:\s*)\S+"), r"\1REDACTED"),
    (re.compile(r"(?i)(cookie:\s*)\S+"), r"\1REDACTED"),
    (re.compile(r"(?i)(set-cookie:\s*)\S+"), r"\1REDACTED"),
    (re.compile(r"eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+"), "REDACTED_JWT"),
]


def main() -> None:
    src = Path(sys.argv[1])
    dst = Path(sys.argv[2])
    text = src.read_text(encoding="utf-8", errors="replace")
    for pattern, repl in REPLACERS:
        text = pattern.sub(repl, text)
    dst.write_text(text, encoding="utf-8")


if __name__ == "__main__":
    main()

cp tests/fixtures/checkout.yml /tmp/checkout.yml.bak
python3 redact_cassette.py tests/fixtures/checkout.yml /tmp/checkout.redacted.yml
# attach only /tmp/checkout.redacted.yml — keep the backup off the prompt

If the file contains a recorded body from billing, identity, or messaging, you should not send the file at all. Write the status code and the field names in your own words, which is usually enough for a coding model to discuss control flow. If the file is a HAR from a browser network panel, assume it holds first-party cookies even when you wanted only the error payload. If the file is a snapshot that embeds headers, regenerate it with header stripping in the test helper, then consider the stripped copy.

When the model only needs to know that a retry happened twice, type that sentence and stop attaching evidence. You do not need a full cassette to teach exponential backoff or a simple timeout constant. When you need schema help, copy keys from one object and replace every value with an obviously fake string. The extra minute of typing is cheaper than rotating every token that would have ridden along.

Remote free inference stays on the far side

Some coding assistants let you iterate on free model access and a free server while you clean a fixture. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers that free model access and a free server, and the convenience does not move the trust boundary back onto your laptop.

You still assemble the working set locally, run the inventory, and only then allow a redacted snippet to cross the network. The free server is a different administrative domain, with disks, operators, and retention you do not configure. Treat it like a shared CI runner you do not own: handy for iteration, and not a private notebook. If you already experiment with that setup, send the redacted cassette, never the original trace archive, and never the CI log that printed the bearer token.

CI logs captured with verbose HTTP clients belong on the runner until you strip them by hand. Browser traces and HAR files from staging hosts are worse when a parent domain shares cookies with production. Snapshot files that include Set-Cookie headers or signed query strings are recordings, even if the test name sounds abstract. Compose dumps and Kubernetes describe output often list pull secrets and internal DNS names you should keep local.

Helpful extras from an agent can leak as well, including recent file lists and absolute paths that contain your laptop username. Git remotes with embedded credentials and crash notes that reprint environment variables belong in the same bucket. Your job is to decide the minimum sentence that states the failure, then stop loading exhibits onto the prompt. Descriptions of a failure can travel; recordings of a failure should stay on the machine that made them.

Limitations, and who should skip this

The script is a text heuristic, so binary Playwright traces, encrypted HAR files, and protobuf recordings will slip past it. It will also flag internal hostnames in public dummy cassettes, which is noisy if you already manufacture fake traffic. It does not consult git, so a file you committed last year is still a prompt leak the moment you attach it. Publication in a repository and transmission to a model remain separate events with separate audiences.

Do not promote this preflight into a compliance program or a substitute for a real contract review. If you handle regulated health or payment data, you need counsel on any remote inference, including free endpoints. Do not treat a quiet scanner as permission to paste more customer narrative from a JSON body. A customer story inside a response body can be sensitive without matching a JWT pattern. Teams that already forbid outbound source should keep that ban and ignore the rest of this workflow.

The habit you want is small enough to survive a bad afternoon in a failing pipeline. Before the prompt leaves, ask whether each attached file is a description of a failure or a recording of one. Descriptions of the failure can travel across the socket without taking live credentials along with them. Actual recordings of the same failure should remain on your side of that trust boundary.

Comments

[Raknaos]

This bites harder than it looks. I run a small fleet of automation boxes and the fixtures folder always ends up being the one place nobody scrubs, because "it's just test data". Authorization headers and internal hostnames survive copy-paste into prompts more often than anyone wants to admit.

We ended up making the scrub step a pipeline gate rather than a code-review norm: a norm gets skipped on the day you're shipping, a gate doesn't. Curious whether you found tooling that does this cheaply, or if it's still manual discipline on your side.