Reproduce a WAF Sample Leak With a Never-Send Fixture

I rebuilt a near-miss as a fixture, not as a named customer outage. A checkout rule fires on a POST that should have passed. The scratch file holds a Cookie, an upstream token, an email, and a comment about an office NAT bypass. The paste key is already halfway down.

That habit is the failure. A model prompt is an export. If I would not attach the scratch file to a public ticket, it does not cross a model boundary. A free host does not edit that rule.

The failure mode is the paste

I write the invariant before any prompt exists. The laptop may hold the raw sample. A local gate may read it. The outbox may not contain a secret. Anything past the outbox is outside the trust zone.

Would you drop that scratch file into a public gist so a stranger could take a look? No? Then a chat box is not a private notebook. It is an egress path with a friendly cursor.

This piece is not a vulnerability report. I did not run a vendor export, and I am not claiming a WAF product leaked anything. The human path is untested until a fixture fails closed.

Trust boundaries, drawn small

Keep four zones, and keep them boring. The workstation is inside: it already has the sample, which you can encrypt and wipe. The gate is inside too: it prints reason codes and refuses to write an outbox file on a hit.

Reason codes name the field. They must not echo the value. A CI log that reprints an Authorization header is the same leak with extra steps. Which log pipeline scrapes stderr on your runner? Know that before you add a helpful debug print.

The outbox is the only artifact allowed to leave. If a file is not in the outbox, it is not a prompt. The hosted model, and any free server you did not build, sit outside. They can discuss a redacted signature. They do not get raw on-call bytes.

scratch file --> local gate --fail closed--> stderr codes (field names only)
                    |
                    +--only if clean--> outbox/ --> hosted assistant

A free server does not collapse that drawing. If you did not provision the box, own the disk, and name the log sink, treat it as a third party. I am not describing a product topology. I am describing the default until a data-flow diagram says otherwise.

What a blocked request quietly carries

People shrug and call it just a WAF sample. Look again. Headers often carry Authorization, Cookie, X-Api-Key, or an upstream token stuffed into X-Internal-Token. The body excerpt can hold an email, a phone, or a long digit run.

The rule comment is worse than the request. That is where bypass notes, office CIDRs, and a copied basic-auth line go to live. The match context is frequently the secret itself, because the engine shows the bytes that fired.

Which of those do you want in a transcript you cannot delete on your own clock? I want none of them. The table is the decision, not the mood.

Field	Send to a model?	Reason
rule_id	Yes	Identifier, not a credential
Header names with values already replaced	Yes	Shape without secrets
Raw Cookie or Authorization	No	Session or bearer material
Rule comment with a bypass CIDR or copied secret	No, until redacted	Topology plus credential
Synthetic body you generated	Yes	Not customer content
Customer body excerpt	No	Content, and often a token

I am not claiming SafeLine, ModSecurity, or any other engine leaks by default. The stand-in schema is waf-sample/v0. I made it up so the fixture has a contract. Map your real export before you trust a green check.

1. Pin the invariant

Fail closed. A sample with a denylisted header, a PEM private-key banner, a bearer-shaped string, an email, or an AKIA access-key shape must not produce an outbox file. Exit 2 is the passing result for that case.

A green job on a leaky sample means the control is installed backwards. Would you celebrate a firewall test that only passes when the deny rule is deleted? Same energy.

Positive fixture: secrets present. Spec: exit 2, no outbox write. Negative fixture: high-risk patterns already gone. Spec: exit 0, outbox written. Hidden fixture: headers look clean, secret sits in rule_comment. Spec: exit 2, hit path under $.rule_comment.

I have not executed this template in the environment that will publish the post. Those exits are the specification, not a measured run. Do not cite them as a benchmark. Python 3.11 or newer, standard library only. I am not pinning a third-party scanner I did not install.

2. Lay down three synthetic samples

Positive. Obviously fake. Still enough to trip the gate. Save it as fixtures/positive_leak.json.

{
  "schema": "waf-sample/v0",
  "rule_id": "checkout-sqli-17",
  "headers": {
    "Authorization": "Bearer EXAMPLE.not.a.secret",
    "Cookie": "session=example-session-value",
    "Content-Type": "application/json"
  },
  "body_excerpt": "email=[e-mail]",
  "rule_comment": "office bypass note removed from this fixture on purpose"
}

Negative. This is what done looks like. Save it as fixtures/negative_clean.json.

{
  "schema": "waf-sample/v0",
  "rule_id": "checkout-sqli-17",
  "headers": {
    "Authorization": "[redacted]",
    "Cookie": "[redacted]",
    "Content-Type": "application/json"
  },
  "body_excerpt": "email field removed",
  "rule_comment": "bypass note removed"
}

Hidden. A header denylist alone will miss it. That miss is the test. Save it as fixtures/hidden_in_comment.json.

{
  "schema": "waf-sample/v0",
  "rule_id": "checkout-sqli-17",
  "headers": { "Content-Type": "application/json" },
  "body_excerpt": "id=1",
  "rule_comment": "copied snippet still had Bearer EXAMPLE.not.a.secret"
}

Do not substitute a real customer capture to make the demo better. The walker also does not understand partial redaction. If an @ remains in a field, that sample is not the negative case, and the gate should stay red.

3. Drop in the gate

Unexecuted template. Read it before you run it. If your samples are gzip, pcap, or screenshots, this file will not save you. It only walks JSON strings.

#!/usr/bin/env python3
"""never_send_gate.py — unexecuted template. Spec, not a measured run.

Stand-in schema: waf-sample/v0. Python 3.11+ stdlib only.
"""
from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

DENY_HEADERS = {
    "authorization",
    "proxy-authorization",
    "cookie",
    "set-cookie",
    "x-api-key",
    "x-internal-token",
}

PATTERNS = [
    ("pem", re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----")),
    ("bearer", re.compile(r"(?i)\bbearer\s+\S{8,}")),
    ("email", re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")),
    ("aws_access_key_shape", re.compile(r"\bAKIA[0-9A-Z]{16}\b")),
]

REDACTED = "[redacted]"


def walk_strings(value, path="$"):
    if isinstance(value, str):
        yield path, value
    elif isinstance(value, dict):
        for key, item in value.items():
            yield from walk_strings(item, f"{path}.{key}")
    elif isinstance(value, list):
        for index, item in enumerate(value):
            yield from walk_strings(item, f"{path}[{index}]")


def header_hits(sample: dict) -> list[str]:
    headers = sample.get("headers") or {}
    if not isinstance(headers, dict):
        return ["headers_not_object"]
    hits = []
    for name, val in headers.items():
        if str(name).lower() in DENY_HEADERS and str(val) != REDACTED:
            hits.append("header:" + str(name))
    return hits


def pattern_hits(sample: dict) -> list[str]:
    hits = []
    for path, text in walk_strings(sample):
        if text == REDACTED:
            continue
        for label, cre in PATTERNS:
            if cre.search(text):
                hits.append(label + ":" + path)
    return hits


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--check", required=True, type=Path)
    parser.add_argument("--outbox", type=Path)
    args = parser.parse_args()
    sample = json.loads(args.check.read_text(encoding="utf-8"))
    if sample.get("schema") != "waf-sample/v0":
        print("schema_mismatch", file=sys.stderr)
        return 2
    hits = header_hits(sample) + pattern_hits(sample)
    if hits:
        print("never_send", file=sys.stderr)
        for hit in hits:
            print(hit, file=sys.stderr)
        return 2
    if args.outbox:
        args.outbox.parent.mkdir(parents=True, exist_ok=True)
        payload = json.dumps(sample, indent=2) + "\n"
        args.outbox.write_text(payload, encoding="utf-8")
    print("ok")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Notice what it does not do. It does not print the match text. It does not call the network. It does not fix the rule. A gate that rewrites secrets into a prompt so the model can help redact has already lost.

4. Assert both directions

Green has to mean something specific. If the job only passes on exit 0, wrap the positive case. Assert exit 2, and assert the outbox path does not exist. A test that forgets the negative path is how a deny-all bug ships. You want both.

python3 never_send_gate.py --check fixtures/positive_leak.json --outbox outbox/pos.json
echo "positive_exit:$?"
# spec: exit 2, stderr starts with never_send, outbox/pos.json absent

python3 never_send_gate.py --check fixtures/negative_clean.json --outbox outbox/neg.json
echo "negative_exit:$?"
# spec: exit 0, stdout line ok, outbox/neg.json written

python3 never_send_gate.py --check fixtures/hidden_in_comment.json --outbox outbox/hid.json
echo "hidden_exit:$?"
# spec: exit 2, a bearer hit on $.rule_comment, outbox/hid.json absent

What does green mean in your pipeline today? If you cannot answer in one sentence, the fixture is not wired yet. Wire the hidden case first. That is the one a header-only cleanup will cheat.

5. Prevent, detect, recover

Phase	Control	Evidence you should demand
Prevent	Only outbox/ may be attached to a prompt. A wrapper refuses the scratch path.	Raw sample path never shows up in the assistant CLI history.
Detect	CI runs all three fixtures on every change to the gate.	Hidden fixture goes red if someone simplifies the walker down to header keys.
Recover	If a raw sample already left, rotate the credential and treat the transcript as tainted.	Incident note lists field names, not a second paste of values.

Asking a model to forget is not recovery. I have no evidence, in this post, that any provider stored or leaked a WAF sample. The recover row exists because you cannot prove the negative after the bytes leave. Rotation does not depend on that proof.

Where an assistant is allowed to sit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is the open-source AI development platform this draft is allowed to discuss. Two availability claims were supplied with the assignment: free model access, and a free server option. I am not pinning a token quota, a GPU shape, a time limit, or a retention promise. Those are time-sensitive, and I will not freeze a number I have not checked against a current primary page.

So what is the workflow, once that disclosure is out of the way? Redact on the laptop. Diff the outbox. Then, if you still want a second pair of eyes on the rule logic, send the outbox only. Ask a narrow question: is checkout-sqli-17 matching the redacted body shape, or is the operator too wide? Do not ask it to look at the original. There is no original on that side of the line.

A free server is still a server. Unless it sits in a network you operate, it belongs in the outside column of the drawing above. Generated rule text is a suggestion. A human diffs it before it is loaded. Would you reload a WAF from a chat transcript with no diff? I would not, even if the model were perfect.

If you evaluate that free model access or that free server option, point either one at outbox/ and read the current data-handling terms before the first real incident. The fixture still works if you never do that. That is the test of whether this piece was worth writing.

Who should skip this

Do not use the script as a DLP product. It misses gzip, unlabeled base64, pcaps, and screenshots. It will also flag a documentation address you meant to publish. That false positive is acceptable for a never-send gate and annoying everywhere else. I did not measure a rate. Do not invent one in a slide.

Do not feed it real customer bodies to see if it works. Synthesize. If blocked payloads are customer content under a retention schedule, the fix is not a smarter prompt. Keep the body off the machine that has a model CLI installed.

Teams that need a signed data-processing agreement, or a path that never leaves the VPC, should not trial a hosted box on live samples while the paperwork catches up. Build the fixture. Then pick a host. If you cannot name the log sink, you are not ready to upload even the redacted file.

What this does not prove

I am not saying hosted models are unsafe for every debugging task. Redacted signatures, public write-ups, and synthetic requests are fair game. I am also not saying a self-hosted assistant becomes trusted because the binary is open source. Open code does not redact a Cookie you pasted. The gate does, or nothing does.

This week's public developer discussion keeps circling one mood: models got faster, and the software around them did not grow a matching boundary. I am not reviewing those posts, and I am not treating their titles as evidence. The boundary I care about is one exit code on a file you control.

The question I want in the retro

Which invariant belongs in CI this week, and which layer should enforce it live?

I would pin the positive fixture to exit 2, on synthetic files, in the same pipeline that lints WAF config. The live layer is a separate argument: an editor hook, an egress filter that blocks Authorization inside outbound prompt bodies, or both. A proxy you can bypass with a personal key is not enough on its own.

The fixture still fails closed when someone is tired and the proxy is off. If you only adopt one line from this piece, adopt that. Exit 2 on a leak is the control. The model is optional. The scratch file is not a prompt.