Canary the Free Tier Before You Trust It

You cannot audit what a free model tier does with your prompt after the request leaves your process. You can audit what your own pipeline put into that request, and that is the only evidence you can collect without a contract, a lawyer, or a vendor's retention page. What follows is a small, reproducible workflow that produces that evidence in about a minute per run, and it will tell you more than any privacy FAQ will.

Flip the direction of the audit

Most teams start with the vendor's data-retention paragraph and try to read it charitably. That paragraph is a claim written by the party with the least incentive to be specific, and it can change between the day you read it and the day you ship. Your side of the boundary is measurable: fixtures, logs, trace exports, build artifacts, cache directories, container layers. So stop asking what the vendor keeps, and start proving what your pipeline sent; then let a leak on your own disk be the alarm you actually control.

The mechanism is old and boring, which is why it works. Miners carried a canary into the tunnel because the bird noticed carbon monoxide before the humans did, and the detection did not require trusting anyone's instrumentation. A canary secret works the same way: it is a synthetic, uniquely shaped string that has no business appearing anywhere outside your fixture directory. If it turns up in a log file, a trace span, or a build artifact, you have a finding with a file path attached instead of a suspicion with a paragraph attached.

This also settles the question of what not to send to a model, because it measures the discipline instead of assuming it. Send the failing test and the relevant diff hunk, not the repository. Send the schema and three synthetic rows, not the production dump. Send the error line and the request id, not the whole session blob. A canary audit tells you whether the code that assembles your payload actually respects that rule, or whether it merely does so in the happy path you remember testing.

Mint canaries you can grep for

The first script creates four canaries per run, each with a fresh 24-character hexadecimal nonce, and records what it minted. Fresh nonces matter more than they look: a hit from this run is unambiguous, while a hit from last week's nonce may just be stale output sitting in a directory you forgot to clean.

# canary_gen.py — mint unique canaries and record them in a manifest
import hashlib, json, pathlib, secrets

OUT = pathlib.Path("canaries")
OUT.mkdir(exist_ok=True)

def mint(kind: str):
    nonce = secrets.token_hex(12)
    shapes = {
        "api_key": f"sk-canary-{nonce}",
        "pii": f"canary+{nonce}@example.invalid",
        "host": f"internal-canary-{nonce}.svc.cluster.local",
        "jwt": f"eyJhbGciOiJub25lIn0.{nonce}.canarysignature",
    }
    return nonce, shapes[kind]

manifest = {}
for kind in ("api_key", "pii", "host", "jwt"):
    nonce, value = mint(kind)
    (OUT / f"{kind}.txt").write_text(value + "\n")
    manifest[kind] = {
        "value_sha256": hashlib.sha256(value.encode()).hexdigest(),
        "nonce": nonce,
    }

(OUT / "manifest.json").write_text(json.dumps(manifest, indent=2))
print(json.dumps({k: v["nonce"][:6] for k, v in manifest.items()}))

Two details keep this honest. The example.invalid domain is reserved by RFC 2606, so a canary address can never accidentally route mail to a stranger, and the JWT shape uses alg: none so it is obviously inert to anyone who decodes it. Keep canaries/manifest.json out of anything you ship, because the manifest contains the needles; treat it the way you treat a test fixture signing key rather than the way you treat a README.

Search everything your process can write

The second script walks the directories your pipeline is allowed to write to, looking for each nonce in three encodings: raw text, standard base64, and URL-safe base64. It prints the kind and path of every hit, and then it exits non-zero, which is the part that turns a report into a gate.

# canary_audit.py — exit 1 if any canary nonce appears in a readable artifact
import base64, json, pathlib, sys

ROOTS = [pathlib.Path(p) for p in ("logs", "build", "artifacts", "traces", ".cache")]
SKIP = {".pyc", ".so", ".png", ".jpg", ".zip", ".gz"}
manifest = json.loads(pathlib.Path("canaries/manifest.json").read_text())

def variants(nonce: str):
    raw = nonce.encode()
    yield nonce
    yield base64.b64encode(raw).decode().rstrip("=")
    yield base64.urlsafe_b64encode(raw).decode().rstrip("=")

hits = []
for root in ROOTS:
    if not root.exists():
        continue
    for path in root.rglob("*"):
        if not path.is_file() or path.suffix in SKIP:
            continue
        try:
            text = path.read_bytes().decode("utf-8", "ignore")
        except OSError:
            continue
        for kind, rec in manifest.items():
            if any(v in text for v in variants(rec["nonce"])):
                hits.append((kind, str(path)))

for kind, path in hits:
    print(f"LEAK kind={kind} file={path}")
sys.exit(1 if hits else 0)

Run it as python canary_gen.py && python canary_audit.py, and wire the second command into CI after your integration tests but before any deploy step. For ad-hoc triage outside the script, ripgrep is faster: rg -uuu --hidden -F -f <(jq -r '.[].nonce' canaries/manifest.json) . will search ignored files too, which is exactly where forgotten copies tend to hide. Rotating a canary after every run keeps the signal clean and costs nothing.

Canary	Boundary it exercises	Where it usually surfaces first
api_key	secret-shaped strings moving between components	request bodies copied into trace spans
pii	fixture data reaching a shared sink	fixture loaders writing to stdout
host	internal topology disclosure	stack traces in error responses
jwt	header and credential copying	debug middleware that dumps headers

Put the gate in front of the call

The last piece is a single send path, because a canary audit only helps if there is exactly one place where payloads leave your code. This function re-checks the outgoing body against the manifest before the request happens, so a fixture accidentally loaded into a prompt fails loudly instead of quietly becoming a support ticket.

# guarded_send.py — refuse to send a payload that still contains a dev canary
import json, os, pathlib, urllib.request

NONCES = [r["nonce"] for r in json.loads(
    pathlib.Path("canaries/manifest.json").read_text()).values()]

def send(payload: dict) -> dict:
    body = json.dumps(payload)
    if any(n in body for n in NONCES):
        raise SystemExit("blocked: dev canary still present in payload")
    req = urllib.request.Request(
        f"{os.environ['MODEL_BASE_URL'].rstrip('/')}/chat/completions",
        data=body.encode(),
        headers={
            "Authorization": f"Bearer {os.environ['MODEL_TOKEN']}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read())

The request path above is illustrative and matches the common chat-completions shape, so check the actual endpoint in your provider's docs rather than copying it blind. The important line is the cheap substring check before the socket opens, because that is where your policy lives. If you are evaluating MonkeyCode's open-source project for a small side build, the free model access and the free server option are the parts that make experimentation cheap enough to bother; the current token allowance is published on the project page, so read it there instead of trusting a number someone quoted in a blog post. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Run the audit against whatever tier you pick, and let the results, not the marketing copy, decide what data you allow through.

Where this method stops

A clean run is a snapshot of the paths you instrumented, not a guarantee about the ones you did not. The scripts above skip archives and compressed files, so unzip your build outputs into a scanned directory first if your packaging step bundles fixtures. The base64 variants catch the most common encoding accident, but tokenization, gzip, and proprietary binary formats can still hide a canary from a substring search. And no canary anywhere can tell you whether a third party retained a request after the response came back, because that observation is simply not available from your side of the wire.

Some teams should not run this at all, and the reason is uncomfortable. If your compliance answer is "no third-party model endpoint for this dataset," then a green canary run does not grant you permission, and treating it as a waiver is exactly the kind of reasoning an auditor will enjoy dismantling. Likewise, if your fixtures are copied from real customer records, you have a data-handling problem that canaries cannot fix; build synthetic fixtures first, then audit. The workflow is for teams that have already decided a tier is plausible and now want to know whether their own plumbing respects that decision. Start with the api_key canary against your logs directory this week, because that single check tends to find at least one trace exporter nobody remembered configuring.