Build a Canary Regression Test for Model-Bound Webhook Data

The 02:14 ticket

Last month a support engineer opened a ticket that looked like a hundred others: one tenant's webhook deliveries were failing, and she wanted a model to explain why. So she copied the failed delivery record — status code, endpoint, body, and the X-Signature header — into a chat box and asked, "what is wrong here?"

The answer was fine. The paste was not. That header is derived from a per-tenant signing secret, and it now lived in a vendor transcript, a browser cache, and a ticket that three people could read.

Nothing exploded. No failed request, no 500, no alarm. That is the worst kind of incident: a secret that crossed a boundary nobody drew.

So let's draw it — and then prove it with a test.

The boundary is not the network

Most "don't send secrets to models" advice is a vibe. The useful version is a specific question: which fields are allowed past the prompt assembler, and what fails the build when someone widens that list?

Boundary	What crosses	Failure mode	Control
Log store → prompt assembler	Structured event records	Whole-record serialization drags in secrets	Field allowlist projection
Assembler → model endpoint	Rendered prompt	Retention you don't control	HMAC pseudonyms + shape-only values
Model output → ticket	Free text	Regurgitated secret in a readable place	Output scan before write

Note what is not on that list: "tell the model not to repeat secrets." A system prompt is a preference, not an enforcement layer. Enforcement lives in code that runs before the bytes leave your process.

Step 1 — Classify data, not payloads

Start with a table somebody can argue with. Mine, for webhook delivery records:

Class	Example field	Send raw?	Why
Secret	signature, authorization, cookie	Never	Rotating a tenant secret is a customer-visible event
Identifier	endpoint, host, bucket	Never	Names your internal topology and tenant mapping
Personal data	customer_email, phone	Tokenized	Correlation is useful; identity is not
Free text	body, error, stack	Synthesized	Where the leaks actually hide
Telemetry	event, status_code, duration_ms	Yes	The reason you called a model at all

Argue about the rows now, in review, with names attached. Not at 02:14.

Step 2 — Project with a default-deny allowlist

Denylists lose because they need you to predict every new field. Allowlists lose nothing but convenience, and convenience is exactly what you want to spend.

# payload_policy.py — starting template, not a validated policy
import hashlib, hmac, re

RULES = [
    (r"(?i)^(event|event_id|status|status_code|attempt|duration_ms|timestamp)$", "allow"),
    (r"(?i)(secret|token|password|signature|cookie|authorization|api_?key)",      "drop"),
    (r"(?i)(host|hostname|endpoint|bucket|arn|ip_addr)",                         "drop"),
    (r"(?i)(email|phone|address|full_?name|card)",                               "tokenize"),
    (r"(?i)(body|payload|message|error|stack|trace)",                            "synthesize"),
]
DEFAULT = "drop"

def classify(key: str) -> str:
    for pattern, action in RULES:
        if re.search(pattern, key):
            return action
    return DEFAULT

def shape(value):
    if isinstance(value, str):
        return f"<str:{len(value)}>"
    if isinstance(value, bool) or value is None:
        return value
    if isinstance(value, (int, float)):
        return 0
    return "<obj>"

def project(record: dict, hmac_key: bytes) -> dict:
    out = {}
    for key, value in record.items():
        action = classify(key)
        if action == "drop":
            continue
        if isinstance(value, dict):
            out[key] = project(value, hmac_key)
        elif action == "allow":
            out[key] = value
        elif action == "tokenize":
            digest = hmac.new(hmac_key, str(value).encode(), hashlib.sha256).hexdigest()
            out[key] = "tok_" + digest[:16]
        else:
            out[key] = shape(value)
    return out

Two details matter more than the code. First, the HMAC key lives outside the prompt path, so pseudonyms correlate within a tenant but resist dictionary attack — an unkeyed hash of an email is reversible in seconds. Second, shape() preserves length and type; that leaks a little, and it is the price of the model still being able to say "this body looks truncated."

Step 3 — Synthesize what the model needs to see

Here is the tradeoff people skip: aggressive redaction produces prompts that are safe and useless. If you replace an error string with [REDACTED], the model can tell you nothing about it.

So keep the skeleton. Replace values with typed placeholders, keep key names, keep ordering, keep booleans. A stack trace becomes file=<str:41> line=0 fn=<str:18> — still enough to spot a pattern like "every failure is in the same function," which is usually the actual question behind the ticket.

Step 4 — Two fixtures, not one

A redaction test that only asserts absence will pass when your projection drops everything. You need a negative control too.

# test_payload_policy.py — pytest, no network
from payload_policy import project

KEY = b"test-only-key-not-a-secret"
CANARY = "CANARY-{}-8f3a1c"

def render(record):
    return repr(project(record, KEY))

def sample():
    return {
        "event": "webhook.delivery_failed",
        "status_code": 502,
        "signature": "sha256=" + CANARY.format("signature"),
        "endpoint": CANARY.format("endpoint"),
        "customer_email": CANARY.format("email") + "@example.test",
        "body": '{"order":"' + CANARY.format("body") + '"}',
    }

def test_canaries_never_reach_the_prompt():
    out = render(sample())
    for field in ("signature", "endpoint", "body"):
        assert CANARY.format(field) not in out, field

def test_useful_fields_survive():
    out = render(sample())
    assert "webhook.delivery_failed" in out   # over-redaction guard
    assert "502" in out
    assert "tok_" in out                      # tokenized, not dropped

def test_pseudonyms_correlate_but_do_not_collide():
    a = project({"customer_email": "[e-mail]"}, KEY)["customer_email"]
    b = project({"customer_email": "[e-mail]"}, KEY)["customer_email"]
    c = project({"customer_email": "[e-mail]"}, KEY)["customer_email"]
    assert a == b and a != c

def test_unknown_field_defaults_to_drop():
    out = render({"some_new_field": CANARY.format("mystery")})
    assert "some_new_field" not in out

That last test is the one that keeps working after the refactor. Every new field a service adds lands in drop until somebody writes a rule and a reason.

Step 5 — Run it against a model you are allowed to break

The hard part of this workflow is not the regex, it is measuring whether your projection destroyed the signal. You need a model endpoint and a throwaway environment, and you should never use production records for that measurement.

This is where MonkeyCode's free model access and free server option are relevant: the operator states that free model access (currently advertised as a 10M-token allowance) and a free server option are available, which is enough to stand up an isolated project, point the same fixture at a real model, and compare answers from raw versus projected payloads. Treat those terms as operator-supplied and check the current page before you rely on them — allowances and server options change, and I have not benchmarked throughput or hardware here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The measurement is simple. Take twenty synthetic records that produce a known correct diagnosis, run each through the model twice — once raw, once projected — and score whether the diagnosis survives. If accuracy collapses, your synthesize rules are too aggressive. If nothing changes, your drop rules are probably not firing.

Step 6 — Detect, then recover

Prevention fails eventually, so make failure loud. Scan rendered prompts before they are persisted, with deliberately dumb heuristics:

# scan a prompt log directory for shapes that should never be stored
grep -rEn 'CANARY-|sha256=[0-9a-f]{64}|Bearer [A-Za-z0-9._-]{20,}|AKIA[0-9A-Z]{16}' ./prompt-logs \
  && echo 'LEAK: rotate before continuing' && exit 1

A pattern match is not proof of a leak, and a clean scan is not proof of safety. It is a tripwire on the specific mistake you already made once.

Found in prompt log	Rotate	Notify
Signing secret	All tenant webhook secrets in that scope	Tenant owner, within the contract window
Session cookie	Invalidate sessions for the affected scope	Security on-call
Tokenized identifier only	Nothing	Nobody — but log the near miss

What this does not cover

Be honest about the edges. Key-name classification breaks the moment a customer pastes a secret into a free-text body field — you get shape-only values, which helps, but not extraction. Canaries only catch values you planted. Self-hosting the model removes network egress but not the prompt log on your own disk. And if your contract requires sending raw customer content to a processor, this gate is the wrong tool; go fix the contract first.

Who should not use this

• Teams with no prompt log store: you cannot detect what you do not keep, and you cannot keep what you have not classified.
• Anyone without a rotation runbook: redaction without recovery is theater.
• Teams sending regulated data where legal review has not signed off on the synthetic-value approach.

The boundary question worth taking to your next review: which of these invariants belongs in CI as a canary test, which belongs in the assembler as an allowlist, and which one are you still pretending a system prompt will enforce?