Scrub the Trace, Keep the Failure: A Redaction Gate You Can Test

Every agent team hits the same wall. A run fails in staging, and someone wants a second pair of eyes.

The trace is 4 MB of prompts, tool arguments, and request headers. Sharing it raw is a leak. Sharing a redacted copy that no longer reproduces the failure is worse.

This post builds a small gate between those two outcomes. A failure signature survives the scrub, and the secrets do not. The mechanism is one policy table, two scripts, and two assertions you can run in CI.

The part you cannot lose is the signature

A trace is only useful if it still says where it broke. Call that the signature: the ordered list of failing spans, each reduced to (seq, name, error_kind).

If the signature is identical before and after redaction, the diagnosis path is intact. If it changes, the scrubber destroyed evidence. That is a testable property, so test it rather than eyeballing diffs.

Write the policy as a table, not a regex

Field names are stable across runs. Values are not. Classify each attribute once, then enforce it.

Attribute	Action	Reason
seq, ts, type, name	keep	ordering, latency, cost attribution
span_id, parent_id	keep	rebuild the span tree
user_id, session, repo, path	hash	joins survive, identity does not
prompt, message, query	hash	free text is where secrets hide
tool, error_kind, exit_code	keep as enum	this is the signature
api_key, auth_header, cookie, env	drop	never needed to debug

Three rules follow from the table. Hash with HMAC so equal inputs stay equal. Drop secrets outright instead of hashing them. Fail on unknown fields so the policy cannot silently rot.

The scrubber is about forty lines

One key, four sets, one function. Adapt the field sets to your own schema before trusting any output.

#!/usr/bin/env python3
"""Field-allowlist scrubber for agent JSONL traces."""
import hashlib, hmac, json, sys

KEY = open("scrub.key", "rb").read()

KEEP = {"seq", "ts", "type", "name", "status", "span_id", "parent_id"}
HASH = {"user_id", "session", "repo", "path", "prompt", "message", "query"}
DROP = {"api_key", "auth_header", "cookie", "env"}
ENUM = {"tool", "error_kind", "exit_code", "http_status", "model"}

def tag(value):
    raw = str(value).encode()
    return "h_" + hmac.new(KEY, raw, hashlib.sha256).hexdigest()[:12]

def scrub(event):
    out = {k: event[k] for k in KEEP if k in event}
    attrs = {}
    for key, value in event.get("attrs", {}).items():
        if key in DROP:
            continue
        if key in HASH:
            attrs[key] = tag(value)
        elif key in ENUM:
            attrs[key] = value
        else:
            attrs[key] = f"<{type(value).__name__}>"
    out["attrs"] = attrs
    return out

for line in sys.stdin:
    if line.strip():
        print(json.dumps(scrub(json.loads(line)), sort_keys=True))

Hashing with a single key keeps cross-run joins alive. Rotate the key and old hashes stop matching. Store the key outside the trace directory and never ship it with the data.

Two assertions on every trace

import json, sys

SENSITIVE = {"user_id", "session", "repo", "path", "prompt", "message",
             "query", "api_key", "auth_header", "cookie", "env"}
ENUMS = {"tool", "error_kind", "exit_code", "http_status", "model"}

def load(path):
    return [json.loads(l) for l in open(path) if l.strip()]

def signature(events):
    sig = []
    for e in events:
        a = e.get("attrs", {})
        failed = e.get("status") == "error" or a.get("exit_code") not in (None, 0)
        if failed:
            sig.append((e.get("seq"), e.get("name"), a.get("error_kind")))
    return sig

raw, clean = load(sys.argv[1]), load(sys.argv[2])
assert signature(raw) == signature(clean), "signature changed"

blob = open(sys.argv[2]).read()
leaks = [v for e in raw for k, v in e.get("attrs", {}).items()
         if k in SENSITIVE and len(str(v)) >= 6 and str(v) in blob]
unknown = {k for e in raw for k in e.get("attrs", {})} - (SENSITIVE | ENUMS)

print(f"signature ok ({len(signature(raw))} failures)")
print(f"leak scan ok ({len(leaks)} hits)")
print(f"unknown attrs: {len(unknown)} {sorted(unknown)}")

Check one is signature equality. Check two is the leak scan: any sensitive raw value of six characters or more appearing in the output fails the job. The third number is the unknown attribute count, which should be zero.

Run it locally first:

python -c "import os;open('scrub.key','wb').write(os.urandom(32))"
chmod 600 scrub.key
python scrub.py < traces/run-0917.jsonl > traces/run-0917.scrubbed.jsonl
python verify.py traces/run-0917.jsonl traces/run-0917.scrubbed.jsonl

A clean run prints 0 hits and unknown attrs: 0. A non-zero leak count means your policy is missing a field. Fix the table, not the test.

The loop you reuse

1. Reproduce the failure and keep the raw trace on the machine that produced it.
2. Scrub, then verify. Fail the build if the signature moves.
3. Hand the scrubbed file to whoever debugs it. They see structure and failure kinds.
4. Diff two scrubbed runs with any JSONL differ to isolate the changed span.
5. Fix, rerun, and compare signatures only. The raw text stays where it started.

Step four is where the gate pays off. Hashed IDs stay stable across runs, so a diff aligns on span_id instead of guessing.

Where a free tier actually helps

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The scrub-and-verify job is CPU-only and bursty. It runs when a trace lands, then idles. MonkeyCode's free server option covers that without a card or a provisioning step, which matters for a side project or a solo on-call rotation.

Free model access helps with the tedious part. Paste your attribute name list, never values, and ask for HASH/DROP assignments plus edge cases you missed. Review the output against the table, then add a leak-scan case for each suggestion.

If you want to run the loop before committing infrastructure, this is enough to process a few hundred traces.

What this does not solve

• Length and type still leak. <str> plus a character count narrows a token.
• Timestamps leak working hours and geography. Coarsen them if that matters.
• Nested lists and dicts flatten badly. Extend scrub() or reject the field.
• Third-party SDK fields land outside the policy. That is what the unknown counter is for.
• error_kind must be set at instrumentation time. Derive it before you scrub, not after.

Hashing is not anonymization. Low-entropy values like a username can be brute-forced from a short hash.

Skip this approach if

Your traces fall under a formal DLP or compliance regime. Use an approved tool and a review process, not a forty-line script.

Also skip it if you need raw text for fine-tuning. This gate is built for debugging, and it is deliberately lossy.