Audit the detection, not the reasoning
If your AI agent handles credentials, you hit a tension fast: compliance wants an
audit trail, and security says don't retain raw reasoning — because the reasoning
trace is exactly where a secret tends to surface even when the answer is clean. Keep
the trace and you're storing credential-bearing logs; delete it and you have nothing
to show an auditor. Most teams recognize this paradox but rarely write down a clean
way out.
Here's the pattern I landed on, refined in the open with operators running this at
deployment scale.
Principle 1 — audit the detection, not the reasoning
The audit artifact does not have to be the trace. It can be the scan result:
"N traces scanned, X findings, redacted." That record is credential-free, so it
satisfies the audit trail while the raw reasoning — the dangerous part — never
persists. You're proving the control ran and what it caught, not keeping the payload.
Principle 2 — keep the fingerprint, not the value
When a finding is recorded, it holds three things and never the raw secret:
type — which credential family (e.g. an Anthropic key, an AWS access key)
scope — the blast radius that type implies (what it could reach)
fingerprint — a salted, truncated, one-way hash of the value
That answers what an auditor actually needs — which credential, what it could reach,
proof it was rotated — without keeping anything exploitable. The authoritative record
of what the value was lives in your secret store's rotation log, not a debug trace.
Containment scope comes from identity + rotation, and the plaintext never sits in
retained logs.
Why the salt matters (an honest correction)
My first instinct was that truncating the hash blocks confirmation attacks. It
doesn't: for a weak, dictionary-sized secret, an attacker who guesses the value can
still confirm it against a short hash. The real defense is a per-deployment salt —
without it, the attacker can't compute the hash offline at all. So the salt is on by
default. Truncation is for correlation (spotting the same secret twice), not secrecy.
The limits, stated plainly
The record still needs integrity. Once you retain anything to feed an audit,
a stored artifact an attacker can edit before it's read is its own surface. A
credential-free fingerprint shrinks that surface; it doesn't erase it. Signing or
hash-chaining the records is the next step, not something to hand-wave.
This is the detective half. Scanning after the fact catches a secret that
surfaced in reasoning but never became an action. It cannot catch what never got
logged. A pre-execution check on the decision node is the complementary half —
capture-independent, and it covers a failure this pattern can't. Different surfaces,
different failure modes; neither replaces the other.
Literal, not semantic. This detects credential strings. A secret paraphrased
with no literal token is out of scope.
Where it's implemented
agentproof-scan emits this record — type, scope, salted truncated fingerprint, masked
value — from both the answer and reasoning surfaces. Apache-2.0. Raw transcripts stay
private; the fingerprint record is credential-free by construction, so it's the part
that's safe to share. [https://github.com/ghkfuddl1327-wq/agentproof]
If you run agents at scale and audit them, I'd genuinely like to hear where this
pattern breaks for you — that's the calibration I can't get from theory.
Top comments (0)