My agent system keeps everything. Session transcripts, tool calls, and
stray observations land in an episodic store on disk, a consolidation
cycle mines them for durable facts, and the whole
thing syncs to a private git repo so every machine shares one memory.
The persistence is the point. It is also the liability.
The secrets you did not choose
The easy half of the problem is explicit secrets: values I decided to
protect, sitting in an encrypted vault, referenced by name. That half
was solved early.
The dangerous class is inline secrets, the ones nobody chose. An API key
pasted into a debugging conversation. A connection string riding along in
a stack trace. An OAuth token surfaced by a browser tool. The agent sees
these all the time, because I see them all the time, and the agent
watches me work.
A stateless agent makes this survivable by forgetting. Whatever it saw
evaporates with the session. A memory system removes exactly that mercy.
The transcript is written to disk, indexed, pushed to a remote, kept. Six
months in, one compromised machine or one leaked repo credential means
every secret that ever appeared in any conversation, exfiltrated at
git-clone speed.
What forced the design was the asymmetry. Scrubbing costs milliseconds
at each write. Not scrubbing costs nothing today, and the bill it runs
up has no ceiling: every day the system operates, a credential leak
exposes more than it would have the day before.
Scrub at the write, not at rest
Four shapes were on the table.
Trusting the private repo and disk encryption is the default posture, the
one you get by doing nothing. It fails on the one axis that matters here:
the residual risk only ever grows.
Encrypting the whole episodic layer at rest protects the data and kills
the layer. My stack is filesystem-first; the daily log is something I
cat, grep, and read in an editor when something looks off. Encryption
at rest taxes every one of those reads, slows the consolidation pass, and
turns a human-readable record into a blob.
Scrubbing twice, once on write and again before the sync push, sounds
like defense in depth. What it actually buys is divergence: the local
store holding things the pushed copy does not, two versions of the truth,
and no single place to point at and say "this is what the system knows."
So: scrub once, at the one point where data enters persistence. The
session-end hook that writes episodic events runs a regex-plus-entropy
scrubber, in-process, before anything touches disk. The budget is thirty
milliseconds for a typical event, and nothing the scrubber recognizes
ever lands in a memory layer in the clear.
The patterns are the ones you would guess. Known prefixes for GitHub,
AWS, and LLM-provider keys. Structural matches for three-segment JWTs,
user:password@host connection strings, and private key blocks. Emails,
always, as PII. And a catch-all at the bottom of the precedence order:
any long character run whose Shannon entropy reaches 4.5 bits per
character gets flagged as a probable secret even when no named pattern
claims it. The code sets the length floor at twenty characters, but the
arithmetic sets it at twenty-three: a shorter run cannot hold 4.5 bits
per character even with every byte distinct. Most specific match wins.
Redacted, but still shaped
The scrubber does not replace a secret with [REDACTED]. It replaces it
with a typed marker:
<REDACTED:github-token:a3f92>
The kind names the pattern class. The five hex characters are a truncated
SHA-256 of the original value, and that truncation is the detail I would
defend hardest.
Because the hash is stable, the same secret produces the same marker in
every event it appears in. So the consolidation cycle can still see
structure. "This token shows up every time that deploy step runs"
survives redaction, because co-occurrence survives, and the cycle never
sees a value to do it.
Five hex characters is twenty bits, which is nowhere near
preimage-resistant. Deliberately so. The marker is a clustering token,
not a verification token, and every bit added to it would leak a little
more information back out of the thing built to remove it.
The sidecar, and failing closed
Sometimes I need the original back. A false positive I want to inspect,
a key I pasted somewhere and then lost track of. So redaction is not
destruction: the originals go to a per-day encrypted sidecar file, keyed
by event and marker, using the same key as the vault. The sidecar is
committed to the repo, which sounds wrong right up until you notice it is
ciphertext. Committed, a new machine can recover originals with the
vault key it already needs for everything else, and an attacker with the
repo holds ciphertext either way.
Two small decisions in the recovery path do more work than their size
suggests. The unredact command prints the recovered value to stderr and
keeps stdout for the confirmation line, so pipes and command
substitutions, the things that mechanically slurp output into transcripts
and sessions, never capture the secret. Recovery is a human verb here; a
value that rode stdout back into a session would be sitting on the write
path to the very store it was scrubbed from. And the system fails closed: no
vault key configured means nowhere to put originals, so episodic writes
that would produce redactions are refused outright. A degraded install is
loud, not silently lossy.
The question that classes a secret
Scrubbing covers what the agent sees by accident. What it may touch on
purpose is a different question, and getting that one right took me
longer than the scrubber did.
The wrong question is "how sensitive is this value?" The question that
actually partitions the vault: may an automated flow ever handle
this?
Shell-class says yes. The agent interpolates $(vault get <key>) into a
command, the value exists in that one command invocation and nowhere
else, and the write-time scrubber guarantees it never persists into any
memory layer. That guarantee gets exercised for real: in the last thirty
days the scrubber caught a vault-held value on its way into the episodic
store 57 times. Redaction is not a nice-to-have here. It is the
precondition that makes shell-class safe to offer at all.
Strict-class says no, and means it. Human-only cold storage. The CLI read
verb refuses strict keys unconditionally. There is no agent-side access
path, and that is a decision, not unfinished work.
It was almost otherwise. The original design promised a generic tool:
agent names a key and a free-form action, the core decrypts in-process,
performs the action, returns the result. The agent never sees the value.
On paper it is zero-knowledge, and it demos beautifully.
I rejected it, and wrote the rejection down as permanent rather than
deferred. The action string is composed by the agent, and the agent is
exactly the party the strict class exists to distrust. Under prompt
injection, "perform this action with my production key" becomes an
exfiltration primitive by construction. The attacker does not need the
value when they can ask the holder of the value to make an authenticated
request to a URL they control. A zero-knowledge property that only holds
while the agent is uninjected is theater. The tool was not even earning
its risk: after seven weeks of daily use, not one real flow had needed
it.
What exists instead is three access paths, ordered by how little the
agent is trusted:
- Shell-class plus redaction. The agent handles the value; persistence is prevented.
- Injection at spawn. Secret references in server config resolve to environment variables when an MCP server launches, so the agent orchestrates the server without ever seeing its credentials. The launcher refuses to resolve strict-class references here, on purpose, so a synced, agent-writable config file cannot be edited into an exfiltration path.
- In-core flows. One class of operations, hardcoded flow by flow, consumes a vault-held token entirely inside the core binary. The agent triggers the flow and sees only the outcome.
Strict-class sits below all three. Nothing automated touches it, ever.
The honest parts
The catch-all pattern has a false-positive bill, and I can put numbers
on it. Over the last thirty days the scrubber emitted 18,344 markers,
and 15,832 of them, 86 percent, came from the catch-all. They cover
7,583 distinct values, against 90 distinct values for every named
pattern combined. Nobody has seven and a half thousand real secrets.
When I went to fetch the exact noise share for this post, the health
check that is supposed to report it turned out not to exist yet. The
decision record promises the check; the code never grew it, which is
its own small lesson about the distance between a consequences section
and a shipped binary. So I built it before the post shipped: it
decrypts the window's sidecars in-process, classifies each value
against known non-secret shapes, and lets only counts escape. The
verdict on a real week: 64 percent of catch-all markers are
identifiably not secrets, and the biggest class by far is long
filesystem paths, which clear the entropy bar constantly. I had guessed
base64 blobs; I was wrong, and I only know that because the check now
exists. The rest stays unclassified on purpose, because a random base64
run is exactly what a real secret looks like, and a classifier that
calls it noise has learned to shrug at leaks. The alternative, an
entropy bar high enough to never misfire, would be high enough to miss
things I care about.
The pattern set is versioned, and old events are not retroactively
re-scrubbed. A pattern I add today protects the future, not the past. The
guarantee is honestly prospective, and I would rather say so than imply
otherwise.
And the sidecar is the only place on disk the captured originals exist.
Lose the vault key, lose the recovery path, permanently. That matches the
posture the vault already had, and I accept it, but it deserves saying
out loud.
What transfers
- Scrub where data enters persistence, not where it leaves. One choke point on the write path beats any number of cleanup passes, each of which creates a second version of the truth.
-
Redact values, keep identity. A typed marker with a stable
truncated hash preserves the analytical structure of your data - what
co-occurs with what - while removing the secret. Plain
[REDACTED]throws both away. - Class secrets by "may an automated flow ever handle this?" Not by sensitivity. The answer cleanly decides which access machinery each value gets, including none.
- Do not let the distrusted party compose the privileged action. Any design where the agent writes a free-form instruction that executes with a secret is an exfiltration primitive waiting for its prompt injection.
- Fail closed, degrade loudly. Refuse to write what you cannot protect, and report your false-positive rate instead of quietly eating it. Then check the report actually exists: mine did not, and I found out only when I needed the number.
The scrubber, the markers, and the sidecar were all in place before the
memory layers filled up, and that ordering is the part I would insist on.
Retrofit redaction onto a store that already holds six months of
transcripts and the past stays exposed no matter what you ship. The time
to decide what an agent is allowed to remember is before it starts
remembering.
Top comments (1)
"Scrub once, at the point data enters persistence" is the right call, and the asymmetry argument (milliseconds now vs an ever-growing bill) is the cleanest way I've seen it justified.
The one place I'd push: regex-plus-entropy catches the secrets that look like secrets. The nasty residual is the inline secret that's structurally boring — a bare bearer token with no prefix, an internal hostname that's only sensitive in context, a customer ID that's PII in your jurisdiction but just a number to a scrubber. Entropy thresholds miss short low-entropy tokens and false-positive on hashes and UUIDs you actually want to keep.
Two things that helped us: scrub at write and keep a quarantine so a missed value can be purged retroactively without rewriting git history (which, with a synced repo, is the part that actually bites). And redact by reference, not deletion — replace with
<scrubbed:apikey>so the consolidation pass still sees that a credential was there and doesn't hallucinate around the hole. Did you consider making the scrubber fail-closed on the write hook?