Hi everyone,
I wanted to share a project I’ve been developing for my own workflow with Claude Code, and hopefully get some feedback and ideas from...
For further actions, you may consider blocking this person and/or reporting abuse
The redaction claim is the part I'd want a number on. Pattern-based scrubbing catches secrets that look like secrets — high-entropy strings, known key prefixes — but the failure mode that actually matters is the one that doesn't match a pattern: a token stored as a plain-looking config value, a credential assembled from two innocuous-looking pieces at runtime, or a secret that's been base64'd or split across files that individually look clean. Have you measured a false-negative rate against secrets deliberately shaped to dodge common patterns, the way you'd fuzz test a parser? A choke point that's confident about what it catches but silent about what it doesn't is the exact "false success" failure mode you mention chasing in the reliability side of this project — worth holding the security side to the same bar.
You were right, and the answer is worse than "not measured" — I went and looked after reading this, and found three false negatives in the layer you were asking about. Thank you for the push.
On what was and was not measured. The redactor's own standard was to measure the checksum against random input before shipping a rule — Luhn passes 9.8% of random 16-digit numbers, Thailand's national-ID checksum passes 10.0% of epoch-millisecond timestamps (the commonest 13-digit run in any log), IBAN's mod-97 passes 1.02% of random alphanumerics. That is entirely a false-positive measurement, and you have put your finger on exactly what it does not cover. There was no false-negative corpus at all.
What a false-negative pass found, within hours:
An IBAN followed by an ordinary word was not redacted at all. The pattern allowed a separator inside the number and a space is a separator, so
wire to DE89370400440532013000 todayconsumed " today" into the match, the length stopped equalling Germany's 22, mod-97 failed, and the rule redacted nothing. 6 of 10 realistic sentences leaked a real IBAN in full. The shape that broke it — a number followed by a word — is the commonest shape in prose, which is precisely why a false-positive corpus of random strings never met it.The one rule that skipped the digit-fold table. A release the day before added folding so an identifier typed in Thai, Arabic-Indic, Persian, Devanagari, Tamil or fullwidth numerals is caught. Six rules read the folded text; IBAN read the raw line, so a fullwidth-digit IBAN never matched. Same commit, same set, one member forgotten.
A tab-separated card number or national ID leaked, which is what a spreadsheet, a TSV export or a copy out of a terminal table produces. The interesting part is why: the separator list existed twice — once in the rules and once, hand-written, in the cheap gate that decides whether to run them at all. Adding the tab to one list changed nothing, because the other one turned the layer off before any rule saw the line. Two lists that must agree, which is the failure this codebase already warns about elsewhere in the same file.
All three are fixed and re-measured: the sentence corpus goes 4/10 → 10/10, the fullwidth IBAN matches, tab-separated cards and IDs are caught, and the false-positive side is unchanged at 0 of 10 ordinary lines and 0 of 2,000 random 13-digit runs.
Now the part that matters more than the fixes. The three cases you actually named — a token stored as a plain-looking config value, a credential assembled at runtime from two innocuous halves, a secret base64'd or split across files — are, in the general case, not solvable by this design and I should not imply otherwise. A value with no shape, no checksum and no naming signal is indistinguishable from data. What the credential half does is narrower than "catches secrets": it catches known key prefixes, high-entropy values in credential-named positions, and credential-shaped values in credential-shaped syntax. A runtime-assembled secret is caught only if the assembled result passes through this choke point looking like one, and a base64 blob with an innocuous name is not caught.
There is also a scope line worth stating plainly, because "catches secrets" invites the wrong reading: this is not a repository scanner. It scrubs the tool's own generated output — the index, the injected block, a command's stdout — on the way to the model. It does not read or rewrite your source files, and it is not a substitute for a secret scanner in CI.
So the honest version of the claim is: it reduces what an assistant sees, measurably, and it will not catch a secret that has no shape. A false-negative corpus is now on the list beside the false-positive one, and if you have shapes you think should be in it, I would genuinely rather have them than not.
Good work — but worth being precise about what this proves. Fixing three found leaks doesn't establish a false-negative rate, it just means those three specific shapes no longer leak. The measurement gap you had for false positives (checksum-against-random-input) needs a false-negative equivalent: generate adversarial variants deliberately — separator injection, digit-folding scripts, delimiter mixing — rather than waiting for someone to notice by reading. On the duplicated separator list: that's a single-source-of-truth bug, and it'll recur elsewhere in the file unless both lists derive from one canonical format table instead of being kept in sync by hand.
Scope first, because a thread this long about redaction gaps starts to read like a repository scanner failing at its job. It isn't one, and it doesn't defend your repository. chamnan sits on exactly one path: what the tool itself sends to the model — the index it generates, the block it injects into context, the stdout of a command it ran. It reduces what goes out through that one pipe. It never reads or rewrites your source files, it is not in the way of a commit, a push or a build, and nothing else in your repository routes through it. So every gap below is a hole in one outbound pipe, and if you run a secret scanner in CI this neither replaces it nor overlaps with it.
Why that pipe is worth a choke point at all, since "not a scanner" only says what it isn't. The goal is that a key or a piece of personal data does not reach a model's backend in the first place. Not because any vendor is careless with it — they publish their handling and I have no reason to dispute it — but because none of it is verifiable from where I sit. What is verifiable, locally, is what left the machine. So the choke point is local, it runs before the request, and it is the one link in that chain I own. Given how regularly an AI product turns up in the news for a leak or an attack, the cheap move is to hand over less rather than to trust harder. Local-first zero trust, applied to the one direction that matters here: outbound.
That is also why the failure mode you named — a
<REDACTED>marker sitting next to a live credential — is the one I care most about here. Missing a secret means the assistant sees what it would have seen anyway; the exposure is unchanged. Marking the line means a reader is told it was handled when it wasn't, and that does change something: it converts an unknown into a false assurance.Both points land, and the first one lands harder than the fixes did.
You're right that three fixes is not a rate. I conflated "found and closed three leaks" with "measured what I miss", which is the same category error you caught the first time, one level up. So I did what you said — pointed something at the layer whose only job was to generate dodging shapes rather than to read code. In a day it found four more, and the first is worse than a silent miss:
A value that contains its own
=. Base64 padding, which every 16-byte key ends in, or aKEY=VALUE;KEY=VALUEconnection string. There is a step in the assignment rule that skips over a type annotation (val apiKey: String = "..."), and it happily read the secret as the type name and stopped at the=inside it. Measured on an Azure Storage connection string: 23 of the 24 characters of the AccountKey came back in the clear, with a<REDACTED>marker sitting immediately after them. That is your "confident about what it catches, silent about what it doesn't", except it isn't silent — it actively tells a reviewer the line was handled. Worse than a plain miss.A list.
api_keys: ["a", "b", "c"]— the assignment rules answer "name, separator, one value" and stop, so the first element was redacted and the other two printed beside it. A YAML block sequence (passwords:then indented- itemlines) was missed entirely. Rotated keys and migration pairs are exactly how that shape occurs.An
alg:noneJWT. The pattern required 8+ characters in the signature segment. RFC 7519 allows it to be empty, which is the classic forgery shape — so the one token most worth flagging was the one token it could not see. Header and claims payload, whole.Go's unquoted form.
var apiPassword string = <value>with no quotes. The type-before-assign shape was wired into the quoted rule and not into the bare one.On the duplicated separator list — you predicted it would recur, and I went and counted. That exact shape, a rule applied to some members of a set and forgotten in the identical ones beside it, is recorded 18 distinct times across 22 files in this codebase, in its own bug comments. Nobody had ever added them up; each was written down where it happened, which is the right place for the reason and the wrong place for the pattern. The Go case above is that shape again — wired into one rule and not into its twin — which makes your prediction good within about eighteen hours.
So the fix isn't a canonical format table — it's that a check now has to derive its own subjects and assert it found them. The two
slug()audits walk everyFunctionDefnamedslugin the shipped modules, assert the population is at least four, and only then assert the property of each. A sweep that reaches nothing fails instead of passing quietly as coverage. That, and the tally is written down in one place now instead of eighteen.Where the number actually stands, plainly. I still do not have a false-negative rate and I am not going to imply one. What exists is a 55-shape labelled corpus: 98.2% recall, 100% precision, 0 of 43 ordinary strings damaged. But that corpus is hand-written, so it measures what I thought to write down — which is your original objection wearing a lab coat. A real rate needs generated variants, and generation is now the thing to build, not another round of reading.
Status, so "fixed" doesn't get read as "shipped". All four above are fixed and measured on my machine; none is committed yet and none is released. Releases are paused while this gets dogfooded, so nothing here is in anyone's hands.
One thing from your first comment did land properly, though, and it's the case you named that I'd have said was unsolvable: a token stored as a plain-looking config value. It still is unsolvable per-value — but not per-column. A CSV or TSV carries its meaning in the header, so a column headed
password,token,iban,aadhaaror a dozen relatives now has its values redacted for the rest of the file regardless of what they look like:hunter2,sk-local-dev, an internal id with no check digit. The header is the evidence, not the entropy. That one is committed and through the suite (3,893 checks). It doesn't generalise to your runtime-assembled or base64'd cases, and I'm not claiming it does — but it moves one of the three from "no signal" to "signal is in the neighbouring cell".Thank you for coming back a second time. The first comment moved the code; this one moved the method, which is the more expensive of the two to be wrong about.
The move from "three fixes" to "a check now asserts it found its subjects" is the real upgrade here — that's the difference between coverage you hope exists and coverage that fails loud when it doesn't. One thing worth turning into a standing gate rather than a retrospective: the 18-instance tally of the duplicated-list bug is exactly the kind of thing that will quietly become 19 the next time someone adds a rule under deadline pressure. Wire the sweep that found those 18 into CI as a structural check — any new rule that reads from a hand-maintained list must prove its list is derived from (or diffed against) the canonical set, not just pass the corpus. Otherwise the corpus catches it eventually, after it's shipped once. Also: the CSV/TSV header-based redaction is a new detection surface with its own false-negative class — header name variants (localized headers, pwd vs password, a header row dropped by a pandas re-export) will escape it the same way the digit-fold table escaped IBAN. Worth running the same "derive subjects, assert found" discipline against the header-matching list itself before calling that one closed.
Both went into the code today, and the second one landed the way you said it would.
The header list. You called the false-negative class before I looked at it, so I tested it the way you'd want rather than the way that flatters it — the real matcher, against variants I did not write it for:
11 of 19 escape. Every localized header, plus the abbreviations and the no-separator compound. The compound rule holds — anything ending in a credential word as its last component matches, and
password_policycorrectly does not — but the vocabulary is English-only and nothing said so. A spreadsheet exported from a Spanish, Thai or German system passes through whole.I'm not going to call that closed by adding eight more words to the list. That's the same hand-maintained shape you flagged, one language wider. What it needs is what you described: the list has to derive its own subjects and assert it found them, and I don't yet have a principled way to enumerate "credential header, any language" the way I can enumerate "every rule that needs a digit separator". So: open, measured, and the number is 11/19. I'd rather say that than ship a longer list and call it coverage.
The structural gate. Partly there, and I should be precise about which part. The separator case is closed the way you'd want — 7 of 7 rules that need a digit separator read the shared
_SEP, 0 hand-write their own, and the suite asserts the population rather than a member. What does not exist is the general form: nothing stops rule number 8 from being written with its own bracket expression, because the check names the two rules that were wrong rather than deriving "every rule that matches a delimited number".Today produced three more instances of exactly the shape you predicted, which is the argument for your version over mine:
That last one is why I stopped writing shape-matchers for a while. A list of specific terms that must never appear caught it on the first run, in a tree four pattern-based checks had already passed. Shape-matching catches the classes you have already been burned by; the next one is the class whose shape you did not think to describe.
Thanks for pushing on the method rather than the fixes. The header number above exists because you predicted it, and I would not have gone looking.