Hi everyone,
I wanted to share a project I’ve been developing for my own workflow with Claude Code, and hopefully get some feedback and ideas from the community on where to take it next.
How it started
Working on large-scale, complex projects, I needed better control over context, token consumption, and security. Before building my own tool, I tested various plugins in the market. As someone who likes to measure and verify metrics independently, I ran tests on actual context usage and noticed that under large, messy repository conditions, results often varied from what was expected.
Since my projects involve sensitive codebases, I wanted a deterministic local pre-processing layer that sanitizes, budgets, and organizes data before anything is passed to Claude Code—ensuring zero unexpected data leaks and absolute predictability.
The struggle & research
What started as a simple helper script turned into a serious engineering effort. I went through over 1,000 research papers and technical docs to refine context retention, token budgeting, and local redaction.
Building reliability tooling is a constant battle against edge cases: silent context loss, false successes, cross-platform path quirks (like Windows process locks), and state drift.
My rule became: Never believe a bug is fixed until a test proves it.
Today, the plugin (chamnan) is at v1.22.1. It's built in pure Python (standard library only, zero external packages) and backed by 3,500+ automated tests.
What it actually does (It's NOT an AI, it's a pre-processing / reliability layer)
- Deterministic Context Budgeting: Pre-indexes and trims repository context before passing it to Claude Code (slashed Quick Index overhead dramatically in real-world tests).
- Local Security & Redaction: Runs local choke-point scrubbing so secrets and internal strings never leave your machine.
-
Agent Continuity / Memory: Keeps track of session decisions, lessons, and project history locally inside
.chamnan/so Claude Code doesn't waste tokens re-discovering architecture across sessions. - Monorepo & Multi-adapter support: Designed to handle nested checkouts, messy directory trees, and OS-specific edge cases cleanly.
Looking for community feedback
I dogfood this daily, but since I built it to share with other developers working on large or privacy-conscious projects, I'd love to hear your thoughts:
- What edge cases or repository structures have broken your Claude Code workflows in the past?
- What safety verifications or context features would you like to see in a local helper plugin?
If you're interested in the architecture or test setup, you can check out the repository here: https://github.com/ArcticFox2029/chamnan
Appreciate any feedback!
Top comments (13)
The redaction claim is the part I'd want a number on. Pattern-based scrubbing catches secrets that look like secrets — high-entropy strings, known key prefixes — but the failure mode that actually matters is the one that doesn't match a pattern: a token stored as a plain-looking config value, a credential assembled from two innocuous-looking pieces at runtime, or a secret that's been base64'd or split across files that individually look clean. Have you measured a false-negative rate against secrets deliberately shaped to dodge common patterns, the way you'd fuzz test a parser? A choke point that's confident about what it catches but silent about what it doesn't is the exact "false success" failure mode you mention chasing in the reliability side of this project — worth holding the security side to the same bar.
You were right, and the answer is worse than "not measured" — I went and looked after reading this, and found three false negatives in the layer you were asking about. Thank you for the push.
On what was and was not measured. The redactor's own standard was to measure the checksum against random input before shipping a rule — Luhn passes 9.8% of random 16-digit numbers, Thailand's national-ID checksum passes 10.0% of epoch-millisecond timestamps (the commonest 13-digit run in any log), IBAN's mod-97 passes 1.02% of random alphanumerics. That is entirely a false-positive measurement, and you have put your finger on exactly what it does not cover. There was no false-negative corpus at all.
What a false-negative pass found, within hours:
An IBAN followed by an ordinary word was not redacted at all. The pattern allowed a separator inside the number and a space is a separator, so
wire to DE89370400440532013000 todayconsumed " today" into the match, the length stopped equalling Germany's 22, mod-97 failed, and the rule redacted nothing. 6 of 10 realistic sentences leaked a real IBAN in full. The shape that broke it — a number followed by a word — is the commonest shape in prose, which is precisely why a false-positive corpus of random strings never met it.The one rule that skipped the digit-fold table. A release the day before added folding so an identifier typed in Thai, Arabic-Indic, Persian, Devanagari, Tamil or fullwidth numerals is caught. Six rules read the folded text; IBAN read the raw line, so a fullwidth-digit IBAN never matched. Same commit, same set, one member forgotten.
A tab-separated card number or national ID leaked, which is what a spreadsheet, a TSV export or a copy out of a terminal table produces. The interesting part is why: the separator list existed twice — once in the rules and once, hand-written, in the cheap gate that decides whether to run them at all. Adding the tab to one list changed nothing, because the other one turned the layer off before any rule saw the line. Two lists that must agree, which is the failure this codebase already warns about elsewhere in the same file.
All three are fixed and re-measured: the sentence corpus goes 4/10 → 10/10, the fullwidth IBAN matches, tab-separated cards and IDs are caught, and the false-positive side is unchanged at 0 of 10 ordinary lines and 0 of 2,000 random 13-digit runs.
Now the part that matters more than the fixes. The three cases you actually named — a token stored as a plain-looking config value, a credential assembled at runtime from two innocuous halves, a secret base64'd or split across files — are, in the general case, not solvable by this design and I should not imply otherwise. A value with no shape, no checksum and no naming signal is indistinguishable from data. What the credential half does is narrower than "catches secrets": it catches known key prefixes, high-entropy values in credential-named positions, and credential-shaped values in credential-shaped syntax. A runtime-assembled secret is caught only if the assembled result passes through this choke point looking like one, and a base64 blob with an innocuous name is not caught.
There is also a scope line worth stating plainly, because "catches secrets" invites the wrong reading: this is not a repository scanner. It scrubs the tool's own generated output — the index, the injected block, a command's stdout — on the way to the model. It does not read or rewrite your source files, and it is not a substitute for a secret scanner in CI.
So the honest version of the claim is: it reduces what an assistant sees, measurably, and it will not catch a secret that has no shape. A false-negative corpus is now on the list beside the false-positive one, and if you have shapes you think should be in it, I would genuinely rather have them than not.
Good work — but worth being precise about what this proves. Fixing three found leaks doesn't establish a false-negative rate, it just means those three specific shapes no longer leak. The measurement gap you had for false positives (checksum-against-random-input) needs a false-negative equivalent: generate adversarial variants deliberately — separator injection, digit-folding scripts, delimiter mixing — rather than waiting for someone to notice by reading. On the duplicated separator list: that's a single-source-of-truth bug, and it'll recur elsewhere in the file unless both lists derive from one canonical format table instead of being kept in sync by hand.
Scope first, because a thread this long about redaction gaps starts to read like a repository scanner failing at its job. It isn't one, and it doesn't defend your repository. chamnan sits on exactly one path: what the tool itself sends to the model — the index it generates, the block it injects into context, the stdout of a command it ran. It reduces what goes out through that one pipe. It never reads or rewrites your source files, it is not in the way of a commit, a push or a build, and nothing else in your repository routes through it. So every gap below is a hole in one outbound pipe, and if you run a secret scanner in CI this neither replaces it nor overlaps with it.
Why that pipe is worth a choke point at all, since "not a scanner" only says what it isn't. The goal is that a key or a piece of personal data does not reach a model's backend in the first place. Not because any vendor is careless with it — they publish their handling and I have no reason to dispute it — but because none of it is verifiable from where I sit. What is verifiable, locally, is what left the machine. So the choke point is local, it runs before the request, and it is the one link in that chain I own. Given how regularly an AI product turns up in the news for a leak or an attack, the cheap move is to hand over less rather than to trust harder. Local-first zero trust, applied to the one direction that matters here: outbound.
That is also why the failure mode you named — a
<REDACTED>marker sitting next to a live credential — is the one I care most about here. Missing a secret means the assistant sees what it would have seen anyway; the exposure is unchanged. Marking the line means a reader is told it was handled when it wasn't, and that does change something: it converts an unknown into a false assurance.Both points land, and the first one lands harder than the fixes did.
You're right that three fixes is not a rate. I conflated "found and closed three leaks" with "measured what I miss", which is the same category error you caught the first time, one level up. So I did what you said — pointed something at the layer whose only job was to generate dodging shapes rather than to read code. In a day it found four more, and the first is worse than a silent miss:
A value that contains its own
=. Base64 padding, which every 16-byte key ends in, or aKEY=VALUE;KEY=VALUEconnection string. There is a step in the assignment rule that skips over a type annotation (val apiKey: String = "..."), and it happily read the secret as the type name and stopped at the=inside it. Measured on an Azure Storage connection string: 23 of the 24 characters of the AccountKey came back in the clear, with a<REDACTED>marker sitting immediately after them. That is your "confident about what it catches, silent about what it doesn't", except it isn't silent — it actively tells a reviewer the line was handled. Worse than a plain miss.A list.
api_keys: ["a", "b", "c"]— the assignment rules answer "name, separator, one value" and stop, so the first element was redacted and the other two printed beside it. A YAML block sequence (passwords:then indented- itemlines) was missed entirely. Rotated keys and migration pairs are exactly how that shape occurs.An
alg:noneJWT. The pattern required 8+ characters in the signature segment. RFC 7519 allows it to be empty, which is the classic forgery shape — so the one token most worth flagging was the one token it could not see. Header and claims payload, whole.Go's unquoted form.
var apiPassword string = <value>with no quotes. The type-before-assign shape was wired into the quoted rule and not into the bare one.On the duplicated separator list — you predicted it would recur, and I went and counted. That exact shape, a rule applied to some members of a set and forgotten in the identical ones beside it, is recorded 18 distinct times across 22 files in this codebase, in its own bug comments. Nobody had ever added them up; each was written down where it happened, which is the right place for the reason and the wrong place for the pattern. The Go case above is that shape again — wired into one rule and not into its twin — which makes your prediction good within about eighteen hours.
So the fix isn't a canonical format table — it's that a check now has to derive its own subjects and assert it found them. The two
slug()audits walk everyFunctionDefnamedslugin the shipped modules, assert the population is at least four, and only then assert the property of each. A sweep that reaches nothing fails instead of passing quietly as coverage. That, and the tally is written down in one place now instead of eighteen.Where the number actually stands, plainly. I still do not have a false-negative rate and I am not going to imply one. What exists is a 55-shape labelled corpus: 98.2% recall, 100% precision, 0 of 43 ordinary strings damaged. But that corpus is hand-written, so it measures what I thought to write down — which is your original objection wearing a lab coat. A real rate needs generated variants, and generation is now the thing to build, not another round of reading.
Status, so "fixed" doesn't get read as "shipped". All four above are fixed and measured on my machine; none is committed yet and none is released. Releases are paused while this gets dogfooded, so nothing here is in anyone's hands.
One thing from your first comment did land properly, though, and it's the case you named that I'd have said was unsolvable: a token stored as a plain-looking config value. It still is unsolvable per-value — but not per-column. A CSV or TSV carries its meaning in the header, so a column headed
password,token,iban,aadhaaror a dozen relatives now has its values redacted for the rest of the file regardless of what they look like:hunter2,sk-local-dev, an internal id with no check digit. The header is the evidence, not the entropy. That one is committed and through the suite (3,893 checks). It doesn't generalise to your runtime-assembled or base64'd cases, and I'm not claiming it does — but it moves one of the three from "no signal" to "signal is in the neighbouring cell".Thank you for coming back a second time. The first comment moved the code; this one moved the method, which is the more expensive of the two to be wrong about.
The move from "three fixes" to "a check now asserts it found its subjects" is the real upgrade here — that's the difference between coverage you hope exists and coverage that fails loud when it doesn't. One thing worth turning into a standing gate rather than a retrospective: the 18-instance tally of the duplicated-list bug is exactly the kind of thing that will quietly become 19 the next time someone adds a rule under deadline pressure. Wire the sweep that found those 18 into CI as a structural check — any new rule that reads from a hand-maintained list must prove its list is derived from (or diffed against) the canonical set, not just pass the corpus. Otherwise the corpus catches it eventually, after it's shipped once. Also: the CSV/TSV header-based redaction is a new detection surface with its own false-negative class — header name variants (localized headers, pwd vs password, a header row dropped by a pandas re-export) will escape it the same way the digit-fold table escaped IBAN. Worth running the same "derive subjects, assert found" discipline against the header-matching list itself before calling that one closed.
Both went into the code today, and the second one landed the way you said it would.
The header list. You called the false-negative class before I looked at it, so I tested it the way you'd want rather than the way that flatters it — the real matcher, against variants I did not write it for:
11 of 19 escape. Every localized header, plus the abbreviations and the no-separator compound. The compound rule holds — anything ending in a credential word as its last component matches, and
password_policycorrectly does not — but the vocabulary is English-only and nothing said so. A spreadsheet exported from a Spanish, Thai or German system passes through whole.I'm not going to call that closed by adding eight more words to the list. That's the same hand-maintained shape you flagged, one language wider. What it needs is what you described: the list has to derive its own subjects and assert it found them, and I don't yet have a principled way to enumerate "credential header, any language" the way I can enumerate "every rule that needs a digit separator". So: open, measured, and the number is 11/19. I'd rather say that than ship a longer list and call it coverage.
The structural gate. Partly there, and I should be precise about which part. The separator case is closed the way you'd want — 7 of 7 rules that need a digit separator read the shared
_SEP, 0 hand-write their own, and the suite asserts the population rather than a member. What does not exist is the general form: nothing stops rule number 8 from being written with its own bracket expression, because the check names the two rules that were wrong rather than deriving "every rule that matches a delimited number".Today produced three more instances of exactly the shape you predicted, which is the argument for your version over mine:
That last one is why I stopped writing shape-matchers for a while. A list of specific terms that must never appear caught it on the first run, in a tree four pattern-based checks had already passed. Shape-matching catches the classes you have already been burned by; the next one is the class whose shape you did not think to describe.
Thanks for pushing on the method rather than the fixes. The header number above exists because you predicted it, and I would not have gone looking.
The 11/19 number is the right way to report it — a longer list dressed as coverage would've been worse than the honest gap.
On enumerating credential headers across languages: instead of growing the vocabulary (an unbounded, moving target), it might be worth scoring the column by value shape instead of header text — a column whose values are uniformly high-entropy, fixed-or-near-fixed length, and drawn from a narrow character set looks like a credential regardless of what the header is named in any language, the same way your checksum-based PII rules don't care what a variable is called. Header text becomes a confidence booster on top of that, not the sole gate.
On the structural gate: the invariant you're describing (derive the population, assert it was found) needs to run over the rule definitions themselves — an AST/grep-level CI check that walks every rule file and fails if any defines its own bracket/separator regex instead of importing the shared one. That's the only way rule #8 gets caught before it ships.
Both of these landed, and the first one is in the source with your framing quoted in the comment beside it — 1.25 credits you by name in the release notes for that thread.
On value shape: it went in as the tie-breaker rather than the gate, and the reason is a measurement I owe you. Shape as a standalone rule got built and rejected the same afternoon, because git commit SHAs and AWS access key ids are indistinguishable on entropy, length and character set. Run it alone and it eats every hash in a changelog while still missing the keys with a human-readable prefix. So the arrangement is the inverse of yours and the same idea: under a key that already says
password, shape decides whether the value is a type name or a credential —password_type = "bcrypt"stays exempt,password_typeholding punctuation and mixed case does not. There is nothing left for it to confuse at that point.Where I think you are still right and I am not there yet: that only helps when a key exists. A headerless column of uniformly-shaped values has nothing to hang off, and I have no answer for it.
On the structural gate — that is the one that paid off, twice, and the second time was this week.
1.26 shipped a check that derives the credential vocabulary from the module and asserts every word in it. It immediately found that a fix I had written for
password = "password"had landed on one member of a set of twelve:secret,credential,apikey,passphrase,auth,cred,keypass,storepass,passwdandsecretkeyall printed their value byte for byte. An earlier exemption answered "that's a label" for every one of them and ran before the rule that would have caught it. The test that caught the original asserted one example; its replacement asserts the population, which is your invariant and it worked exactly as you said it would.What I have NOT built is the part you actually specified — the AST walk that fails a rule file for defining its own bracket/separator regex instead of importing the shared one. Mine derive the population from the module and assert the members; yours derives it from the rule definitions and asserts they share a source. Those catch different things. I counted three places that define their own, which is three chances for rule #8. It is on the list.
1.26 is out, if you want the diff: the credential-vocabulary check is the one worth reading, and
tools/verify_release.pyreprints the numbers on your own machine rather than asking you to take mine.The AST walk for shared-_SEP-import will catch a hand-rolled inline regex but not an indirect one -- re-exported through a wrapper module, imported under an alias, or assembled at runtime from string concatenation. All of those still "import the shared source" by the letter of the check while not actually sharing behavior once someone edits the wrapper. A runtime population check closes that gap the same way the credential-vocabulary check closed the header gap: instrument every call site that actually performs separator-splitting during a real test run, and assert the observed set of call sites matches the expected population derived from the rule definitions, not by reading source, but by watching what code path actually executed. Catches the case where the AST technically imports the right symbol but a decorator or monkeypatch swaps the implementation before it runs.
Ran it as five planted shapes against the real resolver rather than arguing about it, and you're half right, which is the useful half.
Caught already: the inline literal, and
+concatenation, and an f-string. The resolver walksBinOpandJoinedStr, so "assembled at runtime from string concatenation" was covered before you wrote it.Not caught: through a plain name, through
"".join([...]), and through%. Those three walked straight past. So the hole is real, it is narrower than the description though.The part I'd push back on is the fix. There's no separator-splitting call site to instrument — the invariant isn't about behaviour at runtime, it's about whether a compiled pattern spells the literal itself or routes through the shared name. By the time anything executes,
re.compilehas eaten the source string and the provenance is gone.patternwould tell you the final text; it can't tell you which name produced it, which is the only thing the gate cares about.What did work was resolving names in the AST, with one exemption that turns out to be the whole trick: when the resolver meets a name ending in the shared constant's suffix it returns empty rather than the value. Without that, the correct route — the rules that do import the shared name — resolve to a string containing the separator and get flagged as violations. The naive version fails every rule that's doing the right thing.
Also worth saying: the module currently has zero hand-rolled separators, so the check was passing over an empty population and would have kept passing if the resolver rotted. It carries seven planted shapes now, five that must flag and two that must not, and I blinded the Name branch to watch it go red before believing it.
Your instinct generalised further than this thread, incidentally. The same day, a different check that forbids
os.geteuidon Windows missed a call because it matched the literal nameosand the caller was using an import alias. That one took down a CI leg.The os.geteuid alias-miss and the separator resolver's blind spot are the same bug in two costumes: matching the literal token instead of resolving what it's bound to. Worth pulling that out as a shared alias-aware symbol resolver -- given a Name node, walk backward through the module's import table (aliases and from-imports included) to the actual origin module.symbol -- rather than re-solving it per rule. That fixes both at once and closes the whole class before the next rule discovers it independently, the way this thread just did twice in one day.
Ran the five shapes again before answering, and you're right about the gap.
from os import geteuidthen a baregeteuid()walked past all three branches —osisn't in the absent-modules set because os exists on Windows, and the call site is a Name, not an Attribute. Same for theas gidversion. That's closed now, with the seven shapes planted in the file so it can't quietly stop working.The shared resolver I don't think holds, and I went and looked rather than guessing. The two checks walk different tables. Check 90 builds
{asname: module_name}off the import statements. Check 31 builds{target: value}off assignments. A resolver that takes a Name and walks the import table backward fixes 90 and does nothing at all for 31, because the separator variable it's chasing was never imported, it was assigned two lines up. Same symptom, different lookup.So the class is real but it isn't code-shaped. What generalises is the question — "does this rule match the token or what the token is bound to" — and each rule still has to answer it against whatever binds names in its own context. I'd rather have that on a checklist for the next rule than a resolver with one real caller.
One thing I nearly got wrong, since this thread has a habit of catching those. My first instinct was to skip the fix: nothing in the repo uses that import shape, 0 of 70 from-imports come off os or signal. Then I found we'd already answered that question the other way last week on check 31, which was passing over an empty population too, and the answer there was to plant shapes rather than drop it. An empty population isn't evidence the check is pointless, it's the reason the check has nothing proving it works.
The checklist framing is right, and I'd go one step further: the two lookups don't have to stay separate functions, they can be the same utility parameterized by which binding forms it walks. Pass check 90 the import table, check 31 the assignment table, and the walk itself — follow a Name to its binding, repeat until you hit a literal or run out of table — is identical code either way. What's not shareable is deciding which tables apply to which check, and that stays a per-rule checklist item like you said. Also, +1 on treating an empty population as 'nothing proving this works' rather than 'nothing to prove' — that's the same mistake in reverse of trusting a check that's never fired.