chamnan is a Claude Code plugin that keeps engineering context in the repository, so an agent
stops rediscovering the same work every session. Part of that is a redactor: it scrubs what the
tool sends out, before it reaches a model backend.
It scored 98.3% recall on a labelled corpus. That number was true and it was hiding something.
The number was measured over a corpus missing the case it was about
A reader — Peter — made an argument on somebody else's
article: a pipeline can hit strong recall on straightforward payloads while systematically
missing anything harder, and a blended recall number hides that completely.
So I split the figure by the retrieval problem each case poses. A credential can be named by an
assignment (API_KEY=...), positioned under a column header, described in surrounding prose, or
standing bare with nothing but its own characters to go on.
The column-header class had zero cases in it.
That is the class that leaked. A password under a CSV header — nombre,correo,contraseña —
escaped in every language the redactor claimed to handle, English included. The fix shipped. The
measurement that should have caught it had never sampled it once.
An empty class does not show up in a breakdown as a zero. It does not show up at all, which is
how it survived being looked at.
The fix moved the headline the wrong way
The corpus now derives its column cases from the redactor's own list of claimed languages, so a
language it claims is a language it is measured on. Sixteen languages, thirty-six cases.
All thirty-six pass — so the blended figure went up, 98.3% to 99.0%.
That is the same defect from the other side. A headline improved by testing more of what already
works flatters exactly as much as one that hides a weak class. So both numbers are published now:
99.0% overall, 93.8% in the weakest class, and a check fails if either goes missing from the
README or SECURITY.md.
Three more places a number could be confidently wrong
Peter's other arguments turned into three more fixes:
- A rate without its denominator. Every reported rate now carries the count it was computed over — a rate over a shrinking, unwatched denominator produces a confident green with nothing behind it.
- A third state for "cannot decide". A file the parser refuses is not a file nobody documented, and only one of those is fixable by writing a docstring. The coverage line says which is which — and still counts them, because removing them raises the percentage, which would mean a repository scoring better because the tool can read less of it.
- A list of accepted findings that could grow unnoticed. When a self-scan finds something new, the cheapest way to make the check pass is to add it to the accepted list, which silences a real finding exactly as well as fixing one. That list has a ceiling now, and raising it has to appear in a diff.
Checking it
check 4869 / 4869, and 1,466 of 1,466 index claims true, on Linux, macOS and Windows across two
Python versions.
git clone https://github.com/ArcticFox2029/chamnan
cd chamnan && python3 tools/verify_release.py
It runs the suite and the index-claim check on your machine and prints what happened. It refuses
to report a result when the totals line is missing, because a run that dies mid-way prints no
failure lines at all.
One honest limit: the recall corpus is synthetic, built to be hard to index, and is not part
of the repository — so you cannot reproduce those particular numbers from a clone. The suite and
the index claims you can.
MIT. Python standard library only. Nothing leaves the machine.
Top comments (0)