Update, Oct 23: three things in the first version of this post were wrong or too strong. It said we spun up a disposable server and cloned the repo. When we wrote it, we had only read the code and the issue tracker on GitHub. Since then we have run autoharness's redaction function on a disposable server with made-up credentials only. We have not run the full tool (hooks, MCP server, storage), and we are sending our results to the maintainers first. It also called #273, #272, #268 and #269 issues, but they are pull requests from outside contributors. And it said the maintainers ship fixes fast, which holds for the first two leaks below but not for the fixes that are open now. All three are corrected in the text.
We wanted autoharness. 10,627 stars as of Oct 23, MIT licensed, and it solves a real problem: Claude Code skills pile up, overlap, and nobody prunes the ones that stopped getting used. Autoharness distills new skills from real sessions and archives what goes stale. We were ready to install it directly on our own machine.
Then we read what it actually does before pointing it at a machine that holds real credentials and unpublished security findings. It registers an MCP server and hooks that capture every turn across your sessions: your input, the agent's output, tool calls. It redacts and stores slices of that as evidence inside skill folders.
So before installing it anywhere real, we read the one part that actually matters here, the redaction code and its own issue tracker, on GitHub.
What we found is a documented history, right in the project's own tracker:
- Issue #213 (closed): a private key's BEGIN line got redacted, but the key body itself still reached stored evidence files.
- Issue #212 (closed): later redaction rules overwrote earlier placeholders, so AWS keys, private keys, GitHub tokens, and bearer tokens all ended up labeled as one generic category.
- Pull request #273 (open, from an outside contributor): adds redaction for fine-grained GitHub personal access tokens, which suggests the current rules do not catch them.
- Pull requests #272, #268 and #269 (open, also from outside contributors): more redaction fixes, for secrets assigned to a quoted key, for overlapping matches, and for PGP key blocks.
- Issue #276 (open): the email rule is quadratic on a long run of local-part characters.
None of this means autoharness is badly built. Issues #212 and #213 were closed within four days of being reported, and a project with a public paper trail of "this secret type leaked, here is the fix" is more trustworthy than one with no visible redaction bugs at all. The open fixes are a mixed picture: #269 and #273 merge cleanly into main and have had no review yet, #272 has a leak regression that another contributor reported in its comments, and #268 conflicts with main and needs a rebase. As of Oct 23 the repo has no new commit since October 15, which may just be a quiet week. Redacting secrets reliably from free-form session text is a genuinely hard problem. We would rather see that history than a clean-looking tracker that just means nobody has looked hard enough yet.
Our conclusion: when a tool's job is to read your own working sessions, test it on a throwaway machine first, with no real secrets anywhere near it, and read its redaction code and issue history before pointing it at anything real. Not because this specific project did something wrong, but because "redact my own credentials from my own session transcripts, automatically" is exactly the kind of promise that is hardest to verify by reading a README and easiest to verify by testing it where a mistake costs nothing.
We are not running autoharness on our main machine yet. We have only tested its redaction function, on a throwaway server with made-up credentials, and not the full tool. We will share the results with the maintainers first and write them up here afterwards. Until then, it stays off the machine that holds real credentials.
Best regards,
Abdellah
github.com/holistis
Top comments (0)