DEV Community

ooocooc
ooocooc

Posted on Fully Autonomous

How to Audit AGENTS.md and CLAUDE.md for Stale Instructions — Without Deleting Anything

#ai #agents #opensource #productivity

AI coding agents keep improving. Their instruction files usually move in only one direction: they grow.

A new workaround goes into AGENTS.md. A tool-specific rule lands in CLAUDE.md. A Skill is copied for a second setup. Months later, paths move, tools disappear, two rules disagree, and nobody is sure which instruction can be removed safely.

The obvious answer is “clean up the file.” The dangerous part is deciding what clean up means.

Anthropic's own guidance now recommends keeping CLAUDE.md short and periodically removing stale content. That is sensible. But a newer model, a newer runtime, or an old timestamp is not evidence that a specific rule is obsolete.

I wanted an audit that could say:

  • this reference is definitely broken;
  • these two generic Skills are byte-identical complete bundles;
  • this large entry file may belong behind progressive disclosure;
  • this rule might be unnecessary, but it needs a controlled test;
  • this domain rule or authorization gate is outside automatic retirement.

That is why I built Skill Sunset.

The key distinction: evidence versus hypothesis

Skill Sunset uses five conservative outcomes:

Verdict Meaning
MERGE Exact duplication or a conflicting same-name generic Skill needs review
UPDATE A path, reference, tool assumption, or version-coupled instruction is stale
DEMOTE Useful detail should probably move out of always-loaded context
RETIRE A complete, same-name, byte-identical generic Skill bundle is a recoverable retirement candidate
TEST The rule may be obsolete, but behavior must be compared before removal

TEST is intentionally not RETIRE.

For example, “always use this tool” may create context or latency overhead. It may also encode a real reliability requirement. Wording alone cannot decide which one is true.

Run a local, read-only audit

No global installation is required:

npx skill-sunset@latest audit --codex --open
Enter fullscreen mode Exit fullscreen mode

For Claude Code configuration:

npx skill-sunset@latest audit --claude --open
Enter fullscreen mode Exit fullscreen mode

Or audit any bounded directory:

npx skill-sunset@latest audit /path/to/setup --open
Enter fullscreen mode Exit fullscreen mode

The audit is local and read-only. The core does not call an AI API, consume model quota, or read provider credentials. It writes an advisory report bundle to the output directory.

What a real fixture produces

The repository includes a public test fixture with five files. The current 0.2.0 CLI reports:

  • 5 files scanned;
  • 11 findings;
  • 2 UPDATE findings;
  • 1 DEMOTE finding;
  • 1 conservative RETIRE candidate;
  • 7 TEST hypotheses.

The deterministic findings include a broken local Markdown reference and a complete duplicate generic Skill bundle. The hypotheses include unconditional tool use, mandatory subagent routing, and a legacy reasoning scaffold.

Skill Sunset report showing 5 scanned files and 11 audit findings

This split matters. A missing file can be checked directly. Whether removing a reasoning scaffold changes task quality cannot.

Validation before deletion

For a TEST finding, Skill Sunset generates an experiment template. Validation is the default:

npx skill-sunset@latest test .skill-sunset/experiment-template.json --root .
Enter fullscreen mode Exit fullscreen mode

Commands do not run unless --run is explicit. Full environment inheritance requires a second explicit flag, --inherit-env.

Even a passing experiment proves only the acceptance criteria encoded in that experiment. It does not prove that a rule is universally useless.

The safe lifecycle is:

discover → inspect evidence → test if needed → human approval
→ recoverable archive or edit → re-audit → project tests → keep history
Enter fullscreen mode Exit fullscreen mode

No generated finding authorizes deletion.

External command-safety integration

The Skill Sunset audit surface is also recognized by HOL Guard, an open-source runtime safety project for AI agents.

The integration was reviewed and merged through HOL Guard PR #2611, then included in HOL Guard v3.0.18.

HOL Guard's integration deliberately covers the installed skill-sunset audit command while leaving experiment execution and npm/npx launcher policy to their existing controls. That limited scope is useful evidence: safety boundaries are clearer when each tool owns one well-defined surface.

What I want to learn next

The tool is deliberately conservative, so false positives and missed cases matter more than a large feature list.

Try it on a bounded directory:

npx skill-sunset@latest audit /path/to/setup --open
Enter fullscreen mode Exit fullscreen mode

If it catches a real stale rule—or gets one wrong—please share a redacted example in GitHub Issues.

If it saves you one risky cleanup pass, you can star the repository. If not, a concrete counterexample is more useful than a star.

Links

Top comments (0)