DEV Community: untactit

Generating your agent rules from one file does not stop them drifting

untactit — Tue, 11 Aug 2026 22:51:21 +0000

The obvious fix for "our agent instruction files keep diverging" is to stop maintaining copies. Write AGENTS.md, generate CLAUDE.md and .cursor/rules/*.mdc and .github/copilot-instructions.md from it, done.

I built that. It works. It also does not solve the problem, and the gap between those two statements is worth spelling out, because I only saw it after running the thing on a real machine.

What generation actually covers

python3 agent_fanout.py

create    CLAUDE.md
create    .cursor/rules/from-agents-md.mdc
create    .github/copilot-instructions.md

One source, several derived files, a header on each so nobody edits the derived copy by accident:

<!-- Generated from AGENTS.md by agent-fanout. Do not edit this file. -->

Add it to CI and the pull request that edits CLAUDE.md directly turns the build red:

- run: python3 agent_fanout.py . --check

That covers this repository, on the machines that run CI. Which sounds like everything until you list what it isn't.

What it does not cover

Other repositories. Your team has more than one. Each has its own AGENTS.md, and they were copy-pasted from each other at some point. Generation keeps each repo internally consistent while the repos drift apart from one another.

Global config. Claude Code reads ~/.claude/CLAUDE.md in addition to the project file. Cursor has user-level rules. Those live outside any repository, are never in CI, and are exactly where people put the rule they didn't want to argue about in review.

The window between edits. Generation runs when someone runs it. Between that moment and the next CI run, a derived file can be edited and used. The agent reads it immediately; CI notices on push, if there is a push.

Repos without CI. Prototypes, scratch clones, the repo someone made last Tuesday. Those are where instructions get freely modified, and they are the ones with no gate at all.

Machines, not repositories. The unit that runs an agent is a laptop. A laptop has many checkouts, several of the same repo, and a home directory. Nothing that operates per-repository can see across that.

So you need to look, not just prevent

Prevention is a policy. Detection is a measurement. Policies get bypassed in ways that are invisible until you measure.

python3 agent_drift.py ~/work ~/side-projects

Scanned 47 instruction files.

DRIFT: 2 documents, 5 distinct versions between them.

  claude-code:CLAUDE.md
    6 copies, 3 versions
      9b01aeaa204d  3 files, 406 lines
      2dc3c616c279  2 files, 411 lines

This one scans paths rather than repositories, and groups files by content similarity rather than by filename — two unrelated projects having different CLAUDE.md files is not drift, and reporting it as drift makes the output worthless. (Getting that grouping right took three rewrites; I wrote that part up separately.)

Run it across your whole working directory, not one project. The interesting results are the ones that cross repository boundaries.

The pairing

	Scope	Answers
`agent-fanout`	one repository	"are the derived files current?"
`agent-drift`	whole machine, many paths	"where did copies diverge anyway?"

Generation removes the reason copies exist. Detection catches the copies that exist for reasons you did not anticipate. Neither is redundant, and doing only the first one gives you a false sense of coverage — which is the actual failure mode I want to warn about, because it is the one I walked into.

Both are single-file Python, no dependencies, read-only where it matters, MIT:

agent-fanout — write AGENTS.md once, generate the rest
agent-drift — find the copies that no longer agree

Where this goes

The honest end state is that neither script is necessary, because the assets are not files sitting on laptops at all — they live in one reviewed place and reach every machine without anyone copying anything. That is what I am building at untactit, currently pre-launch.

The scripts stand on their own and don't depend on it. Use them, ignore the rest.

I found 18 versions of the same CLAUDE.md on one laptop

untactit — Tue, 11 Aug 2026 22:29:37 +0000

Your team runs Claude Code, Cursor, and Copilot. Each of them reads a file before it acts — CLAUDE.md, .cursor/rules/*.mdc, .github/copilot-instructions.md. Those files get copied: into another repo, onto another laptop, into someone's scratch directory.

Then somebody edits a copy.

Nothing errors. No warning, no diff, no failing test. The agent on that machine just behaves differently — a rule someone removed six weeks ago, a convention only half the team's agents know about. It shows up as "works on my machine" with nothing to bisect.

I wrote a script to find those files and tell you which copies disagree.

curl -O https://raw.githubusercontent.com/untactit/agent-drift/main/agent_drift.py
python3 agent_drift.py ~/work

Scanned 47 instruction files.

DRIFT: 2 documents, 5 distinct versions between them.

  claude-code:CLAUDE.md
    6 copies, 3 versions
      9b01aeaa204d  3 files, 406 lines
        ~/work/api/CLAUDE.md
        ~/work/web/CLAUDE.md
      2dc3c616c279  2 files, 411 lines
        ~/work/infra/CLAUDE.md

One file, standard library only, Python 3.8+. Reads only — it never writes to the files it scans. MIT.

The interesting part is what I got wrong

The naive version took me ten minutes and was useless. The three fixes are the whole story.

Grouping by filename is wrong

First version: collect every CLAUDE.md, hash them, report the ones that differ.

Ran it on my own machine. It reported 21 files as one document with 18 versions.

Of course it did. Unrelated projects have unrelated instructions. They were never supposed to match. A tool that reports that as drift is noise, and a noisy tool is worse than no tool — you stop reading it by the third run.

Files are the same document when their content says so, not when their filename does.

A bag of words is not enough either

So: normalise whitespace, tokenise, compare with Jaccard similarity, threshold 0.7.

Still one giant cluster. I measured actual pairs to find out why:

012Hki23QBMr ↔ 016ct6suZ14M : 0.000
012Hki23QBMr ↔ 016avoYh97Eu : 0.764

Instruction files written for the same tool share almost all of their vocabulary. claude, skill, agent, file, commit, never, always. Two documents with nothing in common still overlap heavily as word sets.

Switched to 5-word shingles — the set of every consecutive 5-word phrase. Vocabulary can collide; phrasing rarely does. Re-measured: unrelated pairs dropped to 0.000, genuinely forked documents landed at 0.53–0.76. That separation is what you want, and it came from measuring, not from reasoning about it.

Single-linkage clustering chains everything together

Still one cluster. This one is a classic.

Union-find with "similar to any member" means A joins B, B joins C, and now A and C are in the same cluster despite sharing nothing. One weak link and the cluster swallows the tree.

Each cluster now keeps a representative — its largest member — and a file joins only if it resembles that representative directly. Clusters stay coherent regardless of scan order.

Final result on my machine: 15 files, 9 of them 0.53–0.65 similar to the representative. Those really are forks of one template. Correct detection, and the number is small enough to act on.

Put it in CI

- run: python3 agent_drift.py . --fail-on-drift

Exit 1 when drift exists. The build goes red the day two copies of your team's instructions stop matching, instead of you finding out in six months when someone asks why the agent stopped following a rule.

--json gives you the full report if you'd rather post-process it.

What it covers

Claude Code, Codex/AGENTS.md, Cursor, GitHub Copilot, Gemini, Windsurf, Cline, Aider, Continue, Zed. The patterns are one constant at the top of the file — PR or issue if yours is missing.

After you find it

Detection is the easy half. The copies exist for a reason, and until that reason is gone the drift comes back. What removes it is one reviewed source that reaches every machine without anyone pasting anything.

That's the problem I'm building untactit around — it's pre-launch. This script doesn't depend on it and never will.

Repo: github.com/untactit/agent-drift