I spent a long time trying to make AI coding agents follow project rules through instruction files: AGENTS.md, CLAUDE.md, a rules folder. For a handful of basic instructions it works. Put your whole development "bible" in there and it falls apart.
The pattern is probably familiar. The agent breaks a rule, apologizes, promises it won't happen again, and breaks it again two changes later. It isn't being lazy. A rule in the prompt is a wish: it competes with everything else in the context, and the longer the session, the weaker it gets.
What actually worked
While building Semitexa, an open-source PHP framework designed for agent-driven development, I moved the rules out of the prompt and into a command. The only instruction the agent gets about rules is this:
Check every code change you make with
ai:verify. If it lets you pass, move forward.
ai:verify looks at what changed and runs the lint and the tests that matter for that diff, plus the structural checks: where logic is allowed to live, what may call what. When it fails, the agent gets a concrete message about the thing it just did.
Three things changed:
- The rules stopped competing for context. Adding a constraint no longer makes the prompt longer, so I can have as many as the project needs.
- Failures became specific. "Handler returns a raw Response" is something an agent can fix in one step. "Follow the architecture" is not.
- The agent got better over the session. The feedback arrives right after the mistake, not at review time, and more and more often the next change passes on the first try.
What the instruction file is still for
I didn't delete AGENTS.md. It got short. Now it's a map: where things live, which commands to run, and the one rule above. Anything that can be checked moved into the check. Anything that can't be checked (naming taste, whether an abstraction pulls its weight, whether the change matches the intent of the ticket) stays with human review.
The trade-off
A check costs more to write than a sentence. Some rules are genuinely fuzzy and never become checks. But every time I noticed myself writing the same review comment twice, it turned out the rule was never fuzzy, just unwritten. Those are the ones worth turning into code first.
Your turn
Where do your project rules live today: in instruction files, in CI, or in reviewers' heads? And has anyone found a good way to measure how often an agent actually follows an instruction file?
Top comments (2)
"A rule in the prompt is a wish" is the line, and the mechanism under it is that the rule had no way to fail. Moving it into
ai:verifydid not make the agent more obedient. It gave the rule a failure mode, and a rule that cannot fail is not a rule regardless of who is reading it.The thing I would fix before this scales is the instruction itself: "if it lets you pass, move forward." That makes
ai:verifythe single point everything now depends on, and it is the one component nothing checks. The failure is not the agent ignoring it. It is a run where the structural checks silently matched nothing — a renamed directory, a glob that no longer resolves, a diff-scoping rule that reads a change as "no relevant tests" — and the command exits zero. The agent is now following your rule correctly and the rules are not running. That is the version of your idea that fails the way AGENTS.md failed, just more quietly.Two cheap fixes, both of which you probably want anyway. Make
ai:verifyfail when it selected nothing to run, rather than pass: a diff that matches no check is a coverage gap, not a clean bill. And make the output state what it ran — "ai:verify 0.4.1, 3 structural checks, 12 tests, 1 file unmatched". Then a pass carries its own coverage, and an agent quoting that pass back at you in a PR is quoting something falsifiable.On the trade-off section: "every time I noticed myself writing the same review comment twice, it turned out the rule was never fuzzy, just unwritten" is the most reusable sentence in the post, and it is doing more work than the tooling is. Twice is a good threshold because it is cheap to detect and hard to argue with.
You found the weak spot I was most worried about. Half of it bit us already: an empty run used to read as a pass. Now ai:verify gives it its own verdict,
nothing_to_verify, withcompleted: false, so it can't be quoted as evidence, and calling it with no scope at all fails outright. The half that's still open is the one you describe: a changed file that no check claims. A pass today doesn't call that out, and it should, so it's going into the backlog in your framing: unmatched files listed in the report, and a change no check covered treated as a gap, not a clean bill. The one-line summary of what ran is a good call too. The JSON already carries counts per status, but the line that gets pasted into a PR is the human one. And yes, "twice" has turned out to be the right threshold for me as well.