There's a question I keep seeing from engineering teams evaluating AI code reviewers: "Is there one that follows our team's own coding rules and standards?" It's the right instinct. Your team has standards that matter, and you want an automated reviewer checking new code against them, not against some generic best-practices list.
The problem is the question treats "our rules" as one thing. It isn't. Rules land on a spectrum of how hard they're actually enforced, and most product pages don't tell you where on that spectrum their custom-rules feature sits. So teams buy "custom rules," point it at their guidelines, and get compliance that's approximate. Then they blame the tool when it's really a mismatch between what they wanted to enforce and how the tool enforces it.
The three shapes "custom rules" actually take
When you dig into how AI reviewers let you encode rules, three distinct mechanisms show up. They differ in how deterministic the enforcement is, and that difference is the whole game.
Prose guidelines the model reads. This is the softest form. You point the reviewer at a standards document, it loads the text into context, and the model tries to follow it during review. CodeRabbit's code guidelines work this way: it auto-detects AGENTS.md, CLAUDE.md, .cursorrules, and the rest in your repo and applies them as review criteria with no configuration. Kodus does the same with plain-language review rules plus syncing rules from Cursor, Copilot, and Claude. The model reads them.
But "the model reads them" is exactly the limitation. A prose rule is interpreted by a probabilistic system. Whether a change complies is an opinion the model forms on a given run, not a verdict. The same change can get flagged one review and pass the next, which is precisely the flaky-judge problem I keep coming back to.
Scoped instructions. A step up, but still prose. These attach guidance to specific paths or globs. CodeRabbit's path instructions let you say "for src/controllers/**, focus on auth and input validation and flag direct DB queries that bypass the ORM." It's more precise because the rule applies only where it's relevant, but the enforcement is still the model's interpretation of that prose.
Structural rules. This is the deterministic end. CodeRabbit's AST-based instructions use ast-grep, which parses code with tree-sitter and matches structural patterns. A rule is YAML describing a node, a pattern, a relationship, or a composite. It either matches or it doesn't. No interpretation, no model opinion. A rule that says "no await inside a for_in_statement without stopBy" is a fact the tool checks, not a suggestion the model weighs.
That spectrum is real, and it's the thing teams should care about.
A concrete structural rule, so you can see the difference
Rather than keep this abstract, here's the shape of a deterministic rule. The whole point of the ast-grep variant is that it operates on syntax, not prose, so there's nothing to misread.
rule:
pattern: await $PROMISE
inside:
kind: for_in_statement
stopBy: end
message: Avoid awaiting inside a for-in loop; collect promises first.
The tool parses the diff with tree-sitter, finds every await expression, checks whether any sits inside a for_in_statement node, and if one does, reports it with that message. You could run that rule over the same PR a hundred times and it would flag the same thing a hundred times. The syntax is either there or it isn't.
Now write the same rule as prose: "don't await inside loops, it serializes requests, use Promise.all instead." A model reading that guideline during review usually catches it. Usually. The word "usually" is the whole problem. On a noisy diff, on a long review where context budget runs thin, or on a refactor that looks different from the textbook case, the model can miss it, and you'd never know until a reviewer or an outage catches it. And if the same PR is rerun, the verdict can change because that's what probabilistic judgment does.
That's the concrete difference you're buying when you pick a custom-rules implementation. It's not "does the tool support our rules." It's "does it know our rule or does it read our rule."
Enforcement strength is the real requirement
Here's the mental model worth taking away: a prose guideline is enforced probabilistically, a structural rule is enforced deterministically. One is "the model tried to follow our standard," the other is "the code either matches the pattern or it doesn't."
That distinction maps onto a finding from the harness-engineering paper in the software-engineering literature, which audited eleven production coding harnesses. One of its cross-cutting observations is that behavioral policy across these systems is migrating from prompt prose to configuration. The reason is exactly this: prose is where behavior degrades unpredictably, and configuration is where it becomes testable. Review tools are heading the same direction, from "we told the model our rules" toward "we encoded the rules."
So when a team asks whether there's a reviewer that follows their rules, the honest follow-up is: which of your rules are structural, and which are judgment?
A rule like "nobody calls process.env directly, it has to go through the config module" is structural. It's a tree-sitter pattern away from being a hard check. A rule like "error messages should be actionable and human" is judgment. It has to be an opinion the model forms. Trying to enforce the second with a deterministic rule will fail, and trying to enforce the first with prose will leak.
Then there's a third category that's worth naming: rules that depend on external requirements rather than syntax. Kodus explicitly handles this by pulling requirements from Jira, Linear, and Notion and checking every PR against them, so a rule like "refund changes need a reason recorded" or "amounts above the limit require approval" isn't a code pattern at all, it's a spec that lives outside the diff. That's a different enforcement problem entirely, and it's easy for a pure syntax-based reviewer to miss. If your team's rules are mostly business rules like this, you have a different requirement than someone whose rules are all code patterns.
What to test before you trust the tool
If you're evaluating reviewers on this, don't read the "custom rules" line on the pricing page. Run a test that distinguishes the mechanisms.
Take two of your real rules: one that's structural and one that's judgment. Encode both in the tool exactly as its docs suggest, then review a PR that knowingly violates the structural one and a PR that violates the judgment one. Check which actually gets flagged, and whether the same structural violation is flagged consistently across multiple runs. Deterministic rules should flag it every time. Prose rules will be noisy.
That noise is the tell. If your team's critical rule is "this specific thing is never allowed," you don't want a reviewer that flags it sometimes. You want one that treats it as a gate, the way you'd want a CI check that actually fails rather than a reviewer that mentions it in a comment and moves on. For the structurally-checkable rules, ask for the code pattern. For the judgment rules, go in knowing you're accepting model interpretation, and pair it with a human read on the sensitive paths.
The tools span this spectrum fully. CodeRabbit gives you all three tiers, from auto-detected guidelines up to ast-grep packages you can share across an org. Kodus lets you write plain-language rules, sync the ones your agents already use, and validate against external requirements. Both are legitimate answers to "follow my team's rules," but they answer it at different enforcement strengths, and both work best when you distinguish which of your rules belong in each tier.
The mistake is not choosing the weaker option for the rules that need it. The mistake is choosing without knowing which enforcement strength you got, then discovering at the first missed violation that your "rule" was a hope, not a check. Ask the spectrum question before you wire a reviewer into your review gate. Sort your rules by whether they're patterns or judgments, pick the mechanism that matches each, and test that the structural ones actually gate while you accept that the judgment ones advise.
Top comments (0)