The AGENTS.md gate: a config file you can't tell is being read
Przemek at szypowi.cz ran the kind of test I wish more people ran before trusting an AI coding agent with their rules: he put a canary word in a project's AGENTS.md, pointed Claude Code at it, and asked for the word back. With telemetry enabled the file loaded. With DISABLE_TELEMETRY=1 or CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 set, it never loaded. No warning, no error, no hint in the session. The model just answered without the project instructions.
The loader ships as a built-in plugin gated behind a remote feature flag called tengu_agents_md_mod, with the off-state as the fallback. When the flag can't be fetched, the local file is skipped. Reading a markdown file from your own working directory needs no network, but in 2.1.277 it waited on a server-side switch.
The fix that actually works is a bit absurd: a one-line CLAUDE.md containing @AGENTS.md. That import path doesn't depend on the flag, so it loads with telemetry off. It costs exactly the extra file the AGENTS.md support was meant to remove.
Why this matters for AI code review
This is not a Claude Code bug report for its own sake. It's the cleanest illustration I've seen of a real problem: the rules file that's supposed to define how the agent behaves is loaded behind a gate you can't observe. And AI code review shipped on each vendor's own rules support has exactly the same failure mode.
When you ask whether an AI code reviewer follows your team's coding standards, the first question is not "what's in the file." It's "is the file even reaching the model?" A reviewer can claim support for custom standards, pass every marketing screenshot, and still silently review against a generic default because the config load is gated behind a flag, a cached rollout, or a setting your team flipped off for other reasons.
The most dangerous part of the szypowi.cz finding is the silence. People who keep a shared instruction file for several agents usually care a lot about what each tool sends home, so they disable telemetry. On Bedrock, Vertex, or a gateway, nonessential traffic is often off by policy. Those are exactly the teams using AGENTS.md in the first place. They all get a feature that is announced as available and then does nothing, and the symptom looks like "the model ignores my instructions." So they rewrite prompts, tune their wording, and blame the model, when the file never reached it at all.
The verification habit
The fix here is a general one, not a Claude Code patch. Whenever a tool's behavior is supposed to be driven by a config file, test that the file actually loads, not that the feature page exists. A canary word is cheap and reproducible: put a unique token in the rules file, ask the agent a question only that token can answer, and confirm it comes back. Do this:
- in a clean, empty directory so nothing else could leak the answer in
- with your production environment variables set (telemetry off, proxy set, whatever your team runs)
- twice, because a flag-gated loader may fetch on the first session and use on the second, which is what the szypowi.cz test had to handle
That last point is worth its own mention: a two-session fetch-then-use pattern means a single green test can still be masking the failure if you only ran once.
This is the same class of problem as a benchmark that can't be reproduced. A self-reported score you can't verify is not evidence, and neither is a feature that loads only under conditions you don't control and can't see. The canary test turns "does it load?" from an assumption into a measured result.
Putting rule-loading on the eval checklist
When you compare AI code reviewers, config loading is rarely on the checklist. The typical comparison looks at coverage, model, pricing, and how many rules it claims to support. What it skips is whether the rules are actually applied to the review. Add these steps:
Ask the vendor how rules reach the model. There's a real difference between a static rule engine, a config file the reviewer reads locally, and a server-side rules service behind a flag. The first two are inspectable. The third can silently fail the way the telemetry gate does.
Run a canary test of your own. Put a rule that would change an actual review decision into the config, submit a PR that violates it, and check the reviewer catches it. If it doesn't, trace whether the config loaded at all before you blame the model.
Check that the rules work in your network. If you're on a gateway, a cloud model endpoint, or a self-hosted setup, the more moving parts between the config and the model, the more ways it can silently not load.
That last question is the one I'd lead with in any vendor conversation, because it directly mirrors the szypowi.cz finding: rules support that depends on a remote flag is rules support you can't trust in the environments where you need it most.
What to do right now
If you run Claude Code with telemetry off and you have an AGENTS.md, add the @AGENTS.md import to your CLAUDE.md or a global user CLAUDE.md, then run the canary test to confirm it loads. That's the workaround this post reproduces, and it costs you nothing but one line.
More broadly, treat every agent's rules-support claim the same way you'd treat a benchmark claim: reproduce it in your own environment before you rely on it. A config file that silently doesn't load is the same family of problem as a benchmark result that can't be reproduced. On my blog I've argued that an AI reviewer judging its own output is a blind spot and that rule-loading is a separate question from generic bug-catching. The telemetry gate is the missing concrete proof for both: you can't verify review quality against rules you never confirmed were loaded.
The rule I keep coming back to is the same one that drives most of what I test: the only result you can trust is the one you reproduced. The szypowi.cz canary test is a five-minute version of that, and it caught a feature silently not working. Run it before you bet your review standards on an agent.
Top comments (0)