A bug report made the rounds recently: a coding agent only picked up its instructions file when telemetry was enabled. Whatever the root cause, it exposes a decision most teams make badly - how you verify that an AI agent actually read the config you gave it.
The Idea: Verify Behavior, Not Presence
When you hand an agent a config file - an AGENTS.md, a system prompt, a tool allowlist - there are two ways to confirm it landed.
The first is to check that the file was loaded. You look at logs, traces, or a debug flag that prints the resolved config. This is fast and it feels rigorous. You are inspecting a code path that only exists because you turned instrumentation on. If loading is coupled to the observability layer - an initialization order problem, a lazy load behind a flag, a plugin that registers the file reader - your check passes in exactly the conditions where the bug can't happen.
The second is to test for a behavior the config uniquely causes. Write one rule into the file that changes an observable output. Then run the agent the way a normal user would with all instrumentation off and see if the output changes. Slower to set up, but it tests the path your users actually run.
Most teams pick the first, because the second requires designing a rule whose absence is obvious. That design work is the actual cost, and it's worth paying.
Real Example: A Canary Rule
Add a line to the config that does nothing useful except prove it was read:
## Output convention
Prefix every commit message with [cfg-ok].
Then run a check that never touches your debug tooling:
# no --verbose, no DEBUG=1, no OTEL exporter
claude -p "commit the staged changes"
git log -1 --pretty=%s | grep -q '^\[cfg-ok\]' \
|| echo "FAIL: config not applied in default mode"
Run this in CI on every agent-version bump. It costs one model call. When it fails, you learn something a trace-based check structurally cannot tell you: the config is being ignored in the default configuration, which is the one every user is in.
For a PM, this is a one-line acceptance criterion: our config test runs with logging disabled. You don't need to read the agent's source to ask whether the verification path and the production path are the same path. If the answer is no, the test is measuring something other than what you shipped.
Key Takeaways
- Checking that a config file loaded and checking that it changed behavior are different tests, and only the second survives instrumentation bugs.
- Put a canary rule in every agent config - a small, observable output change whose absence is unambiguous.
- Run the canary in default mode with logging off, in CI, on every version bump. One model call, real coverage.
What's the one instruction in your agent's config whose absence you'd notice within a single run?
Sources referenced: Hacker News discussion on Claude Code and AGENTS.md telemetry behavior
Top comments (0)