Configured Isn’t Loaded: A 3-Step Test for AI Coding Instructions

Caio Ribeiro — Wed, 15 Jul 2026 01:09:54 +0000

A context file can exist, be synced, and have the right hash — and still never reach the model. It can also reach the model and still fail to improve the task.

That distinction matters because AI coding tools now have different loading gates for files such as AGENTS.md, CLAUDE.md, skills, rules, and MCP tool definitions.

A recent Systima measurement made the boundary concrete. Using a logging proxy, it found that Claude Code 2.1.207 ignored a 72 KB AGENTS.md, then loaded the same content after it was renamed CLAUDE.md, adding more than 20k input tokens per request. The study also found a useful counterexample: lower startup context did not automatically mean lower whole-task cost, because request count and batching changed the result.

So “the file exists” and “the prompt is smaller” are both incomplete success criteria.

The three boundaries

For any instruction source, ask three separate questions:

Configured: Is the intended source present at the path the tool is supposed to read?
Loaded: Is there runtime acknowledgement, an attached-source label, or provider-boundary evidence that the destination actually received it?
Useful: Did a fixed task outcome improve across repeated runs?

A hash proves source identity. It does not prove delivery. A delivery trace proves observation. It does not prove adherence or usefulness.

A small A/B test

Use one harmless, task-specific instruction whose effect is observable — for example: “Before editing, list exactly two target files.”

Then:

Pin the tool version, model, repository state, task, and fresh-session condition.
Run once with the instruction under a filename/path the target does not natively load.
Run again with the same bytes under the target's native source.
Record the source hash, configured path, observed load evidence, request count/input tokens when available, and the task result.
Repeat enough times to separate a loader failure from stochastic non-adherence.

Keep a third state: loaded_but_outcome_failed. Otherwise a successful file-discovery check can hide a behavior failure.

If the source is observably loaded but the outcome still does not improve, stop expanding the prompt. Move hard constraints to a deterministic hook, test, or policy gate.

A browser-only fixture

I added this exact comparison to Pluribus' 30-second context sync demo. It uses the same source hash in two synthetic lanes:

AGENTS.md → configured_but_ignored
CLAUDE.md → loaded_and_task_outcome_accepted

The fixture is intentionally marked synthetic: it demonstrates the measurement shape; it does not claim to capture your runtime.

The open question is more useful than another universal rules-file recommendation: what is the smallest piece of evidence your tool exposes that distinguishes configured from actually loaded?

If you run the A/B, share the tool/version, source label, and whether the task outcome changed. A counterexample is more valuable than a clean demo.

DEV Community: Caio Ribeiro

Configured Isn’t Loaded: A 3-Step Test for AI Coding Instructions

The three boundaries

A small A/B test

A browser-only fixture