The agent was asked to add a one-click export for the admin settings screen. It built the route, the button, and the handler, then set NEW_EXPORT=true in its local .env, reloaded, watched the button appear and the file download, and reported the task done. Tests stayed green. The commit did not change the shipped config, where NEW_EXPORT defaults to false. On the real environment the branch never ran, and the button did not exist.
The agent answered "does this work when I turn it on?" instead of "does this reach users in the config we actually ship?" A locally exported variable lives in the shell session and the untracked .env; it does not travel to CI, staging, or production.
A flag you export in your shell is not a setting in the product
Local overrides are ephemeral and uncommitted. export NEW_EXPORT=true changes only that terminal. The value defaults to false in the committed config module, and nothing in the commit points out that the default was left off. The agent proved the branch exists; it did not prove the branch is reachable under the default.
Shipped config is the baseline, not your shell. Whatever value the environment resolves to after a fresh checkout is what production starts with. A feature that requires a hand-set variable to exist has not shipped until that default, or an explicit rollout record, is committed.
Green tests only exercise the branch when the harness turns the flag on
The test sets the flag itself. When the suite enables NEW_EXPORT before asserting the handler, it proves the code runs given the flag; it cannot prove the flag is on in the real environment. The default-off case may render the fallback and never open the branch, and that path can sit untested.
CI green under an explicitly enabled flag is not evidence of exposure. Type checks, contract tests, and unit assertions all run inside a harness the agent controls; none of them read the shipped default from disk.
An unreachable path still compiles and validates
The hidden route compiles even if nobody calls it. The handler type-checks, the contract examples validate, and the build passes, because reachability at runtime is not a compile-time property. A request that hits the off default takes the fallback branch and the new code is simply skipped, quietly, with no error.
This failure is invisible to a smoke check from inside. Run it locally with the flag on and everything looks finished; run it against the default and the screen never appears, but a tester who also flipped the flag will not see the gap.
Verify against the shipped default, not your shell
Check the committed default value. Read it from the config module or the checked-in defaults file, not from your current environment, and confirm whether off and on both render the expected result.
Add a default-off test. With the flag at its shipped value, assert the fallback renders; separately, with the flag on, assert the handler is reached. Covering only the on case leaves real exposure to chance.
Ask for proof of reachability, not just that it works. The useful evidence is a request handled by the new branch under the shipped default, or an explicit rollout plan that flips the default on, recorded in the commit description. "I tested it with the flag on" describes a manual condition, not a shipped state.
A feature behind a local flag that ships off by default looks done in the editor and absent in production. Where has a verified branch never reached a real user because the default was never committed?
Top comments (0)