In August we wrote a careful instructions file for the coding agents that work in our repository. How to run the checks, what is unsafe to run locally, which database the dev server really talks to, how to name commits. Hard-won knowledge, written down once so nobody had to learn it twice.
For a month, no agent read a single line of it.
How instructions silently disappear
The agent tool we use reads one file automatically at the root of the repository. Anything else only loads if that file imports it with a single line. Our root file did not have that line.
Nothing failed. No warning, no error, no missing-file message. The agents simply did their work without the rules, and because they are capable, most of that work looked fine. The damage was the kind you only notice later: a check skipped here, a convention ignored there.
We found it by asking an obvious question very late: "what exactly is loaded when a session starts?"
The fix was one line. The guard is a test.
Adding the import took ten seconds. The interesting part is what we did next. We wrote a test that fails if that import line ever disappears again. It runs with the rest of the suite.
The general lesson: if a piece of configuration matters and fails silently, it needs a test, not a reminder. We now treat agent instructions like any other config. They are code that can break, so something has to notice when they break.
Errors are read by agents now
Writing that test changed how we write every check. A message like expected false to be true is useless to a human and worse for an agent, which will happily guess at a fix.
Our rule is that a failing check prints the file name and the fix. For example, a docs test fails with the path of the dead link and the file it was found in. A config test fails with the name of the CI job that drifted and the command it should run. When the reader of your error message is another program, the message is the interface.
One command, one exit code
The same principle shaped our quality gate. There is a single command that runs types, formatting, lint and the full test suite. It runs every gate even when an earlier one fails, so one run tells you everything that is wrong, and it finishes by printing the command that re-runs just the broken part.
Two details turned out to matter more than expected:
- It never stops early. An agent that fixes the type error, re-runs, then discovers a lint error, then a test failure, burns three cycles on what one run could have reported.- The exit code is trustworthy. If you pipe the output into something else, the pipeline returns the last command's status, not the gate's. We wrote that trap into the instructions file, which is now actually loaded.## Things that fail without failing The missing import was not our only silent failure. Two others from the same period:
- A CI workflow file that was valid YAML but referenced a job we had deleted. GitHub rejected the whole file at startup, reported a failure with zero jobs and no log, and it stayed red through two more merges. We now run a workflow linter that catches exactly this.- A nightly test run that was red sixteen nights in a row while nobody acted on it. We removed it. A gate everyone ignores teaches the team that red is normal, which is worse than having no gate.The pattern in all three: the system was quiet when it should have been loud. Most of the work was not fixing the bug. It was making sure the next one makes noise. ## If you use coding agents Check what actually loads at session start. Not what you think loads. Then write a test for the part you cannot afford to lose.
Top comments (0)