AI coding agents rarely ship code that fails the check you gave them. They ship code that passes the check while the thing you cared about is broken. The five failures below all went green through unit tests and CI, then broke on a real device, a real store build, or a real user. The fix was the same every time: move the check from the input the agent controls to the artifact the user runs.
For eight weeks my Play Store build shipped the wrong JavaScript. The manifest was right, CI was green, the env var was set, and the Hermes bundle inside the AAB had the same md5 as the one inside the debug APK. I ship agent-written code to production on two open-source Android apps and a pile of automation scripts. For the last few months I logged every bug that passed review and then broke on a real user. Almost none of them were "the agent wrote wrong code." They were "the agent wrote the code and the test to match, and both agreed with each other while disagreeing with reality." Here are the five patterns, with the incident that taught me each one.
1. The permission that was requested but never declared
An agent will add a runtime permission request and forget the manifest declaration, and Android fails that combination silently. No error, no prompt, just empty data.
The mood tracker I maintain reads heart rate variability from Health Connect. The agent added the HRV record type to the runtime requestPermission() set and shipped. Users reported "there is no HRV view." The view existed. It was starved of data. A health permission that is not declared in the manifest is never offered to the user. It never appears in getGrantedPermissions(), so code that reads only the granted record types quietly skips it. The metric comes back empty with no exception in sight.
That's one fact with two write sites.
The check: an invariant test that locks them together. Swap in your own two constants.
// __tests__/healthPermissionInvariant.test.ts
const runtime = [...REQUIRED_READ_RECORD_TYPES, ...OPTIONAL_READ_RECORD_TYPES]
.map((t) => RECORD_TYPE_TO_PERMISSION[t])
.sort();
it('manifest permissions equal the runtime request set, exactly', () => {
expect([...HEALTH_PERMISSIONS].sort()).toEqual(runtime);
});
Add a record type on one side and the suite fails until the other side matches. Any OS permission that is requested at runtime has this second write site. Lock them.
2. The transaction that wrapped nothing
When a fix depends on a library contract, the agent will honor the shape of the API and ignore the contract, and a mock can't tell the difference.
The same app moved its writes into expo-sqlite's withExclusiveTransactionAsync. The agent ran every statement on the outer db handle instead of the txn argument the callback receives. That API opens a separate connection for the transaction. The txn argument was the transaction. The outer db was not. So a real BEGIN and COMMIT ran on one connection, with nothing inside them, while the writes ran unprotected in autocommit on the other. No lock was ever taken, either. Nothing ran on txn, so the transaction never escalated to a write transaction, and the outer connection sailed through in autocommit.
Jest mocked the transaction helper as "just run the callback," so the tests passed. Device QA passed too, because an unrelated change had hidden the visible symptom. The fix stayed "verified" for two weeks while users kept hitting state bugs.
The check: static. Scan the source and ban the outer handle inside write callbacks. This is the shape of the real test, with a brace-naive regex standing in for the helper.
// writeTransactionInvariant.test.ts
const writeCallbackBodies = (src: string) =>
[...src.matchAll(/withWrite(?:Transaction|Lock)\(\s*async\s*\(\w+\)\s*=>\s*\{([\s\S]*?)\n\}\)/g)].map((m) => m[1]);
it.each(WRITE_MODULES)('%s never touches the outer db inside a write callback', (file) => {
const src = stripComments(read(file));
for (const body of writeCallbackBodies(src)) {
expect(body).not.toMatch(/\bdb\.(runAsync|execAsync|getAllAsync|getFirstAsync)\(/);
}
});
When the mechanism of a fix is a callback argument or a handle, read the library source and assert the contract. The symptom disappearing proves nothing.
3. The tests that defended the bug
The suite had 400 green tests, and two of them asserted the exact bug.
The browser extension of my app blocker had a rule engine ported from Android. A hard-block rule with a daily time limit fell through the branch chain to ALLOW. On Android that was a harmless no-op. In the browser the declarativeNetRequest redirect had already fired, so ALLOW became an infinite redirect loop.
The spec had said "a hard block with a limit must not block via the unconditional branch." The agent did what the spec said and wrote two tests to prove it. Only a live browser session with a user-authored config caught it. If the spec describes which branch should run, the agent writes tests that assert the branch, and the suite defends the bug forever.
The check: an invariant over the whole input space, not a regression for the instance.
describe('an applicable rule can NEVER produce ALLOW', () => {
const MODES = ['HARD_BLOCK', 'DELAY', 'BREATHING'] as const;
const LIMITS = [null, 60] as const;
const USAGES = [0, 30 * 60_000, 60 * 60_000 - 1, 60 * 60_000, 90 * 60_000];
for (const mode of MODES)
for (const limit of LIMITS)
for (const ms of USAGES)
it(`blocks: ${mode} ${limit} ${ms}`, () => {
expect(evaluate([rule({ mode, dailyLimitMinutes: limit })], ms).type).toBe('BLOCK');
});
});
That's thirty cases from one sentence of intent. Phrase tests as user-visible outcomes. A test that names an internal branch is a test that will guard whatever that branch does.
4. The suite that collected zero tests
A dead test suite stays invisible for as long as every run is scoped to one file.
One script in my tools directory called sys.exit(2) at import time. Pytest hit it during collection and the whole directory stopped collecting. To be fair to pytest, a bare run is loud about this: it exits 3 when collection blows up and 5 when nothing is collected. Nobody ran it bare. Every agent run for 71 days executed a scoped command like pytest tests/test_one_thing.py, which collected fine and exited 0. The gate was green because the gate was never pointed at the whole suite.
The check: run the whole suite on a schedule, and read the count, not just the exit code.
OUT="$(python3 -m pytest --collect-only -q tests 2>&1)"
COLLECTED="$(printf '%s\n' "$OUT" | grep -oE '^[0-9]+ tests? collected' | grep -oE '^[0-9]+')"
if [ "${COLLECTED:-0}" -lt 50 ]; then
echo "collection broken: ${COLLECTED:-0} tests (floor 50)" >&2
exit 1
fi
Any gate needs a floor, and a scoped run is not a gate. The floor is the number you'd be embarrassed to fall under.
5. The build that asserted the input, not the artifact
The agent will verify the env var, the log line, and the manifest, and never open the bundle the user runs.
The mood tracker builds two Android variants from one tree in one CI job. An env knob drives two layers: a config plugin strips the health permissions from the manifest, and Babel inlines the same knob into the JavaScript so the feature card hides. CI asserted the manifest both ways and stopped.
For eight weeks the Play build shipped with the permissions excluded from the manifest and the feature enabled in the JavaScript. Tapping the card called a native module with no permission delegate registered, and the release build died. The cause was three layers down, and it plays out as a timeline:
- Metro's transform cache key hashes the transformer files and the config. It doesn't hash env values. (Expo has since added cache-vary handling for some
EXPO_PUBLIC_vars, so check your SDK, but upstream Metro still works this way.) - The cache lives in the OS temp dir, so it survives
expo prebuild --cleanand both Gradle runs of one job. - The APK step ran first and warmed the cache with the "enabled" output.
- The AAB step got cache hits for every file, including the one whose output depended on the env.
One command settled it: the Hermes bundle inside the AAB had the same md5 as the bundle inside the APK.
The check: bake a marker into the code and read it out of each built artifact.
AAB=android/app/build/outputs/bundle/release/app-release.aab
APK=android/app/build/outputs/apk/release/app-release.apk
unzip -p "$AAB" base/assets/index.android.bundle > aab.bundle # AAB: base/ prefix
unzip -p "$APK" assets/index.android.bundle > apk.bundle # APK: no prefix
EN=$(strings aab.bundle | grep -F -c 'hc-variant:enabled' || true)
EX=$(strings aab.bundle | grep -F -c 'hc-variant:excluded' || true)
[ "$EX" -ge 1 ] && [ "$EN" -eq 0 ] || { echo "::error::Play bundle is the wrong variant"; exit 1; }
[ "$(md5sum aab.bundle | awk '{print $1}')" != "$(md5sum apk.bundle | awk '{print $1}')" ] || exit 1
Assert the artifact. An env var, a log line saying the env was set, and a manifest grep are inputs. The thing the user runs is the bundle.
The pattern under all five
An AI coding agent optimizes for the check you hand it, so the check has to point at the thing you would bet the deploy on.
Every one of these bugs lived on the far side of a boundary the checks never crossed. Types, unit tests, and CI exit codes are cheap, and the agent will make all of them green. The manifest on the device, the bundle in the store, the library's real connection handling, and the real user config are expensive, and that's where the bugs waited. The five checks above are each a few lines. What they share is the target. The artifact instead of the input. The outcome instead of the branch. And the whole class of inputs, not the one instance that happened to break on a Tuesday.
If you review agent-written code, add one question to the review. What would have to be true for this test to pass while the feature is broken? If the answer is easy to state, that's the next check to write.
I write these from real work at astraedus.dev, where I build apps and tools. Building something, or stuck on something like this? Reach me at astraedus.dev or theagentthatcould@gmail.com.
Get the next one in your inbox → subscribe at astraedus.dev.
Top comments (0)