AI-generated code rarely fails loudly. The failure that scares me is the quiet one: a build report that says a security control is "Applied" when the code behind it was never written.
I hit this on a real build. "PII encryption at rest" was marked done in three places: a schema comment, an architecture invariant, and a compliance sheet. Three documents, all agreeing with each other.
Then I grepped for createCipheriv. Zero hits. The encryption key referenced in the docs was in no environment schema. The address columns were plaintext.
Every automated signal was green. The invariant that should have verified the control was marked "manual," so the linter skipped it. The tests passed because they only test what exists; nothing asserts the absence of a thing that was never written. The compliance sheet passed because it echoes the spec instead of auditing the code. Greenness was self-referential.
Once you look for this pattern you find weirder versions. Another build's invariant linter reported "0 violations" while inspecting zero files: the model had re-derived a glob-to-regex translation and applied the wildcard replacements in the wrong order, so the pattern matched nothing. A clean pass over an empty set. Across four builds I watched, the model's first attempt at that same translation was wrong in three, each differently.
What finally caught the missing encryption was not more tests. It was a fresh-context reviewer agent that gets the spec and the working tree, and deliberately not the builder's notes or report. Its job is to disbelieve: grep for every claimed control, file and line. It flagged the encryption on the first cycle. The remediation added real AES-256-GCM plus an integration test that reads the row straight from the database and asserts the stored value is ciphertext, so the control can never silently evaporate again.
The question to ask an AI build is not "did it pass." It is "what proved it passed." Would a deleted control actually fail something?
I wrote up the full story, with the other failure variants, on the MDLC blog. Disclosure: I build MDLC, the workflow these gates come from.
Top comments (1)
The raw-row assertion catches plaintext, but by itself it only proves that the stored bytes changed. For AES-GCM I’d add a successful decrypt round-trip and a negative test that flips one ciphertext byte and requires authentication to fail. That verifies the authentication path, not just that storage no longer contains the original plaintext.