I had a podcast pipeline I was proud of. It took a transcript, turned it into a two-person conversation with text-to-speech, laid in the music, and produced an MP3 I could publish. I'd built it in one app, and it worked. I loved the output.
So when I started a second app that needed the same flow, I didn't want to rebuild the pipeline. I already had one. I just wanted it over there. So I asked the AI to copy it. And it did.
Here's the part that matters: I didn't just copy it and hope. I checked. I opened a fresh session (a clean one, no memory of the first) and told it to look at the new pipeline and make sure everything was right. It went and looked. It came back and told me everything was good.
Everything looked good. Or so I was told.
Then I loaded the first real transcript and ran it.
It was wrong. Not a little wrong. The voices were wrong. The music didn't come in when it was supposed to. It didn't cut off when it was supposed to. It didn't fade. It just stopped. The words were all there, every one of them, in the right order. But everything that made the first pipeline good (the timing, the production, the feel) was gone.
I walked away from my desk for a bit. It pissed me off, because I'd done what I was supposed to do. I'd asked. It had answered. The check was green. And the check was a lie.
Here's what I think I actually got wrong, and it's not "I trusted the AI." It's subtler than that.
When I asked a fresh session to "make sure everything's good," I got back a confident yes. But the session had no way of knowing what good sounded like. It never heard the first pipeline. It had no stake in whether the podcast was any good. It reported what it could see (the code looked reasonable) and what it could see was almost never the thing I actually cared about.
That's the trap, and it isn't a beginner's trap. I have a whole process built to avoid exactly this: spec, adversarial review, a plan, a build, a code review. And I skipped it, on a task I decided was too small to bother with. The machine will always tell you it's fine. That's not the machine lying; it's the nature of the thing. It has no stake and no memory, and "verified" from something with no stake and no memory is a hypothesis, not a fact.
I pressed play. That's how I found out.
The uncomfortable part is that most of the signals we trust are exactly this. A passing test is a stand-in for correct code. A merged PR is a stand-in for working software. A closed issue is a stand-in for a capability that exists. The agent's "done" is a stand-in for done. Every one of them can be perfectly, beautifully green while the thing underneath is broken. When a machine writes the code, that gap becomes the most dangerous place in your workflow, because the machine will make its proxies green whether the work is right or not.
So the question I keep coming back to: how do you actually know it works? Not "did it run." Not "did the tests pass." Not "did the agent say it checked." How do you know?
This is the opening of a book I'm writing in the open: **Or So I Was Told: How to Verify What Your AI Coding Agent Actually Built, book 4 in my Outgrowing the Vibe Code series. It's early and cheap, and the price goes up as it fills in. Read it here: leanpub.com/orsoiwastold
And if you've hit a version of this (the confident "verified" that wasn't), tell me about it in the comments. I'm collecting them.
Top comments (0)