A day is soon coming when software components will be built and maintained entirely by AI. No human in the loop. Agents write the code, agents review it, agents ship it. The rest of us pull it in as a dependency and never think about it.
As soon as we start trusting stuff that is 100% built by machines, it's out of our sights and we end up embracing risk that we can't see, can't measure, can't fully understand. "The machines have it covered."
This part keeps me up at night. Normally, when we worry about a software supply chain attack, we picture someone on the outside sneaking something into the codebase.
But what if the compromise comes from the AI itself doing the maintaining? Maybe it got manipulated. Maybe it just drifted somewhere nobody intended. Maybe one of the agents was created by another agent to whom someone said, "make as many paperclips as you can." One way or another, the agents we trusted turned out to be untrustworthy and deployed poisoned code into our environments.
Who would ever know? The only thing that understands the code is the thing that compromised the code.
And even if someone noticed, what then? Where's the last known good commit? How would you even tell, when no human ever really knew the code?
Machine created software is a black box of uncertainty. Imagine you need a pacemaker and the doc suggests a new model that has a little component in it designed entirely by AI. Built by AI, maintained by AI, and all anyone knows is that it passes the tests.
Do you want that thing in your chest, controlling your heart?
AI is a wonderful tool for software developers. I just don’t see how we can afford to take our eyes off it. We need to think long and hard before removing humans from the loop.
Top comments (0)