Can you trust software built with AI by someone who can't read the code it produced? The whole debate keeps circling one question: did a human check it? I build these systems for a living, and the question I'd rather answer is a harder one: would anything have caught the mistake before it shipped?
The fear is a proxy
This started as an exchange on X. I had written that "did a human check it?" is the wrong question about AI-built software, and someone sent back a better one:
"AI didn't replace builders — it exposed the real gap: trust. Builders don't just assemble apps; they earn the trust to ship them. What do you think breaks that trust faster: a flaw in the AI's code, or the fear that no human ever looked?"
The fear, clearly. But I think the fear stands in for something else. Nobody actually wants a human to have read every line; people want to know that something would have caught the mistake. Almost everything written about AI and code quality answers the first demand and leaves the second one alone. Trust is the net, not the eyes.
We trust people. Planes tell the whole story.
Trust is anthropomorphic, it attaches to faces. Aviation shows how deep that runs: in a UBS survey, only 17% of passengers said they would board a pilotless plane, and more than half wouldn't buy the ticket even if it were cheaper. An Ipsos poll found 81% of Americans uncomfortable with the idea. People want a human in the cockpit, and no discount changes their mind.
Now look at what actually made flying the safest way to travel, because it wasn't the pilot's eyesight. Checklists, written after crashes, so that no landing depends on someone remembering. Two pilots, because any one person will eventually be tired, sick or wrong. Redundant instruments, mandatory maintenance intervals. Aviation never found better humans; it wrapped ordinary, fallible ones in a structure that catches what they miss. The passenger's trust goes to the pilot, and the catching is done by everything around him.
I think software is at the moment where it has to learn the same distinction.
Meanwhile, the eyes are losing ground
Even where competent eyes exist, the guarantee is dissolving. I've reviewed enough code to vouch for the first half: we miss things. Every incident postmortem I've read includes a change that a competent person approved.
The volume takes care of the second half. Faros AI's engineering report, built on telemetry from twenty-two thousand developers, measured what AI adoption does to the review pipeline: pull requests merged without any review at all are up 31.3%, and median time in review has quadrupled. Reviewers haven't gotten lazier, the queue simply outruns them. So "a human looked at it" is failing on both ends: it was never sufficient, and it's becoming rare.
None of this makes review useless. It just can't be the thing the whole weight of trust rests on, and an automated reviewer doesn't change that: still a pair of eyes bolted on after the fact, just faster ones. The net is whatever was already there before anything shipped.
The less you know, the more net you need
Here's the part the debate skips, and the reason it concerns me directly. All the standard advice about AI-generated code assumes a particular reader: a developer who inspects the output, someone who knows what to look for. For that person, AI is an accelerator with a known failure mode, and their own judgment is the control.
Take that person away. The maker who describes an app in plain language and gets working software back, the exact promise of no-code and vibe coding alike, cannot be the check. It isn't carelessness: checking requires knowing what failure looks like, and that knowledge is precisely what they didn't need in order to build. Telling them "review the output carefully" is telling them to inspect a bridge with no idea what a crack is.
That inverts the usual hierarchy: the people with the least domain knowledge are the ones who need the most protection, and they need it built into the floor they stand on, because there is no crew around them. If trust has to come from somewhere, it can only come from the system underneath, the layer whose job is to carry the judgment the person on top can't supply.
The engine is a net
For me this part was never abstract, because building the net is most of the job. I run frontend engineering at GoodBarber, an app builder, and the engine we build is, among other things, an accumulation of catches. I gave a talk about this at the No Code Summit, transcribed on our blog: the talk was about safeguards, stopping the obvious mistake from shipping without standardizing what people build. Set an ad frequency aggressive enough to degrade the reading experience, and a floor holds. Push a configuration past what the layout can carry, and the engine transforms it instead of rendering the mistake, a mechanism I've detailed in the article on our rendering engine.
None of these catches is spectacular, which is sort of the point. A net is an accumulation of places where a mistake runs out of room, each one settled once, in the structure, on behalf of people who will never know the question existed. Checklists, second instruments, walls an error can't cross: aviation wrote that shape down a century ago, and engines like ours have been writing the software version of it for fifteen years, since long before anyone called this an AI problem.
And it runs deeper than what you see on screen: my colleague Pierre-Laurent has written about privacy guarantees enforced at the binary level, a catch nobody has to remember exists.
AI on top, humans underneath
Put AI on top of that picture, because that's where it's landing. Generation raises the need for the net: more people building without the knowledge to check, and mistakes produced faster than anyone can read them. An AI generating into the void ships whatever comes out; the same AI generating into an engine meets the floors, the transformations and the walls that were already there for humans.
One principle from our own work with this, because it's the most reusable thing I know on the subject: the line between "the AI does it" and "the AI proposes it" shouldn't be drawn by how confident the model sounds, but by whether the mistake can be walked back. What's reversible can run. What isn't, waits for a human. (Deciding what the AI gets to do inside our own tool was a story in itself. My colleague Dominique has told it.)
That, I think, is the durable place of these engines in the AI era: carrying the part of trust that neither the model nor the maker can carry alone. Built by humans, with AI generating on top.
Trust is a property of the system
So, back to the fear. Promising that a human looked won't cure it; that promise is already breaking inside professional teams, and it was never available to the maker building alone. What cures it is catches you can name.
Deciding which mistakes deserve a catch, and which are cheap enough to let through, is judgment work, the kind that doesn't get automated away when generation gets cheap. And every check nobody formalizes doesn't disappear. It joins the bill someone pays later, usually at the worst time.
Some will read all this as giving up on humans. I see it as the opposite: taking what careful people do and making it survive their absence. The rule I come back to is the one I'd offer anyone building with AI today: you don't fully hand off what you can't undo.
Mathieu Poli — Head of Frontend Engineering @ GoodBarber.
I teach and write about frontend engineering, product design, and AI — and everything that happens when the three meet.
X: @hellomathieup · LinkedIn: hellomathieup
Photo: Maya Alexa G. Romero / Unsplash
Top comments (0)