I wanted to test a simple question:
If 20 AI agents get the same requirements and build independently, how similar will the applications be?
I chose a small payment service. It accepts a payment message in JSON or XML, validates it, and transfers money between accounts. The goal was not to prove that AI is good or bad at coding. I wanted to see what happens when we let agents turn one shared brief into working software without human code review steering or correcting their implementations.
The results were more different than I expected.
The setup
First, I wrote one shared set of requirements for the payment application. I also prepared an AGENTS.md with instructions for an agent to implement the requirements autonomously.
Then an orchestration agent created 20 numbered implementation folders. Each folder received the same requirements and instructions. One agent was assigned to each folder and worked in isolation from the other implementations. The agents could choose their own design and implementation details. There was no cross-agent feedback or human code review during the build.
Once all 20 implementations were complete, I compared their source and project evidence. The static analysis looked at areas such as requirements coverage, validation and edge cases, code complexity, security signals, possible bug candidates, and deployability. Those signals help direct a review; they are not a substitute for a full code audit.
The same messages, in the same order
To compare runtime behavior, I generated a fixed fixture of ten accounts and a corpus of 250 JSON plus 250 XML payment messages. The messages cover expected successful payments, failures, validation errors, boundary conditions, and edge cases.
For every implementation, I reset the same ten fixture accounts to the same opening balances. Then I sent the messages one at a time in the same order: sample 001 as JSON, sample 001 as XML, then JSON and XML for sample 002, continuing through sample 250.
Balances were not reset between messages. A committed transfer changed the account state for later requests. The runner recorded each response and continued after request failures. At the end, it saved the final account balances.
That gave us 500 request attempts per implementation, or 10,000 attempts across the 20 implementations. Each record includes details such as HTTP status, payment status and response code, response body, elapsed time, expected case, and timestamp.
The results were scary
The 20 applications did not behave identically on the same stateful sequence. Their 2xx response totals ranged from 244 of 500 for implementation 17 to 260 of 500 for implementation 08.
That is not an accuracy score. A 2xx response includes both a completed transfer and a replay, and later outcomes depend on the balances left by earlier messages. The point is that the same brief did not lead to identical observable behavior.
The ending balances differed too. For example, ACC-1001 started at 500.000 JOD and ended between 3.576 and 4.816 JOD across the runs. ACC-2002 started at 100.000 JOD and ended between 2,991.375 and 3,014.825 JOD.
Those differences do not, on their own, tell us which implementation is correct. They tell us that independent builds made different choices—or interpreted some part of the requirements differently—and that we need review and independent checks to understand which outcomes are acceptable.
There were also four transport failures: implementations 12 and 13 each had two on sample 073. The runner recorded them and continued through the remaining requests. Implementations 06 and 14 also had an extra development-seeded account outside the shared ten-account fixture; their final snapshots identify it. These details matter when interpreting the comparison.
“Wouldn’t 20 human developers do the same thing?”
Yes. Different developers can interpret the same requirements differently and produce different systems. That part is not new, and this experiment does not show that AI is uniquely inconsistent.
The change is the speed and volume of implementation. A team of developers might deliver 20 features over several days, with review, QA, issue reports, fixes, and another test cycle in between. Agentic coding can compress the implementation work into hours. The review and validation work does not automatically shrink at the same rate.
That creates a practical question: who reviews and tests what the agents produce?
If another team does it, can that team keep pace with the agents? If the agents review their own code and write their own tests, how do we know those tests reflect the requirements rather than the agent’s own interpretation of them?
An agent can run its tests successfully and still miss a requirement. The code and tests can share the same blind spot. Independent acceptance tests, code review, and validation against the real requirements still matter—especially when the software moves money.
The bottleneck is control
AI agents do not remove the need for an engineering process. They make code arrive faster, which can move the bottleneck to review, testing, and deciding whether a change is safe to ship.
For this experiment, the important question was not which implementation “won” by producing the most 2xx responses. The question was whether the same requirements, given to independent agents, produced equivalent behavior—and what differences a team would need to catch before release.
The experiment is small: 20 implementations, one payment application, and one ordered message corpus. It is not a representative benchmark of every model, prompt, team, or application. The static findings are review signals, and the local performance figures are not a uniform production benchmark.
But the result is a useful reminder: faster implementation is not the same as faster assurance. AI may take us from driving one car to piloting a much faster aircraft. We need controls that work at the new speed, while keeping people accountable for the decisions that matter.
And if you still want to let an AI agent roam through your repository while your eyes are closed, at least prepare precise, carefully thought-out acceptance criteria—and verify independently that they are met.
Top comments (0)