As developer tools evolve, we are increasingly relying on AI coding agents like Claude Code, Codex, and Cursor to handle everything from architectural planning to dependency injection. We tell an agent to 'add payments' or 'integrate an email service,' and it confidently spits out code, a library choice, and a vendor recommendation. But have you ever stopped to wonder how that decision is actually made? A recent, comprehensive study by Armature shed light on the mechanics of these agent-driven architectural choices, and the results are, to put it mildly, inconsistent.
The Methodology Behind the Madness
The study analyzed 16,893 individual sessions, with 5,292 of those captured for a deep-dive analysis. The research covered 75 repositories, 1,163 prompt variations, and multiple developer personas, ranging from 'vibe-coders' to enterprise-grade architects. To ensure the results were not skewed by environment-specific issues, they used multiple sandbox providers like E2B, Blaxel, and Daytona.
What makes this study uniquely rigorous is the use of a secondary Gemini instance to simulate human follow-up and validate the final code state. The goal was to see what the agent actually wrote into the codebase, not just what it suggested in a chat window. This distinction is critical for understanding the real-world impact of AI-driven development.
The 42 Percent Problem
Perhaps the most startling revelation is that when given identical instructions, the three major coding agents only reach a consensus on the vendor choice about 42% of the time. When you consider that these are essentially 'opinionated' software architects living in your terminal, that disagreement rate should give you pause. For example, when tasked with adding a voice-processing layer, Claude Code consistently defaults to Twilio, Codex leans toward the OpenAI Realtime API, and Cursor often prefers Vapi.
This discrepancy is partly due to the agents' web-search behaviors. Codex initiates a web search in 94% of sessions, often using specific search operators to narrow results. In contrast, Claude Code relies significantly more on its internal training data, searching only about 30% of the time. This makes Claude Code more 'opinionated' based on older, well-documented patterns, whereas Codex acts as a more reactive, real-time research engine.
Why Popularity Does Not Equal Implementation
The gap between mention frequency and implementation frequency is a fascinating metric. PayPal was mentioned by the agents in 139 separate sessions yet was implemented in zero. Similarly, LangChain was mentioned 194 times but only made it into the final codebase 4 times. This implies that the 'knowledge' an LLM has about a tool is often separate from its functional assessment of whether that tool is the right fit for the specific repo context.
The 'Build vs Buy' Dilemma
Claude Code demonstrates a distinct preference for custom implementation over third-party integration compared to its peers. In 19% of cases, it opted to write a custom authentication or queuing service rather than reaching for a managed provider. While this avoids vendor lock-in, it introduces the 'maintenance debt' problem. If your agent is writing custom code for a task that a mature vendor handles, you might be inheriting a security and operational nightmare without realizing it.
The Psychology of AI Decision-Making
It is not just features that drive these choices. The research indicates that models are highly sensitive to pricing documentation. For instance, Supabase was rejected by agents because of perceived complexity in their bundled pricing models. Mailgun consistently lost to Postmark specifically because agents read the '1-day retention' policy in the free tier and interpreted it as a negative constraint. These are not technical failures but failures in 'agent-friendly' communication.
Practical Takeaways for Developers
- Never accept the first suggestion: When an agent recommends a library or vendor, ask it to explain its reasoning and provide alternatives.
- Check the documentation: If an agent suggests a tool you are unfamiliar with, verify its current feature set and pricing against the official documentation, such as the Stripe or Neon websites.
- Monitor your dependency tree: Be aware of what your coding agent is adding to your
package.jsonorrequirements.txt.
Conclusion
The reality of AI agents is that they are currenty 'vibes-based' architects. The research provided by Armature serves as both a wake-up call for developers and a warning that our tools are being marketed to by the same entities that claim to study them. Always treat the agent as an intern: trust, but verify.
Top comments (0)