For years, "AI adoption" meant one thing: how many developers were using the tools. Now that agents write much of the code, that number no longer tells you anything useful.
Amazon made this point publicly last week. According to GeekWire, the company revised its software engineer role guidelines to emphasize results for customers over how much AI engineers use. AWS vice president Marc Brooker said that for the vast majority of Amazon developers, AI agents now write almost all the implementation code that used to be written by hand.
If a company at that scale is rewriting its playbook, smaller IT teams should pay attention.
What Amazon actually said
The guidance, published on Amazon's site on October 27, makes three points worth borrowing.
Usage is not the goal. Tom's Hardware reports that Amazon wants engineers to stop chasing token counts, a habit now nicknamed "tokenmaxxing." Heavy AI usage does not automatically mean better work.
Outcomes are the yardstick. Brooker told CNBC that Amazon looks at feature velocity, meaning product launches and successful changes made in production.
Judgment matters. The memo says an engineer who can explain where AI doesn't help their work is showing exactly the judgment the company wants. In other words, knowing when not to use AI is a skill.
Why usage metrics mislead
Counting tokens, prompts or AI-assisted pull requests feels objective, but it rewards activity instead of value. A developer can generate thousands of lines in an afternoon and still ship nothing customers want. Worse, when usage is rewarded, people optimize for the metric. They paste in bigger prompts, run longer agent loops and open more changes, and your AI bill grows while delivery stays flat.
There's a quality risk too. Faster code generation shifts the bottleneck from writing to reviewing. If review capacity stays the same while output doubles, defects slip through. Amazon reportedly reviewed the stability of its retail platform in March after several site outages, a reminder that speed without safeguards has a cost.
What to measure instead
Use outcome metrics that don't care how the code was produced:
- Lead time: how long from an idea to a change running in production?
- Change failure rate: what share of releases cause incidents or rollbacks?
- Time to restore: how fast do you recover when something breaks?
- Review time and rework: is AI-generated code being merged cleanly, or sent back repeatedly?
- Customer-facing results: features shipped, tickets resolved, retention.
These are the same DORA-style measures good teams already track. AI doesn't replace them; it makes them more important. Delivery teams at a mobile app development company tend to see this quickly, because app releases are judged by crash rates and user reviews, not by how fast the code was typed.
Practical steps for your team
Keep humans accountable for what ships. The developer who merges the change owns it, whether they wrote it or an agent did. Make that explicit in your engineering guidelines.
Strengthen review, don't skip it. Add automated tests, linting and security scanning as gates, and set expectations for reviewing AI-generated code with the same care as any other.
Document where AI falls short. Amazon treats AI's limits as findings worth reporting. Have your team keep a short log of tasks where the tools failed, such as legacy systems, unusual domain rules or tight performance work. This saves other developers from repeating dead ends.
Prototype fast, harden slowly. AWS's Velocity Labs took an open-source model from concept to public release in under a week, according to GeekWire. Fast prototypes are valuable, but speed in an experiment is different from speed in production. Keep a clear line between the two.
Watch the cost per outcome. Instead of total token spend, track what it costs to deliver a feature or fix a bug. That tells you whether AI is actually paying for itself.
Mobile teams have extra reasons to be careful
Mobile releases are harder to roll back than web deployments. App store review cycles, device fragmentation and users who delay updates all raise the price of a bad release. Teams building apps with AI help should lean even harder on testing across devices, staged rollouts and crash monitoring, so faster coding doesn't turn into faster mistakes.
Conclusion
AI is now a normal part of software development, and nobody serious is arguing otherwise. The open question is how to judge the work. Amazon's answer is to stop counting how much AI developers use and start asking what they deliver. Measure outcomes, protect code quality, and treat knowing AI's limits as part of the job. Teams that do will get the speed benefits without the hidden costs.
Top comments (0)