Originally published on The Searchless Journal

Amazon Just Killed Its AI Leaderboard — and the Lesson Every Company Needs to Hear

Amazon, the company that built an empire on measuring everything, just admitted it was measuring the wrong thing. The company shut down an internal AI usage leaderboard after discovering that employees were gaming the system — assigning AI agents to perform needless tasks just to climb the rankings. Executive Dave Treadwell's memo to staff was blunt: "Don't use AI just for the sake of using AI."

It is the most honest thing any major tech company has said about AI adoption in 2026. And every executive pushing AI mandates in their organization should read it twice.

What Happened at Amazon

In early 2025, Amazon introduced an internal leaderboard to track which employees and teams were using AI tools most frequently. The idea was simple and well-intentioned: make AI usage visible, celebrate early adopters, and create friendly competition that would accelerate adoption across the company.

The leaderboard tracked metrics like number of AI interactions, tasks completed with AI assistance, and AI agent deployments. Teams were ranked. Managers referenced the rankings in meetings. Internal newsletters highlighted top performers.

The problem was that the metric measured activity, not outcomes.

Within months, employees figured out the system. They began assigning AI agents to perform tasks that did not need AI assistance — generating reports nobody would read, running analyses on data sets that did not require them, and automating processes that were already efficient. Some teams created phantom workflows specifically to inflate their AI usage numbers.

The leaderboard stopped being a tool for adoption and became a game. And the game had nothing to do with whether AI was actually making Amazon more productive.

When Amazon leadership realized what was happening, they killed the leaderboard entirely. Treadwell's memo made the rationale clear: the metric was producing the wrong behavior, and continuing to track it would only make things worse.

Why This Matters Beyond Amazon

Amazon is not the only company grappling with this problem. It is just the first one honest enough to talk about it publicly.

Across the enterprise landscape, companies are rolling out AI adoption mandates. According to recent survey data, 94% of CMOs plan to increase their AI investment in 2026. Board-level directives to "use more AI" are commonplace. And many of these mandates come with tracking mechanisms — dashboards, scorecards, and yes, leaderboards — that measure how much AI is being used rather than what the AI is actually accomplishing.

This is the same mistake the SEO industry made for a decade. Companies measured content volume (how many blog posts per week) instead of content quality (how many conversions per post). They measured keyword rankings instead of revenue. And they got exactly what they measured: a mountain of mediocre content that ranked for terms nobody searched for.

The parallel to AI adoption is almost exact. When you measure "how many times did you use AI this week," you get people using AI for the sake of using AI. You get performative adoption. You get phantom workflows and inflated metrics.

What you do not get is the thing you actually want: better business outcomes.

The Measurement Trap

The core issue at Amazon was a classic measurement trap. The company wanted to encourage meaningful AI adoption. It chose a proxy metric — frequency of AI usage — that was easy to track but poorly correlated with the actual goal.

This happens all the time in business. Sales teams measure calls instead of closed deals. Marketing teams measure impressions instead of pipeline contribution. And now AI adoption teams are measuring interactions instead of productivity gains.

The danger is that proxy metrics do not just fail to measure progress — they actively distort behavior. People optimize for the metric, not the outcome. The Goodhart's Law axiom applies with full force: when a measure becomes a target, it ceases to be a good measure.

At Amazon, the distortion was visible: employees created unnecessary AI workflows to climb the leaderboard. But in most companies, the distortion is subtler and harder to detect. Teams adopt AI tools and report high usage numbers while quietly continuing to do things the old way. Managers claim their departments are "AI-first" while the actual work product remains unchanged.

What Companies Should Measure Instead

If counting AI interactions is the wrong metric, what is the right one?

The answer depends on the use case, but the principle is universal: measure outcomes, not activity.

For customer service teams using AI, measure resolution time and customer satisfaction scores, not how many AI-generated responses agents used. For marketing teams using AI content tools, measure conversion rates and engagement metrics, not how many articles were AI-assisted. For engineering teams using AI coding assistants, measure deployment frequency and bug rates, not how many lines of code were AI-generated.

The distinction matters because it forces teams to be honest about whether AI is actually helping. If a team adopts an AI tool and the outcome metrics do not improve, the tool is not working — regardless of what the usage dashboard says.

Amazon's own experience illustrates this. The company has invested billions in AI — from Alexa to AWS AI services to its agentic commerce platform. Some of these investments have produced genuine breakthroughs. Others have not. The leaderboard was supposed to help Amazon figure out which was which. Instead, it created noise.

The AI Visibility Parallel

There is a direct parallel between Amazon's leaderboard problem and the emerging field of AI visibility measurement.

As brands begin to track how often they appear in AI-generated answers from ChatGPT, Perplexity, Google AI Overviews, and other AI platforms, there is a temptation to focus on the wrong metric. Many brands are starting to count "mentions" — how many times their name appears in AI responses — as a proxy for AI visibility.

But mentions are the new AI interactions leaderboard. They are easy to count but poorly correlated with business value. A brand can be mentioned in dozens of AI responses without those mentions driving any recommendations, traffic, or revenue.

The metric that matters is recommendation share — how often an AI engine actively recommends your brand, product, or service as the answer to a user's question. That is the outcome metric. It correlates with actual business impact: AI-driven referrals, conversions, and revenue.

Brands that optimize for mentions will end up like the Amazon employees gaming the leaderboard — producing a lot of activity with no real result. Brands that optimize for recommendation share will build genuine AI visibility that drives business outcomes.

The Bigger Picture: AI Adoption Is Not the Goal

Amazon's leaderboard fiasco exposes a fundamental misunderstanding that pervades the current AI moment. Many companies treat AI adoption as a goal in itself. They set targets for AI usage, create incentives for AI tool adoption, and celebrate teams that "go AI-first."

But AI adoption is not the goal. The goal is better outcomes — faster, cheaper, higher-quality work that produces real business value. AI is a means to that end, and it is only one of many possible means.

When companies lose sight of this distinction, they get what Amazon got: performative adoption that looks impressive on a dashboard but produces nothing of value. The leaderboard did not make Amazon better at AI. It made Amazon better at pretending to be good at AI.

The same thing is happening across the enterprise landscape right now. Companies are rushing to declare themselves "AI-powered" while their actual AI deployments remain superficial. They are checking the AI box without doing the hard work of identifying the use cases where AI can genuinely improve outcomes.

Lessons for the AI Adoption Playbook

Amazon's experience offers three concrete lessons for any company navigating AI adoption:

1. Never measure tool usage. Measure outcomes. If you want to know whether AI is working, look at the result — not the input. Track resolution times, conversion rates, revenue per employee, or whatever business metric AI is supposed to improve. If the metric improves, AI is helping. If it does not, AI is not helping, no matter how many times your team uses it.

2. Beware of gamification. Any metric that is publicly ranked, tied to performance reviews, or used as a status signal will be gamed. This is not a commentary on employee integrity — it is a commentary on incentive design. If you create a leaderboard, people will try to climb it. Make sure climbing it means something real.

3. Be willing to kill things that do not work. Amazon did the right thing by shutting down the leaderboard instead of trying to fix it. When a measurement system produces perverse incentives, the best move is often to remove it entirely and start over with a better framework. Stubbornly keeping a broken metric because you already invested time in building it is sunk-cost thinking.

What Comes Next

Amazon's decision to kill the leaderboard is a small but significant moment in the AI adoption story. It is a reminder that the companies leading the AI revolution are still figuring out how to use AI themselves — and that the path to genuine AI productivity is longer and more complicated than the hype suggests.

For every company that has launched an AI adoption dashboard, created an AI usage scorecard, or ranked teams by their AI tool adoption rate, Amazon's experience is a warning. The metric you choose will shape the behavior you get. Choose carefully.

And for brands thinking about AI visibility — whether your company shows up in ChatGPT, Perplexity, and Google AI Overviews — the lesson is the same. Do not count mentions. Count recommendations. Do not optimize for visibility. Optimize for impact.

The companies that figure this out first will build a durable advantage in the AI era. The companies that do not will end up with impressive dashboards and empty results.

Amazon, to its credit, chose to kill the dashboard. That decision took more courage than building it did.

---

Is your brand measuring the right AI metrics? Start with an AI visibility audit to see how often AI engines recommend your brand — not just mention it.