A team can double the number of AI-generated lines in a week and still deliver no more useful software. It might even deliver less: more review work, more defects, more rework, and a larger bill. The uncomfortable part is that a dashboard full of activity can make the opposite story look convincing.
That tension sits at the center of Amazon’s newly published guidance on the software developer role in the age of AI. The memo, written by Terk Terkowitz, vice president and distinguished engineer at Worldwide Amazon Stores, asks engineers to learn and use AI. It also warns that token consumption, adoption rates, code changes, and reviews are deeply flawed if managers treat them as targets rather than imperfect signals. The company says those inputs can still correlate with valuable outputs, but should be used with care and judgment. Amazon’s memo is more nuanced than a simple “stop using AI” headline suggests.
The important question is not whether developers should use AI. It is whether a company can tell the difference between useful adoption and performative activity. Amazon’s guidance offers a good starting point, but the measurement problem is harder than swapping one dashboard for another.
What Amazon is actually saying
The memo makes two claims that need to be held together.
First, AI is likely to become central to software engineering. Amazon encourages engineers to learn the tools, apply them in daily work, and keep improving their practice. The memo calls for hands-on understanding, not secondhand opinions.
Second, the company says measures such as tokens consumed, adoption rates, code changes, and reviews are “deeply flawed” when detached from outcomes. As targets, they can encourage higher costs or unnecessary complexity. The memo gives “tokenmaxxing” as the case where increased consumption is treated as a goal in itself. It also warns that measuring lines of code can conflict with simplicity.
That does not mean Amazon considers those measurements useless. Its memo says input metrics have correlated with outputs that matter, and recommends looking carefully at both the 10th and 90th percentiles of adoption. The first can reveal teams that are not finding effective uses. The second can show whether teams with heavy usage are seeing delivery gains to match.
This distinction matters. A metric can be informative as a signal and damaging as a quota. If a manager asks why adoption is low on a particular team, the answer may expose a training gap or a workflow where the tools genuinely do not help. If the manager demands that every engineer hit a token target, the measure has become the work.
Why usage metrics are tempting
Usage is easy to count. Outcomes are not.
A provider can report tokens, requests, active seats, or generated suggestions almost immediately. A team can count pull requests, changed lines, or review comments. These numbers are legible, comparable, and available before a release has reached customers.
But the distance between “the tool was used” and “the customer received more value” is long. A generated patch may be rejected. An accepted patch may need extensive revision. A merged change may not solve the intended problem. A feature may ship faster and still introduce a reliability issue that takes weeks to surface.
That gap is a measurement problem, not an argument against using AI. It is also not solved by measuring only a different proxy. If token count is a weak stand-in for productivity, replacing it with lines of code or pull request count may merely move the incentive to another easy-to-game number.
The 2025 METR randomized controlled trial is a useful caution about scope. It studied 16 experienced open-source developers completing 246 tasks in mature projects they knew well. With the early-2025 AI tools in the study, developers took 19% longer on average, even though they expected AI to make them faster. METR’s paper does not prove that current tools slow every developer or every task. It shows why broad adoption claims cannot substitute for measuring the specific work, people, and tools involved.
METR later ran a follow-up with late-2025 tools, but said the newer data was an unreliable signal because developers who most wanted AI were less likely to participate or submit tasks they preferred to do with AI. In that follow-up, METR estimated an 18% slowdown among returning participants, with a confidence interval spanning from 38% slower to 9% faster, and a 4% slowdown among newly recruited developers, with a confidence interval spanning from 15% slower to 9% faster. Both intervals include no effect. METR said the selection effects made the results difficult to interpret and likely understated benefits for the developers and tasks missing from the study. METR’s 2026 design update explains those caveats. The takeaway is not that AI still slows developers; it is that a single metric or study cannot settle the question for every workflow. That is exactly the kind of nuance a usage dashboard hides. It can show that a team used an assistant more. It cannot, on its own, tell you whether the assistant helped with that team’s actual task mix.
A better unit of analysis: the task, not the token
A practical evaluation should start with a meaningful unit of work. That could be a bug fix, a small feature, a migration step, a test suite improvement, or a support issue. The definition should be stable enough that “done” means something consistent before and after AI enters the workflow.
For each task, record a small set of signals:
- Outcome: Was the intended user or operational problem resolved?
- Time to an accepted result: Include prompting, checking, editing, and rework, not only the first generated draft.
- Quality: Did tests pass? Were review findings, regressions, or follow-up fixes needed?
- Human effort: How much focused developer and reviewer time did the task consume?
- Context: What kind of task was it, how familiar was the code, and what tool or workflow was used?
This is not a universal productivity formula. It is a way to avoid collapsing different work into a single count. A one-line configuration change and a complex production incident should not be treated as equivalent just because both produce one pull request.
Teams can then compare like with like. For example, they might examine several similar maintenance tasks completed with and without AI assistance, while noting differences in developer familiarity and review requirements. The comparison will not be a perfect experiment. It can still be more informative than asking whether AI usage went up.
The key is to preserve enough context to explain the result. If AI helps draft repetitive tests but slows work in a deeply familiar legacy module, that is not contradictory. It is evidence that usefulness depends on the task and workflow.
Keep adoption data, but do not turn it into a quota
There is a reasonable place for usage data. If nobody on a team has tried an approved tool, that may be worth investigating. Maybe access is broken. Maybe the team has not had time to learn. Maybe its work is unusually constrained. Usage can identify a question to ask.
It should not answer the question by itself.
A manager can use adoption data as a prompt for a conversation: “Where did this help? Where did it create extra work? Which task would you not use it for again?” That makes usage a diagnostic input. A required weekly token number makes it a performance target, even if leadership says otherwise.
Amazon’s attention to the 10th and 90th percentiles points toward a better practice: look at the outliers and investigate why they differ. Low usage might reveal a workflow that has not been adapted or a legitimate lack of fit. High usage should invite the same scrutiny: are those teams delivering more, or merely generating more activity? Do not assume the answer from the chart.
Averages can conceal both situations. A team-wide mean may look healthy while a few groups are struggling to adopt the tools, and a few others are spending heavily without a corresponding benefit. Percentiles do not explain the cause, but they help identify where a closer look is worthwhile.
A lightweight review cycle for engineering teams
A team does not need a measurement department to test whether AI helps. It does need a review cycle that resists easy gaming.
1. Pick one workflow. Choose a recurring task with a clear completion condition. Avoid beginning with “all engineering work.” Narrow scope makes the result interpretable.
2. Define the quality bar first. Specify the tests, review expectations, or operational checks that make the task acceptable. Do not lower the bar after seeing a faster draft.
3. Count the whole loop. Include time spent describing the task, waiting, reading generated output, correcting it, reviewing it, and handling rework. A fast first response is not the same as a fast accepted outcome.
4. Record negative cases. Note when the tool was not useful and why. Amazon’s memo explicitly treats an engineer’s explanation of where current AI does not help as useful judgment, not failure. Without this record, a team may remember only the impressive examples.
5. Review outcomes with context. Compare similar tasks and separate changes in scope, experience, and codebase familiarity. Treat a small sample as a learning signal, not a company-wide proof.
6. Revisit the measure. If people start optimizing the metric instead of the work, stop using that metric as a target. Keep it only if it still helps explain outcomes.
This cycle creates a practical decision: expand the workflow, adjust it, or leave it alone. The result can be “AI helps here,” “AI needs a different review pattern,” or “the current tool adds no value for this task.” All three are useful findings.
The “Renaissance Developer” is more than a tool user
Amazon describes the desired developer as a “Renaissance Developer,” combining AI tools with human judgment and domain expertise. The label is broad, but the memo gives it a concrete meaning: understand the product and customers, learn the tools directly, connect inputs to outputs, and exercise judgment about where AI belongs.
That is a stronger standard than simply asking engineers to use more AI. More importantly, it puts responsibility on leaders as well as individual contributors. Managers need to understand the technology well enough to recognize when a workflow is useful, and to connect adoption to delivery rather than treating usage as proof of progress.
There is a tension in the memo worth noticing. Amazon argues that its input metrics correlate with output, while also warning that they can cause harm when made targets. That is plausible: a measure may track a trend at the organizational level and still be a poor individual performance score. A company-wide relationship does not establish that every engineer should have a personal target for tokens, code changes, or tool usage.
The memo does not supply a complete measurement method, and its recommendations are written for Amazon’s own SDE and SDM organizations. Other teams should not copy its percentile thresholds or assume they will observe the same correlation. The transferable idea is more modest: inspect where AI use is low or high, ask what explains it, and verify whether outcomes moved too.
A checklist before adopting an AI productivity metric
Before putting a number on a dashboard, ask:
- Does this measure customer or operational value, or only activity?
- Can an engineer increase the number without improving the result?
- Does it account for review, correction, and rework?
- Are we comparing similar tasks and codebase familiarity?
- Could the metric reward unnecessary code, requests, or complexity?
- Do we record cases where AI was not useful?
- What decision will we make if the number rises but outcomes do not?
If the answers are unclear, the metric is not ready to become a target. It may still be useful for exploration, but label it as an input and say what it cannot prove.
Amazon’s memo is a welcome correction to the idea that more AI activity automatically means more productivity. Its strongest point is not that tokens do not matter. It is that tool usage is only a clue. The work still has to solve the right problem, pass the quality bar, and deliver something useful.
I write about software engineering and AI tools. Subscribe if you want more practical, evidence-led breakdowns like this. Have you seen AI usage metrics help a team improve, or push people toward activity instead of outcomes? Share what happened.
Save the checklist for the next time someone proposes a token, adoption, or code-volume target. Start with the result you want, then choose the smallest measure that can help you understand whether the tool contributed.
Top comments (0)