Measure Shipped Work, Not Tokens

Andi Smith — Thu, 16 Jul 2026 00:00:00 +0000

If everyone on your team already uses AI every day, what does your AI adoption metric actually tell you?

Nothing, is the honest answer. And yet usage is still the number most companies are chasing. Earlier this year, Meta ranked its 85,000 employees by token consumption on an internal leaderboard. The winner earned the title of Token Legend after burning 281 billion tokens in a month. Uber blew through its annual Claude Code budget by spring. Amazon engineers reportedly delegated pointless tasks to agents just to climb the internal charts.

There's a name for this now: tokenmaxxing. And it's like the time where we tried to measure productivity through lines of code all over again. Tokens measure an input. They tell you the machine is running, not that it's producing anything you'd want to keep.

The Metric That Matters

So how do we measure that AI is making our our working lives better? How do we measure the progress we are making utilising AI to improve our development speeds and processes? And how do we ensure that we continue to improve the way how we use AI?

I've been thinking a lot about this and I think it is summed up with:

The percentage of shipped work completed end to end by agents.

End to end means from a human intent through build, test, deploy and verify, with humans reviewing by exception rather than gating every step. It measures what the AI delivers, not how often you touch it.

The real value is the question it forces. What class of work do you not yet trust agents with, and what would it take to trust them? That turns AI strategy from a spend conversation into an engineering conversation.

Verification Is the Gap, Not Throughput

Throughput is largely solved. Trust is not. Research across 22,000 developers found task completion up 34% with heavy AI use, but bugs per developer up 54% and code churn up 861%. Throughput measures what shipped. It doesn't measure what survived.

The companies furthest ahead understood this early. Spotify's background agent Honk merges around 1,000 PRs every 10 days, and it works because every change is verified against builds and tests before a PR is even opened, on a codebase standardised over 15 years.

There's also a structural risk to design against. When AI writes the ticket, the code and the review, errors correlate. The fix is independent checks the authoring model never touches. Tests generated from real incidents, security scanning on every PR, canary rollouts to a small slice of users.

As your confidence in verification grows, the class of work you are confident you can allow the AI to do end-to-end also increases. You can move from menial low risk tasks to more complex areas of the system.

Foundations plus verification.

That's the whole playbook. Get the foundations right so you can build the right verification layer and have confidence in the output - then tokens are just the electricity bill.

Undefined Roles

Andi Smith — Sat, 07 Mar 2026 00:00:00 +0000

With the acceleration of AI, the traditional boundaries between product, engineering and UX are collapsing.

For as long as I can remember, in most businesses these disciplines have operated in clearly defined lanes. Product owned the vision, engineering built it, and UX made it usable. Each role had its specialists, its processes, and its seat at the table. In the start-up world, the lines have always been a bit more blurred with team responsibilities, but with this new wave of agentic AI even larger teams are going to find that this model is under pressure.

AI is compressing the skill gap in ways that would have seemed unlikely even six months ago. The cost of writing code is lower than ever. Interfaces can be prototyped quickly, thrown away and restarted with almost no penalty. Product decisions can be validated without a full team behind them. The bottleneck is no longer technical execution. Instead, the bottleneck is thinking time.

This is already playing out. Engineers are now making product calls. Designers are prototyping in code. Product managers are shipping without engineering dependency. The lines aren't just blurring — they're genuinely dissolving. A single person can now explore an idea from concept to working prototype in a single day.

This has real implications for how we build and hire. The most valuable people aren't necessarily deep specialists anymore. They're people who can move fluidly across disciplines, hold the bigger picture in mind, and make good decisions with incomplete information. T-shaped skills are giving way to something harder to define but increasingly recognisable when you see it. None of this means specialisation is dead. But the weight is shifting.

So if you are working in product engineering, what should you do?

The best place to start is by getting curious about the disciplines adjacent to yours. If you're an engineer, spend time understanding why product decisions get made. If you're in product, get closer to the technical constraints. If you're in UX, understand the business outcomes you're designing for.

You don't need to become an expert in everything — but developing fluency across the stack, both technical and human, is increasingly the differentiator. The people who will thrive in this new world are those who lean in rather than wait to see how it plays out.

Change is coming whether we're ready or not. The right response isn't anxiety. It's curiosity. Bring a product mindset, an entrepreneurial spirit, and a genuine openness to working differently.

Nobody has this fully figured out yet. It's a work in progress - for all of us. The real advantage right now is curiosity.

DEV Community: Andi Smith

Measure Shipped Work, Not Tokens

The Metric That Matters

Verification Is the Gap, Not Throughput

Undefined Roles