Chapter 14 - What to measure when your team delegates to agents
The Agentic Development Lifecycle - @praveen govindaraj
Stop Counting PRs. Measure the Queue.
AI agents can generate code faster than your team can review it. So how do you measure real productivity?
A team hit its merged-PR target, but pull requests grew by a third.
Nobody cheated. The metric simply rewarded output instead of outcomes.
When code generation becomes cheap, review capacity becomes the constraint.
Measure the queue
Instead of counting pull requests alone, track:
• Review wait time: Time from PR creation to first human review.
• Incoming PRs: How much work enters the queue.
• Batch size: Whether changes are becoming harder to review.
If arrivals increase slightly but waiting time rises sharply, your review process may be approaching saturation.
Fewer, larger PRs won't necessarily fix the problem. They can reduce the item count while leaving the actual review workload unchanged.
Measure delegation quality
Track a few additional metrics:
• First-pass acceptance: How often agent changes are accepted without human edits.
• Rework rate: How often merged agent-authored code needs subsequent changes.
• Harness coverage: Whether controls for known failure modes have been tested.
• Cost per accepted change: Include compute, tokens, and human review time.
Keep the four DORA metrics, too: deployment frequency, lead time for changes, change failure rate, and time to restore service.
Be cautious with metrics such as lines of code generated, PR counts, and self-reported “time saved.” They can look impressive without proving that delivery has improved.
Try this week
For four weeks, track incoming PRs alongside time to first human review.
Record PR size, too, so fewer but larger changes don't disguise the workload.
Watch how waiting time responds when arrivals increase. If the queue grows faster than the incoming work, investigate review capacity before asking agents to produce more.
Measure the constraint, not the output.
📖 Read the full chapter: Chapter 14: Metrics for Delegation
I'm exploring The Agentic Development Lifecycle one chapter at a time, in plain words.
Discussion: What AI productivity metric does your team track—and does it measure delivered value or just generated output?
Top comments (1)
Time to first review also needs a view of the PRs that have not received one yet. If the slowest items remain open, an average calculated only from reviewed PRs can improve while the backlog gets worse. I'd keep the age of the oldest unreviewed change and the share waiting past an agreed threshold beside arrival rate, split by change size. That makes a four-week observation less dependent on which work happened to leave the queue.