DEV Community: Jamie

We tried to measure AI's actual impact on our codebase. Here's why it's so hard.

Jamie — Tue, 07 Apr 2026 10:11:54 +0000

Everyone's seen the stats. "55% faster." "40% more code." "3x productivity." They end up in pitch decks and team retrospectives, and nobody really questions them because the conclusion feels right -- AI tools do feel helpful when you're using them.

But when we actually tried to find those numbers in real commit histories and PR patterns, we hit a wall.

Not because AI isn't changing how people write code -- it clearly is. But because measuring how much, and whether it's actually good, turns out to be a genuinely messy problem.

The obvious metric is the wrong one

The first instinct is output. More commits, more PRs, more lines of code. And yes, those numbers do go up.

But so do some less flattering ones:

Average commit size grows -- more lines per commit, which correlates with harder-to-review changes
PR cycle times don't improve, and sometimes get worse -- reviewers are spending longer on more code written in a less familiar style
Commit message quality drops -- more "update logic", less actual context

None of this means AI is making things worse. It means raw output metrics don't capture what's actually happening.

The before/after problem

Comparing pre- and post-AI adoption sounds straightforward. In practice, you're also comparing different project phases, team compositions, architectural decisions, and a dozen other variables that moved at the same time. Almost every "AI made us X% faster" claim, when you look at the methodology, is comparing an enthusiastic adoption period against an uncontrolled baseline.

What you can actually observe

After looking at a lot of repositories, the measurable impacts fall into a few categories:

Code homogeneity increases. AI-assisted codebases tend to become more internally consistent -- the same patterns repeat. Good for cognitive load, but the same mistakes replicate everywhere too.

Review burden shifts. The bottleneck doesn't disappear, it moves. Code output goes up, but someone still has to review it. AI-generated code tends to look correct even when it isn't, which makes subtle bugs harder to catch.

Test coverage quietly degrades. AI-assisted PRs consistently ship proportionally less test code than production code. The feature lands fast, the tests get deferred.

The uncomfortable ones

Knowledge erosion. We've seen repositories where contributor breadth increases but contributor depth decreases. Bus factor metrics that look healthy, where none of the contributors could confidently explain the module without re-reading it. The metric looks fine. The codebase is fragile.

Architectural drift. AI-generated code is only as good as the context it had when generating. Over months, this creates "dialects" within the same repo -- different patterns for the same operations, because different sessions had different context.

The productivity paradox. More code, but not proportionally more capability. The codebase grows faster than the product does. AI accelerates accidental complexity because the cost of writing code drops while the cost of understanding and maintaining it stays constant.

What to actually track

If a single metric or simple before/after comparison won't cut it, here's what we think gives a more honest picture:

PR cycle time (not PR volume)
Review comment density (not approval speed)
Revert and hotfix rates (not commit counts)
Test-to-production code ratio over time
Contributor depth per module, not just breadth

And watch for: commit sizes trending upward without corresponding feature complexity, declining review thoroughness as volume increases, and the growing gap between code output and test coverage.

AI coding tools are genuinely useful -- we use them ourselves. But the rush to quantify impact has produced a lot of misleading numbers, and optimising for the wrong metrics leads you somewhere you don't want to be.

Curious whether others are tracking any of this, or whether most teams are still pointing at velocity and calling it done. What signals have you found actually tell you something meaningful?

How to Spot a Burning-Out Codebase Before It Burns Out Your Team

Jamie — Fri, 27 Mar 2026 11:45:38 +0000

The Codebase Knows First

It usually starts with a vague feeling. Standups feel heavier. Estimates keep slipping. Engineers who used to volunteer for hard problems are suddenly very interested in being assigned simple ones.

By the time it surfaces as a people problem — someone handing in notice, a team missing a milestone — the warning signs were already sitting quietly in the commit history, often for months.

Codebases don't burn out. But they accumulate the conditions that burn out the people working inside them. Learning to read those signals is one of the highest-leverage things an engineering manager can do.

The Codebase as a Team Health Mirror

A repository's activity patterns are a direct reflection of how a team is functioning. Commit cadence, PR review turnaround, contributor distribution — these aren't just engineering hygiene metrics. They're a record of how your team is spending its energy, who's carrying the load, and where friction is building.

The advantage of looking here is that it's objective and early. People are often reluctant to surface burnout or frustration until it's already critical. The repo doesn't have that filter.

5 Warning Signs to Watch

1. Sustained Commit Velocity Drop

A single slow week is noise. A four-week declining trend in commit frequency — especially when nothing obvious changed (no holidays, no planned focus sprint) — is a signal worth investigating.

This often means work has become harder, not less. Engineers are spending more time fighting existing complexity, debugging unclear ownership, or simply losing motivation to push things forward.

2. Bus Factor Creep

Pull up the contributor breakdown for your core repositories. If one person is responsible for 60% or more of recent commits, you have a bus factor problem — and potentially a burnout candidate.

High bus factor isn't just a knowledge-risk. It means one engineer is carrying disproportionate cognitive load, probably fielding most of the questions, and likely feeling increasingly isolated. When they leave or burn out, the knowledge goes with them.

3. Stale PR Pile-Up

Open PRs that sit untouched for a week or more are a sign that review culture has broken down. This usually happens for one of two reasons: reviewers are overwhelmed and deprioritising review work, or the codebase has become so complex that reviewing feels like too much to take on.

Either way, the PR queue becoming a graveyard is demoralising for the engineers opening them. It signals that their work isn't being seen.

4. Inactivity Spikes Around Specific Contributors

Look for individual contributors whose commit frequency suddenly drops — not to zero, but noticeably below their baseline. This is different from someone taking leave. It's the pattern of someone who hasn't quit but has mentally checked out.

These engineers are often still delivering on assigned work but have stopped taking initiative, stopped reviewing PRs proactively, stopped contributing outside of what's strictly required.

5. Shrinking Contributor Diversity Over Time

A healthy codebase sees contributions spread across the team. When you notice the same two or three names appearing on every recent commit — and others fading out — it's worth asking why.

Sometimes it's a natural consequence of specialisation. Often, it's a sign that the codebase has developed informal gatekeepers, or that the onboarding cost of contributing has become too high for newer team members to bother.

What To Do When You See These Signals

The goal isn't to fix the repository. The repository is just the evidence. The goal is to have better conversations, earlier.

Use the signals as a starting point, not a conclusion. "I noticed PR review times have been climbing over the past three weeks — what's making reviews feel hard right now?" is a much more productive conversation opener than a retrospective about why a deadline was missed.

Rotate ownership deliberately. If bus factor is creeping up, create explicit opportunities for knowledge transfer. Pair reviews, rotating on-call, architecture walkthroughs — not as a process exercise but because the team genuinely needs the coverage.

Cut scope when velocity drops. A sustained velocity drop rarely means people aren't working hard enough. It usually means the work itself has become harder than planned. The answer is almost never to push harder — it's to remove friction, defer non-critical work, or acknowledge that the timeline needs to change.

The engineers on your team will tell you when they're struggling, eventually. The codebase will tell you sooner.

I'm building RepoShark to surface exactly these kinds of signals automatically from your repositories. If you're an engineering leader who wants earlier visibility into codebase health, I'd love to hear from you.