You Have a Review Ceiling. Measure It Before It Measures You.

I sat in on Margaret-Anne Storey's DORA community session last week, and she put a name on the thing I'd been circling since April. It isn't technical debt.

Her ACM Queue piece splits software health into three debts. Technical debt is the familiar one: implementation choices that make tomorrow's change harder. Intent debt is the missing rationale, the goals and constraints that say what a system is even for, which now has to be legible to agents and not just to people.

Cognitive debt is the one that stopped me. It's the erosion of shared understanding, the state where nobody on the team can confidently explain how the system works or predict what a change will break.

Read that again if you review pull requests for a living.

I closed a thirteen-post retrospective last month admitting I couldn't answer one question: how many AI-generated pull requests a week can a review process absorb before it stops working as a control? I still don't have that number. What I have now is a name for what you accumulate while you don't have it, and a way to find yours.

Approval velocity measures motion

Every metric most teams watch gets better as review collapses.

Merge rate climbs. Time-to-approve drops. The throughput chart looks terrific right up until the incident review, because a reviewer who has quietly become a rubber stamp is indistinguishable from a fast reviewer in every dashboard you own today. Cognitive debt doesn't announce itself as a red number. It shows up as green ones, arriving faster.

I know this failure mode from the inside. Two months of green CI on conformance checks that had never once passed, on my own project. A human audit caught it. No metric I was watching came close.

What you need to measure is detection. Almost nobody does.

Mutation testing, pointed at the reviewers

We solved this once already, for test suites.

Mutation testing injects known bugs into code and checks whether the tests catch them. A suite that passes everything might be thorough or might be asleep, and mutation testing is how you tell which. Nobody finds this exotic anymore.

Run the same play against review.

Start from a real baseline. The delivery-layer metrics I'd instrument first haven't changed since I wrote them down, so I won't re-list them here. Two weeks of honest numbers beats a quarter of arguing about which numbers to collect (the arguing is more fun, which is why teams pick it).

Then seed canaries. Plant a small, steady stream of known-defective changes into the review queue under ordinary conditions. An off-by-one. A swallowed exception. A permission widened exactly one notch, the kind of diff that reads fine at 4:40 on a Thursday. Tell the team the program exists and never which PRs are in it, the way regulated shops have run surveillance testing against their own controls for decades.

How many canaries, and what that precision buys

More than instinct suggests. Detection rate is a proportion, so your error bars shrink with the square root of the count. Thirty canaries in a window gets you within 18 points at 95% confidence. A hundred narrows that to 10. Those are worst-case widths, and they tighten as detection climbs away from a coin flip.

Ten tells you nothing you didn't already believe.

I'd start at one canary per twenty pull requests, measured in four-week windows. A team reviewing 200 PRs a week hits 40 canaries a month at that rate, which is enough to compare one window against the next. Seed heavier and the signal arrives sooner, paid for in reviewer hours spent on defects you planted yourself. Seed lighter and you'll still be waiting on your first usable number at Thanksgiving.

Be honest about what that precision buys, because it isn't much. At 40 canaries a window you can see a collapse, detection dropping from 80% to 50%. You cannot see a 5-point drift, and anyone who tells you otherwise is selling something. This method finds cliffs. Gentle slopes stay invisible until they stop being gentle.

That's arithmetic. I haven't run it at a real team's volume, and the first person who does should publish what the numbers actually looked like.

Where your curve breaks

Now raise the load. As AI-generated PR volume climbs, detection against the canaries holds, then sags, then drops off a cliff, and the knee of that curve is your ceiling. That number is yours. It won't match anyone else's, and a vendor can't sell it to you.

detection
rate      *----*----*
                     \
                      \        <- the knee: your ceiling
                       *
                          *
                              *
          ---------------------------> AI PRs per week

Past the knee you aren't reviewing. You're taking on cognitive debt at a rate nobody is tracking, and you'll settle the balance during an outage.

The objections are all reasonable

Seeding defects sounds dangerous. The canaries live behind a flag that never deploys, same as any dark launch.

It sounds like distrusting your engineers. You're auditing a process, which is what the word audit means (nobody reads a fire drill as an accusation).

And yes, it's work nobody budgeted for. So was every control your shop now treats as table stakes, back when somebody first proposed it and got told there wasn't time.

Set against that: you're already running the experiment. Every team that switched on AI-assisted coding is raising the load on its reviewers right now, today, with no curve and no instrument. The only open question is whether anyone is reading it.

What I want from you

I'm building this into AIEOS so the answer is a query rather than a research project. One operator's numbers aren't an industry answer, though they're one more data point than anyone has published.

So measure it. Publish the curve, anonymized, methodology attached. Fifty teams publishing detection curves would teach this industry more about engineering in the AI era than every framework announcement of the last two years, mine included.