My AI-vs-human commit comparison had a control group of one day

Ars Technica covered a Harvard working paper (Fiona Chen, James Stratton) built on about 300M work events from 700+ companies. After teams adopted coding agents, PRs went up 23% and code output about 30%, but issue resolution didn't move in a statistically significant way. Review was the bottleneck: time to merge rose 49%, and revision requests nearly doubled.

I wanted to see if my own repos looked like that. So I ran one comparison on two of my side projects (Repo A and Repo B below), both built with Claude since June 4.

AI commit = the message has a Co-Authored-By: Claude trailer (245 commits). Everything else is "no trailer" (65 commits). Metric: share of commits whose subject starts with fix or revert.

	AI commits	No-trailer commits
subject starts with fix/revert	38%	17%

More than double. Same direction as the study, though not the same metric. I almost wrote the post right there. Then I tried to break it.

Check 1: who writes fix: in the subject?

69% of the AI commits use feat: / fix: style subjects. Among the no-trailer commits it's 18%. An agent that stamps fix: on every fix will get counted more by any metric that reads subjects. Widening the match to any subject containing fix, bug, revert or their Korean equivalents already shrinks the gap: 43% vs 26%.

Check 2: who are the no-trailer commits?

I had been treating them as the human side of the comparison without looking at them. 57 of the 65 landed on a single day, June 5. Their subjects look like this:

• Phase 0-1: Add core modules (logging, pipeline, batch)
• Phase A: Semantic cache + batch processing integration
• Enhancement Level 2: Sentiment analysis & topic classification

That reads like scaffolding day. Both repos also start on June 4 with an AI-trailer commit, so there is no pre-AI period in either one to use as a baseline. I can't say who wrote the no-trailer commits, and that is the problem: my "control group" is one day of work whose authorship I don't know.

Check 3: commit size

AI commits touch more files: 3.1 on average, against 1.7. If I count "the same file was touched again within 72 hours" per commit, bigger commits get more chances to overlap. Per commit it said 74% vs 68%. Counting per file instead, so that commit size drops out:

Measure	AI commits	No-trailer commits
subject starts with fix/revert	38%	17%
subject contains fix/bug/revert (or Korean equivalents) anywhere	43%	26%
a touched file is edited again within 72h (per file, both repos combined)	46%	47%

The last row ignores subjects and commit size, and it shows no gap.

Where that leaves the number

I can't tell from two repos whether AI commits get fixed more or less. I'm not saying the study is wrong either: it's a tiny sample and nothing here shows cause. The 2x I started with came from three things I wasn't measuring on purpose: a naming habit, a control group made of one day, and commit size.

Before I trust a gap between AI and non-AI work again, I want answers to these:

• Who is actually in the comparison group, and when did they commit?
• Does the label I'm counting come from a habit of the thing I'm measuring?
• Are the two groups doing the same size of work?

Run it on your repo

Standard library only. It compares commits after the first AI commit. Change the co-authored-by: claude string if you use another tool.

import subprocess, sys
from collections import Counter
from datetime import datetime

repo = sys.argv[1] if len(sys.argv) > 1 else "."
out = subprocess.run(
    ["git", "-C", repo, "log", "--no-merges", "--name-only",
     "--pretty=format:\x1e%ct\x1f%B\x1f"],
    capture_output=True, text=True).stdout

commits = []  # dict: time, AI trailer, subject, files touched
for chunk in out.split("\x1e")[1:]:
    t, msg, files = chunk.split("\x1f")
    commits.append(dict(t=int(t), ai="co-authored-by: claude" in msg.lower(),
                        subj=msg.strip().split("\n")[0], files=set(files.split())))
commits.sort(key=lambda c: c["t"])

if not any(c["ai"] for c in commits):
    sys.exit("No commits with an AI trailer found. Check for Co-Authored-By: Claude in messages.")
start = min(c["t"] for c in commits if c["ai"])   # only compare after the first AI commit
commits = [c for c in commits if c["t"] >= start]

pairs = {True: [0, 0], False: [0, 0]}             # [files, files edited again within 72h]
for i, c in enumerate(commits):
    for f in c["files"]:
        pairs[c["ai"]][0] += 1
        if any(c["t"] < d["t"] <= c["t"] + 72 * 3600 and f in d["files"] for d in commits[i + 1:]):
            pairs[c["ai"]][1] += 1

for ai in (True, False):
    g = [c for c in commits if c["ai"] == ai]
    if not g:
        continue
    fix = sum(c["subj"].lower().startswith(("fix", "revert")) for c in g)
    day, k = Counter(datetime.fromtimestamp(c["t"]).date() for c in g).most_common(1)[0]
    n_f, hit = pairs[ai]
    print("AI        " if ai else "no trailer", f"{len(g)} commits | subject starts with fix/revert {fix / len(g):.0%}"
          f" | {sum(len(c['files']) for c in g) / len(g):.1f} files per commit"
          f" | file re-edited within 72h {hit / max(n_f, 1):.0%}"
          f" | busiest day holds {k / len(g):.0%} of the group")

On my two repos:

Repo	Group	Commits	fix/revert subject	Files per commit	File re-edited in 72h	Busiest day
Repo A	AI	145	40%	2.9	47%	10%
	no trailer	16	25%	1.6	52%	62%
Repo B	AI	100	36%	3.4	44%	23%
	no trailer	49	14%	1.8	46%	96%

The last column is the one I wish I had printed on day one. If your comparison group has most of its commits on one day, stop and look at it before you trust any ratio.

If you run it, what does your busiest-day column look like?