Last week I published a small open-source tool — a linter for the instruction files coding agents read (CLAUDE.md, AGENTS.md, skills). Then two things happened in two days, and together they changed how I think about reviewing contributions.
The first pull request wasn't from a human
A PR appeared. Clean diff, two tests, and a scoping decision argued in a comment rather than silently made. The description ended with this:
This pull request was implemented and validated autonomously by OpenAI Codex using the
LunaMeerkatsaccount. No human review or authorship is being claimed.
The change itself was a real bug in my code: my file walker skipped any directory whose name started with .git, which also swallowed .github. Since .git was already in the ignore list, that prefix test only ever excluded .github — leaving the Copilot instruction-file support unreachable since the day it shipped. Documented in my README, typed, wired up, and dead.
What made it easy to merge wasn't the code, it was the reasoning. The PR claimed its change introduced no new findings, and argued why: every other consumer of that walker is path- or basename-scoped, so widening the walk couldn't reach them. I verified it the hard way anyway — diffed the finding sets before and after on three real repositories, identical on all three — but I was checking a stated argument, not guessing at intent.
Then a human read the source and filed eleven issues
A day later someone went through the code and opened eleven issues. Not drive-by complaints: each one had a file:line diagnosis, a repro, and often the fix. Two were serious. An unreadable Makefile took the whole scan down with an unhandled exception — while the package.json branch four lines above it was already wrapped in a try. And --fix wrote replacements through String.replace with a string pattern, so a $& or $` in a path spliced surrounding text into the user's file instead of the text I'd shown them. A linter that silently corrupts the file it's checking is about the worst thing I could have shipped.
All eleven are fixed. Two releases in a day, each regression test verified to fail on the unpatched source first.
What I took from it
Disclosed agent contributions can clear the bar, and the bar shouldn't move. I reviewed that PR line by line, tested the claim it made about itself, and merged it on its merits. If it had been sloppy I'd have closed it, exactly as I would for a human. What made it reviewable was the disclosure plus an argument I could check.
The most valuable thing a project receives isn't a star — it's someone who reads the source. Mine had nine stars when a stranger sat down with it and found eleven real defects. No metric I track would have told me that much.
And the irony is doing work. A tool that exists to stop agents from trusting stale information had its own bugs found by an agent and a careful human, in that order.
The project is MIT and runs entirely locally: driftlint.
Curious how others are handling this: if a PR discloses that an agent wrote it, does that change your review, and should it?
Top comments (2)
The detail that matters is that the PR argued its scope claim instead of asserting it. You verified anyway and got the same answer, but you were checking a stated argument rather than reverse-engineering intent, and that is a different and much cheaper job. It's also the review standard I'd want from humans.
The
.gitprefix swallowing.githubis a lovely bug. Documented, typed, wired up, and unreachable since the day it shipped, which is the exact failure where everything looks healthy because nothing ever had a reason to report.Both halves land, and the second one has a sequel.
On the PR: what made it cheap was that the scope claim was falsifiable. "Shortcut reference links are out of scope," asserted, leaves a reviewer nothing to do but guess whether that was a decision or an oversight — and guessing at intent is unbounded work, because there's no state you can reach where you're done. With the reason attached, the job collapses to one question with an answer sitting in the repo. I still checked. It still cost less than reading the diff would have.
The caveat I'd add is that an argument can be confidently wrong, and a well-argued wrong claim is more expensive than an unargued one, because it recruits you into its frame before you've decided whether the frame is right. The trade is still overwhelmingly good — a wrong argument is at least a wrong thing you can point at.
On the .git prefix: I shipped the same bug again this week, ent.
build, out, bin, dist, target, obj get skipped as build outp a repo — except inside .claude/, where they're ordinary skill names. So a skill at .claude/skills/build/SKILL.md was never opened, and the scan came back clean, because the only thing with a reason to speak
was the file that never got read. I found it because I happe build and the test failed for the wrong reason.
Same shape as .git/.github: an exclusion whose blast radius s mental model, sitting in a reporting tool where didn't lookand nothing to report produce byte-identical output.
The generalizable fix, for anything that reports on a discovered set: assert the discovery separately from the judgment. Nearly all of my tests
asked "given this file, is the finding correct?" None asked ll?" There are assertions of the second kind now. They're theonly kind that can catch a surface going dark, because the first kind needs the surface to already be lit.