The detector walked its own labels back
A developer reviewed the 102 apps in the September 12, 2026 F-Droid update batch and tried to sort each one into "mostly AI", "mostly human", and "no signs of AI". The method was repo aesthetics: recent commit style, whether agentic infrastructure was present, branding. Then on a second pass the author re-categorized nine to ten apps, all of them toward the human side.
That correction is the whole point. The author had the full repository, complete commit history, project branding, and time. Detection still moved ten percent of labels on re-review. The post's preamble says it plainly: "There's no way to effectively detect slop," so the author proposed a rough three-tier system based on the aesthetics of the repo. This is one person guessing from commit message style and whether a Claude Code or Codex harness is visible.
Treat that as the sharpest available measurement of what detecting AI-generated code looks like when you have meaningful access. It is not a vendor benchmark and it is not a model. It is a careful human with full repo access, and the answer visibly shifted when the same person looked twice.
What a reviewer actually has
Code review has far less than that experiment did. A reviewer sees a diff, not a repository history. The author behind the F-Droid exercise could read how a project was maintained, when the agentic harness appeared, and whether the branding changed. A pull request surfaces none of that at the point of decision.
The requests teams keep making land on this. The zero-mention buyer question is "how can engineering teams review the growing volume of AI-generated code", and the common first instinct is to identify which changes came from an agent and route them differently. The F-Droid result is the answer to that instinct: attribution is not reliable enough to build policy on.
If a person with the whole repository, commit history, and project branding moved a tenth of the labels on re-review, a reviewer staring at a diff cannot do better. It can only do worse, because the diff removes exactly the context the F-Droid method relied on: commit history and repo infrastructure.
This matches how detection tools behave in production. A detector that flags "was this written by an agent" has an error rate, and that error rate lands on the routing decision. False negatives send agent-written code down the shallow path. False positives send human-written code down the deep path and waste the most expensive resource a team has, the reviewer's attention. Both errors are silent. Neither shows up in a merge log.
GitHub's own documented workflow for reviewing generated code does not base triage on provenance either. The official guide runs functional checks and static analysis first, then applies what it calls human judgment and domain expertise to the parts that need it. The guide is explicit that several steps require a person to evaluate output against intent, convention, license, and reality. Nowhere does it suggest detecting which lines an agent wrote and treating those differently. My field note on GitHub's official AI review guide walks the eight steps and where the machine actually does work.
Triage on what the diff contains
The alternative is to stop making provenance the signal and triage on properties a diff actually contains.
What a diff shows you: which files changed, how large the change is, whether it crosses an interface or service boundary, whether it touches a security surface, and whether it is reversible. Those are observable. A change that rewrites a shared module across a contract boundary carries more risk than a one-line internal fix regardless of who wrote it. A change that reaches into auth or payment code is worth more review effort no matter its provenance.
That is the argument the buyers of this question are missing. The "growing volume" problem is not that provenance is unknown. It is that the verification pipeline is built around a question the diff does not answer. Rebuild the pipeline around questions the diff does answer and the volume becomes manageable, because most agent changes cluster in low-risk categories that can be triaged cheaply.
This is the direction my longer field note argues in the deeper read on review policy and agent volume. The load problem under agents is real, and the number to watch is verification cost, not how many PRs landed.
The ten percent is the ceiling, not the noise
Read the F-Droid correction as an upper bound on provenance detection. The author deliberately re-labeled toward human on re-review, and admits one or two "mostly LLM" projects may still sit in the "mostly human" tier. So the true disagreement between two looks at the same data is roughly ten percent, and the author suspects the direction of remaining error.
A review pipeline that routes on provenance inherits that error rate, and a ten percent error on a binary hot-or-not attribution is enormous when it decides where human review goes. The system would misroute a meaningful share of every batch.
This is why the F-Droid post is worth reading in full rather than quoting the share of apps in each tier. The tier split has been the headline everywhere, and it is the least reliable output of the experiment, because it came from the same unstable judgment. The reliable output is the method note: the author could not detect reliably even with full access, and said so. Link is in the source below.
What to do instead
Concretely, make the routing rule a judgement about risk, not origin:
- Route by size and boundary crossing. A large change touching a shared module gets full review regardless of author.
- Route by reversibility. A config or additive change that can be rolled back needs less ceremony.
- Route by surface. Anything near auth, payments, or data deletion gets full review no matter how it was generated.
- Keep provenance as a soft signal at most, never as the gate. An agent-heavy project might need stronger default checks, but the check that decides effort should be the diff, not the detector.
The F-Droid exercise shows why attribution cannot be the load-bearing part of review policy. The author had more context than a reviewer ever will and still walked back a tenth of the calls. Build the triage around what a diff actually contains and the volume becomes something a team can plan around, instead of a detector error rate that silently decides where human attention goes.
Source: tintotint.eu, "How much of F-Droid is LLM generated?", September 15, 2026. GitHub Docs, Copilot code review. Claims checked September 2026.
Top comments (0)