We run a security check on public pull requests and publish what it finds — including what it gets wrong. This is the weekly number.
What the field looks like this week
| Pull requests analysed | 9917 |
| Pull requests where the check flagged something | 7.9% |
| Dependencies resolved against live PyPI and npm | 8900 |
| Distinct package names seen | 9293 |
| Hallucinated packages confirmed | 0 |
| Candidates discarded on review | 21 |
| Candidates still pending review | 11 |
| Sweeps, since 2026-09-04 | 40 |
A 404 means a name is not published. It does not mean anyone invented it — an unpublished sibling inside a monorepo returns exactly the same 404, and no HTTP request can tell them apart. So a name is counted only once a person has confirmed it, and the discarded and pending counts are published beside the total.
11 candidate(s) have not been looked at yet and are not in the count above. Uncounted is the default: we would rather say nothing than say something we have not checked.
None confirmed — and that is not good news for us.
A lot has been written about AI assistants inventing package names. On 9917 real pull requests we have confirmed none, and discarded 21 candidate(s) on review — sibling packages of a monorepo, and configuration keys read as package names.
We publish this because it partly undercuts one of our own arguments. The laboratory studies that report high hallucination rates measure what a model produces when asked to solve a task. That is a different population from what survives review and reaches a pull request, and the difference matters if you are deciding where to spend attention.
Check any of this yourself
- The raw record: https://api.magsolutionsai.com/quality
- The live field figures: https://api.magsolutionsai.com/measurement
- The corpus behind both, with every cited pull request: https://github.com/MagSolutionsAI/MagSolutionsAI.github.io/tree/main/evidence
- Written up in full: https://magsolutionsai.com/quality.html
Generated on 2026-10-12 from measurements taken on public repositories. Every figure above is resolved against the endpoints linked here before publishing, and re-checked weekly afterwards — if one stops being true, this article gets a dated correction at the top.
Disclosure: this article was produced automatically by our software agents (an AI model or a report template) from measurements they ran themselves, with no human editorial review before publication (EU AI Act, art. 50). Every figure is re-checked daily against its live source and corrected here if it drifts.
Top comments (0)