Result first: a made-up but plausible package name, fastjson-schema-validator-pro-xq7, returns "No matching distribution found" from pip and E404 from npm. A real one, requests, lists its versions. That one-second check catches a hallucinated dependency before install ever runs.
pip index versions <name> # lists versions if the package exists on PyPI
curl -s -o /dev/null -w '%{http_code}\n' https://pypi.org/pypi/<name>/json # 200 = exists, 404 = not on PyPI
npm view <name> version # E404 if the package isn't on the npm registry
Why this matters (peer-reviewed, USENIX Security 2025): Spracklen et al. generated 576,000 code samples with 16 code-generating LLMs. They found the average share of hallucinated packages was "at least 5.2% for commercial models and 21.7% for open-source models," with 205,474 unique hallucinated names. The authors' repository explains the risk: an attacker can publish a malicious package under a name that models tend to hallucinate.
So "it exists" is not the same as "it's safe." A hallucinated name may already have been registered by someone. Before you install a package an AI suggested:
- Check that it exists (the commands above).
- Check that it is the package you meant: open its project page and source repository from the registry, not from the AI's answer.
- Prefer packages you or your team already use, and pin versions in your lockfile.
Do this now: add a rule to your agent's instructions: "Before suggesting an install, run the registry check and show its output."
Sources
- USENIX Security 2025 paper page, Spracklen et al., "We Have a Package for You!": https://www.usenix.org/conference/usenixsecurity25/presentation/spracklen
- arXiv version: https://arxiv.org/abs/2406.10279
- Authors' code and data repository (risk explanation): https://github.com/Spracks/PackageHallucination/
From Penloom Studio's free AI lessons: penloomstudio.com
Top comments (0)