A 404 is the easy half. Scoring a package that actually exists is where we got it wrong.

Someone left this on my last post:

I'd also treat the "404 means stop" rule as only the first gate. A package resolving successfully doesn't establish that it is trustworthy. For newly registered packages that closely match common model-generated names, provenance and publication history become useful signals too.

The deeper lesson is that agents need pre-action verification, not just error recovery. Once the package is installed, the security decision has already been made.

Both correct, and the first one sent me to look at our own data. We publish a list of package names that LLMs invent. 1,885 of the 1,911 entries simply do not exist — those are easy. 26 of them answer 200.

I checked those 26 by hand. Almost all of them were wrong.

The one that stung

name        : opentelemetry-configuration
author_email: OpenTelemetry Authors <[e-mail]>
repository  : github.com/open-telemetry/opentelemetry-python
summary     : OpenTelemetry Python Declarative Configuration (experimental)

We had the official OpenTelemetry Python package on a public list of slopsquatting candidates. Not because anything about it is suspicious — because it was 41 days old when we looked, and our rule said 14–60 days is medium.

The same rule caught cdx-rs, which is a cd replacement for the terminal:

crate  : cdx-rs
desc   : A fast and interactive cd command alternative for directory navigation
repo   : github.com/Cinnamon-jp/cdx
created: 2026-08-30

A model suggested cdx-rs as a Rust crate for parsing CycloneDX SBOMs. The name was already taken by something unrelated, published five days earlier. We scored it high.

Age was a proxy for a question I wasn't asking

The reasoning behind an age window is sound: an attacker who watches what models hallucinate registers the name afterwards, so a fresh registration under a hallucinated name is suspicious.

Read that sentence again and the actual signal is right there. Afterwards. Not "recently" — after the hallucination.

I had been measuring the wrong interval. created_at against today tells you how new a package is, which is a fact about the calendar. created_at against the first time any model produced that name tells you whether the registration could have been a response to it, which is the thing I actually wanted to know.

And we already stored it. Every finding carries hallucinated_by, with a first_seen per model, because the dataset is meant to be evidence rather than a count:

{
  "model": "gemini-2.5-flash",
  "task": "Rust crate for parsing CycloneDX SBOM",
  "first_seen": "2026-09-04T..."
}

cdx-rs was published 2026-08-30. First sighting 2026-09-04. It predates the hallucination by five days — nobody registered it in response to anything. The new rule is four lines:

if first_hallucinated and info.created_at < first_hallucinated:
    return "low", "registered before the name was ever suggested"
age_days = (now - info.created_at).days
if age_days < 14:
    return "high", "registered after the name was suggested"
if age_days < 60:
    return "medium", "young"
return "low", "established"

Both false positives drop to low and come off the list. A name registered after the first sighting keeps its score, and now the status string says why, which it never did before.

This is not airtight. Two people can invent the same obvious name independently, and "registered after" is correlation. But it is a far better question than "is this new", and it costs nothing — the data was already in the row.

npm's 200 means more than one thing

Three entries were labelled registered but never published: native-websocket, node-ics, sql-injection-detector. That status is meant for a name someone claimed and left empty — dangerous precisely because an existence check passes while the holder can publish anything under it later.

Except:

curl -s https://registry.npmjs.org/native-websocket | jq '.time'

{
  "created": "2020-11-29T...",
  "modified": "2020-11-29T...",
  "unpublished": { ... }
}

These were published and then taken down — in 2016, 2020 and 2022. npm leaves a tombstone: 200, no versions, and time.unpublished set. My check was versions == 0, which cannot tell an empty squat from a grave.

It matters because the two have different mechanics. An empty registered name is held by someone who can publish to it today. A tombstone is a name that was real — which is plausibly why a model learned it in the first place — and whether anyone can claim it depends on the registry's unpublish policy, not on the holder's intent. They are medium now, with a status that says what they are.

If you are doing this yourself: don't read versions alone. time.unpublished is the field.

A score is a statement about a moment

The last problem had no clever cause. opentelemetry-configuration was 41 days old when we scored it. It is 77 days old now. Nothing re-ran the classifier, so it sat on a public list for a month under a judgement that had expired.

Every finding that resolves is now reclassified on each cleanup run, and anything that drops to low is withdrawn — kept at its URL with a notice, because other people link to these IDs, but out of the list and out of the API.

The finding here is boring and general: if your scoring function reads a clock, something has to re-run it. A pipeline that only ever classifies on insert will accumulate verdicts that were true once.

What I still don't have

The comment's other word was provenance, and that is the honest gap. We score the name and the registry's timeline. We do not check who published it or from where.

Both big registries now expose this:

• npm publishes provenance attestations — a signed statement that a package was built by a specific GitHub Actions workflow from a specific repository.
• PyPI supports PEP 740 attestations on the same idea.

"Registered after the hallucination, by an anonymous account, with no attestation" is a very different package from "registered after the hallucination, built from a public repo with 400 stars by a workflow you can inspect." Right now we score those identically, which is the most useful thing anyone has told me about this project in a while.

That is the next signal. Until it lands, the list says what it can support and not more.

---

The data is here — CSV and JSON, daily, CC BY 4.0. The gate is one call and needs no key:

curl -X POST -H "Content-Type: application/json" \
  -d '{"packages": ["pkg:npm/fast-jsonwebtoken"]}' \
  https://vdb.ai.kr/v1/ai/check-packages

Which is the commenter's last point, and the reason any of this exists: the verification has to happen before the install, not after the error.

Comments

[Bhavin Sheth]

The shift from “is this package new?” to “was it registered after the hallucination?” is a really strong improvement. I also like the distinction between an empty package name and an npm tombstone—small registry details like that can completely change the security assessment.

[jj1423]

Thanks! And your second sentence aged well — I hit another one the next day.

crates.io treats - and _ as the same name, so a lookup for cdx_rs comes
back as the crate cdx-rs — the one I'd just withdrawn. I was storing a
finding per spelling, so the twin sat there while I congratulated myself for
removing it. Now I just ask the registry what it calls the thing.

The provenance check landed too (npm attestations, PyPI PEP 740). It matches
zero rows right now — everything left that resolves is an empty name, a
tombstone, or crates.io. Which tells you something about the bucket.

[Tae Kim]

The "score is a statement about a moment" section is the one that's staying with me. I had essentially the same gap with entity merge scores: the classifier ran on insert and I never set up periodic re-scoring, so half a year of flags in the trusted bucket were stale. We only caught it because a downstream mismatch looked wrong enough to trace back. I'm now curious what the right re-run cadence looks like for something like your false-positive cases, where the score improved as a package aged but nothing automatically promoted or demoted it.

[jj1423]

Same gap here, honestly — the re-scoring existed but only ran when someone ran it. As of today it's a daily job, ahead of the dataset publish, so the site and the published dataset carry that day's verdicts.

How I picked "daily": find the narrowest window your score depends on and re-score well inside it. Ours flips at 14 and 60 days, so a day of lag is fine. Inputs that change outside your clock (for us, a scope gaining an owner or a build attestation appearing) get picked up by the same sweep.

What I haven't done is give each verdict its own expiry date. At our size a full sweep is cheap, so it wasn't worth it yet.