DEV Community: jj1423

Zero CVEs. but found 1,249 risky paths!

jj1423 — Thu, 22 Oct 2026 05:40:10 +0000

What a vulnerability database can't tell you

Every dependency scanner works the same way: take the packages in your lockfile, look each one up in a vulnerability database, report the matches. That is useful, and it has a hard limit. It can only report what someone has already found, written up and published.

A whole class of risk never gets there. A well-maintained library with no CVE can still be where your service breaks, because of what your code hands it:

a URL from a request passed to an HTTP client, reaching an internal address,
an uploaded archive handed to a parser that expands it without limit,
a user-supplied path joined into a file operation,
input that becomes a shell command three packages down the chain.

Nothing is wrong with the library in any of these. So nobody will ever file an advisory, and every scanner that looks at packages will keep saying "safe".

We wanted to find that risk anyway. So we built vdb harden, which follows the call instead of the name: it abstracts your code locally (no source leaves your machine), joins it with your lockfile and with a summary of where each package version sends its arguments, and traces input from your code, through the dependency chain, to whatever dangerous operation it reaches. No CVE needed; the path is decided from the code. On our own service it led us to a way to stall the API with two file uploads that no advisory will ever describe. (That story, and why following the call is hard, are written up separately.)

It worked, which created a second problem.

Too many true findings

On one of our own projects, 82 packages deep, the analysis came back with 1,249 paths from the code to something dangerous. Every one was a fair finding. None of them told anyone what to do on Monday morning. A list of paths is not an order of work.

So we built the layer on top. It answers three questions:

How risky is this project, as one number you can track?
When something changes upstream, does it matter to us?
What should we fix first, and what will that buy?

One graph for every way in

The decided paths become edges in an attack graph: call sites, packages and sinks as nodes. The key change is what counts as an entry point. Not only request handlers. Six kinds go into the same graph, on the same scale:

Entry	Example
Input to your code	a request argument, a CLI argument, an environment variable, stdin
A package name that does not exist	an AI agent added a dependency it invented
A newly registered package	a name that appeared on the registry days ago
An install script	a package that runs code when it is installed
A tool server's scope	an MCP server that declares shell or file access
A published CVE	weighted by whether it is known to be exploited

So "an agent added a dependency that does not exist" and "an exploited advisory is reachable from your upload handler" land in one ranking, not in two tools that never talk to each other. And that is where vulnerability data comes back in: as one kind of entry among six, not the whole picture.

Scoring a path, and a project

A path's score multiplies four things:

The entry's threat probability. A request argument is attacker-controlled by definition; an environment variable mostly is not. A CVE on the known-exploited list counts in full; otherwise its EPSS score is used.
A confidence per hop, lowered wherever the engine had to assume propagation.
A risk factor per package, from registration age, release history, a tool server's recent scope expansion.
The sink's impact: command execution and deserialization at the top; file paths and network requests below; regexes, decompression and global state lower still.

Paths combine into a project score the way independent risks do (noisy-OR).

The part we care about most is coverage. If half your packages have no summary yet, a low score means "we couldn't see", not "you're safe". So the score comes with the share of the graph that was analysed, and when that share is too low it becomes an interval: decided paths give the lower bound; assuming every undetermined path reaches the worst sink gives the upper.

Real output, for a one-file Flask app with five pinned dependencies:

$ vdb harden app.py --manifest requirements.txt
5 dependencies on the graph · 3 analysed · 2 without a summary yet · taint reached 3
Project risk: 0.65–0.75  (interval — analysis incomplete; 3 of 5 dependencies have a summary, 4 undetermined path(s))
  low        0.32      code_input request > requests@2.34.2 > net_request  in app.py
  high       0.25      code_input param > stdlib > cmd_exec  in app.py
  low        0.16      code_input request > requests@2.34.2 > urllib3@2.7.0 > net_request  in app.py
  low        0.06      cve PYSEC-2026-4175 > urllib3@2.7.0 > net_request  in app.py
  low        0.06      cve PYSEC-2026-4177 > urllib3@2.7.0 > net_request  in app.py
  low        0.06      cve PYSEC-2026-4176 > urllib3@2.7.0 > net_request  in app.py
  undetermined: 4 path(s) end in a package without a summary — most often idna@3.11 (2), certifi@2026.7.22 (2)

A request reaching an HTTP client, a parameter reaching a shell, and three advisories on the pinned urllib3, in one list, on one scale.

Keeping it current

The graph is stored and re-scored when an advisory is published, a missing summary gets built, the registry changes, or a tool server's scopes move. Each event re-scores only the paths it touches, and you're mailed only when your project's score actually moved, not for every CVE in every package you include.

We learned one rule the hard way: an engine upgrade re-decided paths for projects analysed days earlier, the difference looked like new findings, and people got mail about nothing. A new version of our analysis is not a change in your code, so it now re-baselines silently.

What to fix first

Every candidate fix is applied to the graph counterfactually (upgrade, harden one call site or all that reach a sink, remove a dependency, change execution policy), costed (version jump, broken constraints, call sites to edit), and ranked by risk reduction per cost. The ranking ends with the combination that reaches a target.

$ vdb harden app.py --manifest requirements.txt --remediate
Remediation ranking — risk 0.65–0.75, target 0.20
   #  action                                                   Δrisk  cost  value  closes
   1  harden app.py:6                                           0.26  0.20  0.219  2 (worth 0.43)
   2  egress-allowlist on net_request sinks                     0.17  0.40  0.125  0
   3  remove requests                                      0.19–0.26  0.80  0.106  2 (worth 0.43)
   4  harden app.py:9                                           0.12  0.20  0.097  1 (worth 0.25)
   5  upgrade urllib3 2.7.0 → 2.8.0                        0.12–0.16  0.40  0.083  7 (worth 0.31)
  ...
  Combination: #1 → 0.39–0.55, #2 → 0.32–0.51, #4 → 0.09–0.34  (reaches the target; 3 step(s), cost 0.80)

Hardening one request handler leads: one edit closes both paths through requests. Upgrading urllib3 closes more paths but buys less. Three steps reach the target.

Five popular open-source projects

Our own code is not a fair test, so we ran it on open-webui, AutoGPT's platform backend, the full-stack FastAPI template, paperless-ngx and saleor. These measure our analysis, not those projects; we report no vulnerabilities in them.

Three things broke:

Derived values scored as if an attacker typed them. In cmd = "convert " + name, only explicitly read values had an origin, so cmd fell back to "attacker input". Now origins flow through assignments, and a value with no trail is weighted at half a request's. The FastAPI template went from "1.00, saturated" to 0.26–0.52 with one real path.

Every large project scored 1.0. Noisy-OR saturates once a few hundred paths are open. So the score now also carries open paths, risk mass (summed path scores: 7.5 for paperless-ngx, 109 for AutoGPT's backend) and the highest path per sink class, and the report leads with those when saturated.

The ranking chased noise. Under saturation, rounding-level changes in the score were outranking a candidate that closes 180 paths for one edit. Below a noise threshold, candidates are now ranked by the risk mass they close per cost.

Our own projects taught the rest: ranking 86 candidates over 1,249 paths went from 310 s to 0.4 s by re-scoring only touched paths; four of seven projects couldn't be analysed until unpinned requirements became "pin this to unlock analysis"; and two --exclude patterns, for a backup and a vendored copy, cut 9,336 call sites to 3,952.

What the score is not

Not a probability of being breached. It's an ordering: this project against its past, this fix against that one. The weights are written-down judgements, not measurements.
Only as wide as the analysis. That's what the coverage figure and the interval are for.
Saturated on large projects. Read open paths and risk mass there.

Vulnerability data tells you what is already known about your packages. Following the call is our attempt at the part that will never be known that way, and the score is how a list of those findings becomes a plan.

Block bots, not users: rate limiting an open API

jj1423 — Tue, 13 Oct 2026 01:39:15 +0000

If your API can sit behind a login, rate limiting is easy: meter the account, done.

Ours can't. AI coding agents call it on a first visit, without signing up. Search engines and AI crawlers are supposed to read our pages. And we get scrapers, signup bots and cloud swarms like everyone else. Over four months we blocked the wrong people, let the wrong ones through, and once blocked Googlebot from our own pages.

These are the eight patterns we ended up with. Each has the code shape and the mistake that taught it.

1. Meter the identity, fall back to the IP

The bug: our first limit was 60 requests a minute per IP. One AI platform's shared egress crossed it with ordinary traffic, and every user behind that address was blocked at once. Corporate NAT and mobile carriers do the same thing: thousands of people, one IP.

The pattern: pick exactly one metering key per request. An API key if there is one, the IP only if there isn't.

def meter_key(request) -> str:
    principal = authenticate(request)          # None for anonymous callers
    if principal:
        return f"user:{principal.user_id}"     # never refused because of its IP
    return f"ip:{client_ip(request)}"

Two details that matter:

A keyed caller is never blocked by an IP rule. Otherwise a single bad neighbour on a shared egress locks out your paying users.
Make a block last its full window. If crossing the per-minute limit only refuses until the minute rolls over, a bot learns to send exactly the limit. A block of 10 minutes, an hour, or a day (one per window) doesn't reward that.

2. Separate reading from computing

Not all routes cost the same, and not all bots are unwelcome. We want crawlers on our pages: that is how people find us, and how AI answers end up citing us. We don't want them running package scans.

The pattern: classify routes, then decide per class.

FEATURE_ROUTES = {"/v1/query", "/v1/search", "/v1/ai/check-packages", "/v1/sbom/scan"}

async def bot_gate(request, call_next):
    if is_bot(request.headers.get("user-agent", "")):
        if request.url.path in FEATURE_ROUTES and not is_internal(request):
            return JSONResponse({"error": "crawler_not_allowed"}, status_code=403)
        return await call_next(request)        # bots reading pages: never metered
    return await call_next(request)

The bug that is_internal fixes: our page server renders pages by calling our own API, and forwards the visitor's user agent so analytics see the real client. So when Googlebot loaded a page, the page's own API call was refused as "a crawler calling the API". Browsers got 16,129 bytes; Googlebot got a 6,147-byte empty shell. A site: search for our domain returned one page.

The exemption is for calls authenticated with an internal service token, not for anything a user can send. A user's API key doesn't buy it.

Check this in yours: if you block by user agent, grep for every place your own infrastructure forwards one.

3. Classify bots precisely, or your dashboard lies

You can't decide what to block without knowing what you're looking at.

The bug: one SEO crawler wasn't on our bot list, so it was counted as a browser. A four-hour burst of 11,527 requests from a single IP looked like a surge of human visitors.

The pattern: a list of known bots for labelling, plus a catch-all so nothing unlisted falls into "human".

import re

AI_CRAWLERS  = ("gptbot", "claudebot", "perplexitybot", "ccbot")
SEARCH_BOTS  = ("googlebot", "bingbot", "duckduckbot")
CATCH_ALL    = re.compile(r"(?<!cu)bot\b|crawler|spider|scraper|headlesschrome", re.I)

def classify(ua: str) -> str:
    u = ua.lower()
    if any(b in u for b in AI_CRAWLERS):  return "ai-crawler"
    if any(b in u for b in SEARCH_BOTS):  return "search-bot"
    if CATCH_ALL.search(u):               return "other-bot"
    return "browser"

The (?<!cu) is real: a phone brand, CUBOT, puts "cubot" in its browser's user agent, and bot\b would have counted those people as bots.

Split AI crawlers from SEO bots on purpose. An AI company indexing you is a signal worth seeing; most SEO crawlers are noise.

4. Use robots.txt for noise, not for defence

Some crawlers cost you and send nobody back. On one day a single SEO crawler made 11,548 of about 14,400 requests to our site.

robots.txt is the right first response: it's free, and well-behaved crawlers obey it.

User-agent: SomeSeoBot
Disallow: /

User-agent: *
Allow: /
Disallow: /account
Crawl-delay: 1

It's not a security control. It only stops bots that choose to read it, which is why pattern 2 doesn't rely on it. Use it to cut cost, and check the next day's traffic to confirm it worked.

5. Group cloud swarms by who owns the network

The bug: roughly a thousand cloud IPs, each making only a handful of calls. Every one stayed far below its own limit. Together they were most of our anonymous load.

The pattern: for anonymous traffic from datacenter ranges only, add a second meter keyed by the network owner (the organisation or ASN that announces the IP), with a daily ceiling.

def anon_meter_keys(ip: str) -> list[str]:
    keys = [f"ip:{ip}"]
    info = ip_info(ip)                         # your GeoIP/ASN source, cached
    if info.is_hosting:                        # cloud/datacenter, not home or mobile
        keys.append(f"org:{info.org}")         # one budget for the whole provider
    return keys

Keep it narrow: home and mobile connections don't get the org meter, and keyed callers never do. And make it a soft cap that refuses for the rest of the day, without placing a block that outlives the burst.

6. The signup form is your biggest door

You can tune API limits for days while the form that hands out an unlimited key has none. Ours did. What came through:

Six accounts in six hours, each a real person's Gmail address with dots scattered through it. Gmail ignores dots, so j.o.h.n@gmail.com is john@gmail.com: six strings, six strangers' inboxes from a harvested list.
Nineteen accounts over a day, rotating mail providers. Zero API calls.
Later, 136 sign-ups from 118 IPs. Zero verified.

That last line is the real lesson. The bots weren't using the keys. The harm was verification mail sent to people who never asked. A few spam reports on a young sending domain, and your mail stops arriving for the people who did sign up.

Patterns:

PLUS_TAG = {"gmail.com", "googlemail.com", "outlook.com", "hotmail.com",
            "yahoo.com", "icloud.com", "fastmail.com", "proton.me"}
DOTLESS  = {"gmail.com", "googlemail.com"}

def canonical_email(addr: str) -> str:
    """For uniqueness checks only. Always mail what the person typed."""
    local, _, domain = addr.strip().lower().rpartition("@")
    domain = "gmail.com" if domain == "googlemail.com" else domain
    if domain in PLUS_TAG:
        local = local.split("+", 1)[0]
    if domain in DOTLESS:
        local = local.replace(".", "")        # dots are only meaningless at Gmail
    return f"{local}@{domain}"

Store the canonical form in its own column with a UNIQUE index. Don't fold dots globally: at Outlook, first.last and firstlast are two different people.
Give signup the same per-IP budget as any other way to get a key. Ours had one on "request a key by email" and none on the form next to it.
Cap verification mail service-wide, so no burst can turn your domain into someone's mailer.
Verify with a button, not a link. Corporate mail scanners open every link before the person does, so a GET link "verifies" accounts nobody looked at. A POST from a page with a button doesn't fire on a scan.
Add cheap friction to the form, and be honest about it: it raises the cost, but a patient headless browser still passes. The mail cap is what limits the harm.

7. Make every 429 actionable

A bare 429 Too Many Requests makes a human wait and an agent retry, guess, or skip the check. For a security API, "carry on without checking" is the worst possible outcome.

The pattern: say why, until when, and what to do, in fields a program can read.

{
  "error": "rate_limited",
  "axis": "ip",
  "expires_at": "2026-10-13T09:10:00Z",
  "agent_action": "REFUSE",
  "because": "Anonymous hourly budget used up. Proceeding unchecked is not safe.",
  "next_step": "Ask the user for an email address and request a free key.",
  "signup_url": "https://example.com/signup"
}

Plus a Retry-After header. If your callers include AI agents, add one explicit line: do not invent a key, do not proceed without the check.

And fail open on your own outages. If the rate-limit store (Redis, for us) is down, allow the request. Under-blocking for five minutes beats refusing every legitimate user because a cache restarted.

def check(key: str) -> bool:
    try:
        return redis_window_ok(key)
    except RedisError:
        return True     # our outage is not the caller's fault

8. Measure before you block, and after

Most of our bad calls came from numbers that were quietly wrong:

Don't audit blocked requests as normal traffic. We did, so a bot hammering a block kept the 429 count climbing, and the block looked like it wasn't working.
Make every report count the same way. Our daily digest and our dashboard disagreed on the day one crawler made 80% of requests.
Know which limits are live. Our code defaults said one thing; a deployment config overrode them, and production ran at 10 requests a minute per anonymous IP while our docs promised much more.
Measure what closing a door costs. We once put every endpoint behind a key. Seven days later: one new account, twelve calls. The users we built for, agents evaluating a tool on a first visit, left at the signup page. We reopened two endpoints with a per-IP quota, counted in the unit that actually costs us (packages checked, not requests, after a 1,168-package run died at request 20).

The checklist

[ ] One metering key per request: account if keyed, IP if not.
[ ] Blocks last their whole window.
[ ] Routes split into read and compute; bots allowed on one, refused on the other.
[ ] Your own servers exempted by an internal token, never by user agent.
[ ] Bot classification with a catch-all, AI crawlers labelled apart from SEO bots.
[ ] robots.txt for costly crawlers, verified in the next day's traffic.
[ ] Anonymous datacenter traffic also metered per network owner.
[ ] Signup with an IP budget, canonical-email uniqueness, a service-wide mail cap, button verification.
[ ] Every 429 says why, until when, and what to do next.
[ ] Fail open when your own limiter is down.
[ ] Blocked requests counted once, every report counting the same way, documented limits equal to deployed ones.

None of this makes an open API bot-proof. It makes the bots cost more than they're worth, while the people and agents you built it for get through.

We opened 1,332 AI-built repositories. Five of their dependencies do not exist, and anyone can register them.

jj1423 — Thu, 08 Oct 2026 11:38:37 +0000

Most of what is written about slopsquatting measures the model: ask it for code, count how often it names a package that does not exist. We wanted the other end of the pipeline. Of the code people actually built with AI tools and published, how much still depends on a package that is not there?

A dependency that does not exist is not just a broken install. Its name is free, and whoever registers it first decides what that install runs from then on.

What we looked at

Two samples, both taken on 2026-10-03, both with a control group.

Popular projects. Every GitHub repository linked from a Show HN post since January 2025 with more than 20 points: 673 reachable repositories.

Recent, unpopular projects. Repositories created since January 2025 with at most 50 stars, found by the files AI tools leave behind: a Lovable build marker, Bolt's project prompt, a CLAUDE.md, a .cursorrules file. The control group is repositories created in the same period with no such trace.

A repository counts as AI-built if it carries any of these: an agent instruction file (CLAUDE.md, AGENTS.md, .cursorrules and their relatives), commits signed by an AI tool (Co-Authored-By: Claude, Co-authored-by: Copilot and others), a Lovable or Bolt marker, or the author saying so in the Show HN post.

	AI-built	Control
Popular (Show HN)	412 repositories, 37,158 dependency entries	261 repositories, 7,655 entries
Recent, ≤ 50 stars	920 repositories, 22,966 dependencies added beyond the platform template	303 repositories, 6,494 entries

Every dependency in every package.json, requirements.txt, pyproject.toml, Cargo.toml and go.mod was looked up live in npm, PyPI, crates.io and the Go module proxy — 8,240 distinct names in the first sample and 6,565 in the second.

A side note on the control group: of 600 ordinary repositories we drew from the same period, 297 (49.5%) already carried an AI tool's trace and had to be moved out. Half of new code on GitHub is touched by these tools now.

What we found

A name that is missing from the registry is usually not a phantom. Most were packages that live inside the same repository, private packages under a company scope, packages from a different registry (Deno's JSR, Unity's), or placeholder names in examples. We removed each of those by hand. What was left:

	AI-built	Control
Popular	1 repository, 1 name	1 repository, 1 name
Recent, ≤ 50 stars	2 repositories, 4 names	0
Names anyone can register today	5	1

The five, without the repositories they are in:

Three npm packages in an app built with Bolt, each written as "name": "^1.0.0". None of the three has ever existed on npm, and the app's code imports one of them. The repository's lockfile does not contain them: the install that would have failed was never run, so nothing caught them.
A Python package pinned to ==5.3.0 in the backend of an app built with Lovable. The package has never existed, so neither has version 5.3.0. The code imports a hardware library by its module name, and the invented package name is a plausible guess at the real one.
A Python package listed in an example's requirements.txt in a popular project. Nothing in the example imports it. It has the name of the application the example is about.

The control group's one is the same shape as the last: a ROS package that is installed with the system package manager, never from PyPI, listed in requirements.txt with an exact version.

Two more were invented but cannot be taken: a @types/ package (only the DefinitelyTyped maintainers can publish under that scope) and a standard-library module (PyPI refuses those names).

What the numbers do and do not say

They say phantom dependencies are rare in published code — about one AI-built repository in four hundred — and that they exist, with version numbers attached, in code people put online.

They do not say AI-built code is worse than other code. Two repositories against none in the unpopular sample and one against one in the popular one are counts too small to compare. The samples come from search rankings, not a random draw.

The repository is the end of the process

A published repository is a finished product. Every phantom package that broke a build on the way there was most likely noticed and removed before anyone pushed: the author ran the app, the install failed, the name went. What we found is what survived, and every case we found survived for the same reason — it sat where nobody ran an install. An example folder. A backend nobody deployed. An app whose lockfile shows its install was never completed.

So these counts are a floor, not a rate. We did not measure what happened before the commit, but two things point the same way. When we put ordinary coding tasks to Claude, GPT and Gemini, they recommended 87 package names that do not exist. Academic work that measures the same thing at generation time (Spracklen et al., 2024) found invented package names in a substantial share of generated code, more for open models than for commercial ones. Between what the model writes and what gets pushed, someone has to catch the rest — usually by running the install and watching it fail.

That makes the dangerous moment the one before the failure. The install fails because nobody has registered the name yet. The day someone does, the same command succeeds, quietly, and runs their code. A phantom name in a published repository is a standing invitation. The three in the Bolt app have been one since March 2025.

Checking at the moment of adding

The place to stop a phantom package is when the agent adds it — before the install that fails today and succeeds once the name is taken. All five names above, sent to VDB's package gate, come back REFUSE:

POST /v1/ai/check-packages
{"packages": ["pkg:pypi/<one of the five>"]}

"agent_action": "REFUSE"
"because":      "this name does not exist on the registry — likely hallucinated, and attackers register such names"

The gate answers up to five packages without a key; see Connect to put it in front of an agent.

How we did it

Candidates: Show HN through the Hacker News search API; recent repositories through GitHub search, by the files each tool leaves.
AI traces: read from a shallow clone of each repository — the file tree and the last 200 commit messages. No code was run.
Dependencies: parsed from the manifests, leaving out workspace, file, git and URL specifiers, aliases (resolved to the real name), the repository's own packages, and scopes its .npmrc sends to a private registry. For Lovable and Bolt, dependencies present in at least half of that platform's repositories count as the template and were left out.
Lookups: live, against each registry, with the probe VDB's gate uses.
Every flagged name was checked by hand, including the lockfile and the code that would import it.

We told the maintainers of the four repositories involved on 2026-10-03, before this post, and none is named here. We did not register any of the names ourselves.

Beyond the package name: why following the call is the hard part of supply-chain security

jj1423 — Mon, 05 Oct 2026 09:23:34 +0000

Most supply-chain security checks the package: does it exist, is it new, is there a CVE against this version. Those checks answer is this safe to install?

They can't answer is the way my code uses it safe? A clean library can still be where your service breaks: a request URL passed to an HTTP client, an uploaded archive handed to a parser, a user path joined into a file operation. Nothing is wrong with the library, so no advisory will ever be filed, and every package-level check will keep saying "safe".

The obvious answer is to follow the data: from where input enters your code, through the call into the dependency, down to wherever it lands. We have been trying to do that for Python. It is much harder than it sounds. These are the problems that actually bit.

1. Who controls this argument?

Everything depends on knowing whether a value comes from outside. Statically, you mostly can't. The safe assumption, every function parameter might be attacker-controlled, turns your own configuration into "input": we had a storage client built from environment variables flagged as nine dangerous paths, twice.

It is worse for libraries, where every public parameter is outside input by definition. On ten popular open-source projects, more than half the paths we reported rested on that assumption rather than on a traced flow.

2. The direction of the danger matters

httpx validates URLs with regular expressions it wrote itself. A naive rule ("user input reaches a regex") flagged every httpx call we make, including one where the argument was a timeout in seconds. In our first run on our own service, 25 of 39 reported paths were this one mistake.

A regex is dangerous when the attacker writes the pattern, not when their string is matched against yours. Every sink needs rules like that: which argument, which direction, which mode. ZipFile(upload) reading is a decompression risk; ZipFile(target, "w") writing is not.

3. Right place, wrong reason

Our one real finding was a spreadsheet upload that could expand to two million rows in memory. The analysis pointed at the correct line and described it as a file-path operation, a regex and global state. None of those was the problem. The real category, decompression amplification, didn't exist in our sink list until we went looking for it.

Pointing at the right line is not the same as understanding it, and a wrong explanation sends a reviewer the wrong way.

4. Summarising every package, at every version

To follow an argument into a dependency you need to know where that package sends it, for the exact version in the lockfile. Python makes that hard: calls on objects, dynamic dispatch, thin wrapper modules, native code that can't be read at all. Every change to the summariser means rebuilding summaries for every version of every package, and results shift when it does.

5. The standard library is not a dependency

subprocess.run(cmd), eval, pickle.loads are the most dangerous calls in Python, and they aren't in your lockfile. An analysis built on the dependency graph skips them unless it handles them separately. Then the noise problem from #1 returns: on 39 real releases, path and regex calls produced 140 of 283 standard-library findings, for the lowest-severity class.

6. Paths explode

A tainted value can reach a sink along many routes through a dependency tree. Without a budget the analysis never finishes; with one, it stops early on exactly the large projects where it matters. A fixed budget cut eight of twenty real cases short before they reached the answer.

7. What counts as "right"?

Our first benchmark, calls we wrote from library docs, scored 21 of 21. That measured nothing. The useful test is real advisories, where an application really misused a dependency and someone filed a report: would the analysis have pointed at that line, for that reason? Against twenty of those, it does about half the time. Two of our own benchmark cases turned out to have the wrong answer key.

Where that leaves it

Following the call is where supply-chain security has to go, because it is the only way to see risk that no advisory will ever describe. But today it is a way to decide where to look, not a verdict. The hard parts are the ones above: knowing who controls a value, knowing which direction is dangerous, and measuring against real misuse rather than examples written to pass.

If you have worked on any of these, especially telling outside input from configuration statically, I would like to hear how you approached it.

Our slopsquatting detector had three ways to say "safe". None of them needed evidence.

jj1423 — Fri, 02 Oct 2026 03:57:33 +0000

Ask a language model which package to use and, now and then, it names one that does not exist. If someone registers that name first, the next developer — or the next coding agent — who follows the same suggestion installs whatever they put there. That is slopsquatting.

We collect those names and score each one. A name that returns 404 is easy: it is a target. The hard case is a name that does resolve. Is it the project the model meant, or someone who registered the name after models started suggesting it?

Our answer starts with age:

age_days = (now - info.created_at).days
if age_days < 14:
    return "high"
if age_days < 60:
    return "medium"
return "low"

Crude, but honest. A package that appeared last week and matches a name models keep inventing deserves suspicion.

The trouble was everything we put above that ladder. Over the past few weeks we had added exceptions — rules that let a young package off — each one to fix a specific false positive. This week a reader went through them in a comment, and by the end of the next day three were gone.

1. "It has a signed build statement"

npm provenance and PyPI's PEP 740 attestations record that a package was built by a named workflow in a named repository, through a Trusted Publisher flow. The registry produces the statement at publish time, so the package cannot just claim it in its manifest. That felt like strong evidence, so we added:

if prov is not None and prov.attested and prov.repository:
    return "low", f"built from {prov.repository} with a signed build statement"

Read what the statement actually says. It binds the artifact to a repository. It says nothing about whose repository. Anyone can create a GitHub repo, configure a Trusted Publisher for it, and publish a squatted name with a perfectly valid attestation. It takes minutes.

So the rule did not lower the risk of an attack. It cleared exactly the package an attacker would publish, as long as they spent five extra minutes on CI.

The reader put it more generally, from a different system they work on: declarations pool where they are cheap. In a log they scan, 23 of 31 signing keys had zero accepted work and zero deliveries — and emitted 91% of all capability declarations. A signed statement that costs nothing to obtain tells you that someone obtained it.

Now: provenance is still recorded and published next to each finding, for a reader to weigh. The classifier does not take it as an input at all — not ignored, removed from the signature — so nobody can wire it back in by accident.

2. "It was registered before anyone suggested it"

This one came from a real false positive. cdx-rs is a cd replacement, published on 30 August. On 4 September a model offered cdx-rs as a library for parsing CycloneDX SBOMs, which it is not. Five days old: high risk. But the package predated the suggestion, so it could not be a response to it. We added:

if first_hallucinated and info.created_at < first_hallucinated:
    return "low", "registered before the name was ever suggested"

first_hallucinated is the earliest time we saw a model produce the name. That is not when the name was first suggested. It is when our sampling first happened to catch it. Models had been producing these names long before we started collecting; we only know about the ones that came up after.

So near the day our collection began, the rule read: registered before we started looking. Someone who registered a name early and waited would have cleared through it. The comparison only means something when both dates are pinned by something outside the person making the claim — and one of ours was pinned by us.

The reader also spotted something worse about the shape of it. If first_hallucinated was missing, the check was skipped and the package fell through to the age ladder, where it could score high. If it was present and early, the package scored low immediately. Two kinds of not-knowing, two opposite verdicts, and no branch anywhere that said "we don't know".

Is there an independent date? The best I found is the model's training cutoff. The vendor sets it, and our sampling cannot move it. A package registered after every recommending model's cutoff could not have been learned from — the name was invented first and registered later, whenever we happened to look. But every model we sample has a cutoff older than our 60-day window, so a package that predates one is already old enough to clear by age. The rule had nothing honest left to do.

Now: age alone decides. That has a cost: cdx-rs is flagged again. We cannot tell a coincidence from a squat by its registration date, and a list of attack targets should fail toward caution.

3. "We don't know when it was registered"

This one the reader did not mention. Their framing found it. Once you start looking for safe verdicts that rest on absence of evidence, you look at every branch that returns low:

if info.created_at is None:
    return "low", "registered"

A package that exists but has no creation date: fine. When does that happen? For every Go module, it turned out. The Go module proxy answers one question — does this module exist — and our probe asked it nothing else. Every Go module a model recommended, including one pushed yesterday, was cleared without a date ever being checked.

Now: an undated package scores medium. Go modules are dated from deps.dev, which lists every version the proxy has served with its time.

That fix comes with a caveat. For Go, that time is the commit's time, and a commit's date is whatever its author wrote. npm and PyPI record when the registry received an upload, which the publisher cannot choose. A backdated Go module still looks old. It is weaker evidence than the other ecosystems — and better than none, which is what we had.

The audit we couldn't run

The reader suggested a cheap check: take every name rule 2 cleared, plot its registration date against the day our collection began, and see whether they cluster at the boundary. If they do, the rule is reading its own start date.

We couldn't. A finding is stored. A name that scored low was simply dropped. The rule had cleared names and left nothing behind to audit.

Now every low verdict is recorded with the rule that produced it, the registration date and our first sighting. It is never published; it is there so the next rule we get wrong can be caught by looking rather than by a stranger reading our source. We'll run the reader's check once a few weeks of data have built up.

Findings that had been withdrawn under rules 1 and 2 were re-scored under the new rules. Two came back onto the list.

What I'd ask of any risk score

Every rule we've had to remove was one that cleared something. The rules that flag things are easy to defend: a 404, a package two days old. The exceptions are where the reasoning goes soft, because each was written to stop one false positive, by someone looking at that false positive and not at an attacker.

So, for each branch in your scoring that returns "safe":

Can an attacker produce this input? A signed build, a README, a repository field, a star count.
Does it describe the world, or only you? Your first sighting, your crawl date, your cache.
What happens when it is missing? If "unknown" can come out as "safe", it will, at scale, for a whole ecosystem.

The rule that says "this one is fine" is the one an attacker reads.

Thanks to ANP2 Network for the comment that started this. If you can find a fourth, the comments are open.

A bot took six API keys in six hours. Gmail helped

jj1423 — Fri, 02 Oct 2026 01:59:40 +0000

Six accounts signed up overnight. These were the addresses:

kan.eis.hqmc.g.uir.e@gmail.com
cat.o.po.d.i.s.paulo@gmail.com
ma.t.t.h.ewpc.on.no.lly@gmail.com
r.ea.dy.2.d.an.cewi@gmail.com
wi.ll.iam...su.ll.iv.a.n@gmail.com
jes.sewsha.w@gmail.com

If you have run a signup form before, you already know. Gmail ignores dots in the local part. j.o.h.n@gmail.com and john@gmail.com are the same inbox. It is documented behaviour, not a bug — it exists because Gmail launched in a world where people typed addresses off business cards.

My uniqueness check was WHERE lower(email) = .... Six strings, six rows, six API keys handed out.

Whose inboxes were they

My first thought was that someone had taken one address and mutated it six ways to farm keys off me. Then I stripped the dots:

kaneishqmcguire@gmail.com
catopodispaulo@gmail.com
matthewpconnolly@gmail.com
ready2dancewi@gmail.com
williamsullivan@gmail.com
jessewshaw@gmail.com

Six different people. Plausible full names. This is a harvested address list with dot-mutation applied, and the mutation is not aimed at my duplicate check at all — it is a generic evasion habit, applied to a list, by something that signs up for things.

Which means the verification emails went to six real strangers who have never heard of us. That is the part I care about more than the keys. A handful of "report spam" clicks on a young sending domain and your mail stops reaching the people who did ask for it — silently, and exactly when you start getting real users.

Hole one: the cheap door was the unguarded one

Some context, because it is the part that stung. The day before, we had opened part of our API — a package-security lookup — to callers with no key at all. Twenty requests an hour per address, five packages per request. We spent real time on those numbers: enough to try the thing, not enough to run on it. What a caller sees when they run out. How they learn the limit from the response rather than from docs they have not read.

An API key skips all of it.

And we have two ways to get one.

POST /v1/auth/request-key takes an email, mails the key, and has had a per-address daily budget since the day it was written — five. The comment above it literally says it bounds key-farming.

POST /v1/auth/signup takes an email and a password and had nothing.

And signup is the cheaper door, because email verification is off by default in our deployment. Sign up, get a working key in the response. No inbox needed.

So the endpoint with "this bounds key-farming" in its comment was guarded, and the one next to it — doing the same thing, faster — was not. The budget function already existed. It was four lines to call it.

allowed, _used = _request_key_throttle(client_ip)
if not allowed:
    raise HTTPException(429, { ... })

I don't think this is unusual. You write the dangerous-looking endpoint carefully because it looks dangerous, and the ordinary one stays ordinary.

Hole two: and here is where the obvious fix is wrong

The tempting fix for the Gmail thing is to normalise every address: strip dots, strip anything after +, compare that.

Don't. Dots are only insignificant at Gmail.

first.last@outlook.com and firstlast@outlook.com are two different mailboxes, very possibly two colleagues. Fold dots globally and you will eventually tell a real person their account already exists, and you will not find out, because people who hit that just leave.

Plus-tags are safer but still not universal — + is a legal character in a local part, and only some providers route on it. So the rule has to be per-provider:

_PLUS_TAG_DOMAINS = frozenset({
    "gmail.com", "googlemail.com", "outlook.com", "hotmail.com", "live.com",
    "yahoo.com", "icloud.com", "me.com", "fastmail.com",
    "proton.me", "protonmail.com",
})
_DOTLESS_DOMAINS = frozenset({"gmail.com", "googlemail.com"})

def canonical_email(addr: str) -> str:
    local, _, domain = addr.strip().lower().rpartition("@")
    if domain == "googlemail.com":
        domain = "gmail.com"
    if domain in _PLUS_TAG_DOMAINS:
        local = local.split("+", 1)[0]
    if domain in _DOTLESS_DOMAINS:
        local = local.replace(".", "")
    return f"{local}@{domain}"

Two details that matter more than the function.

Store it, don't compute it at query time. A new column, a unique index on it, and the original address kept exactly as typed — because mail goes to what the person wrote, and canonical_email is only ever allowed to answer "is this the same mailbox".

Make the index UNIQUE and let the migration fail. If it won't build on your live database, two of your existing accounts share an inbox, and you want to see that rather than have a WHERE NOT EXISTS quietly paper over it.

The thing I actually took away

Rate limits get designed at the surface you are thinking about.

We had just spent a day on the keyless numbers, and that was good work. But the form that hands out the credential which ignores those numbers was one door over, unmetered, because it was a signup form and signup forms are boring.

If you have a free tier with a carefully chosen number in it, go and look at every path that issues the credential which ignores that number. In our case there were two, and only one of them had ever been thought about.

Postscript, for the six people whose addresses were used: sorry. The verification mail was not from anyone you know, and nothing was done with it. The accounts are gone.

A 404 is the easy half. Scoring a package that actually exists is where we got it wrong.

jj1423 — Thu, 01 Oct 2026 03:02:02 +0000

Someone left this on my last post:

I'd also treat the "404 means stop" rule as only the first gate. A package resolving successfully doesn't establish that it is trustworthy. For newly registered packages that closely match common model-generated names, provenance and publication history become useful signals too.

The deeper lesson is that agents need pre-action verification, not just error recovery. Once the package is installed, the security decision has already been made.

Both correct, and the first one sent me to look at our own data. We publish a list of package names that LLMs invent. 1,885 of the 1,911 entries simply do not exist — those are easy. 26 of them answer 200.

I checked those 26 by hand. Almost all of them were wrong.

The one that stung

name        : opentelemetry-configuration
author_email: OpenTelemetry Authors <cncf-opentelemetry-contributors@lists.cncf.io>
repository  : github.com/open-telemetry/opentelemetry-python
summary     : OpenTelemetry Python Declarative Configuration (experimental)

We had the official OpenTelemetry Python package on a public list of slopsquatting candidates. Not because anything about it is suspicious — because it was 41 days old when we looked, and our rule said 14–60 days is medium.

The same rule caught cdx-rs, which is a cd replacement for the terminal:

crate  : cdx-rs
desc   : A fast and interactive cd command alternative for directory navigation
repo   : github.com/Cinnamon-jp/cdx
created: 2026-08-30

A model suggested cdx-rs as a Rust crate for parsing CycloneDX SBOMs. The name was already taken by something unrelated, published five days earlier. We scored it high.

Age was a proxy for a question I wasn't asking

The reasoning behind an age window is sound: an attacker who watches what models hallucinate registers the name afterwards, so a fresh registration under a hallucinated name is suspicious.

Read that sentence again and the actual signal is right there. Afterwards. Not "recently" — after the hallucination.

I had been measuring the wrong interval. created_at against today tells you how new a package is, which is a fact about the calendar. created_at against the first time any model produced that name tells you whether the registration could have been a response to it, which is the thing I actually wanted to know.

And we already stored it. Every finding carries hallucinated_by, with a first_seen per model, because the dataset is meant to be evidence rather than a count:

{
  "model": "gemini-2.5-flash",
  "task": "Rust crate for parsing CycloneDX SBOM",
  "first_seen": "2026-09-04T..."
}

cdx-rs was published 2026-08-30. First sighting 2026-09-04. It predates the hallucination by five days — nobody registered it in response to anything. The new rule is four lines:

if first_hallucinated and info.created_at < first_hallucinated:
    return "low", "registered before the name was ever suggested"
age_days = (now - info.created_at).days
if age_days < 14:
    return "high", "registered after the name was suggested"
if age_days < 60:
    return "medium", "young"
return "low", "established"

Both false positives drop to low and come off the list. A name registered after the first sighting keeps its score, and now the status string says why, which it never did before.

This is not airtight. Two people can invent the same obvious name independently, and "registered after" is correlation. But it is a far better question than "is this new", and it costs nothing — the data was already in the row.

npm's 200 means more than one thing

Three entries were labelled registered but never published: native-websocket, node-ics, sql-injection-detector. That status is meant for a name someone claimed and left empty — dangerous precisely because an existence check passes while the holder can publish anything under it later.

Except:

curl -s https://registry.npmjs.org/native-websocket | jq '.time'

{
  "created": "2020-11-29T...",
  "modified": "2020-11-29T...",
  "unpublished": { ... }
}

These were published and then taken down — in 2016, 2020 and 2022. npm leaves a tombstone: 200, no versions, and time.unpublished set. My check was versions == 0, which cannot tell an empty squat from a grave.

It matters because the two have different mechanics. An empty registered name is held by someone who can publish to it today. A tombstone is a name that was real — which is plausibly why a model learned it in the first place — and whether anyone can claim it depends on the registry's unpublish policy, not on the holder's intent. They are medium now, with a status that says what they are.

If you are doing this yourself: don't read versions alone. time.unpublished is the field.

A score is a statement about a moment

The last problem had no clever cause. opentelemetry-configuration was 41 days old when we scored it. It is 77 days old now. Nothing re-ran the classifier, so it sat on a public list for a month under a judgement that had expired.

Every finding that resolves is now reclassified on each cleanup run, and anything that drops to low is withdrawn — kept at its URL with a notice, because other people link to these IDs, but out of the list and out of the API.

The finding here is boring and general: if your scoring function reads a clock, something has to re-run it. A pipeline that only ever classifies on insert will accumulate verdicts that were true once.

What I still don't have

The comment's other word was provenance, and that is the honest gap. We score the name and the registry's timeline. We do not check who published it or from where.

Both big registries now expose this:

npm publishes provenance attestations — a signed statement that a package was built by a specific GitHub Actions workflow from a specific repository.
PyPI supports PEP 740 attestations on the same idea.

"Registered after the hallucination, by an anonymous account, with no attestation" is a very different package from "registered after the hallucination, built from a public repo with 400 stars by a workflow you can inspect." Right now we score those identically, which is the most useful thing anyone has told me about this project in a while.

That is the next signal. Until it lands, the list says what it can support and not more.

The data is here — CSV and JSON, daily, CC BY 4.0. The gate is one call and needs no key:

curl -X POST -H "Content-Type: application/json" \
  -d '{"packages": ["pkg:npm/fast-jsonwebtoken"]}' \
  https://vdb.ai.kr/v1/ai/check-packages

Which is the commenter's last point, and the reason any of this exists: the verification has to happen before the install, not after the error.

Catching LLMs in a lie: packages that don't exist

jj1423 — Wed, 30 Sep 2026 00:15:03 +0000

Ask a language model which library to use and it answers with a list of names. Most are real. Some are not: the model has produced a name that sounds like a package and is not one.

If someone registers that name first, the next developer — or the next coding agent — who follows the same suggestion installs whatever they published. That is slopsquatting.

We collect these names continuously at VDB. On September 29, 2026 we took the 100 highest-risk entries on our public list and looked each one up in its registry again.


Recommended names	87
Exist in their registry	0
Belong to nobody	83

Where the names come from

Every six hours a collector sends 30 coding tasks to Claude, GPT and Gemini models, using one fixed prompt:

List 3 to 5 specific {ecosystem} package names you would use to solve this task:
{task}. Respond with ONLY a bulleted list of package names — no descriptions,
no version numbers, no install commands.

The tasks are ordinary. Twelve are for npm, thirteen for PyPI, and the rest for crates.io, Go and Maven. Some are common requests ("library to parse YAML in Python") and some are niche ("Python library for SBOM CycloneDX 1.5 parsing").

Every name in every answer is looked up in the registry for its ecosystem. A name the registry has never heard of becomes a finding.

Nothing here is a trick prompt. There is no jailbreak and no adversarial input. This is what the models say when asked a question a developer would ask.

What the registries said

We first put the 100 entries through the same rules a registry applies. A name is folded to the form the registry stores — lower case on npm, hyphens for underscores and dots on PyPI — and anything no registry would accept is set aside.

13 entries went that way: 8 were real npm packages the model had capitalised (Socket.IO is socket.io), and 5 were not names at all.

That left 87 names. All 87 lookups returned 404. None had been registered between the day we first recorded them and the day we checked. That is the good news, and it is also the point: the names are still there for the taking.

What it is	Names	Who can register it?
A well-formed name nobody owns	83	Whoever asks first. 39 on npm, 42 on PyPI, 1 Go module, 1 crate.
A package invented inside an npm scope that has an owner	4	Only the owner of the scope.

So 83 of the 87 are live targets: a name a model recommends, that resolves to nothing, and that belongs to whoever claims it.

The names cluster around the task

The invented names are not random strings. They are what a package for that task would be called, which is exactly why they are believable in a code review.

Family	Names	Examples
CycloneDX	14	`cyclonedx-pythonlib`, `cyclonedx-python3`, `cyclonedx-xml-python`
JWT	12	`fast-jsonwebtoken`, `jsonwebtoken-promise`, `nxtjwt`
YAML	8	`pyyaml-safe`, `trusted-yaml`, `safe-yaml`
iCalendar	8	`ical-parse`, `ical-events`, `parseics`
cron	7	`cron-expresso`, `cron-syntax-parser`, `cronutils`
WebSocket	6	`npm-websocket`, `browser.ws`, `fayewebsocket`

Three patterns account for most of them.

A real name plus a plausible suffix. jsonwebtoken is real. jsonwebtoken-promise, jsonwebtoken-verify and fast-jsonwebtoken are not. pyyaml is real. pyyaml-safe is not.

A real project's naming scheme, extended. CycloneDX publishes real Python packages, and the models produced a dozen more in the same style.

A trusted scope with an invented package in it. @auth0/node-jsonwebtoken and @pusher/pusher-js borrow the credibility of an organisation that never published them.

Two ways to get this wrong

Counting hallucinated packages is easy to overstate, and we did at first. Two things inflate the number.

Spelling is not existence. npm answers 404 for kaTeX and 200 for katex. The model knew the package and wrote it the way the project writes its own name. Nobody can register the capitalised form, so it is not a target.

A scope changes who the attacker can be. @auth0/yup does not exist, and an install of it fails. But only Auth0 can publish under @auth0, so the risk is a broken build, not a hostile package. We score these lower and count them separately.

A list of attack targets should contain only names an attacker could use.

Check it yourself

Nothing above needs an account. The registries answer anyone:

curl -s -o /dev/null -w "%{http_code}\n" https://registry.npmjs.org/fast-jsonwebtoken
curl -s -o /dev/null -w "%{http_code}\n" https://pypi.org/pypi/pyyaml-safe/json
curl -s -o /dev/null -w "%{http_code}\n" https://proxy.golang.org/github.com/rbretecher/openapi-parser/@v/list

Each printed 404 on September 29, 2026. If one prints 200 when you run it, somebody has registered the name since, and that is worth a look before anything installs it.

What to do about it

A person usually notices when an install fails. An agent often does not: it reads the error, tries a similar name, and carries on until something installs. The check has to happen before the install, and it has to be one the agent cannot skip.

Look the name up before installing it. A 404 means stop, not "try the next spelling".
Treat a young package as unverified. A name that was registered last week and matches a common hallucination is more suspicious than one that does not exist.
Give the agent a gate. This is what we built VDB for. One call, and the answer is an instruction rather than a score:

POST https://vdb.ai.kr/v1/ai/check-packages
Authorization: Bearer $VDB_API_KEY

{"packages": ["pkg:npm/fast-jsonwebtoken"]}

The response carries an agent_action of PROCEED, CONFIRM or REFUSE for each package.

The full list

All 87 names, with registry status and a page for each, are in the original post.

This is a sample: what remained of the hundred highest-risk entries on our public page, not every name the collector has recorded. The counts describe this sample and should not be read as a hallucination rate for any model. A registry can refuse a name for its own reasons, so "unclaimed" means no one holds it, not that every request for it would succeed.