Every few weeks another npm or PyPI package turns out to steal tokens on install. The usual advice is "check before you install", but few people have a tool that does the checking, and fewer know how well that tool works.
I build vetcode, an open-source scanner for malicious packages, supply-chain attacks and leaked secrets. Before telling anyone to use it, I wanted to know how it compares with the best free tool in this space. That tool is GuardDog from Datadog. A recent paper (NPMAnalysis) ranked it first among 13 npm malware detectors.
This post covers how I measured, every result I got, and where vetcode still loses.
The setup
Data. Malicious npm packages come from NPMAnalysis. Malicious PyPI packages come from the Datadog malicious-software-packages dataset. The clean side is real, popular packages from the npm and PyPI registries.
Fresh samples only. Each run draws a random sample with a new seed. I tune rules on one seed, then confirm on a seed that has never been looked at. Every number below comes from a confirmation seed. A scanner tuned and scored on the same samples will always look good.
Same packages for both tools. Both scanners read exactly the same tarballs. Nothing is installed or executed; both tools only read files.
GuardDog gets its best verdict. GuardDog has no single yes/no answer, so I scored it four ways: any issue, any threat rule, risk score ≥ 5, and risk score ≥ 7. It is credited with whichever gives the best F1 on each sample. vetcode is scored one way only: "flagged at severity high or above".
Metrics.
- F1 combines precision and recall into one number.
- Precision is the share of flagged packages that really are malicious.
- Recall is the share of malicious packages caught.
- False alarms is the share of clean packages wrongly flagged.
False alarms matter more than they look. A tool that flags 1 in 5 popular packages gets switched off.
npm: eight fresh samples
| Sample | vetcode F1 · P · R · FA | GuardDog F1 · P · R · FA |
|---|---|---|
| seed 14 (500 + 500) | 93.3 · 96.4 · 90.4 · 3.4% | 92.8 · 95.7 · 90.0 · 4.0% |
| seed 16 (500 + 500) | 94.4 · 97.6 · 91.4 · 2.2% | 93.3 · 95.4 · 91.2 · 4.4% |
| seed 18 (150 + 150) | 92.6 · 97.8 · 88.0 · 2.0% | 93.2 · 94.5 · 92.0 · 5.3% |
| seed 20 (500 + 499) | 95.1 · 98.3 · 92.2 · 1.6% | 93.0 · 96.0 · 90.2 · 3.8% |
| seed 21 (500 + 498) | 94.1 · 97.2 · 91.2 · 2.6% | 94.2 · 97.4 · 91.2 · 2.4% |
| seed 23 (150 + 150) | 93.5 · 96.5 · 90.7 · 3.3% | 94.6 · 96.5 · 92.7 · 3.3% |
| seed 26 (500 + 499) | 96.1 · 98.3 · 94.0 · 1.6% | 93.0 · 93.5 · 92.4 · 6.4% |
| seed 27 (150 + 150) | 95.5 · 98.6 · 92.7 · 1.3% | 93.9 · 95.2 · 92.7 · 4.7% |
vetcode wins five of eight. GuardDog wins three, by 0.1 to 1.1 F1 points. Two of its wins are on the small 300-package samples, where one package moves F1 by about half a point.
The more consistent difference is false alarms. vetcode flags 1.3–3.4% of clean packages; GuardDog flags 2.4–6.4%. On the latest 1,000-package sample that is 8 clean packages against 32.
PyPI: seven fresh samples
| Sample | vetcode F1 · P · R · FA | GuardDog F1 · P · R · FA |
|---|---|---|
| seed 12 (300 + 216) | 87.4 · 93.5 · 82.0 · 7.9% | 81.3 · 83.0 · 79.7 · 22.7% |
| seed 15 (300 + 216) | 84.1 · 93.1 · 76.7 · 7.9% | 81.1 · 82.9 · 79.3 · 22.7% |
| seed 17 (300 + 216) | 86.4 · 93.4 · 80.3 · 7.9% | 78.3 · 78.5 · 78.0 · 29.6% |
| seed 19 (150 + 150) | 89.0 · 95.4 · 83.3 · 4.0% | 80.0 · 80.0 · 80.0 · 20.0% |
| seed 22 (300 + 152) | 89.2 · 96.5 · 83.0 · 5.9% | 85.0 · 87.9 · 82.3 · 22.4% |
| seed 24 (150 + 150) | 89.0 · 95.4 · 83.3 · 4.0% | 80.0 · 80.0 · 80.0 · 20.0% |
| seed 28 (150 + 150) | 88.6 · 95.4 · 82.7 · 4.0% | 79.6 · 79.9 · 79.3 · 20.0% |
Here the gap is clear. GuardDog's PyPI rules fire on many ordinary packages: 20–30% of popular packages get flagged at its best setting.
Seed 15 is the one I learned most from. GuardDog caught more malware there (79.3% vs 76.7%). Two kinds of package were behind the gap:
- beacons hidden inside functions;
- droppers that write an
.exe.
I added rules for both, and recall went back above GuardDog's on the next fresh sample.
Disguised malware
Malware authors know scanners read source code, so they rewrite it. I wrote a script that rewrites caught samples the same way, without running anything. Then I checked how many each tool still catches. The table shows the share of previously caught samples still caught:
| Rewrite | vetcode npm | GuardDog npm | vetcode PyPI | GuardDog PyPI |
|---|---|---|---|---|
strings split and joined with +
|
100% | 99.2% | 100% | 95.9% |
\xNN escapes |
100% | 99.2% | 100% | 93.4% |
built from character codes (String.fromCharCode, chr) |
100% | 98.4% | 100% | 90.1% |
| strings written backwards | 100% | 97.6% | 100% | 90.9% |
whole file base64-packed into eval/exec
|
100% | 92.2% | 100% | 100% |
| gzip, then base64 | 100% | 95.0% | 99.2% | 98.3% |
| code pushed 400 spaces off screen | 100% | 100% | – | – |
moved into dist/
|
99.2% | 100% | 100% | 99.2% |
vetcode decodes these tricks before its rules run, and it labels what it found "(hidden)". It does this statically: joins split strings, decodes escapes, rebuilds character-code strings, reverses reversed ones, and unpacks base64 or zlib payloads up to three layers deep.
Building this benchmark also found a real bug: one of vetcode's own regexes could hang for minutes on a 500 KB base64 line. Measuring hostile input finds things that clean input never does.
Things the datasets barely contain
Public datasets are full of 2021–2023 typosquats. Newer techniques are rare in them, so I keep a separate set of 15 hand-built cases from recent campaigns. Each one sits next to a clean look-alike that must not be flagged:
- GlassWorm's invisible Unicode payloads;
- code hidden off-screen in
dist/; - payloads inside images;
- protestware and time bombs;
- homoglyph package names;
- a Rust
build.rsthat reads~/.ssh; - a Visual Studio project that runs encoded PowerShell at build time;
- a malicious agent skill in the style of ClawHavoc.
vetcode catches 15 of 15 with no false alarms on the look-alikes.
Speed
On the same 30 PyPI packages (659 MB), GuardDog took 1,104 seconds and vetcode 299. Since then vetcode has become another third faster. It has zero dependencies: Node 18+ and nothing else.
What this does not show
- The datasets favour GuardDog. Many packages in them were found by GuardDog in the first place, which helps it on recall.
- Samples are small. They range from 300 to 1,000 packages, and one or two F1 points between seeds is normal variation. That is why I publish every seed, not the best one.
- Rules alone are not the whole story. vetcode is designed to run inside an AI coding agent (Claude Code, Codex, Cursor, Copilot and others). The agent re-reads each finding before reporting it. None of the numbers above include that step; they are the scanner alone.
- Agent skills are hard for everyone. On 204 malicious and 132 clean agent skills, vetcode's rules catch 29.9% with 3.0% false alarms. Cisco's skill-scanner catches 23.5% with 15.9%. Most malicious skills are plain-English instructions, which rules cannot judge.
Try it, or check my work
Scan a project, a package you are about to install, or a repo you just cloned:
npx @openyard/vetcode .
Or give it to your AI agent as a skill:
npx skills add openyard-cloud/vetcode
It also runs as an MCP server and as a GitHub Action that comments on pull requests.
Every script used here is in bench/. If you run it and get different numbers, or find a malicious package vetcode misses, please open an issue. That is the most useful thing you can send me.
vetcode is MIT-licensed and part of OpenYard, a small community of open-source projects.
Top comments (2)
Really appreciate you publishing the losses too — the PyPI false-alarm gap (20–30% vs ~4–8%) is the number that matters most in practice, since that's what gets a scanner switched off. The seed-15 story was a nice touch as well: two missed patterns, then rules added and recall recovered on the next fresh sample. That's how benchmarking should work.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.