Every "awesome" list makes the same promise: a curated shelf of plugins you can install without reading the source. In August we decided to test that promise against our own pipeline. On a single day — 2026-08-28 — we ran twenty community-recommended DSH plugins through the same verification ring we apply to every submission on whaleharness.com. Seven earned a spot on our shelf. Of the other thirteen, eleven are ones we tried and do not recommend; two are still candidates for us — one a bundle pick, one under review — not rejections.
This is not a victory lap, and it is not another recommendation list. It is the first in a series of original write-ups built from our own audit data — a verification report. We are publishing the method, the numbers, and three real rejection cases, because the gap between "a list says it is good" and "safe to install" is wider than most people assume. A recommendation tells you a plugin exists and someone liked it. A verification tells you what it touches, what it runs, and what it is allowed to do. They are not the same product.
How we test before we recommend
Our review loop has three stages, and every plugin must survive all three.
First, an audit ruling. Every plugin on our site carries a public audit record, and the corpus behind today's numbers is audit.json — 2,521 records of rulings we have made over time. Nothing ships without one.
Second, static analysis of the GitHub source. We scan for red lines: child_process and eval, calls to external endpoints, and shell scripts — especially installer scripts that execute code before you have had a chance to read it.
Third, a clean-room Docker sandbox. We add the plugin, run dump-config to see exactly what it injects into the runtime, then boot it under whaleharness-review:latest and watch what it actually does.
Each stage exists to catch a different class of failure: the audit carries policy history, the static scan catches red lines, and the sandbox catches what the code actually does at runtime.
Only a plugin that passes all three stages is considered for our shelf. That is the part recommendation lists do not show you. Their job ends at "this exists." Ours starts there.
Three ways to fail our shelf
The rejections in our four batches clustered into three shapes. Here is one example of each, named by repo and technical reason. This is a quality report, not a critique of people.
RED-LINE — hisence999/dsh-vision. Its install.sh runs curl | bash, a pattern we flag on sight. Then it goes further: it uses awk to rewrite the allowlist inside the host's own dsh-host-apiproxy/lib/index.js. Editing the host's source from an installer is out of bounds — a plugin should extend the runtime, not patch it. The package is also non-standard in form: private:true with no cordis bundle, so it could not load in a normal pipeline even if the installer were clean.
License — deep-whale-day-night-theme. This one hurt, because technically it was perfect. The sandbox passed twice, the static scan found zero red lines, and the NOTICE file carried a proper three-party attribution chain. But it ships under CC BY-NC-SA 4.0. The NC clause forbids commercial use, and the SA clause forces any derivative to keep the same license. Our shelf allowlist is MIT, Apache, and BSD. A technically excellent plugin behind a non-permissive license still does not ship here.
Form — tokentracker. It is not a DSH plugin at all. It is a standalone CLI plus a desktop app (tokentracker-cli, which tracks 34 AI tools). There is no cordis.patch.yml and no dsh.bundle, so it cannot install into a standard pipeline. No red lines, no license problem — it is simply a different kind of software wearing a plugin-shaped recommendation.
What the numbers say
All four batches completed on 2026-08-28. Batch one: 6 recommended, 2 accepted (agent-teams, mic-input). Batch two: 5 recommended, 1 accepted (dsh-context). Batch three: 4 recommended, 2 accepted (dsh-sql, dsh-finreport). Batch four: 5 recommended, 2 accepted (vision-any, plugin-agent-workflow). That is 7 accepted out of 20 tested.
Coverage matters as much as the acceptance rate. Every one of our 174 listed plugins has passed the audit. When we cross-referenced a single community list — Herdeny's awesome-dsh-plugins-2026, which claims 132 entries — against our system, 143 of 157 were already audited: a 91% overlap. That residue, where "popular" and "verified" diverge, is exactly where installers get hurt. The overlap is reassuring, but the useful signal lives in the 9% not yet audited — and in how the shared 91% was ruled.
One ecosystem note for August: zhayujie, the author of chatgpt-on-wechat, shipped a new project called CowAgent — an independent Agent Harness stepping into the space. Worth an observation, not yet an evaluation.
We verify, you install. That is the whole brand, and it is checkable: audit.json and categories.json are public. No private scores, no hand-waving.
Your pre-install checklist
Before you install anything from a recommendation list, ask four questions:
- Form. Is there a cordis.patch.yml, and does package.json look like a standard published plugin — or is it a CLI app in disguise?
- Installer. Does anything run curl | bash? If an install script touches files outside the plugin's own directory, stop.
- License. Is it MIT, Apache, or BSD — or does a CC clause (NC or SA) quietly change what you are allowed to do?
- Evidence. Is there a public audit record, or just a name on a list?
Appendix: batch results
| Batch | Recommended | Accepted | Not on shelf | Accepted plugins |
|---|---|---|---|---|
| 1 | 6 | 2 | 4 | agent-teams, mic-input |
| 2 | 5 | 1 | 4 | dsh-context |
| 3 | 4 | 2 | 2 | dsh-sql, dsh-finreport |
| 4 | 5 | 2 | 3 | vision-any, plugin-agent-workflow |
| Total | 20 | 7 | 13 | — |
Note: batch 1's two non-rejections (dsh-memory-vault as a bundle candidate, dsh-whale-girl under review) are counted as not-on-shelf, not as rejections. The 11 outright rejections are the ones covered by the three cases above and the same class of reasons.
Top comments (0)