DEV Community

Moby
Moby

Posted on Originally published at whaleharness.com

How We Verify DSH Plugins

#opensource #security #deepseek #ai

DeepSeek Harness (DSH) plugins are just npm packages that patch an agent profile — small, powerful, and easy to ship. That ease is exactly why an open plugin store needs a verification process you can actually check. At WhaleHarness we run one on a simple bet: trust neither the author nor the reviewer — trust the record.

This post explains how we verify plugins, with the live numbers and public artifacts behind them. Everything below is on the site and machine-readable; none of it is a marketing claim.

The two trust failures a plugin store actually faces

A plugin directory worries about two different kinds of failure. The first is a bad plugin: something that calls out to the network, reads credentials off disk, or fakes what it returns to the model. The second is a bad review: when authors wait on a private queue, they cannot tell whether their submission is being judged fairly or silently dropped.

Most stores fix the first problem with manual review and hope the second never comes up. We inverted the design: the review itself is public.

Step 1 — a public submission box

Every submission lands in a public HTTP PUT endpoint:

https://whaleharness.com/submit/whalepod2026/<name>-<version>.tgz
Enter fullscreen mode Exit fullscreen mode

It accepts only .tgz / .tar.gz, only a single file, and only up to 5 MB — anything else returns 404. Everything that lands is publicly readable through an autoindex. There is no account, no queue, and no black box to wait inside. An author who claims "I was never told why it failed" is refuted by the public directory; a store that claimed rigor but shipped a dangerous plugin would be refuted by the same directory. The record is the authority, not either party.

Step 2 — automated checks first, one veto

Every submission runs mechanical checks before any human looks at it:

  • Structure — a standard npm package with package.json, a dsh.bundle.patch declaration, and a cordis.patch.yml that inserts only the plugin's own id. The plugin name must be unique.
  • Dependencies — peerDependencies may list only official @deepseek-ai/* packages. Any other npm dependency must be justified in the README and is assessed by a human.
  • Danger patterns (any single hit vetoes) — network exfiltration, eval / child_process, reading credentials or sensitive paths, or content that impersonates the store or fakes provenance.

These checks are a floor, not a guarantee. Whatever passes still gets installed, booted, and called end-to-end in a real DSH with a throwaway DSH_HOME — in an isolated low-privilege sandbox that contains a honeypot credential, a fake secret planted where a malicious plugin would look. A plugin that tries to steal has nothing to gain and leaves evidence.

Step 3 — a five-tier ecosystem audit, every six hours

The store is one thing; the wider ecosystem is another. Every 6 hours an automated pipeline audits third-party DSH plugin repos, not just our listings. Each repo is pinned to a repo + commit + version and given a tiered verdict:

  • RED-LINE — a safety red line was hit; a design problem that needs a rewrite or an appeal
  • FORMAT-ISSUE — fixable; the rejection note says exactly what to change
  • PASS — passes static review
  • UNEVALUATED — cannot be evaluated (e.g. missing package.json)
  • EXCLUDED — out of scope

Live tally (generated 2026-08-22 06:43 UTC, straight from audit.json): 1,271 repos audited — 171 RED-LINE · 388 FORMAT-ISSUE · 505 PASS · 1 EXCLUDED · 206 UNEVALUATED. Among the 1,065 evaluated, the pass rate is 47.4%. Most failures are format issues with a published fix path — the audit page tells every author exactly what to change. The pipeline is fully public: machine-readable JSON, a human-readable report, an Atom feed, per-repo badges, and a 24-hour appeal SLA on GitHub Discussions.

Step 4 — reproducible build, pinned by sha256

Every store listing is built reproducibly from public source (source.repo + commit) and pinned by sha256 in plugins.json. What you download is what we verified. The store's 165 plugins are all boot-verified and checksum-pinned; the ~1,271 audited repos are static-reviewed, and the depth of verification is stated per plugin in agent.json — installers are told the verification level before they install, not after.

Real cases, all on the record

  • whale-breathe by kwawa — the first community submission to be rejected and then fixed. The rejection note said exactly what was wrong; the fixed version passed and shipped.
  • whale-digest, same author — the first submission to pass on the first try.
  • dsh-whale-fortune — our own reviewer was wrong once (a schema rule that only applied to one registration path). We revised the rule and published an apology next to the review. Process bugs get fixed publicly too.
  • Anionex/dsh-vision-toolkit#33 — a false positive corrected on appeal within the SLA.

Honest limits

Static review does not boot your plugin; sandbox verification does, but it cannot prove a plugin is "safe" in any absolute sense — it makes the process checkable by anyone. Curation does not scale: a human still reads every dependency note. Review is promised within 72 hours, not minutes. And a public record cannot make a plugin safe — it can only make the process checkable. That is the whole point.

Try it from the other side

Authors can submit without touching a terminal: install whale-submit and ask the model to pack a directory and PUT it to the box. Everything — submissions, verdicts, build log — is public:

航 (Hang), growth operator at WhaleHarness, a bilingual plugin store for DeepSeek Harness. All numbers above are from the live site as of 2026-08-22 06:43 UTC; the pipeline re-runs every 6 hours and the live count is authoritative.

Top comments (0)