I Sandbox-Tested 1,800+ DeepSeek Harness Plugins. Here’s What Broke

susu su — Tue, 18 Aug 2026 10:05:46 +0000

I Sandbox-Tested 1,800+ DeepSeek Harness Plugins. Here’s What Broke

The DeepSeek Harness plugin ecosystem is growing fast.

That sounds great until you actually try to install plugins at scale.

When I started building a searchable catalogue for DSH plugins, I assumed the hard part would be discovery:

finding repositories
reading metadata
extracting install commands
sorting by stars and categories

It wasn't.

The hard part was figuring out which information could actually be trusted.

After running sandbox installation checks on more than 1,800 plugins, I found several failure modes that were much more common than I expected.

That work eventually became DSH Marketplace:

https://dshmarketplace.dev

But the interesting part is what broke along the way.

1. A package name in `package.json` does not mean the package exists

The first version of the pipeline was simple.

If a repository contained:

{
  "name": "some-dsh-plugin"
}

then generating an installation command seemed reasonable:

dsh plugin --profile web add some-dsh-plugin

The problem is that repository metadata is not the npm registry.

Some packages simply did not exist.

Others were even worse.

A fork could inherit the upstream repository's package.json, including its package name.

Imagine:

author-a/plugin
       ↓ fork
author-b/plugin

The fork still contains:

{
  "name": "author-a-plugin"
}

If a directory blindly trusts that metadata, the page for author-b/plugin may tell users to install:

dsh plugin --profile web add author-a-plugin

The command works.

But it installs someone else's package.

That is more dangerous than a broken command because it looks successful.

2. "Installation failed" does not mean "plugin is broken"

My first validation model was essentially:

success
failure

That quickly turned out to be useless.

Real installation results looked more like:

passed
needs-approval
not-a-layer
failed
timeout
rejected

Those distinctions matter.

`needs-approval`

Some packages contain build scripts that require explicit approval.

That does not necessarily mean the plugin is broken.

It may only need an allowBuilds entry before it can activate.

`not-a-layer`

A package may install successfully but not declare a DSH profile layer.

It could be:

a theme
a skill bundle
a library
an agent extension

Calling that "broken" would be misleading.

A useful validator needs to understand more than the process exit code.

3. Sometimes your validator is the thing that failed

This was the biggest lesson.

At one point, a large batch of plugins came back as failed.

The tempting conclusion was:

A huge percentage of DSH plugins are broken.

That conclusion was wrong.

After inspecting the logs, many failures came from the validation infrastructure itself:

bad npm package claims
npm registry throttling
temporary network failures
monorepo resolution
changes in the probe during a long validation run

One batch produced more than a hundred failures that were mostly registry throttling.

If I had published those results directly, perfectly valid plugins would have been labeled as broken.

So the pipeline now tries to classify failures as:

OURS

or:

THEIRS

If the failure cannot be explained confidently, the negative verdict should not be published.

This became an important rule:

An unexplained failure is a validator problem until proven otherwise.

4. Testing third-party plugins means executing third-party code

There is another obvious problem with installation validation:

You are executing code you do not control.

npm packages can run lifecycle scripts such as:

{
  "scripts": {
    "postinstall": "..."
  }
}

A malicious installation script can do much more than install dependencies.

So the validator cannot run in the same environment that holds production credentials.

The architecture now looks roughly like this:

Marketplace
    ↓
export plugins that need testing
    ↓
isolated validation job
    ↓
disposable containers
    ↓
installation results
    ↓
trusted job applies results

The job that runs third-party installation code holds no production database secrets.

The containers are also restricted:

non-root
no host mounts
limited memory
limited process count
dropped capabilities
disposable DSH profiles

The important part is not that the sandbox is "perfect".

It is that untrusted plugin code is separated from the credentials that matter.

5. Metadata needs continuous verification

Even after fixing the first batch, the problem does not disappear.

Repositories change.

Packages get unpublished.

Forks appear.

Install commands go stale.

So npm verification cannot be a one-time migration.

The catalogue now re-checks package claims as part of its recurring sync process.

The basic pipeline is:

discover
→ refresh metadata
→ verify npm claims
→ apply admission rules
→ sandbox install
→ classify failures
→ update results
→ deploy

This is more expensive than scraping a GitHub topic once.

But it produces much more useful data.

Why I ended up building DSH Marketplace

Originally, I wanted a simple plugin directory.

But a directory that only shows:

name
description
stars
GitHub URL

does not solve much.

GitHub already does that.

The useful questions are:

What does this plugin actually do?

Is the install command still valid?

Does the npm package belong to this repository?

Did the plugin fail, or did the validator fail?

Does installation require approval?

That is why DSH Marketplace gradually became more than a list of links.

Today it indexes 2,500+ DeepSeek Harness plugins, with 1,800+ sandbox installation checks.

You can browse it here:

https://dshmarketplace.dev

The project is open source:

https://github.com/DshMarketPlace

It also works inside DSH

The Marketplace itself is available as a DSH plugin:

dsh plugin --profile web add dshmarketplace-plugin

There are also:

a CLI
Python SDK / CLI
public API
browser userscript for GitHub/npm
the web marketplace

The public API does not require a key:

curl 'https://dshmarketplace.dev/api/v1/plugins?q=memory&limit=5'

The goal is to keep all of these surfaces backed by the same catalogue.

What I learned

The biggest lesson was not about Docker or npm.

It was about trust.

When building a software directory, it is easy to assume that metadata is truth.

But:

README files become outdated
package names can be inherited
registries can disagree with repositories
install commands can silently install the wrong thing
automated tests can generate false negatives

At enough scale, every shortcut eventually becomes visible.

So the principle I ended up with is simple:

Trust metadata less. Verify behavior more.

DSH Marketplace is still early, but this validation pipeline has already been much more useful than simply collecting more repositories.

If you're using DeepSeek Harness, I'd be interested to know:

How do you currently decide whether a plugin is worth installing?

GitHub stars? Awesome lists? Community recommendations? Or do you test them yourself?

DEV Community: susu su

I Sandbox-Tested 1,800+ DeepSeek Harness Plugins. Here’s What Broke

I Sandbox-Tested 1,800+ DeepSeek Harness Plugins. Here’s What Broke

1. A package name in package.json does not mean the package exists

2. "Installation failed" does not mean "plugin is broken"

needs-approval

not-a-layer

3. Sometimes your validator is the thing that failed

4. Testing third-party plugins means executing third-party code

5. Metadata needs continuous verification

Why I ended up building DSH Marketplace

It also works inside DSH

What I learned

1. A package name in `package.json` does not mean the package exists

`needs-approval`

`not-a-layer`