DEV Community

susu su
susu su

Posted on

I Sandbox-Tested 1,800+ DeepSeek Harness Plugins. Here’s What Broke

#ai #deepseek #opensource #devtool

I Sandbox-Tested 1,800+ DeepSeek Harness Plugins. Here’s What Broke

The DeepSeek Harness plugin ecosystem is growing fast.

That sounds great until you actually try to install plugins at scale.

When I started building a searchable catalogue for DSH plugins, I assumed the hard part would be discovery:

  • finding repositories
  • reading metadata
  • extracting install commands
  • sorting by stars and categories

It wasn't.

The hard part was figuring out which information could actually be trusted.

After running sandbox installation checks on more than 1,800 plugins, I found several failure modes that were much more common than I expected.

That work eventually became DSH Marketplace:

https://dshmarketplace.dev

But the interesting part is what broke along the way.

1. A package name in package.json does not mean the package exists

The first version of the pipeline was simple.

If a repository contained:

{
  "name": "some-dsh-plugin"
}
Enter fullscreen mode Exit fullscreen mode

then generating an installation command seemed reasonable:

dsh plugin --profile web add some-dsh-plugin
Enter fullscreen mode Exit fullscreen mode

The problem is that repository metadata is not the npm registry.

Some packages simply did not exist.

Others were even worse.

A fork could inherit the upstream repository's package.json, including its package name.

Imagine:

author-a/plugin
       ↓ fork
author-b/plugin
Enter fullscreen mode Exit fullscreen mode

The fork still contains:

{
  "name": "author-a-plugin"
}
Enter fullscreen mode Exit fullscreen mode

If a directory blindly trusts that metadata, the page for author-b/plugin may tell users to install:

dsh plugin --profile web add author-a-plugin
Enter fullscreen mode Exit fullscreen mode

The command works.

But it installs someone else's package.

That is more dangerous than a broken command because it looks successful.

2. "Installation failed" does not mean "plugin is broken"

My first validation model was essentially:

success
failure
Enter fullscreen mode Exit fullscreen mode

That quickly turned out to be useless.

Real installation results looked more like:

passed
needs-approval
not-a-layer
failed
timeout
rejected
Enter fullscreen mode Exit fullscreen mode

Those distinctions matter.

needs-approval

Some packages contain build scripts that require explicit approval.

That does not necessarily mean the plugin is broken.

It may only need an allowBuilds entry before it can activate.

not-a-layer

A package may install successfully but not declare a DSH profile layer.

It could be:

  • a theme
  • a skill bundle
  • a library
  • an agent extension

Calling that "broken" would be misleading.

A useful validator needs to understand more than the process exit code.

3. Sometimes your validator is the thing that failed

This was the biggest lesson.

At one point, a large batch of plugins came back as failed.

The tempting conclusion was:

A huge percentage of DSH plugins are broken.

That conclusion was wrong.

After inspecting the logs, many failures came from the validation infrastructure itself:

  • bad npm package claims
  • npm registry throttling
  • temporary network failures
  • monorepo resolution
  • changes in the probe during a long validation run

One batch produced more than a hundred failures that were mostly registry throttling.

If I had published those results directly, perfectly valid plugins would have been labeled as broken.

So the pipeline now tries to classify failures as:

OURS
Enter fullscreen mode Exit fullscreen mode

or:

THEIRS
Enter fullscreen mode Exit fullscreen mode

If the failure cannot be explained confidently, the negative verdict should not be published.

This became an important rule:

An unexplained failure is a validator problem until proven otherwise.

4. Testing third-party plugins means executing third-party code

There is another obvious problem with installation validation:

You are executing code you do not control.

npm packages can run lifecycle scripts such as:

{
  "scripts": {
    "postinstall": "..."
  }
}
Enter fullscreen mode Exit fullscreen mode

A malicious installation script can do much more than install dependencies.

So the validator cannot run in the same environment that holds production credentials.

The architecture now looks roughly like this:

Marketplace
    ↓
export plugins that need testing
    ↓
isolated validation job
    ↓
disposable containers
    ↓
installation results
    ↓
trusted job applies results
Enter fullscreen mode Exit fullscreen mode

The job that runs third-party installation code holds no production database secrets.

The containers are also restricted:

  • non-root
  • no host mounts
  • limited memory
  • limited process count
  • dropped capabilities
  • disposable DSH profiles

The important part is not that the sandbox is "perfect".

It is that untrusted plugin code is separated from the credentials that matter.

5. Metadata needs continuous verification

Even after fixing the first batch, the problem does not disappear.

Repositories change.

Packages get unpublished.

Forks appear.

Install commands go stale.

So npm verification cannot be a one-time migration.

The catalogue now re-checks package claims as part of its recurring sync process.

The basic pipeline is:

discover
→ refresh metadata
→ verify npm claims
→ apply admission rules
→ sandbox install
→ classify failures
→ update results
→ deploy
Enter fullscreen mode Exit fullscreen mode

This is more expensive than scraping a GitHub topic once.

But it produces much more useful data.

Why I ended up building DSH Marketplace

Originally, I wanted a simple plugin directory.

But a directory that only shows:

name
description
stars
GitHub URL
Enter fullscreen mode Exit fullscreen mode

does not solve much.

GitHub already does that.

The useful questions are:

What does this plugin actually do?

Is the install command still valid?

Does the npm package belong to this repository?

Did the plugin fail, or did the validator fail?

Does installation require approval?

That is why DSH Marketplace gradually became more than a list of links.

Today it indexes 2,500+ DeepSeek Harness plugins, with 1,800+ sandbox installation checks.

You can browse it here:

https://dshmarketplace.dev

The project is open source:

https://github.com/DshMarketPlace

It also works inside DSH

The Marketplace itself is available as a DSH plugin:

dsh plugin --profile web add dshmarketplace-plugin
Enter fullscreen mode Exit fullscreen mode

There are also:

  • a CLI
  • Python SDK / CLI
  • public API
  • browser userscript for GitHub/npm
  • the web marketplace

The public API does not require a key:

curl 'https://dshmarketplace.dev/api/v1/plugins?q=memory&limit=5'
Enter fullscreen mode Exit fullscreen mode

The goal is to keep all of these surfaces backed by the same catalogue.

What I learned

The biggest lesson was not about Docker or npm.

It was about trust.

When building a software directory, it is easy to assume that metadata is truth.

But:

  • README files become outdated
  • package names can be inherited
  • registries can disagree with repositories
  • install commands can silently install the wrong thing
  • automated tests can generate false negatives

At enough scale, every shortcut eventually becomes visible.

So the principle I ended up with is simple:

Trust metadata less. Verify behavior more.

DSH Marketplace is still early, but this validation pipeline has already been much more useful than simply collecting more repositories.

If you're using DeepSeek Harness, I'd be interested to know:

How do you currently decide whether a plugin is worth installing?

GitHub stars? Awesome lists? Community recommendations? Or do you test them yourself?

Top comments (0)