DEV Community

Cover image for DeepSeek Harness vs Hermes Agent: Tied First, One Flaw
Shamuddin
Shamuddin

Posted on Originally published at aitecharchive.com

DeepSeek Harness vs Hermes Agent: Tied First, One Flaw

DeepSeek Harness (DSH) is the better pick for repository code changes on open models, and Hermes Agent, Pi or OpenCode is the better pick when background services must keep running after the agent exits. The Harness Leaderboard added DSH on 9 November 2026 and, per its changelog, ranked it in every view on 11 November. It is statistically tied for first in all five evaluations, yet it places only 6th of 10 on Terminal-Bench 2.1 because of a single cleanup habit.

TL;DR

  • DSH is tied for first in all five board evaluations, where "tied" means the 95% bootstrap interval of the paired difference to the leader covers zero (Niklaus announcement).
  • Averaged over three models, DSH is 2nd on DeepSWE v1.1, 0.5 points behind Hermes Agent, and 6th on Terminal-Bench 2.1.
  • The gap comes from DSH shutting down every background program it started when a task ends. Three Terminal-Bench tasks grade a server after the agent exits.
  • DSH averaged $0.53 per solved trial on the board.
  • The harness is MIT-licensed and still a developer preview, so expect breaking changes.
  • Last verified: 13 November 2026.

How does DeepSeek Harness compare on the Harness Leaderboard?

The board, run by Hugging Face researcher Joël Niklaus, holds the model and reasoning effort fixed and swaps only the harness. It scores Terminal-Bench 2.1 and DeepSWE v1.1 with three attempts per task, according to the Space. The five evaluations are DeepSeek-V4.1-Flash and GLM-5.3-Flash at low and high effort, plus Flint.

Where Result
Five board evaluations Statistically tied for first in all 5
DeepSWE, averaged over 3 models 2nd of 10, 0.5 points behind Hermes Agent
Terminal-Bench 2.1, averaged 6th of 10
GLM-5.3-Flash at high effort 5th of 10, tied with leader

Source for the table: the board changelog and Niklaus's announcement. A 5th-place rank that is still tied with the leader is a reminder that the field is tight. Before DSH arrived, the top five harnesses sat within 2.2 points of each other, according to CodeMago's write-up of the launch week.

Why does DeepSeek Harness lose ground on Terminal-Bench?

DSH shuts down every background program it started once it finishes a task, as Niklaus reports. Three Terminal-Bench tasks ask the agent to start a server, for example a model inference API, and the grader then tests that server after the agent has exited. DSH passed 2 of 36 attempts on those tasks, while OpenCode, Pi and Hermes Agent passed 36 of 36.

Those three tasks cost DSH between 1.5 and 1.7 points per evaluation. That is more than its 1.3-point gap to the leader on DeepSeek-V4.1-Flash at high effort (announcement). In other words, the flaw is a behavioural default, not a lack of capability, and it is the whole reason DSH does not top the Terminal-Bench view.

What can you do if your server must outlive the agent?

This section is our own analysis, not a claim from the board or its author. Given the confirmed behaviour, there are two sensible workarounds:

  1. Start long-lived services under systemd or with nohup and setsid, so they sit outside the agent's own process group and are not swept up at cleanup.
  2. State the persistence requirement explicitly in the prompt, for instance "leave the server running after you finish".

We have not tested either against DSH, so treat them as a starting point and verify with a quick task before relying on them. If a workflow depends on this often, the simpler route is to choose a harness that already passes this class of task.

Is DeepSeek Harness cheap to run?

DSH averaged $0.53 per solved trial on the board (Niklaus). The board lists DeepSeek-V4.1-Flash at $0.28 per million input tokens and $1.12 per million output tokens, and GLM-5.3-Flash at $0.15 per million input tokens (Space). The harness itself costs nothing, so your bill is the model API.

One caveat on scoring: 20 of DSH's passes were overturned, for example for fetching upstream code, which the board counts as a fail (board method). DSH also ran with web search off, because its search needs a DeepSeek account key the test machines do not hold.

Does the harness matter as much as the model?

Independent research suggests it does. An arXiv study of 66 model-harness configurations found model rankings reverse across harnesses. On Terminal-Bench 4, Claude leads GPT by 7.94 points in OpenHands but trails it by 30.16 points in PI. The same paper reports DSH as the best harness for GLM-5.3 and DeepSeek V4 Pro on that benchmark.

The board's own launch data points the same way: swapping the harness moved scores by up to 24 points for weaker harnesses, per CodeMago. For more on how agent platforms differ in practice, see our open-source coding agent platform comparison and the Claude Code tool-calling comparison. For the model side of the same question, read SWE-2 Flash vs GLM-5.3 Flash.

How do you install and run DeepSeek Harness?

These steps come from the DeepSeek Harness repository.

  1. Install Node.js, version 22.19 or later, or 24 and above.
  2. Run npx @deepseek-ai/dsh web.
  3. Open the Web UI at http://127.0.0.1:3080 and add a DeepSeek API key under Settings, then Models.
  4. Start a session in the folder you want DSH to work in.
  5. To build from source, you need Node ^22.19 or 24+ and pnpm 11, then run pnpm install && pnpm run build && pnpm dsh web.

Session logs stay local by default. Setting DSH_TELEMETRY_MODE=FEEDBACK_ONLY shares a log only when you submit feedback, and FULL uploads continuously. The repository carries a SAFETY.md notice that you should read before running an agent with file and shell access. DSH is a developer preview (the board tested 0.2.2-rc.1), and the repo warns that compatibility-breaking changes are expected. Its architecture treats everything as a plugin on Cordis, and profiles are ordered lists of plugin bundles managed with dsh plugin.

The project is MIT-licensed, with 279,576 stars and 34,424 forks at the time of checking, created on 13 August 2026 (GitHub). Popularity is not evidence of quality, which is why the independent board result matters more.

Which harness should you choose?

Choose DSH when the job is repository code changes on open models, the kind of work DeepSWE measures. Choose Hermes Agent, Pi or OpenCode when servers, watchers or dev APIs must survive after the agent finishes. Whichever you pick, verify results before trusting them, as we argue in verify-first vs build-first agentic engineering, and plan the task before handing it over, as in this plan-first coding workflow.

The board is also still moving. Cline is listed as the next harness in preparation (changelog), so rankings may shift. For a week-one view of the board's totals and caveats, bokonon.ai's analysis is worth reading.

FAQ

Q: What is DeepSeek Harness?
A: DeepSeek Harness (DSH) is an open-source coding agent from DeepSeek AI, released under the MIT licence. It uses an "everything is a plugin" architecture built on Cordis, according to its GitHub repository.

Q: Is DeepSeek Harness free?
A: The harness is free and open source, but you pay for the model API. The Harness Leaderboard measured an average of $0.53 per solved trial (Niklaus).

Q: Why does DeepSeek Harness fail some Terminal-Bench tasks?
A: It shuts down background programs it started when it finishes, while three tasks grade a server after the agent exits. DSH passed 2 of 36 attempts on them, per the announcement.

Q: Which models can run in DeepSeek Harness?
A: It talks to models through an OpenAI-compatible endpoint. The board used DeepSeek-V4.1-Flash, GLM-5.3-Flash and Flint, per the Space.

Q: Should I use DeepSeek Harness or Hermes Agent, Pi or OpenCode?
A: Use DSH for repository code changes. Use Hermes Agent, Pi or OpenCode when background services must stay alive after the agent exits.

Q: Does DeepSeek Harness send my data anywhere?
A: Not by default. Session logs stay local unless you set DSH_TELEMETRY_MODE to FEEDBACK_ONLY or FULL, as described in the repository.

Corrections: none yet. This article was researched and drafted with AI assistance and edited by a human; see our editorial policy for details.

Top comments (0)