- Hermes IDE: Download free · Star it on GitHub — one console for every coding agent, on macOS, Windows and Linux
- Book: AI Agents Pocket Guide · AI in TypeScript — the 5-book series
- Me: xgabriel.com | GitHub
Meta's chart says Muse Wonder beats Claude Opus 5.5 at long coding
jobs: 78.8 to 74.2 on DeepSWE, 53.1 to 46.3 on SWE-Marathon
(model card).
Artificial Analysis
gives Wonder 52 on its Intelligence Index. Opus 5.5
scores 58.
Both cost $4 per million input tokens and $20 per million output
(Wonder's price per
TechCrunch,
Opus's per the same Artificial Analysis article).
Meta's chart and the independent index can both hold, because they
measure different things, and neither ran on your code.
Meta launched Muse Wonder on October 12, 2026
(Meta newsroom).
In the Hacker News thread,
one commenter called it "much worse than Opus 5.5" after an
image-to-HTML test. That is one prompt. Nothing below is a Muse
Wonder result. What follows is the setup to get your own: the same
ticket handed to two agents in two git worktrees, and a script that
scores both.
Why the charts disagree
Wonder wins five of Meta's 14 rows against Opus 5.5, mostly
long-horizon coding, legal agent work and Meta's internal memory
test. It loses nine, including ProgramBench, where Opus leads 70.5
to 50.0. The row-by-row view is in
the scorecard breakdown.
Meta's own footnotes tell you how far to trust any single row. For
competitor runs it did in-house, Meta says they "use common settings
where practical but remain best-effort: prompts, tools, and runtime
may not be optimally tuned for proprietary models." And from the
limitations section of the
model card:
"Performance can vary with quantization, serving configuration,
prompts, tools, and the agent harness."
No leaderboard can hold prompts, tools and harness fixed for your
setup, but a bake-off on your own repo does exactly that.
One ticket, two agents building the same thing, one scale to weigh them.
Step 1: one ticket, one API key
Choose a real ticket with a test that fails today (or write one
first), a fix that touches a few files, and an outcome you can judge.
Save the prompt to a file outside the repo so every agent gets the
same words, and name the test command in it. "Make
npm test -- slug pass without changing the tests" gives the agents
and you the same finish line.
Next, create a key for the Meta Model API and export it under the
name Meta's coding-agents docs
use. Put it in your shell profile (or load it from your secrets
manager) so any new terminal has it:
export MODEL_API_KEY="your-key-here"
The API model ID is muse-wonder-1.0. The API does not serve Muse
Wonder Full, the stronger sibling: it is for Muse Code subscribers
only
(TechCrunch).
Step 2: session A, Claude Code on Opus 5.5
I build Hermes IDE, a free desktop console
for coding agents, so the setup below uses it
(download it to follow along). The
plain-git version follows session B.
Press ⌘N (Ctrl+Shift+N on Windows and Linux), pick Claude Code in the
agent chip, pick Opus 5.5 in the model chip, paste the prompt and
press Enter. Claude Code uses its normal sign-in. The task starts in
its own git worktree on a new hermes/<task> branch, cut from the
branch you are on.
Step 3: session B, the same Claude Code on Meta's model
Meta documents three environment variables that point Claude Code at
its API: ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN and
ANTHROPIC_MODEL. The docs also tell you to set the Opus, Sonnet and
Haiku aliases and the subagent model to the same ID, "so no path
falls back to a Claude model". Skip those and some requests ask for
a Claude model ID instead of Muse Wonder, and session B stops being
a clean Muse run.
Seven variables make a long command, so put them in a wrapper. Save
this as claude-muse somewhere on your PATH and chmod +x it:
#!/usr/bin/env bash
# claude-muse: Claude Code against Meta's Model API
export ANTHROPIC_BASE_URL="https://api.meta.ai"
export ANTHROPIC_AUTH_TOKEN="$MODEL_API_KEY"
m="muse-wonder-1.0"
export ANTHROPIC_MODEL="$m"
export ANTHROPIC_DEFAULT_OPUS_MODEL="$m"
export ANTHROPIC_DEFAULT_SONNET_MODEL="$m"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="$m"
export CLAUDE_CODE_SUBAGENT_MODEL="$m"
exec claude "$@"
The base URL has no /v1, and the key goes in AUTH_TOKEN, not
API_KEY. Both details come from Meta's docs. Their worked example
uses muse-spark-1.3; muse-wonder-1.0 is the Wonder ID.
Now press ⌘N again and pick Custom agent ("Any command you
type"). Name it Claude Code · Muse Wonder and set the command to
claude-muse. The hint under the field says it plainly: "Hermes
starts this command in the session's terminal, as if you typed it."
Because it's a real shell, an env-prefixed one-liner works too. When
the CLI is up, paste the same prompt.
Sessions A and B now run the same harness on the same ticket. What
differs is the model and the API serving it. Meta's footnote on its
SWE-Marathon row calls that row "not a controlled common-harness
comparison". Your A-versus-B run is a common-harness one.
Real Hermes window, test build with stand-in agents. The same task in two sessions side by side, each on its own worktree branch. In your bake-off, the Custom agent takes the second slot.
No Hermes? Plain git does the same
The isolation part is one command per agent. From your repo:
git worktree add -b bakeoff/opus ../bakeoff-opus
git worktree add -b bakeoff/muse ../bakeoff-muse
Then run claude in ../bakeoff-opus (pick Opus 5.5 with /model)
and claude-muse in ../bakeoff-muse, each in its own terminal.
You lose a single place to watch both, and the review step later.
Hermes worktrees are plain git branches too, so nothing here locks
you in.
Optional step 4: session C, Meta's own muse CLI
Session B tests the model inside Anthropic's harness. If you want
Meta's harness too, install Muse Code. Meta's installer creates a
command named muse in ~/.local/bin. Read the script first if you
prefer, then run:
curl -fsSL https://dev.meta.ai/install.sh | bash
Add one more Custom agent with the command muse. Muse Code is a
subscription: the Power Usage plan
is $50 a month and includes Wonder Full. Meta's product page lists
macOS and Windows. One early report is negative: in the same Hacker
News thread, a commenter said the Muse Code sandbox "never worked
across Mac or Linux". Check it on your own machine before you pay
for a plan.
Step 5: score both worktrees with a script
When both agents say they're done, score them the same way. This
script takes the base branch, a test command, and any number of
worktree paths. For each one it prints the branch, whether the tests
pass, how many files changed and how many lines went in and out
against the point where the branch left the base. Save it as
compare-worktrees.sh next to your repo and chmod +x it:
#!/usr/bin/env bash
# Score agent worktrees against the branch they started from.
# usage: compare-worktrees.sh BASE "TEST COMMAND" WORKTREE...
set -u
base=$1
test_cmd=$2
shift 2
for wt in "$@"; do
branch=$(git -C "$wt" rev-parse --abbrev-ref HEAD)
fork=$(git -C "$wt" merge-base "$base" HEAD)
# count untracked files before the tests can add any
new=$(git -C "$wt" ls-files -o --exclude-standard | grep -c .)
(cd "$wt" && bash -c "$test_cmd") >/dev/null 2>&1
code=$?
if [ "$code" -eq 0 ]; then result=PASS; else result=FAIL; fi
# working tree vs fork point: committed and uncommitted edits
stats=$(git -C "$wt" diff --numstat "$fork")
files=$(printf '%s' "$stats" | grep -c .)
added=$(printf '%s\n' "$stats" | awk '{s+=$1} END {print s+0}')
removed=$(printf '%s\n' "$stats" | awk '{s+=$2} END {print s+0}')
echo "== $wt"
echo " branch: $branch"
echo " tests: $result (exit $code)"
echo " files: $files changed, $new untracked"
echo " lines: +$added -$removed vs $base"
done
It diffs the working tree against the fork point, so it counts edits
an agent made but never committed. Brand-new files git doesn't track
yet show up in the "untracked" count instead of the line totals. That
count runs before the tests, so files the test run writes don't
inflate it. Anything the agent's own test runs left behind still
counts unless your .gitignore covers it.
The output below is not from Muse Wonder or Opus 5.5. It comes from a
throwaway toy repo (git 2.52, Python 3.14, __pycache__/ in
.gitignore) with two hand-written fixes for a broken slugify()
standing in for agent output. Fix A rewrote the function with a
regex. Fix B stripped punctuation, still produced double hyphens on
extra spaces, and left a TODO in the test file:
$ git worktree add -b bakeoff/agent-a ../agent-a
Preparing worktree (new branch 'bakeoff/agent-a')
HEAD is now at 759ecee Initial commit
$ git worktree add -b bakeoff/agent-b ../agent-b
Preparing worktree (new branch 'bakeoff/agent-b')
HEAD is now at 759ecee Initial commit
$ ../compare-worktrees.sh main "python3 -m unittest -q" \
> ../agent-a ../agent-b
== ../agent-a
branch: bakeoff/agent-a
tests: PASS (exit 0)
files: 1 changed, 0 untracked
lines: +5 -1 vs main
== ../agent-b
branch: bakeoff/agent-b
tests: FAIL (exit 1)
files: 2 changed, 0 untracked
lines: +6 -1 vs main
In Hermes the worktrees are plain git too, so git worktree list in
your repo prints the paths to pass in.
Fill in the scorecard
The script covers the first three rows. The rest you note while you
watch:
| A: Claude Code, Opus 5.5 | B: Claude Code, Muse Wonder | C: muse (optional) |
|
|---|---|---|---|
| Tests pass (exit code) | |||
| Lines +/- vs base | |||
| Files touched | |||
| Wall time to "done" | |||
| Cost | |||
| Review comments before you'd merge |
Wall time.
Artificial Analysis
measured Wonder's time to first token at 19.27 seconds at launch,
against a 5.52-second median for comparable models. An agent makes
many calls per task, so expect that to show.
Cost. Meta's docs warn that agents pointed at a custom provider
often show $0 cost, because they lack pricing metadata. Don't read
cost from session B's terminal. Take the run's input, cached and
output token counts and do the math: input millions × $4, cached
input millions × $0.20 (the cached rate
Handy AI
reports), output millions × $20. The per-token prices match Opus
5.5, but per-task cost can still differ. Artificial Analysis counted
130M output tokens for Wonder to run its index, against an 85M
median. For session A, Hermes shows an estimated cost per session.
Step 6: review in Hermes, land the winner
Press ⌘G (Ctrl+Shift+G on Windows and Linux) to open the Review Desk
and switch to By turn. You read each agent's work turn by turn
instead of as one final diff. Risk flags mark lockfiles, CI
workflows, secrets, new binaries and edits an agent makes to its own
config.
From hermes-ide.com: a recreation of the 2.0 app with demo data. The Review Desk by turn, with a line comment for Claude Code and a drafted commit message.
A line comment goes back to the agent that wrote the line. Count
those comments: that is the last row of your scorecard. Hermes knows
less about a Custom agent than a built-in one, so if session B's
turns look thin, read it By file, or fall back to the script's
numbers and a plain git diff against your base branch.
Land the winner from the Land sheet: commit, open a pull request or
merge locally, with undo afterwards. Then close the other session.
By hand, remove the losing worktree with
git worktree remove ../bakeoff-muse (or ../bakeoff-opus).
Where one bake-off falls short
One ticket is an anecdote. Run three to five tickets of different
shapes before you call it, because a model that wins a refactor can
lose a bug hunt.
Hermes has no status signals for Custom agents. It doesn't know the
command's approval model, so the attention inbox can't tell when
session B or C is waiting on you. Watch their terminals. Session A
keeps full status.
You're also testing Muse Wonder, not Wonder Full. The API doesn't
serve Full, so only session C, on a plan that includes it, reaches
it. And session B is Anthropic's harness on Meta's model: fair for
comparing models, maybe not how Meta tuned Wonder to work, which is
why session C exists.
Hermes IDE is not affiliated with or endorsed by Meta, Anthropic,
OpenAI or Nous Research.
Run the bake-off
Pick the ticket you'd hand an agent this week, run it through both
sessions, and land the one you'd rather review again.
I build Hermes IDE. It is free, source-available, and runs on macOS,
Windows and Linux. It ships often: 2.2.10 came out on October 12,
- Download it here, and if it settles a model argument on your team, star the repo. It's built in the open, and a star helps other developers find it. If the Custom agent setup misbehaves with Meta's API, or you want status signals for it, open an issue.
If you're designing the agents themselves, the
AI Agents Pocket Guide
covers the patterns behind them.
Top comments (1)
tr.ee/dev-to