"Verified data.json is fetched live and all four requested metrics are shown." Muse Spark Open sent that line from inside Claude Code 16 seconds after we asked for a small dashboard page. The verification was a text search for a few expected words. Muse never rendered the result, and every bar on it was empty. DeepSeek V4.1 Flash needed 2 minutes 29 seconds, loaded its page in a headless browser (Chrome with no window, driven from a script), studied the screenshots and delivered a dashboard that worked.
Meta made the Muse Spark Open weights public on 15 October, and both models went through identical coding tests on the 16th. The full Muse Spark Open vs DeepSeek V4.1 Flash comparison has the video, every table and the run records. Here we look at which numbers fool you when you plug models into coding agents.
A 16-second "done" with an invisible bug
The bug survives a quick read of the code. The fill only gets a colour from rules written as descendant selectors, CSS that matches an element only when it sits inside another one. Muse placed the bar beside the row instead of within it:
<div class="row c0">...</div>
<div class="bar"><div class="fill"></div></div>
<style>.c0 .fill { background: ... } /* matches nothing */</style>
The width animates on an element with no background. Open the page and you notice within a second. Grep the file for strings and you never will.
DeepSeek slipped too, in a milder way. One card footer calls two-task totals "each". The figures are right; the wording is not.
Tokens per second measures a service, not a model
Models read and write in tokens, chunks of a word, so speed is quoted in tokens per second. In the speed round, Muse on Baseten put code on screen in 2.5 seconds, streaming 455 tokens per second. DeepSeek streamed 197 per second and burned its whole 1,500-token allowance on private reasoning, ending with no code. Artificial Analysis, taking the median across every provider, finds the opposite: 179 tokens per second for DeepSeek, 102 for Muse.
Baseten serves an NVFP4 copy of Muse, the weights squeezed into 4-bit numbers so they take less memory and move faster. Its usage data also reports speculative decoding, where cheap guesses at upcoming tokens get checked by the big model in one pass. Each probably explains part of the lead; the test cannot separate them. Benchmark a hosted model and you are benchmarking its host.
The other trap is when you start the stopwatch. Time to first token, the usual latency number, counts reasoning tokens that never reach your screen. In a separate morning test, DeepSeek's first token showed up in under a second, yet its first readable words took 3 to 6 seconds. Muse needed about one. If your agent should feel responsive, time the first answer token on its own.
Cost per task, not price per token
LiveBench, a public test set, supplied eight coding problems from AtCoder. Each model solved seven, and both failed on the same problem. Muse produced roughly 88,000 output tokens against 59,000 for DeepSeek, at a higher rate per token. At list prices that put the eight problems at 0.38 USD for Muse and 0.07 USD for DeepSeek. In small Claude Code jobs on a toy banking library, both passed every hidden test, at 0.148 USD per task for Muse and 0.016 USD for DeepSeek. The DeepSeek numbers assume peak pricing, so treat them as ceilings.
Reasoning is billed as output, so a low token price means little until you know how long a model thinks.
The shim's settings are part of your experiment
Claude Code sends requests in Anthropic's format, while Baseten accepts only OpenAI's. A LiteLLM proxy translated between the two, and its config set Muse's reasoning effort, the knob for how long a model thinks, to medium. DeepSeek has its own Anthropic-compatible endpoint and kept its default effort, high. The dashboard round was therefore uneven. Whether Muse at high effort would have tested its own page is an open question, and we did not run it.
Record every setting a translation layer applies and count each as a test variable. The proxy config and the Claude Code setup for each model sit in the engineering appendix of the Muse vs DeepSeek article.
What to collect on your own shortlist
Before committing to a coding model, we would measure:
- time to the first answer token, not the first token
- output tokens per solved task, reasoning included
- cost per task at the price and hour you will actually pay
- whether the agent opens what it built before it says it is done
Limits: each test ran once, on a single day: eight problems and three Claude Code jobs, on public problems either model may have seen in training. The speed result describes Baseten, not Muse in general. Public weights do mean you can run Muse on your own hardware, but the FP4 weights take about 408 GB, and we have not tried. We had already run DeepSeek V4.1 Flash on two graphics cards in an earlier test.
Credit
Meta built Muse Spark Open and opened its weights, and Baseten had it serving on test day. DeepSeek made V4.1 Flash, released the weights and runs it on its own API. The problems and their tests come from AtCoder; the LiveBench team assembled the set on top of LiveCodeBench. Artificial Analysis maintains the independent scores. Anthropic builds Claude Code, and without the LiteLLM project's proxy, Claude Code could not have reached Muse. We wrote the test scripts, graded the answers and configured each model.
For the per-problem timings, the price breakdown and the exact prompts, read the complete Muse Spark Open and DeepSeek V4.1 Flash test on inferencekey.com.
Top comments (0)