Inception Labs recently bumped their free tier to 100 million tokens, which triggered my curiosity: what happens if you run a diffusion-based language model (which generates text in parallel instead of one token at a time) through a complete software development lifecycle?
My expectation going in was honest and low: a diffusion model, while absurdly fast, simply doesn't have the reasoning capability to be useful in any part of an SDLC. I had no illusion this would work. I ran it anyway ...because free tokens.
The results were not surprising, even though I had low expectations, I had hope that the model would perform.
Key Insights
Mercury 2 averaged 60.1/100 across Ship-Bench's five SDLC roles, passing only 1 of 5. The failures were not evenly distributed: upstream document stages scored in the high 70s, while implementation and verification collapsed into the 30s.
The upstream stages set up the downstream failure. Architecture (77.8), design (78.4), and planning (73.3) produced structurally fluent but thin artifacts, and that lack of upfront reasoning help compounded directly into the implementation phase's overall struggle (31.3, with every core user flow broken).
The practical lesson isn't "diffusion models aren't useful." Their speed has real value for routine, low-reasoning tasks, and Inception's own product split (Mercury for generation, Mercury Edit for latency-sensitive code suggestions) already points at where they fit. I am likely going to evaluate it for DumbQuestion.ai simply due to its raw speed.
Setup
This was a single-model experimental run, not a head-to-head. There is no comparison arm: just one model, one pipeline, and a hypothesis.
| Item | Value |
|---|---|
| Machine | Intel Mac Mini |
| Runtime | macOS Sequoia |
| Ship-Bench run | evals_july2026_mercury2 branch |
| Benchmark task | Standard Ship-Bench task: a local-first article/knowledge-base app |
| Item | Configuration |
|---|---|
| Harness | GitHub Copilot CLI v1.0.77 |
| Model | Mercury 2 (Inception Labs), a diffusion-based reasoning LLM |
| Backend | Inception Labs direct API, 100M free-token tier |
| Item | Value |
|---|---|
| Judge harness | Claude Code |
| Judge model | Opus 5 |
| Evaluation mode | LLM judge with live-search version verification, independent re-execution of the app, plus human operator review |
A note on the model
Mercury 2 is not a conventional autoregressive LLM. It's built on a diffusion architecture: instead of decoding one token at a time, it drafts a noisy version of the full response and refines it in parallel passes. Inception markets it as the world's fastest reasoning language model, with independent trackers listing a 128k context window and tool-use support (Artificial Analysis). The speed claims are real and well documented, which is precisely why this experiment was interesting. Speed is only useful if the output is worth having quickly.
Ship-Bench Context
Ship-Bench evaluates models across five SDLC roles: Architect, UX Designer, Planner, Developer, and Reviewer. Each phase produces artifacts that feed the next stage, which makes it a test of handoff quality across a realistic workflow, not just isolated output quality. A model that writes a weak spec hands a weak foundation to its own implementation phase. That compounding proved important in this run.
Overall Results
| Role | Score | Verdict | Gates |
|---|---|---|---|
| Architect | 77.8 | FAIL | 4/5 passed |
| UX Designer | 78.4 | PASS | 5/5 passed |
| Planner | 73.3 | FAIL | 3/5 passed |
| Developer | 31.3 | FAIL | 2/5 passed |
| Reviewer | 39.9 | FAIL | 2/5 passed |
| Average | 60.1 | 1/5 passes |
The shape of the table demonstrates the collapse. The document-producing stages cluster in the mid-to-high 70s. The moment the model had to do rather than describe (implement code, then verify it), the scores fall off a cliff. Ship-Bench's pipeline structure means those stages are also causally linked: the Developer inherited the Planner's broad chunks and the Architect's lean specs, and the Reviewer inherited the Developer's broken app.
Gate Failures
| Role | Gate failure | Practical severity |
|---|---|---|
| Architect | Frameworks gate: 6 of 10 pinned dependency versions were stale; one core dependency abandoned | The doc claimed versions were "verified via live web search" (they were not) |
| Planner | ≥70% right-sized chunks (actual: 40%); MVP-only scope (stretch features scheduled inside MVP) | Directly shaped the implementation phase's cost and failure concentration (iteration 4 was huge) |
| Developer | MVP flows barely work; styles never compiled; implements UX bare bones spec | Three unhandled 500s on primary routes; the app was effectively non-functional |
| Reviewer | Flows verified; defect list complete; evidence attached | Verified via curl instead of a browser; missed all three 500s and the total absence of CSS |
Two of these gate failures share a root cause worth naming: confabulated diligence. The architecture doc asserted a version-verification process that never happened. The developer phase's iteration summaries asserted "all tests passed" over suites that, in the final tree, cannot even initialize. More on the nuance of that second one below.
Architect
Can the model turn a product brief into a concrete technical plan with clear decisions and minimal unresolved ambiguity?
| Score | Verdict | Output | Eval |
|---|---|---|---|
| 77.8 | FAIL (gate) | architecture.md | eval |
LLM judge summary: Decision-dense and immediately actionable, with the search layer specified at DDL resolution (FTS5 virtual table, sync triggers, bm25 ranking), the strongest area of the document. It missed the 75-point bar's gate requirement on version currency: TypeScript, Prisma, Zod, Playwright, and SQLite were all pinned behind current stable, and the chosen markdown editor (React-MDE) has been unmaintained for roughly five years. The judge's remediation list was "narrow and mechanical."
Human notes: I think the judge is a little generous here. The document covers all the required sections, but it was extremely lean throughout: section presence over section depth. That said, other than the confabulated version-currency claim, it could still be used as a base.
Practical takeaway: A structurally sound skeleton with one fabricated claim of diligence: usable as a starting point, not as a plan.
Designer
Is the design specific enough to implement from: flows, states, layout, interaction detail?
| Score | Verdict | Output | Eval |
|---|---|---|---|
| 78.4 | PASS | design-spec.md | eval |
LLM judge summary: Implementation-grade handoff: real CSS values, Tailwind class strings, an executable Zod schema, named components at concrete paths, and a contrast claim that independently verified (5.56:1 actual vs. 5.0:1 stated; conservative). Deductions for a self-contradiction (the Draft badge is red in the token table, gray in the prose), no create-mode route, loading states defined only for buttons, and zero visual artifacts of any kind.
Human notes: The design spec had the same lean quality as the architecture. It covered the design system, but the screen descriptions and layouts (with no wireframes) left the coding agent to implement the barest of interpretations.
Practical takeaway: The LLM judge misses the critical importance of the wireframes leaving the development role to take the least tasteful path.
Planner
Can the model convert the specs into an executable delivery sequence: right-sized chunks, sane dependencies, testing integrated rather than deferred?
| Score | Verdict | Output | Eval |
|---|---|---|---|
| 73.3 | FAIL | backlog.md + iterations | eval |
LLM judge summary: Only 2 of 5 chunks were right-sized (40%, against a 70% gate). Iterations 2 and 3 were "model chunks": clean vertical slices of exactly one feature. But Iteration 1 spent 18 steps on scaffolding with no runnable result, Iteration 4 bundled three features into one chunk, and the plan contradicted its own stretch phasing: tags and draft/published status were declared post-MVP, then scheduled inside the MVP path.
Human notes: This was a critical failure in practice, not just on the rubric. The iterations were too broad and lacked task-level detail. Coupled with the lean design spec, the coding agent was forced into constant judgment calls. A stronger model (if I'd used Fable as the coding agent, it would have figured it out) fills those gaps with its own reasoning. A smaller model like Mercury 2 needs all the upfront guidance it can get, and its own upstream stages didn't provide it.
Practical takeaway: The plan's failure is literally visible in the billing data: the overloaded Iteration 4 consumed half the run's entire cost.
Developer
Can the model implement the backlog into a working MVP, staying aligned to the prior artifacts?
| Score | Verdict | Eval |
|---|---|---|
| 31.3 | FAIL | eval |
LLM judge summary: Every MVP flow failed. The article detail page returned HTTP 500 for every article (a Next.js 16 async-params bug: the code read params.id synchronously). Search returned 500 unconditionally because the FTS5 table was never created by any migration. The edit page hung on "Loading…" forever. Tailwind was never loaded, so the entire app rendered as unstyled serif HTML. Effective test coverage: 0%. The Jest environment package was missing, no Playwright config existed, and tsc reported 8 errors including literal ntest typos.
Human notes: My expectations were already low, and the run managed to come in under them. The model genuinely struggled with tool use: it repeatedly tried to run commands as if they were named tools ("ls tool does not exist") before retrying through the actual Shell tool, and it would try to edit files that never existed, in folders that didn't exist. It did eventually recover from these fumbles, fortunately. It also shipped a malformed package.json (a missing delimiter) which I had to fix myself just to get the app to run. Most models would have caught that the moment they tried to run or test.
One nuance worth being fair about: it did run unit and e2e tests during the run. The iteration summaries claiming "all passed" weren't necessarily fabricated: my working theory is the final iteration broke things after earlier checkpoints went green, and nothing re-ran the suite on the final tree. A competent developer's last act is re-running the tests before declaring done. Mercury 2 treated verification as a milestone to check off rather than a gate to exit through.
Practical takeaway: The gap between "writes plausible code" and "operates a development loop" is enormous, and this run sat entirely on the wrong side of it.
Reviewer
Does the reviewer actually close the loop: run the app, catch defects, make the correct ship/no-ship call?
| Score | Verdict | Output | Eval |
|---|---|---|---|
| 39.9 | FAIL | qa-report.md | eval |
LLM judge summary: A structurally excellent report (severity tiers, reproduction commands, an ordered fix list, and an unambiguous NO-SHIP) undermined by its method. Verification was done via curl against endpoints rather than in a browser, so it missed all three HTTP 500s in the required flows, the complete absence of applied CSS, and the missing /login route. Two sections contained claims affirmatively contradicted by re-execution.
Human notes: I actually rate the QA role as relatively the strongest of the five. It listed many real failures and committed to a negative result. Its testing was lean, but it didn't try to be overly positive: it was willing to say no.
Practical takeaway: Right verdict, wrong method. For a QA role, reaching the correct ship decision counts for something, but an audit that's wrong about why in several specifics is not one you can trust at scale.
Token and Cost Analysis
The quality collapsed, but the economics deserve their own look.
| Metric | Value |
|---|---|
| Input tokens | 12.66M |
| Output tokens | 56.1k |
| Estimated total cost (market rate) | $3.21 |
| Actual cost (100M free-token tier) | $0.00 |
| Cost per average score point | ~$0.053 |
Market-rate equivalent at Mercury 2 list pricing ($0.25/M input, $0.75/M output). I didn't capture cache-hit numbers, so treat $3.21 as an upper bound: the real metered figure would likely be far lower.
Three observations. First, Iteration 4 (the chunk the judge flagged as overloaded) consumed half the run's input tokens and half its total cost. Second, the input/output ratio is roughly 226:1; for a model whose entire pitch is fast parallel output generation, output was largely irrelevant. The run was dominated by context ingestion (consistent agentic development). Third, the whole five-role pipeline cost less than a coffee but really bad tasting coffee.
App Comparison
No screenshots exist for this run: it wasn't worth taking any. The rendered application was unstyled default-browser HTML (Tailwind was never compiled into the page), the home route still served the iteration-1 scaffold text, and no navigation existed anywhere in the app. The visual comparison is not "provisional"; it's simply absent, which is itself the finding.
Interpretation
My hypothesis going in was that a diffusion model, however fast, lacks the reasoning capability to be useful in any part of an SDLC. The run largely confirmed it, but the shape of the confirmation is more useful than a flat "no."
Mercury 2 can clearly produce documents about software work. The upstream artifacts were structurally fluent but consistently lean in substance, and occasionally decorated with claims of diligence that didn't survive contact with execution. When the pipeline demanded judgment under ambiguity (planning detail), operation of a live toolchain (implementation), and empirical verification (QA), the performance collapsed.
The compounding effect is what I'd emphasize. A strong model can recover from a weak handoff: it fills spec gaps with its own reasoning. Mercury 2 needed the strongest possible guidance from its upstream stages and got its own thinnest work instead.
But I am not walking away from diffusion models. The speed is real, and it's mind-melting. What's missing is the depth of reasoning that multi-step agentic SDLC work demands, not capability writ large. For routine tasks where latency is the feature and the reasoning burden is low, this architecture is genuinely compelling. Inception clearly agrees: they ship Mercury Edit, a code-specific model aimed at small edits and suggestions in latency-sensitive scenarios. That's the right shape for what this technology currently does well. I'll absolutely be exploring Mercury for applications where its speed is the point (like in DumbQuestion.ai).
Verdict
This run showed that diffusion-based models, at least at Mercury 2's current capability level, are not ready to run an end-to-end software development lifecycle, and the bottleneck is reasoning, not speed or cost.
If your goal is agentic, multi-stage SDLC automation where each phase must operate on and verify the work of the last, use a conventional reasoning/autoregressive model. If your goal is fast turnaround on bounded, low-reasoning tasks (autocomplete-adjacent edits, quick suggestions, routine transformations), the diffusion architecture's speed makes it worth your attention today.
Top comments (0)