Claude Haiku 5.5 vs GPT-6 Luna: Cost, Benchmarks

Claude Haiku 5.5 vs GPT-6 Luna is the small-model match-up Anthropic picked for its own launch chart. So we ran both on our two agentic coding benchmarks. Haiku won both. On Forge 1.0, an Atlassian Forge app built with no internet, it scored 0.9661 to Luna's 0.395, fourth of 31 runs and 0.0106 behind Claude Opus 5.5. On Gauntlet 7.2, a payments app, it scored 0.661 to 0.4797. Both models list at $0.10 per million input tokens and $0.50 per million output. Haiku's two runs billed $1.62. Luna's billed $0.46.

Same 150-call budget, both through OpenRouter, nobody touching the keyboard. Same sticker price. The bill still came out 3.6 times higher for Haiku. Most of that isn't Haiku being chatty. Its price steps up five times once a request's prompt passes 100,000 tokens, and our agent resends the whole conversation on every call. In these runs 64 of Haiku's 111 calls were over the line. One run crossed eight calls early because of a bug in our own agent, and I'll show you the call. Some things differed between the two models' runs: Luna ran a week earlier, on an older build of our agent and scorer, on a different machine. I come back to all of it.

On our payments app the two are close once you take out one failed step each. On our Forge build they aren't close at all. And the price on the sticker isn't the price of an agent.

Below: the scores and how to read them, the bill call by call, what each model built and where it broke, what other benchmarks say, what could be wrong with my numbers, and which model I'd use for what. The full design of both benchmarks is in Claude Sonnet 5.5 vs Opus 5.5, and small-vs-big inside one family is in Qwen3.8 Flash vs Max Prime.

Claude Haiku 5.5 vs GPT-6 Luna: the scores on both boards

These are the boards as I read them on Saturday 10 October, 10:11 EEST. The Gauntlet board is the current word, not this table. The Forge 1.0 board is frozen, so its column won't move. On the Forge page, switch the version to Forge 1.0 to see it. Haiku's bigger siblings and the Forge leader are there for scale.

model	Gauntlet 7.2	rank of 34	Forge 1.0	rank of 31	bill, both runs
Claude Haiku 5.5	0.661	15	0.9661	4	$1.62
GPT-6 Luna	0.4797	22	0.395	16	$0.46
Claude Sonnet 5.5	0.7984	3	0.5508	12	$6.79
Claude Opus 5.5	0.7926	6	0.9767	2	$11.60
GPT-6.1 Sol	0.7978	4	0.9769	1	$2.00

The names are OpenRouter's: anthropic/claude-haiku-5.5 and openai/gpt-6-luna. Every run is one model driving goose, our fork of the open-source agent Block started, with a 150-call budget and no human help. Bills are what OpenRouter's per-generation records add up to, and they're lower bounds. The run pages hold every check: Haiku's Gauntlet run and Forge run, Luna's Gauntlet run and Forge run.

How to read these scores

Each benchmark is a few hundred automated checks, grouped into tiers like storage, UI or 3D interaction. 88% of a run's score is its tier scores. I call that the core score below. The other 12% is excellence, which here mostly means economy: how few API calls the app spends. A critical is a failure bad enough to multiply the whole run, by 0.6 at full severity. A band is a group of must-pass checks, and failing one puts a ceiling on the run, 0.699 or 0.799, whatever else it did. Forge pins reasoning effort to medium and cuts the building model off the internet. Gauntlet runs each model at its default effort, which both vendors document as medium, with the network open. The exact rule is on the methodology page.

One caveat, said once for the whole piece. Each model has one run per board. One run can't separate a model from its luck, and one early decision in a Forge build can move a run a whole band. Read everything below as what happened in these runs and why the scorer says so. It isn't a law about the models.

What we found in the four runs

1. The 100k step, not the token count, drove most of the bill gap. 88% of Haiku's Gauntlet prompt tokens and 75% of its Forge prompt tokens were in requests over the line. My re-pricing model reproduces both real bills within 5.5%.
2. One tool result did a lot of that damage. Haiku tested its own services with one shell command that chained fourteen curl calls, and the output came back at 188,758 bytes on 14 lines. Goose said the output was over its limit and passed it all through. The prompt jumped from 57,094 to 148,478 tokens in one call.
3. Haiku's Forge app passed every core check, and its economy was nearly Opus's. The whole 0.0106 gap to Opus is two Jira calls on the backfill, 16 against Opus's 14, plus a rounding's worth on the live path.
4. Luna's Forge app wrote a perfect live ledger and an empty history. The scorer's live event rows were 22 of 22 exact; its first scheduled backfill recorded 0 of 62 past changes. Every row in its sprint modal says "event", never "reconcile".
5. Luna's manifest had two problems that bite at runtime: no scope for the Jira Software call that lists a sprint's issues, which the site answered with a 401, and no llm module for the Forge LLM call its code makes.
6. On Gauntlet, Luna's run spent all 150 calls, at $0.0015 a call. Haiku declared itself done after 39 calls, at $0.028 a call, writing about ten times more output per call.
7. Haiku's Gauntlet ceiling came from one numbering choice in a hidden pick buffer. It drew every tower part right in all 8 inspected poses and still failed the band.

Haiku vs Luna cost: the same list price, 3.6x the bill

Here are the two price lists side by side, from the vendors' own pages, read 10 October.

	Claude Haiku 5.5	GPT-6 Luna
released	7 October 2026	22 September 2026
input / output per million tokens	$0.10 / $0.50	$0.10 / $0.50
cache read	$0.01	$0.01
when the price steps up	prompt over 100,000 tokens	prompt over 272,000 tokens
price above the step	$0.50 / $2.50, cache read $0.05	2× input and cache, 1.5× output
context window / max output	1M / 128K	1.05M / 128K
default reasoning effort	medium	medium
knowledge cutoff	June 2026	18 May 2026

Anthropic's pricing page is explicit about what counts: "A request's prompt length counts all of its input tokens, including cache reads and cache writes. Each request is priced on its own: a request over the threshold pays the higher prices even when part of its prompt is a cache hit…" OpenAI's Luna page sets its step at "more than 272K input tokens". Haiku 5.5 is also the one current Claude model the pricing page leaves out of "the full 1M token context window at standard pricing". Sonnet 5.5 and Opus 5.5 have no step.

And here is what the four runs did.

run	calls of 150	how it ended	$ billed	$ per call	output tokens per call	wall time
Haiku 5.5 · Gauntlet	39	stopped itself	$1.1046	$0.0283	11,778	36.6 min
GPT-6 Luna · Gauntlet	150	used the whole budget	$0.2315	$0.0015	1,142	33.5 min
Haiku 5.5 · Forge	72	stopped itself	$0.5186	$0.0072	2,871	20.1 min
GPT-6 Luna · Forge	144	n/a	$0.2239	$0.0016	n/a	12.5 min

Each run includes one tiny side call our harness makes to another model, and the bills are lower bounds. Luna's Forge token counts and stop reasons are in files I couldn't reach for this piece; more on that below. Wall time is the agent's own time, before scoring. Luna made 150 calls in less time than Haiku made 39: about 13 seconds a call against 56.

GPT-6 Luna benchmarks: 150 calls at $0.0015 each

OpenAI calls Luna "Our most efficient model for focused, high-volume tasks", and that's how it worked. On Gauntlet it read 12.9 million prompt tokens over 150 calls and wrote 1,142 output tokens a call. Haiku wrote 11,778 a call, and 80% of its billed output was reasoning. Luna made 150 short calls. Haiku made 39 long ones.

GPT-6 Luna Pro, the same model in OpenAI's pro reasoning mode, has a Gauntlet run too. The Sonnet vs Opus piece covers it: 0.1343 for $1.90, more than eight times plain Luna's bill for less than a third of its score. Luna Pro has no Forge run, and nothing on our boards ran plain Luna at higher effort.

Claude Haiku 5.5 pricing over 100k tokens: where the step bites an agent

Goose resends the whole conversation on every call: the system prompt, the task, every file it wrote and every tool result so far. Prompt caching makes the repeated part cheap, but it still counts toward the 100k. In our runs the prompt only grew, and once one call crossed the line every later call did too.

On Gauntlet, 24 of Haiku's 39 calls were over 100k, starting at call 16, and those 24 calls carried 88.1% of its prompt tokens and 65.8% of its output tokens. On Forge, 40 of 72, starting at call 33, carrying 75.4% of the prompt tokens. I re-priced every call from the telemetry at Anthropic's two tiers, using each run's measured cache-read share, 95.7% and 97.7%. The model lands within 5.5% of the real bill on both runs, $1.1533 against $1.1046 and $0.5469 against $0.5186. By that model, the same calls without the step come to about $0.30 on Gauntlet and $0.19 on Forge. Luna billed $0.23 and $0.22. Roughly level. Against the real bills, the step cost roughly 3.7 times on Gauntlet and 2.7 times on Forge.

Caching doesn't get you out of it. Over the line, a cache read costs $0.05 instead of $0.01, and almost all of Haiku's prompt was cache reads.

Luna's Gauntlet calls averaged 86,061 prompt tokens, a long way under its 272K step. I don't have its per-call maximum, so I won't claim none crossed.

One of the first comments in the Hacker News thread on Haiku 5.5's launch day said the 100k cutoff "will be quickly exceeded if you are doing anything with Agents". Our two runs crossed it at calls 16 and 33.

At least one comparison page ranking for this pairing computes its cost gap, 2.7×, from Artificial Analysis's per-task costs, and says itself that AA prices Haiku at the under-100k rates, so its real cost would be higher on any task that crossed. That makes 2.7× a floor. Ours is two billed agent runs with the step in.

Part of Haiku's bill was our bug

The Gauntlet chart has a cliff at call 16. The prompt went from 57,094 tokens to 148,478 in one call. No other call in either run jumped even 15,000.

I went looking in goose's request log. The cliff isn't a file. It isn't reasoning either. While testing its app, Haiku ran one shell command that chained fourteen curl calls against its own services. Two of them returned long JSON on a single line each: its notifier's list of processed events, 154,712 characters, and its event feed, 31,212. Goose's shell tool has a 50,000-byte output limit. The whole output was 188,758 bytes on 14 lines, and goose's notice to the model said so: "Output exceeded 50000 byte limit (188758 bytes total). Full output saved to…". Then it sent all of it anyway.

The cause is a few lines in goose's shell tool. It builds its preview by counting lines, and 14 lines is under its 50-line preview. The notice was true. The cut never happened. That's our bug, not Haiku's. I've filed it in goose's queue.

What did it cost? If the output had been cut to 50,000 characters, about 67,000 fewer tokens would have ridden along in each of calls 16 to 39. By the same re-pricing model, that run crosses 100k at call 24 instead of 16, 16 calls over the line instead of 24, and comes to about $0.48 instead of $1.15. That assumes Haiku would have done the same work afterwards, which I can't test. Forge has no jump like it; that run crossed the line by plain growth.

What keeps Haiku under the step, and what I haven't tested

Goose compacts the conversation at 80% of the model's context window by default, set by GOOSE_AUTO_COMPACT_THRESHOLD. On Haiku's 1M window that's around 800,000 tokens. These runs peaked at 194,766, so compaction never fired. To keep Haiku under the step on that window you'd need a threshold under 0.1. I haven't run that. Compaction throws away context the model may need. So I don't know what it does to the score.

What I'd do first is cheaper. Cap tool output for real, so one shell command can't put 91,000 tokens in the conversation. That's the fix I've filed.

Is Claude Sonnet 5.5 worth it instead?

Sonnet 5.5 lists at $2 and $10, twenty times Haiku's sticker. No step, though. Its bills on the same two benchmarks were $5.34 and $1.45, 4.8 and 2.8 times Haiku's. On Gauntlet that bought 0.7984 against 0.661. On Forge, the board says 0.5508, but that number has known scorer problems I come to below. In these agent runs, Sonnet cost a few times Haiku, not twenty.

Claude Haiku 5.5 vs GPT-6 Luna on Forge: 0.9661 to 0.395

Forge 1.0 asks for a real Atlassian Forge app. At its heart is a scope ledger that records every issue added to or removed from a running sprint, and around it sit a dashboard widget, a sprint action with a table and a summary comment, a Rovo action, and the Forge LLM explaining the creep. It's graded on three seeded Jira sites. The build has no internet. Whatever the model knows about Forge is what it brought.

Haiku lost only economy, and nearly matched Opus there

Haiku's Forge run has 66 checks and two of them are below 1.0. Both are economy. On the live-event path it made 91 Jira calls where the scorer's optimum is 24: one read per relevant change plus one forced retry. That's 3.79 times the optimum, and it scores 0.4028. On the backfill it made 16 calls against an optimum of 15, and that one extra call drops the row from 1.0 to 0.75. That's the whole loss. The final is 0.88 × 1.0 + 0.12 × 0.7176 = 0.9661.

Opus 5.5 got the same perfect core. It spent 90 calls on the live path against an optimum of 27, a slightly better 3.33 times, and 14 on the backfill. So the 0.0106 between them is almost all two backfill calls. That's also about what the review says Forge 1.0's scoring faults can move among the top four, up to 0.01. Don't read an order into it. The scorer graded Haiku's app on three seeded sites and they agreed to within 0.0012.

The app itself is small. It's 1,352 lines across the manifest, backend and three UI surfaces, 141 of them manifest. Two storage entities, each indexed with exactly one range attribute, which is the rule Qwen3.8 Max Prime broke. An issue trigger, a queue consumer, an hourly scheduled trigger for the backfill, the widget, the sprint action, a Rovo action and an LLM module. The kit's pre-deployment check said "would deploy and run", 0 lint errors.

Luna's ledger never learned the past

Luna's app did the hard live part well. Sprint changes arriving as Jira events landed in the ledger exactly: 22 of 22 on the scorer's live event-rows check, though only 3 of 25 changes that touch several sprints at once. It read the ledger the right way, 28 of 28 reads through the sprint index, and a re-run with nothing new wrote nothing.

What it never did was import history. Not one row. Forge 1.0 seeds each site with sprints that already have changes in them. An app installed mid-sprint has to backfill those from Jira on its first scheduled run. After that first run Luna's ledger held 0 of the 62 past changes the scorer's answer key expected. It found 0 of 12 removals to the backlog and healed 0 of 4 deliberately dropped events.

That backfill check is two things at once in Forge 1.0. It's a critical, labelled "data loss — changes silently missing", which multiplies the run by 0.6. And it belongs to the 0.699 "working ledger" band. Luna earned 0.6583 before the multiplier and 0.395 after it. The ceiling never came into play. The multiplier did.

I checked every capture of Luna's sprint modal, the two on the run page and the six from the original scoring before the re-score, and the Source column says "event" on every row. Haiku's says "reconcile".

Several of Luna's other zeros fit the same hole. Widget numbers: 0 of 4 views exact. Ledger table: 0 of 3. The Rovo action's answers about active sprints: 0 of 3. Re-estimated sprints: 0 of 3. The scorer's own map lists those rows as downstream of a failed backfill, and a sprint total computed from a ledger with no history is wrong in exactly that way. I haven't traced each row in Luna's run, so read it as consistent with, not caused by.

Forge's docs make a neighbouring point. Async events are "guaranteed to be delivered at least once within a defined retention window", and product events can be retried, so even an app that is listening can see an event twice. History from before the install never arrives as an event at all. A ledger has to go and read it.

Luna's manifest was missing two things

The scopes check says Luna's manifest had no scope for one Jira Software call its app made: the one that lists a sprint's issues. The site answered it with a 401. That's the natural call for a backfill to make. Is that why the backfill came back empty? I can't prove it without Luna's code. The same check lists four declared scopes no observed call used, one of them the comment-write scope.

The manifest check flags the second problem: Luna's code uses @forge/llm, and its manifest declares no llm module. Forge refuses that call. That explains why the scorer never saw Luna's app call the Forge LLM, and why its explain feature scored 0 of 5.

The scorer also never saw Luna's app post its summary comment, so those checks were skipped for lack of anything to check. I can't tell you why. A bug in our test harness, confirmed on 9 October, drew feature-flag badges over apps' own buttons so the scorer's clicks could miss. It hit other runs. Luna isn't named among them. Even at its worst it would be worth a few hundredths here, and it doesn't touch the backfill.

Luna's widget looked finished and didn't get one sprint fully right

Luna's UI is the nicer of the two. Cards, a live indicator, a legend that reads. The visual tier agrees: Luna scored 1.0 on visual, same as Haiku. Function is a different tier. Luna 0.48, Haiku 1.0. Luna's chart bars were off their own linear scale by up to 70 pixels, and its ledger table's sort got 0 of 6 toggle states right.

Haiku owned up to a curl

Each run ends with the model's own closing note. Haiku's Forge note is candid. It lists what it didn't verify: "The sprint-action sort toggle, row selection, router links and the explain button are coded but not exercised in a browser." Then this: "I used curl against the dev site once early on, which STARTER.md says the app must not do. I stopped and used only the forge-dev tools after that." The dev site is the local seeded Jira. The sandbox blocks the internet, and the accuracy review records the sandbox denying two of Haiku's reads outside its workspace. Its 0.9661 was checked against all of that and called clean. The things it coded and never clicked all passed when the grader clicked them.

Haiku 5.5 vs Sonnet 5.5 and Opus 5.5 on Forge: the small one matched the top on every core check

The practical line first. On Forge, Haiku matched Opus 5.5 on every core check for $0.52 against $4.26. On Gauntlet, Sonnet and Opus were about 0.13 ahead.

Anthropic's launch post says where Haiku should sit: "Sonnet 5.5 and Opus 5.5 remain better choices for complex agentic coding tasks like those measured by Terminal-Bench 4.0. By contrast, Haiku 5.5 is best suited to more narrowly scoped tasks…" On Gauntlet that's what we saw. Haiku's core score, 0.9349, is well under Sonnet's 0.9879 and Opus's 0.9864. On Forge the gap closed: Haiku's core was 1.0, Opus's was 1.0, and Opus finished 0.0106 ahead on economy.

The board also puts Haiku 0.4153 above Sonnet 5.5. Don't read that as Haiku knowing Forge better than Sonnet. The accuracy review Haiku's score triggered confirmed two Forge 1.0 defects that hit Sonnet's run: the button overlay above, and the exactly-once comment check firing its ×0.6 on zero comments or a missing flag. With both fixed, the review puts Sonnet at about 0.86 or above. Sonnet hasn't been re-scored, so 0.5508 is still what the board says. The same review called Haiku's 0.9661 clean. Its one finding that touches Haiku is that the backfill economy step from 1.0 to 0.75 is too steep, and fixing it would lift Haiku a little.

Claude Haiku 5.5 benchmarks on a payments app: 0.661, behind Sonnet's 0.7984

Gauntlet 7.2 is normal programming. It's a payments console with a ledger service, a notifier, an approval workflow, webhooks from a vendor that misbehaves on purpose, money in four currencies including JPY and KWD, and a 3D field of towers, one per payment. The public internet is full of apps like it.

Both had the same 0.669 ceiling from the same band. It decided Haiku's score and never touched Luna's. The critical did that.

The towers were right and the numbering was its own

Haiku drew every part of every tower correctly in all 8 inspected poses, and its pick function returned the right payment every time. But the scorer reads a hidden pick buffer to check which payment each pixel belongs to, and the contract says that number is the payment's arrival index plus one. Haiku's buffer used its own internal index. That one choice failed checks in the 0.699 band and capped a run that had earned 0.8417 at 0.669.

Haiku sent the payment; the table didn't re-sort

Haiku's critical is the user journey. Log in with a role token, create a draft, submit, approve. Then see the states in the drafts list, the notification, and the payment in the table. It did six of seven. The payment really was created, €1,250.00 to Probe Rig F1, and the ledger returned it. The scorer then clicked the table's Date header to sort newest first and found the next request still asked for oldest first. In Haiku's code the sort is a button inside the header cell, and the reload skips while any row is saving. Whether the click missed the button or the reload was skipped, I couldn't determine. 6 of 7 is 0.8571, a multiplier of 0.9429.

Luna's approval journey stopped after the login step

Luna's journey row lists one completed step: the role token. The scorer's browser bot tried to fill in the draft form and waited six seconds for a new draft to appear in the list. None did. The journey ends there. A failed draft step zeroes the whole row, and a primary flow that can't complete is a critical at full severity, ×0.6. The reject journey scored 0 for the same reason.

Why no draft appeared, I don't know. The parts of the journey aren't in the published data. Luna's run page has no workflow screenshot either: the bot takes one at exactly this failure, but the run page keeps five images and Luna's five went to renders, sync, error and mobile.

Take out one penalty each and they're level

Luna earned 0.7996 before its multiplier and 0.4797 after. Haiku earned 0.8927 before its multiplier and 0.8417 after, and the ceiling cut that to 0.661. If I remove both multipliers and apply the same ceiling rule, the arithmetic gives Haiku 0.6636 and Luna 0.6589. That isn't a score anyone published. It says the 0.1813 on the board is almost all one failed approval step.

Where Luna was better: picking, coasting and streaming

19 Gauntlet checks differ between the two runs, and Luna won 9. Most are 3D mechanics. Its pick buffer agreed three ways, GPU, pixel and analytic, where Haiku's agreed on 1 of 6 picks. After a flick its field coasted and settled the way the contract asks. Haiku's coasted only 1.379°, short of the 3° the check looks for, and took 3.5 seconds to settle against a 923 ms budget. A streamed batch of updates became visible in 222 ms in Luna's app and 592.9 ms in Haiku's. Those timings were measured on different Macs, which I come back to.

Haiku won the other 10: both workflow journeys, the pick render pass after every change (Luna 0), the inspector framing (Luna framed the tower in 0 of 8 poses), the replay of a committed update (Luna 0), the presentation text (Luna 0 of 4 groups readable), the per-currency collar widths, status badges, idempotency keys on retries and the API's error envelope. Luna's wins are mostly how the 3D field picks, coasts and streams. Haiku's include both approval journeys, the inspector and the update animation.

What other benchmarks say about Haiku 5.5 and Luna

Most independent boards point the same way: Haiku ahead on quality and dearer per task, with one tie and a couple of evaluations where Luna leads.

On Vals AI's Terminal-Bench 4.0 board, Haiku 5.5 scored 35.35% at $3.05 a test, and Luna 13.64% at $0.35, both with the same mini-SWE-agent harness. That's 8.7 times the cost per test for 2.6 times the score. Vals also runs an app-building benchmark, Vibe Code Bench v1.1, where Haiku scored 90.44%, third of 110. Artificial Analysis puts Haiku at 43 on its Intelligence Index at max effort and Luna at 38, at $0.21 and $0.07 per task. Its launch analysis says Haiku uses about 162k output tokens a task to Luna's 50k, and has Luna ahead on knowledge accuracy, 44% to 36%. Luna also leads its AutomationBench, though AA says an over-refusal issue held Haiku's 35% down and expects it to rise. On WebDev Arena, where people vote between two generated web apps, the two sit at 1587 and 1581 with overlapping intervals, a tie.

The vendors' own charts are in the launch posts. Anthropic's table has Haiku 5.5 at 39.2% on Terminal-Bench 4.0 against a public-leaderboard 16.4% for Luna. OpenAI's GPT-6 announcement says Luna at max effort scores 66.6% on DeepSWE v1.1, "comparable to Claude Opus 5 and Fable 5 at medium effort". They used different harnesses, efforts and tasks, so I don't put them in one table.

What could be wrong with these numbers

1. The runs weren't on the same setup. Luna's two runs were built on 2 and 3 October, on an older goose build and on our laptop, and re-scored on scorer release 3.0.92. Haiku's were built and scored on 9 October on the Mac Studio, on 3.0.108. That goose release also changed how a reply cut at the output limit is handled, which Haiku hit two or three times. The timing checks Luna won were measured on different hardware.
2. Luna's build files live on that laptop, and I couldn't reach them for this piece. So three things are unknown: why its draft never appeared, its Forge token counts and stop reasons, and whether the button-overlay bug hit it.
3. The bills are lower bounds from OpenRouter's records. Haiku's Gauntlet bill misses one call the harness couldn't price, about 65,000 output tokens. The re-priced Haiku figures are a model, not a bill. It reproduces the real bills within 5.5%, which is why I print it.
4. Forge 1.0 has confirmed scorer defects, and its scores stay as published. They inflate Haiku's lead over Sonnet. They can move the top four by up to 0.01 and leave the fifth about 0.03 too low, so the order of the top five isn't reliable. That includes Haiku's fourth place. They don't touch the backfill failure that decided Luna's score.
5. Forge 1.0 is finished. The evening Haiku's run published, Forge 1.0 runs were stopped so we could build a harder Forge 2.0. Its first six results, Luna's among them, didn't survive an audit, and they stay invalidated. The repaired Forge 2.0 started publishing on 12 October, with two models so far, GPT-6.1 Sol and MiMo v2.6 Flash. Neither Haiku nor Luna has a result on it. Read the Forge result as a Forge 1.0 result.

The honest verdict on Claude Haiku 5.5 vs GPT-6 Luna

In these runs Haiku 5.5 was the better agentic builder of the two, and on Forge 1.0 it wasn't close. It built a Forge app that passed every core check Opus 5.5 passed, for $0.52. Luna built one that looked better and never imported a sprint's history.

My reading of Luna's Forge run, labelled as a reading: it didn't fail on the parts every web app has. It passed the kit's lint and deploy stages, storage scored 1.0 and visual 1.0. It failed where a Forge app differs: reading a product's history through a scheduled job, and declaring the scopes and modules its calls need. Haiku's knowledge cutoff is June 2026 and Luna's 18 May, a few weeks at most. I don't think that explains a 0 of 62.

On our payments app it's closer than the board says. One failed approval step is most of the 0.18 gap, and Luna's 3D mechanics were the better ones.

The thing I didn't expect is the money. Two models at the same list price, and Haiku's agent runs cost 2.3 to 4.8 times Luna's. By our re-pricing, most of that is the 100k step, which these goose runs crossed at calls 16 and 33, the first of them eight calls early because of our own truncation bug. Priced from the sticker, these two runs would have been under-budgeted by about three times.

The cheapest LLM for coding agents: count the bill per working app

If I were picking a cheap model to build something on its own, I'd pick Haiku 5.5. Then I'd do two things. Cap tool output for real, because one oversized result puts every later call over the line. And budget about three times the list price unless I'd measured my agent staying under 100k. Haiku built a Forge app that passed every core check for $0.52. That's the number to compare, not $0.10 per million.

If Haiku isn't enough, Sonnet 5.5 has no step, so in our runs it cost 2.8 to 4.8 times Haiku rather than twenty. On our payments app it bought about 0.14 more.

If I needed thousands of focused, small calls, a classifier, a router, a reviewer of one diff at a time, I'd test Luna first. Under 100k tokens both list at the same price, but Luna wrote about a tenth of Haiku's output per call here, and Artificial Analysis measured about a third of Haiku's tokens per task. These runs don't test small calls either way. What they show is narrower: left alone with a whole app, Luna used every call on one and never imported the history on the other.

For Forge 1.0-sized apps: Haiku 5.5, Opus 5.5 or GPT-6.1 Sol. All three passed every core Forge 1.0 check. Haiku did it for an eighth of Opus's bill. One catch if your app calls the Forge LLM. Atlassian's Forge LLM model list offers Haiku 4.5 in its Haiku tier, not 5.5, as I write this. That's the model your app calls at runtime, not the one that writes it.

The Gauntlet board updates as runs land. The Forge 1.0 board is frozen, and the repaired Forge 2.0 board has only just started to fill. Every run page has the full checks, the screenshots and the graded recording.

---

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.