DEV Community

Grain64
Grain64

Posted on

GPT-6 Luna vs Claude Haiku 5.5: cost per correct answer on 4 tasks

Short answer from four public runs: on pass rate these two cannot be told apart on these rows, and neither is cheaper everywhere. GPT-6 Luna cost about 1.5x less per correct answer on 77-intent banking classification; Claude Haiku 5.5 cost about 1.2x less on SMS spam and the same within a hair on review sentiment; on JSON extraction both are borderline: pass rate at or above the bar, interval low edge under it. Haiku answered in about 250 ms at the median against about 390 ms for Luna on every task.

Cost per correct answer here means total spend over every attempt, retries included, divided by the answers that matched the label. Prices are the 5 Nov 2026 snapshot, all calls at temperature 0.

Task (rows, bar) Luna: pass, $ per success Haiku 5.5: pass, $ per success GPT-6.1 Terra: pass, $ per success
Banking intent, 77 labels (150, 70%) 82.0%, $0.000090 80.0%, $0.000138 84.0%, $0.000727
Review sentiment (300, 88%) 98.0%, $0.000024 97.3%, $0.000023 98.0%, $0.000165
SMS spam (300, 90%) 97.0%, $0.000026 97.3%, $0.000021 98.3%, $0.000163
JSON extraction (136, 55%) 55.1%, $0.000401 61.0%, $0.000577 51.8%, $0.003926

You can check any cell with a calculator: Luna on banking spent $0.011092 for 123 correct answers, $0.011092 / 123 = $0.00009. Haiku spent $0.016501 for 120, $0.016501 / 120 = $0.000138.

The pass rates overlap. The 95% intervals for the two models overlap on all four tasks (banking: 75.1-87.3% vs 72.9-85.6%; spam: 94.4-98.4% vs 94.8-98.6%). Two points of pass rate on 150 or 300 rows is noise, so the choice between them is a price and speed choice, not a quality one, on classification.

The price gap moves with the task. If you pick by the rule "cheapest model whose interval's lower edge clears your bar", it looks like this:

const clears = m => wilsonLow(m.successes, m.n) >= bar;
const pick = models.filter(clears)
  .sort((a, b) => a.cost / a.successes - b.cost / b.successes)[0];
Enter fullscreen mode Exit fullscreen mode

Between these two, by that rule: banking goes to Luna ($0.00009 vs $0.000138), spam to Haiku ($0.000021 vs $0.000026), sentiment is a coin flip at $0.000024 vs $0.000023. Neither passes the low-edge rule on extraction: both come out borderline, with lower edges (46.8% and 52.6%) under the 55% bar. Haiku is higher on that task, 61.0% vs 55.1%, at about 1.4x the price per success.

The one that costs more did not win extraction. GPT-6.1 Terra cost 6 to 10 times Luna per correct answer on these runs and was lower than both on extraction (51.8%, below its bar). A bigger tier is not a safer default when the task is structured output.

One other model in the same runs was cheaper still. Mistral Small (2603) cleared sentiment at $0.000019 and spam at $0.000016. It was below the bar on banking intent (68.0% against 70%).

The verdict page uses a different rule: the cheapest model that clears the bar and also matches the strongest model (GPT-6.1 Sol) on the same rows. That is why it names Terra for banking, and Luna for extraction, where it matches Sol within noise. Which rule fits depends on whether you need to match the top tier or just clear your own bar. Every row, prompt and price behind these tables is public, and the same run on your own examples, with a pass bar you set, is free during early access (3 runs a month, up to 100 examples, your own OpenRouter key): https://clearbar.grain64.com/verdicts/?utm_source=devto&utm_campaign=devto-luna-vs-haiku

All four runs send each row as one short call with a short system prompt. If your calls share a long prefix that your provider caches, the cost order could change. Have you measured a cache hit rate on either model, and did it change which one came out cheaper per correct answer?

From Clearbar, by Grain64.

Top comments (0)