Updated September 23, 2026: added GPT-6 Luna, released on September 22, and autocomplete results. Two statements in the first version are no longer true and are corrected below: that GPT-5.6 cannot do autocomplete in DynoTable, and that Bedrock's Converse API cannot reach GPT-5.6.
On July 30, OpenAI cut GPT-5.6 Luna's price by 80%, and the same cut landed on Amazon Bedrock. Seven weeks later GPT-6 Luna arrived at about half of that new price. Every comparison page updated a pricing table. None of them answered the question that matters: at that price, is Luna actually good enough to drive an agent?
We had a way to answer it. DynoTable's AI agent runs real agent loops against DynamoDB (tool calls, PartiQL queries, SQL routing, staged writes) and every model we ship is gated by an Eval suite that scores those loops end to end. So we pointed the suite at both Lunas: 201 test cases, three runs each, against a live database, with every call priced from the tokens it actually used.
The verdict:
- GPT-6 Luna scored 93.02%, GPT-5.6 Luna 93.94%, and Amazon Nova 2 Lite (the cheap model we ship as our tuning floor) 96.00%, on the same 201 cells with the same scorers.
- GPT-6 Luna is the cheapest model we have measured: $0.00025 per call, 2.4× less than GPT-5.6 Luna and 12× less than Nova 2 Lite.
- Both Lunas now do autocomplete. GPT-6 Luna scores 96.97% on it, second only to Claude Haiku 4.5.
A new generation that ties its predecessor on agent work and halves the bill. The numbers are below, plus the caching behavior the pricing tables miss and the transport quirks that cost us debugging time.
What this benchmark is (and is not)
These are not public benchmark scores. Our suite drives the production agent through real DynamoDB sessions: pick the right tool for a plain question, read an item by composite key, stage a write for human review, respect PartiQL's no-JOIN boundary, route a GROUP BY to the SQL surface instead, ask a clarifying question only when the request is genuinely ambiguous, refuse a "delete everything". A scorer asserts the outcome of each session, we learned the hard way what happens when a scorer asserts tool names instead.
Four honesty notes before any chart:
- "Verified" in our model picker is a coverage claim, not a quality bar. It means the eval suite runs against that model. Both Lunas are verified and scored below our floor model, both facts are published here.
- Nova 2 Lite and Claude Haiku 4.5 are the two models this app ships against, so months of prompt tuning have gone toward them specifically. Neither Luna got any tuning. Read the gap with that in mind.
- GPT-6 Luna was measured seven weeks after the others. Same 201 cells, same scorers, but our test data was regenerated in between, and two of its cells still expected names from the old copy. In both, GPT-6 Luna read the current row correctly and was marked wrong. We left the score as measured rather than re-grade one model, so its 93.02% is slightly understated. The 0.92-point gap between the two Lunas is inside the run-to-run noise of this suite: read it as a tie.
- Three runs per case absorb flukes, not variance. One run going the other way moves a single case by 33 points, so no claim below rests on one.
The headline: two Lunas vs Nova 2 Lite
Measured on the 201 test cases all three models ran, after we fixed a bias in how we were grading, the same yardstick, held to all three:
| Model | Score | Cells | Trials |
|---|---|---|---|
| Amazon Nova 2 Lite | 96.00% | 201 | 579 |
| GPT-5.6 Luna | 93.94% | 201 | 579 |
| GPT-6 Luna | 93.02% | 201 | 579 |
The gap to Nova 2 Lite is real and we are not smoothing it. GPT-5.6 Luna refused a task it should have attempted, routed an indexed top-N query through a full-table pass, and once deferred to the user instead of finishing an export. GPT-6 Luna's misses are its own: asked to read a product, it suggested opening the table instead; asked to raise a price, it worked out the new value and then asked the user to open the row rather than staging the change. Those are agent-behavior losses, not formatting noise, and they are why Nova 2 Lite remains our floor.
But the shape of the result is the story. Two untuned models, on a benchmark tuned for someone else's models, landing within two to three points of the incumbent at a fraction of the cost.
What every call actually cost
Sticker prices say Luna should be cheap. The token counts say it is cheaper still. Here is every call we made to these models, priced at current Bedrock rates:
| Model | Calls | $/call | vs GPT-6 Luna |
|---|---|---|---|
| GPT-6 Luna | 2,519 | $0.000246 | 1× |
| GPT-5.6 Luna | 5,144 | $0.000591 | 2.4× |
| Amazon Nova 2 Lite | 6,303 | $0.003022 | 12.3× |
| Claude Haiku 4.5 | 2,654 | $0.003275 | 13.3× |
| GPT-5.6 Terra | 751 | $0.006458 | 26.3× |
Two notes on the GPT-6 Luna row. Its price, $0.11 in / $0.55 out per million tokens, is derived rather than read off an AWS price table: AWS had not published a GPT-6 price row when we measured, so we used OpenAI's list price plus the 10% Bedrock adds on in-region and geo inference profiles, which is what we called. And its calls include a batch of short autocomplete calls that pull the average down a little; GPT-5.6 Luna's include autocomplete calls from its own August run too. Terra appears for cost only: we ran it across far fewer test cases than the others, and a score from a smaller sample does not belong beside a full one.
On sticker prices alone, GPT-5.6 Luna ($0.22 in / $1.32 out per million tokens) is already cheaper than Nova 2 Lite ($0.33 / $2.75) on both axes, and GPT-6 Luna halves it again. But the measured gap per call is wider than the stickers explain. The difference is caching.
The implicit prompt caching nobody configures
An agent loop re-sends a large, mostly-static prompt on every step: system instructions, tool schemas, conversation so far. Whether you pay full price for those repeated tokens is the single biggest cost lever in agent workloads.
On Bedrock's Converse API, prompt caching is normally opt-in: you place cache breakpoints in the request, and models without cache support simply pay full rate. The OpenAI models behave differently: caching is implicit and on by default. No breakpoints, no request changes, nothing to configure. It held on both transports we measured: GPT-5.6 Luna on Bedrock Mantle, GPT-6 Luna on Converse.
Share of input charged at the discounted cache rate:
| Model | Cache-read share of input |
|---|---|
| Claude Haiku 4.5 | 95.5% |
| GPT-6 Luna | 95.0% |
| GPT-5.6 Luna | 94.5% |
| Amazon Nova 2 Lite | 71.8% |
Across 5,144 calls, GPT-5.6 Luna paid full price on fewer than six input tokens per call. The static prompt prefix was billed at the read rate almost every step, without a cache breakpoint anywhere in our requests. GPT-6 Luna did the same on Converse; its full-price average is higher, 72 tokens per call, only because its short autocomplete prompts have no prefix worth caching. Nova 2 Lite, on explicit Converse caching with a 0.25× read rate (vs 0.1× for the Lunas and Haiku), left 28% of its input at full price, and that, more than the stickers, is why the per-call gap is 12× and not 3×.
AWS shipped explicit cache breakpoints for GPT-5.6 the same day as the July price cut. Our numbers are the default behavior, what you get before reading that post. For an agent loop with a stable prefix, the default already captures about 95%.
Two transports, budget a day for these
When we first measured GPT-5.6, it was the first Bedrock family the Converse API could not reach: it lived only on the bedrock-mantle.{region} endpoint, speaking the OpenAI Responses protocol. That has changed. AWS now publishes us. and global. inference profiles for GPT-5.6 that Converse accepts, and GPT-6 launched on Converse. The bare GPT-5.6 model ID still goes to Mantle, and four things there cost us real debugging time:
-
temperatureis rejected, not ignored. OpenAI's own endpoint silently drops unsupported sampling parameters; Mantle returns400 unsupported_parameter. Our first full run failed everywhere on this before a single token was generated. -
The base URL is
/openai/v1, not/v1. The wrong path also returns a 400, one that reads misleadingly like an entitlement problem. - A fresh account gets 401s until the AWS Marketplace subscription for the model is active. The error text talks about authorization, not subscriptions; if you hit it, check Marketplace first.
- Asking for a reasoning summary is a 400. When you set a reasoning effort, the OpenAI SDK asks for a detailed reasoning summary by default, and Bedrock now refuses the parameter, so every GPT-5.6 turn at low, medium or high effort failed until we sent an explicit "no summary". Nothing is lost by it: Bedrock returns no reasoning text for OpenAI models on either transport.
None of this is exotic once you know it, but no pricing table mentions any of it.
The full ladder: 12 models, one instrument, one day
After the July price cut we re-ran every model we tracked in one session (12 models, 39 scenarios, 3 runs each, $20.49 of Bedrock spend) so every row below is measured the same way on the same day. The table uses the 36 scenarios all twelve models ran. The three autocomplete scenarios are left out because at the time our autocomplete could not reach the GPT-5.6 models; that is fixed, and autocomplete has its own section below.
| # | Model | Score | $/agent run |
|---|---|---|---|
| 1 | Claude Haiku 4.5 (1) | 100.00% | $0.0100 |
| 2 | Claude Sonnet 4.5 (ceiling ref) | 99.69% | $0.0290 |
| 3 | Amazon Nova 2 Lite (1) | 97.22% | $0.0091 |
| 4 | GPT-5.6 Luna | 93.60% | $0.0024 |
| 5 | GPT-5.6 Terra (reference) | 92.82% | $0.0224 |
| 6 | MiniMax M2 (2) | 92.81% | $0.0178 |
| 7 | Nova Pro (Gen 1) | 90.66% | $0.0150 |
| 8 | Nova Lite (Gen 1) | 90.36% | $0.0013 |
| 9 | DeepSeek V3.2 (2) | 90.17% | $0.0508 |
| 10 | Ministral 8B (2) | 89.43% | $0.0104 |
| 11 | Ministral 3B (2) | 88.53% | $0.0082 |
| 12 | Nova Micro | 86.96% | $0.0008 |
(1) The two models this app ships against, months of prompt tuning went toward them specifically, and Haiku's perfect score should be read with that squarely in view. (2) No Bedrock prompt caching, so every step of an agent loop pays full input rate, which is how DeepSeek ends up costing 5× Claude Haiku per run while scoring seven points below it.
An agent run here is a full multi-step session (tool calls, queries, staged writes) and the dollar column is what one such session costs, averaged over the 100+ we ran per model.
Four readings we'd defend from this table, and one we wouldn't. GPT-5.6 Luna is the only model in the top half whose cost sits in the bottom half, the price-performance outlier of the ladder. Terra, the mid-tier GPT-5.6 at nearly ten times Luna's measured cost per run, lands 0.78 points below its cheaper sibling, that gap is inside the noise band, so the honest reading is a tie, and a tie is the damning version: on this workload, paying 10× for the bigger variant bought nothing we could detect. The caching column of the economy matters more than the sticker: Nova Lite Gen 1 at $0.0013/run is the budget surprise, and the no-cache tier is uniformly mispriced for agent work. And the two models we tuned for hold ranks 1 and 3, which is the tuning bias made visible, not a neutral ranking. What we would not defend: reading any two adjacent rows as meaningfully different, with three runs each, neighbours can swap on a single one going the other way.
Where GPT-6 Luna lands. It is not in the table because it did not exist on the day the table was measured, and a row from another day would break the one thing the table promises. On the same 36 scenarios, measured on September 23, it scored 90.12%. Two of those scenarios are the stale-data cells described above, where it answered correctly and was marked wrong. The rest of the gap is the misses named above, plus an email lookup where it searched with the wrong prefix, found nothing and stopped. Treat it as indicative: in its predecessor's range, not above it, at less than half the price.
One more thing the ladder run measured: our own suite, again. The three autocomplete scenarios initially scored 0% for all four no-cache models (36 runs of zero) and every one was rejected before the model generated a token: we were sending a request field those models don't accept. Same bug class we published about before, new field. Those 36 zeros measured our bug, not the models, so they are excluded from every number above, which raised those four scores by about 4 points each.
Then we fixed the builder and re-ran exactly those cells. DeepSeek V3.2 (which the broken suite had at 0%) scored 100% on all three autocomplete scenarios. The other three landed mid-range. Had we published the first run uncorrected, we'd have reported a model as incapable of autocomplete when it is, measurably, the best of the four at it. A benchmark that won't audit itself will happily rank vendors on its own bugs.
Autocomplete: both Lunas can do it now
When this post first ran, the GPT-5.6 models could not drive DynoTable's inline autocomplete at all: that path spoke only to Converse, and GPT-5.6 lived on Mantle. We moved autocomplete onto the same model layer the agent uses and measured again. Autocomplete here is the gray ghost text in the query editor: given the statement so far, the table schema and some stored values, complete the line.
| Model | Autocomplete score | Median time per completion |
|---|---|---|
| Claude Haiku 4.5 | 98.48% | 698 ms |
| GPT-6 Luna | 96.97% | 482 ms |
| Amazon Nova 2 Lite | 96.21% | 525 ms |
| Nova Micro | 81.82% | 345 ms |
That is 22 autocomplete cases, three runs each, measured on September 23 with reasoning off. GPT-5.6 Luna scored 96.08% on the 17 cases the suite had in August; it has not been re-run on the five added since, so it gets no row here.
Getting GPT-6 Luna to that number taught us two things.
It would rather say nothing than guess, and that is correct. On its first run it left 7 of 18 cases blank on every try. Six of the seven were cases where the right completion was a table or column name the prompt never showed. Our prompt already said "emit nothing rather than guess", and GPT-6 Luna was the first model to take it literally. We decided it was right: autocomplete should never invent an identifier you did not write and your schema does not contain. The scoring changed to accept an empty completion there, and the prompt stayed.
It also refused to pick from values it was shown. Given a column and its four stored values, it still emitted nothing, reading "pick one of these" as the guess it had been told not to make. One sentence fixed it: choosing a candidate the statement or the schema actually shows is not a guess. Every model went up or stayed flat on the next run (Nova Micro by 8.6 points), and GPT-6 Luna's value-completion cases went to 100%.
We tried the obvious lever first. Raising GPT-6 Luna's reasoning effort to low or medium moved its score by half a point and roughly doubled its latency, so autocomplete runs with reasoning off. The whole final autocomplete run, four models on 22 cases, cost five cents.
What transfers if you're picking a cheap agent model
- Benchmark on your workload, not on leaderboards. Luna's public scores and our agent scores rank models differently. The only number that predicts your bill and your failure rate is one measured on your own loops.
- A new generation is not automatically a better agent. GPT-6 Luna tied GPT-5.6 Luna on our work and halved the cost. That is a good trade, but it is a price story, not a capability one.
- Price the loop, not the token. Cache behavior differs per model family: read rates, write premiums, and whether caching is implicit or needs breakpoints. Two models with similar stickers can differ 5× per call.
- A model being 2 points behind may not matter, where it loses does. Read the failing behaviors, not the aggregate. A wrong refusal is recoverable; a silent wrong answer is not. A blank autocomplete is better than an invented column name.
- Audit your test data, not just your scorers. Two of GPT-6 Luna's "failures" were answers checked against data we had since replaced. Expected values drift like everything else.
- Record what every call costs, from day one. Every number here exists because we kept that record. Estimating from published prices would have missed the caching story entirely.
GPT-6 Luna is in DynoTable's default model set, and GPT-5.6 Luna remains a verified, selectable model. Bring your own Bedrock credentials and pick either in the model picker. Claude Haiku 4.5 and Nova 2 Lite remain the models we tune against, and the gap is real. But if your bill is the constraint, two or three points is a very cheap trade.
Download DynoTable · How the AI agent works · The eval story behind our scorers
Top comments (0)