- Book: AI That Ships
- The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Three launch tables went up in the same September 2026 week, and
every one of them has its own vendor on top. Anthropic shipped
Claude Fable 5.1. Google released Gemini 3.8 Flash. OpenAI shipped
GPT-6 Astra and called it the start of the AGI era.
OpenAI's launch numbers show Astra beating Fable 5.1
on almost every row. Artificial Analysis, which runs its own
evaluations, puts Fable 5.1 first on overall intelligence and on
its coding agent index. LLM Stats scores the same two models and
gets a different ordering again.
None of those tables is lying. They measure different things, under
different settings, and the disagreement between them is the most
useful information in this whole launch cycle. So this comparison
does the unglamorous thing: it goes through the numbers, says who
measured each one and how, and ends with the only question that
matters for your system — which model for which work.
The tale of the tape
| GPT-6 Astra | Claude Fable 5.1 | Gemini 3.8 Flash | |
|---|---|---|---|
| Vendor | OpenAI | Anthropic | |
| Context window | ~1M tokens | 1M tokens | 1M tokens |
| Max output | 128K | 128K | — |
| Input price / 1M | $10.00 | $10.00 | $0.75 (promo) |
| Output price / 1M | $50.00 | $50.00 | $3.75 (promo) |
| Output speed | ~87 tok/s | ~67–69 tok/s | ~305 tok/s |
Two prices need footnotes before anything else. Gemini 3.8 Flash's
$0.75/$3.75 is promotional through 31 December 2026, after which
Google lists $1.50/$7.50 — still a fraction of the other two.
Astra also has a fast tier at $20/$100, and Fable 5.1 prices cache
reads at $0.25 per million tokens, which matters more than it
looks. The speed row is measured throughput from
Artificial Analysis,
not a vendor claim.
Prices move. Check the current pages before you budget:
OpenAI,
Anthropic,
Google.
What OpenAI's launch table says
Astra's vendor numbers are the loudest, so start there. From
OpenAI's launch material, as reported by
The New Stack
and broken down by
Vellum:
- FrontierMath Tier 4: 97.6%, against Fable 5.1's 87.8%.
- ARC-AGI-3: 99.9%, effectively saturating a fluid-intelligence benchmark that most models struggle to score double digits on.
- ExploitBench: 100%, with Fable 5.1 at 70%.
- GPQA Diamond: 96.0%.
- OSWorld 2.0: 72.6%, at roughly 40 minutes per task — the strongest computer-use score in the launch-week coverage.
- AutomationBench: 41.4% vs Fable 5.1's 31.4%.
- BenchCAD: 95.9% vs Fable 5.1's 84.3%.
- Terminal-Bench 4.0: 57.7% vs Fable 5.1's 55.8%.
On research-grade math, abstract reasoning, offensive security and
computer use, Astra is ahead of everything, sometimes by a wide
margin. Those results are real and worth taking seriously.
They also come with footnotes that most of the coverage skipped.
Vellum's breakdown lists them. Per that write-up, OpenAI funded
FrontierMath's development and has exclusive access to part of the
problem set. The ARC-AGI-3 score was produced under OpenAI's own
provider adapter harness. The BenchCAD comparison against Claude
used modified evaluation settings, and the comparison on
ExploitGym, a separate exploit benchmark, ran with time limits
removed. And the production model you can actually call is the
safety-hardened version, which refuses the advanced cyber work the
100% ExploitBench score was measured on.
The numbers are not fake. They are vendor numbers:
measured by the party with the most to gain, under settings you
cannot fully reproduce. That is exactly the situation where you
want a second opinion.
What independent measurement says
Artificial Analysis
runs the same evaluation suite across every major model,
independently of the vendors. Its launch-week read:
- Intelligence Index: Fable 5.1 scores 66 at maximum effort. Astra scores 61. Gemini 3.8 Flash scores 59 in its high configuration.
- Coding Agent Index: Fable 5.1 at ~70, Astra at ~67, Gemini 3.8 Flash at ~61.
So the model that loses almost every row of OpenAI's launch table
wins the independent aggregate. Both facts hold at once, because
they are not measuring the same thing. OpenAI's table leans on
frontier academic benchmarks: Tier 4 math, saturated reasoning
suites, exploit development. Artificial Analysis weights a broad
mix that leans closer to the work most systems do: code, tool use,
instructions, long context.
Two individual results make the split concrete. On Humanity's Last
Exam with tools, Fable 5.1 scores 65.0% against Astra's 57.2% —
tool-heavy research is a Fable strength. On DeepSWE v1.1, Astra's
74.1% is barely ahead of Gemini 3.8 Flash's 73.8%, a model costing
a thirteenth as much per token.
LLM Stats
weights things differently again and puts Astra ahead overall,
60.7 to 56.8, while still showing Fable 5.1 ahead on individual
agentic rows. Three scoreboards, three orderings.
Benchmarks still earn their keep. An aggregate score answers the
question "good at what the aggregate weights", and nobody's
aggregate weights your workload. When two
credible measurements disagree this much, the winner for you lives
in the individual rows that resemble your traffic, and in your own
eval suite — nowhere else.
The economics are the actual differentiator
Capability-wise, these three models are closer than any launch
table admits. The pricing is not close at all.
Astra and Fable 5.1 list identical per-token prices: $10 in, $50
out. Gemini 3.8 Flash lists $0.75 in and $3.75 out until the end of
- Run a moderate workload through all three and the shape is obvious. Say 1,000 requests a day, 4,000 input tokens and 800 output tokens each — a back-of-envelope calculation from list prices, no one's measured bill:
type Price = { inPerM: number; outPerM: number };
const MODELS: Record<string, Price> = {
"gpt-6-astra": { inPerM: 10, outPerM: 50 },
"claude-fable-5-1": { inPerM: 10, outPerM: 50 },
"gemini-3-8-flash": { inPerM: 0.75, outPerM: 3.75 },
};
function dailyUSD(p: Price, reqs: number): number {
const inTok = (reqs * 4_000) / 1e6;
const outTok = (reqs * 800) / 1e6;
return inTok * p.inPerM + outTok * p.outPerM;
}
for (const [id, price] of Object.entries(MODELS)) {
console.log(id, dailyUSD(price, 1_000).toFixed(2));
}
gpt-6-astra 80.00
claude-fable-5-1 80.00
gemini-3-8-flash 6.00
Eighty dollars a day against six. Over a month that is roughly
$2,400 against $180 for the same request volume. A 5-point gap on
an intelligence index has to buy a lot of correctness to justify a
13x bill, and on high-volume routes it usually does not.
Between the two premium models the list prices are identical, so
the tiebreakers are structural. Fable 5.1's cache reads at $0.25
per million tokens mean a long-running agent that keeps re-reading
the same large context pays a fortieth of the input price on the
repeated part. Agent loops carry a large stable prefix: system
prompt, tool definitions, a repository map. For those, cache
pricing is the difference between a context strategy that scales
and one that does not. On the other side, Astra's ~87 tokens per
second beats
Fable 5.1's ~67–69, and Gemini 3.8 Flash's ~305 embarrasses both.
If a human is watching the tokens render, that ordering is felt on
every single request.
Which model for which work
Read the individual rows instead of the aggregates and the three
models sort themselves with surprisingly little overlap.
GPT-6 Astra for research-grade math and science, computer-use
agents, and security work — the rows where its lead runs 10
points or more. The 40-minutes-per-task OSWorld result also makes
it the current pick for long autonomous desktop tasks, if your
harness can handle a call that runs that long.
Claude Fable 5.1 for long agentic coding runs and tool-heavy
research, where both the independent coding index and the
tools-enabled benchmarks put it first, and where cache-read
pricing compounds over every loop iteration. If your workload is
"an agent working in a repository for an hour", the independent
numbers say this is the default.
Gemini 3.8 Flash for everything high-volume: classification,
extraction, summarisation, chat, and any route where latency is
user-visible. It sits a few index points below the other two and
costs a thirteenth as much, at four times the throughput. Most of
the requests most systems serve belong here, and the DeepSWE
result says it is no longer only a small-model tier.
Which means the real answer to "which model won the week" is the
boring one: you route. Cheap and fast by default, escalate to a
premium model on the work shapes where its lead is measured in
double digits, and let cost per successful task decide where the
boundary sits, not a launch table. The escalation-router pattern
is small: call the cheap model, validate the output against
something deterministic, escalate on failure, and refuse any
request that would blow a per-request cost ceiling. It applies
unchanged with these three models in the slots.
Every number above is days old, from launch-week material, and
some of it will be stale within a quarter: promotional prices
expire and models get patched. Treat this post as a snapshot of
September 2026 and a method for reading the next launch table.
The ranking has a shelf life. The method does not.
If this was useful
The uncomfortable part of every model launch is that no vendor
table, and no independent index, can tell you whether the swap
helps your system. An eval suite over your own traffic, cost per
successful task on every route, and a router that escalates on
signal — that is the machinery that answers it, and it is what
AI That Ships builds, in TypeScript, end to end.
It is book 5 of AI in TypeScript, a five-book series that runs from your first LLM call through to agents you can leave running in production.
Top comments (0)