Every frontier model launch now ships with the same artifact: a benchmark table with suspiciously few footnotes. GPT-6 Astra's is no exception — and it's a genuinely interesting table, because its strongest and weakest numbers are strong and weak in different ways. Reading it well is more useful than arguing about it.
The table itself
From OpenAI's announcement and third-party coverage since the September 3 release:
| Benchmark | GPT-6 Astra | Comparison | Who verified it |
|---|---|---|---|
| ARC-AGI-3 | ~98.6–99.9% | 87.8% (Claude Fable 5.1) | Vendor-reported |
| DeepSWE v1.1 (agentic coding) | 74.1% | 72.7% (GPT-5.6 Sol) | Vendor-reported |
| OSWorld (computer use) | ~72.6% | prior frontier ~70–72% | Third-party coverage |
| ExploitBench | 100% | 78.5% (Sol), 70% (Opus 5) | Vendor-reported |
| ExploitGym | 42.4% | 30.3% (Sol) | Vendor-reported |
| Agentic long-task accuracy | ~68% | +10 pts over 2nd place | Vendor-reported |
Look at the last column before you look at any other. Most of the table is vendor-reported.
The one number that can't be faked
Before the launch table existed, there was the August 1 reveal: ten math problems previously considered human-level, solved at roughly $2,000 in tokens each, with every proof mechanically checked by the Lean 4 proof assistant. Formal verification is the rare benchmark property that's immune to training-data contamination — a proof either compiles or it doesn't. That record remains Astra's most tamper-proof credential, and nothing in the skeptical coverage has dented it.
The pattern in the rest of the table
Two things stand out when you stare at it for a minute:
First: Astra's lead is largest exactly where independent verification is hardest. ARC-AGI-3 and ExploitBench are expensive and awkward to re-run, and they're where the margins are enormous. Agentic coding — DeepSWE, where practitioners can check results daily on their own repos — is where the lead shrinks to 1.4 points, which is inside run-to-run noise for real codebases.
Second: the security numbers double as a safety statement. ExploitBench at 100% is the same capability profile that got the model rated Critical on cybersecurity before launch and delayed its release into isolated testing. Impressive and unsettling are the same number here.
The independent pushback, fairly stated
- Artificial Analysis has both a launch review and an evaluation noting Astra costs ~75% more than GPT-5.6 Sol on a blended-price basis, and questioning its score-per-dollar — per benchmark point, it's the most expensive OpenAI model to date.
- Vellum's writeup notes Astra leads "in OpenAI's table" — a phrase chosen carefully, because several of those numbers haven't replicated independently.
- Practitioner threads on Reddit argue Astra is "actually behind Fable, and even Opus" on real coding work. Anecdotal — but it rhymes with the pattern above.
None of this means the scores are fake. The honest reading: Astra is clearly the strongest OpenAI model; whether it's the strongest model overall depends on which benchmark — and whose run — you trust. The full roundup with sources is on the TeamoRouter blog, and the flagship-vs-flagship question gets its own treatment in Fable 5.1 vs GPT-6 Astra.
Which numbers should actually move your decisions
If you're deciding whether to route production traffic to Astra, rank the table like this:
- Cost per solved task, not per token. $10/$50 pricing, plus a reported double-rate tier past ~272K input tokens, means a 2x quality win can still be a 5x cost loss on volume work.
- Your own repo. DeepSWE at 74.1% vs Sol's 72.7% tells you nothing about your codebase. Ten to twenty representative tasks, run twice each, beats every table above.
- Computer use — if that's your use case. OSWorld ~72.6% is the one genuinely corroborated, practically-visible lead. Desktop automation is where Astra's advantage is hardest to dispute.
- Cache economics. Cache reads at $1.00/M vs $10 fresh input mean agent loops with stable system prompts see very different real costs than the sticker implies.
A 30-minute evaluation you can actually run
python
from openai import OpenAI
client = OpenAI(api_key="sk-...", base_url="[https://api.teamorouter.com/v1](https://api.teamorouter.com/v1)")
# run the SAME 10-20 tasks against:
# model="gpt-6-astra"
# model="claude-fable-5-1"
# model="deepseek-v4-pro" # your cost baseline
Top comments (0)