DEV Community: Olivia Hayes

GPT-6 Astra: How I’d Evaluate It for Agents, Coding, and Production Costs

Olivia Hayes — Wed, 09 Sep 2026 02:04:07 +0000

The interesting question about GPT-6 Astra isn’t whether it tops every leaderboard. It doesn’t. I’d ask whether it can finish enough difficult work to justify its price.

OpenAI released Astra on September 3, 2026, as the successor to GPT-5.6 Sol. Its strongest reported improvements involve computer use, coding agents, scientific workflows, long-context retrieval, and professional artifacts—not a uniform jump in general reasoning.

That distinction changes how I’d evaluate it. For a chatbot, token price and answer quality might dominate. For an agent operating across a repository, browser, and terminal, I care more about successful completion, failed iterations, and human rescue time.

These notes draw on OpenAI’s published results and Artificial Analysis testing, rather than my own hands-on benchmarks.

Start With the Integration Constraints

Before looking at scores, here’s the API model specification I’d use for planning:

Property	GPT-6 Astra
Model ID	`gpt-6-astra`
Context window	1.05M tokens
Maximum output	128K tokens
Input	Text and images
Output	Text
Reasoning effort	`low`, `medium`, `high`, `xhigh`, `max`
Fine-tuning	Not supported

The large context window is useful, but it is not flat-price memory. Requests above 272K input tokens move to a more expensive pricing schedule for the full request.

There’s also no native audio or video input. If those modalities are central to an application, I wouldn’t treat Astra as a drop-in replacement for a model that supports them.

A minimal Responses API call

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-6-astra",
    reasoning={"effort": "high"},
    input=(
        "Review this repository architecture. Identify the highest-risk "
        "design issue, explain the evidence, and propose a migration plan."
    ),
)

print(response.output_text)

The Responses API is the natural starting point for tool-heavy work. Astra supports function calling, structured outputs, web search, file search, code interpreter, hosted shell, apply patch, computer use, MCP, and tool search.

I wouldn’t automatically set reasoning to max. The official guidance recommends the lowest effort that meets your evaluation target. That gives you a quality-cost curve to measure rather than a single expensive default.

Availability is staged

The September 3 rollout was scheduled to reach ChatGPT Plus, Pro, Business, and Enterprise users, the OpenAI API, and AWS over the following days. That does not mean every account received access simultaneously.

Enterprise access was off by default at launch; administrators can enable Astra per workspace.

The Upgrade Is Mostly About Staying on Task

I read Astra’s positioning as an attempt to improve the whole execution loop: reason, act, inspect the result, revise, and continue.

That matters because generating plausible code is only one part of an engineering task. An agent also has to navigate tools, preserve constraints, recover from mistakes, and deliver something usable.

Computer interaction gets faster and more reliable

On OSWorld 2.0, OpenAI reports 72.6% for Astra versus 65.7% for Sol.

In the same latency simulation, task completion took roughly 40 minutes for Astra versus about 75 minutes for Sol—a reduction of around 47%. With an updated Codex harness, OpenAI also reports 1.9× faster completion on Mind2Web.

The harness qualification matters. I’d treat those figures as results for the evaluated system, not a promise that swapping one model name will make every browser agent 1.9× faster.

Finished artifacts get more attention

OpenAI describes training aimed at documents, spreadsheets, presentations, data analysis, design work, and template adherence.

It also reports better handling of requirements that change mid-task. For an enterprise agent, that is more useful than it sounds: a steering message should refine the work without silently discarding the original objective.

Codex can preserve notes beyond the active window

For long coding sessions, Astra introduces a way for Codex to preserve and retrieve notes after the active context fills.

Earlier approaches relied heavily on compaction. A compressed summary can lose details such as why a fix failed or which constraints were already tested. OpenAI says notes across context windows improve continuity during large refactors and debugging.

I’d still test this with actual long-running issues. A larger context window and cross-window notes solve related, but different, problems.

Read the Benchmarks by Workload

OpenAI’s comparison is much easier to interpret once general-intelligence scores are separated from tool-using execution.

Benchmark	GPT-6 Astra	GPT-5.6 Sol
AutomationBench	41.4%	18.1%
OSWorld 2.0	72.6%	65.7%
Terminal-Bench 4.0	57.9%	37.3%
Terminal-Bench Science 0.1	64.6%	22.4%
FrontierMath Tier 4 v2	97.6%	83.0%
ExploitBench	100.0%	78.5%
SRE-Bench, one attempt	88.0%	55.9%
MRCR v2, 512K–1M	96.3%	73.8%
AA Intelligence Index v4.1.1	61.2	60.9

The large changes are in business automation, terminal work, scientific tasks, reverse engineering/SRE work, and very-long-context retrieval.

The AA Intelligence Index barely moves: 60.9 to 61.2. OpenAI’s launch table also puts Claude Fable 5.1 ahead at 65.7.

Other headline launch results include 99.9% on ARC-AGI-3, alongside 97.6% on FrontierMath Tier 4 and 100% on ExploitBench. Those are striking results, but they don’t establish universal superiority—or a settled definition of AGI. OpenAI’s product pages do not define Astra itself as AGI.

Independent testing makes the cost story more specific

Artificial Analysis reports an Intelligence Index score of 61, equal to GPT-5.6 Sol at max effort.

Its coding-agent results are more encouraging. Astra improves on the Coding Agent Index while using roughly three times fewer tokens than Sol at max effort in the tested Codex harness. That allowed higher scores at about the same cost per coding-agent task.

General-intelligence economics were less favorable:

Astra used about 10% fewer output tokens than Sol at max effort.
Its per-token price was 2.5× higher.
Artificial Analysis estimated about 75% higher cost per task at max effort.

It also reported an AA-Omniscience hallucination-rate reduction from 92% to 51%, with accuracy increasing by four points.

My takeaway is narrow: the evidence for a major upgrade is strongest when the model must act and iterate. For ordinary reasoning or bulk text generation, the premium is harder to justify.

Budget for Completed Work, Not Just Tokens

Here is the published API pricing schedule. All rates are per million tokens.

Processing / context	Input	Cached input	Cache write	Output
Standard, short context	$10	$1	$12.50	$50
Standard, long context (>272K input)	$20	$2	$25	$75
Batch / Flex, short context	$5	$0.50	$6.25	$25
Batch / Flex, long context	$10	$1	$12.50	$37.50
Fast mode, short context	$20	$2	$25	$100
Fast mode, long context	$40	$4	$50	$150

Sol’s Standard short-context rates are $4 input and $20 output per million tokens. Astra’s $10/$50 rates are therefore 2.5× higher.

But I wouldn’t multiply an existing agent’s bill by 2.5 and call that a forecast. Lower token use can offset some of the premium, as the coding-agent testing illustrates. Fewer retries and less manual repair can also change the economics.

For a production evaluation, I’d track:

Total cost per successful task
Failed iterations and abandoned runs
Human review and correction time
Quality of the final artifact or merged change
Performance at different reasoning-effort settings

Routine classification, extraction, simple drafting, and chat still look like poor default destinations for premium tokens unless task-specific evaluations show otherwise.

Where the Alternatives Still Make Sense

I wouldn’t retire the other models just because Astra is newer.

Metric	GPT-6 Astra	GPT-5.6 Sol	Claude Fable 5.1	Gemini 3.8 Flash
Context window	1.05M	1.05M	1M	1.048M
Maximum output	128K	128K	128K	65.5K
Input modalities	Text, image	Text, image	Text, image/PDF	Text, image, audio, video, PDF
AA Intelligence Index	61.2	60.9	65.7	58.7
AutomationBench	41.4%	18.1%	31.4%	—
Terminal-Bench 4.0	57.9%	37.3%	55.8%	19.1%
DeepSWE 1.1	74.1%	72.7%	67.4%	73.8%
FrontierMath Tier 4 v2	97.6%	83.0%	87.8%	—
HLE with tools	57.2%	—	65.0%	—
HealthBench Professional	63.4%	60.5%	58.1%	52.1%
Standard input price	$10/M	$4/M	$10/M	$0.75/M
Standard output price	$50/M	$20/M	$50/M	$3.75/M

My routing preference would depend on the failure mode

Astra: I’d prioritize it for long coding changes, browser or desktop automation, research spanning tools, technical artifacts, and scientific or operational workflows. Its premium is most defensible when failed runs require expensive human intervention.

Claude Fable 5.1: I’d keep it in evaluations centered on long-form reasoning. Its 65.7 AA Intelligence Index and 65.0 HLE-with-tools scores exceed Astra’s 61.2 and 57.2 in OpenAI’s comparison.

Gemini 3.8 Flash: I’d consider it for high-volume multimodal extraction, audio/video understanding, and routine agents. Its broader native inputs and introductory $0.75/$3.75 pricing put it at a very different operating point.

GPT-5.6 Sol: I’d retain it wherever Astra’s execution improvements don’t compensate for the higher token rates.

A unified multi-model API such as CometAPI can be useful for running these comparisons through one integration, but the decision should still come from your own task evaluations.

Cyber Capability Changes the Deployment Design

Astra is the first broadly deployed OpenAI model to reach the Critical cybersecurity capability threshold under its Preparedness Framework.

OpenAI says that, with appropriate tools and access, it can identify previously unknown security flaws and develop exploitation strategies across hardened systems. Reported launch results include:

ExploitBench: 100%
ExploitGym: 42.4%
SRE-Bench, one attempt: 88.0%

That capability comes with restrictions an application needs to handle.

OpenAI warns that production safeguards can slow, pause, or stop legitimate work, especially in higher-risk cyber contexts. ChatGPT or Codex may request user review before continuing; an API task can stop.

The default deployment also refuses more advanced exploit-creation requests. OpenAI’s Daybreak program provides separate, vetted access for some defensive workflows.

There’s a monitoring caveat, too. The system card describes Astra as better aligned overall than Sol, while noting that its written reasoning can be harder to monitor under adversarial evaluation conditions. OpenAI uses broader misalignment monitoring around tool-using Astra inference.

For me, that makes least-privilege tools, approval boundaries, audit logs, and explicit agent permissions part of the architecture—not cleanup work after the demo.

Three Workflows Worth Testing

Repository-scale engineering

I’d give Astra a constrained development environment, a concrete issue or migration goal, tests, and scoped tool access.

Then I’d judge the resulting change rather than the elegance of its explanation: does it pass review, how many iterations failed, and how much human time did it consume?

The Terminal-Bench, DeepSWE, database-migration, computer-use, and long-context results make this a sensible evaluation target. They do not replace repository-specific testing.

Visual game-development tasks

Playco tested Astra inside Playbot, an AI-powered IDE working directly in Unity and Godot. OpenAI reports 50% fewer manual fixes than with the previous model, plus three themed prototypes built from one grey-box foundation.

This is a useful example because the job extends beyond code generation. It involves spatial layout, playing the game, testing changes, identifying bugs, and iterating inside a visual environment.

Legal and financial review

Astra’s professional-work positioning includes polished documents, spreadsheets, and analyses.

The useful capability here is maintaining state across many files, cross-checking evidence, and producing a finished artifact—not merely answering one question about a document.

I’d still require expert validation, approval controls, and least-privilege data access. Better artifact generation does not remove those responsibilities.

My Adoption Rule

I’d adopt Astra where an execution-focused evaluation demonstrates a better outcome per dollar: fewer failed agent runs, less human correction, or better completed artifacts.

I would not adopt it solely for the 1.05M-token context window, saturated benchmark scores, or a newer model name. Long context costs extra, fine-tuning is unavailable, native audio/video input is absent, and cyber safeguards can interrupt work.

The practical opportunity is narrower—and more useful—than “best model at everything”: an agent engine that may earn its premium by actually finishing the difficult job.

Claude Fable 5.1’s Cache Pricing Is More Interesting Than the Headline Benchmarks

Olivia Hayes — Tue, 08 Sep 2026 03:06:34 +0000

Claude Fable 5.1 is easy to frame as another frontier-model release.

It has a 1M-token context window, up to 128K output, text and image input, adaptive thinking, and the usual collection of strong coding and knowledge-work benchmarks.

But the number that caught my attention wasn’t a benchmark score.

It was $0.25 per million tokens for cache reads. That is a big change from the previous $1 per million cache-read price, and it matters much more once you stop thinking about isolated prompts and start thinking about long-running agents.

Agents keep paying for the same context

A lot of agent workloads are repetitive in a very specific way.

The task changes from turn to turn, but a huge amount of the context doesn’t.

A coding agent might keep carrying the same repository map, system instructions, tool definitions, project documentation, and previously loaded files across dozens of interactions.

A research agent may repeatedly reuse the same source set while asking new questions.

A document workflow might keep the same policy manuals or reference material in context for an entire session.

That is where cache pricing starts to matter.

If the stable part of the prompt can be reused cheaply, the economics of a 30-turn or 50-turn workflow can look very different from what the headline input price suggests.

Fable 5.1’s official base pricing is still $10 per million input tokens and $50 per million output tokens, so this is not a cheap model in the usual sense.

But base token price isn’t the whole story anymore.

A simple pricing comparison can be misleading

Imagine two models.

One has cheaper normal input.

The other has much cheaper cache reads.

Which one costs less?

There’s no useful answer without knowing the workload.

If you send independent prompts all day, cache pricing might barely matter.

If you run a coding agent for an hour and most of its context is stable between turns, it can matter a lot.

The same goes for model quality.

A model with cheaper caching can still lose economically if it needs more retries, makes more bad tool calls, or takes much longer to finish the job.

And a model that looks expensive per token can still be cheaper overall if it completes the task cleanly on the first attempt.

That’s why I wouldn’t compare Fable 5.1 with another frontier model by looking at the pricing page alone.

I’d measure the whole agent session.

What I’d actually record

For long-running agent workloads, I’d want to track:

uncached input tokens
cached input tokens
output tokens
number of tool calls
failed tool calls
retries
total runtime
human corrections
whether the final result was actually accepted

Then I’d calculate cost per accepted task.

That number is much closer to what I actually care about in production.

A $2 run that fails is not cheaper than a $4 run that works.

And a model that saves money through caching can lose that advantage very quickly if it keeps wandering into unnecessary tool calls.

The 1M context window makes caching more important, not less

Fable 5.1 supports a 1M-token context window by default.

It’s tempting to look at that and think context management is becoming less important.

I think the opposite is true.

The larger the context gets, the more expensive it becomes to repeatedly move the same information through the model.

A huge context window is useful, but I still wouldn’t throw an entire repository or document archive into every request unless the task really needs it.

I’d still use retrieval.

I’d still prune irrelevant history.

I’d still summarize old state.

And if a large prefix really does need to stay around, I’d pay close attention to whether it can be cached.

A million-token window gives you headroom.

It doesn’t make context architecture irrelevant.

This is where Fable 5.1 starts to make sense to me

I wouldn’t use a model like Fable 5.1 for every request.

Simple extraction, short summaries, routine classification, and predictable automation usually don’t justify premium-model pricing.

The model gets more interesting when the work is both difficult and long-lived.

Things like:

multi-hour coding sessions
agents working across large repositories
long research workflows
document-heavy professional tasks
tool-using agents that retain a lot of state
workflows where failed attempts are expensive

Those are exactly the places where cheap cache reuse and strong agent behavior can compound.

Anthropic also positions Fable 5.1 around demanding, long-horizon coding and knowledge work rather than routine traffic.

That seems like the right way to think about it.

There are migration details I’d test before moving traffic

One thing I wouldn’t do is assume Fable 5.1 is a drop-in replacement just because the API shape looks familiar.

The current model documentation notes some behavior that can affect agent integrations.

For example, forced tool choice using a named tool or any is not supported in the same way, and some workflows need to be tested around thinking blocks, edited conversation histories, and model switching.

Those sound like implementation details until your agent depends on one of them.

So before migrating production traffic, I’d run the same tool-heavy workflows I already have and verify that the model behaves correctly end to end.

Not just that the first response looks good.

I’d compare Fable 5.1 on sessions, not prompts

This is probably the biggest change in how I’d evaluate it.

A single prompt hides most of the interesting economics.

A 40-turn coding session exposes them.

That’s where cache reuse, tool reliability, retries, context growth, and output length all start interacting.

I’ve been using CometAPI for these kinds of side-by-side tests because Fable 5.1 can be accessed through the same broader API setup as other models. That makes it easier to keep the surrounding workload unchanged and compare the model rather than the integration.

For me, that is the interesting part of Fable 5.1.

The 1M context window is useful.

The benchmark improvements are useful.

But cutting cache reads to $0.25 per million tokens changes the economics of a particular class of workload in a way that is much easier to feel once you’re running agents for hours instead of prompts for seconds.

That’s the number I’d benchmark first.

Disclosure: This post is adapted from research originally published by the CometAPI team.