GPT-5.6 Sol Scores Higher. Why Does My Weekly Limit Vanish Faster?

A higher score can describe the engine, the harness, or the whole machine. Those are not the same claim.

I have been using GPT-5.6 Sol for a few days, and my reaction is genuinely mixed.

The context budget shown in my Codex sessions is larger than what I was seeing with GPT-5.5 — roughly 356K versus 250K in my setup. That sounds like a clean upgrade. In practice, my weekly allowance feels like it disappears much faster. Sol also feels substantially slower than GPT-5.5 did. It asks me more questions at Max than GPT-5.5 did at xhigh, and Ultra is not my thing so far.

Ultra felt less like a controlled workspace and more like opening a door into a room full of agents doing something. I could see activity, but I did not feel that I had a clean control plane for understanding it, stopping it, or deciding which intermediate result I trusted.

That experience pushed me into a larger question: when a model company or leaderboard says one model is better, what exactly became better?

Answer Snapshot

Claim	What it usually means	What it does not tell me
“1M context window”	The model or product accepts a large maximum token budget under stated conditions	How reliably it uses every token, when the product compacts, or how fast my allowance drains
“Higher Intelligence Index”	A better weighted result across a published mix of agentic, coding, general, and scientific evaluations	Whether the product is faster, less annoying, easier to steer, or better for my exact workflow
“Higher Agentic Index”	Better final outcomes on tool-using, multi-step tasks in a specified harness	That the system uses a swarm, exposes its plan, or validates its own workers
“Higher Coding Index”	Better performance on the coding benchmarks inside that index	That the full coding assistant is better at repo navigation, review, interruption, or cost control
“Ultra”	A multi-agent product mode that splits suitable work across subagents	That more parallel work will automatically be more correct or more controllable

My current conclusion is simple: a benchmark score is a measurement contract, not a model review. If I do not know the tasks, tools, settings, grader, and aggregation rule, the headline number is almost useless to me.

The Context-Window Numbers Are Not Actually Contradicting Each Other

The first thing I had to untangle was my own context-window comparison.

OpenAI's current API pages list both GPT-5.6 Sol and GPT-5.5 with 1.05M-token context windows. That clearly does not match the roughly 356K and 250K figures I saw in Codex. Elsewhere, OpenAI's ChatGPT Business model-and-limits page documents a 272K window for GPT-5.6 Sol.

That is not necessarily an error. It means “the context window” can refer to several different layers:

Layer	The question it answers	Example
Model maximum	What can the API model accept under its published limits?	OpenAI currently lists 1.05M for both GPT-5.6 Sol and GPT-5.5
Product-exposed window	How much does this ChatGPT, Codex, or workspace surface make available?	A product can expose less than the API maximum
Session budget	How much space remains after system instructions, tool definitions, files, messages, and output reserve?	The number I see inside an active coding session
Compaction threshold	When does the agent summarize history instead of continuing to grow it?	A user-configured threshold can be far below the maximum
Effective context	How much information can the model use reliably for this task?	Task-dependent, and usually smaller than the advertised maximum

This distinction matters because I use Claude Code's 1M-capable models, but I set CLAUDE_CODE_AUTO_COMPACT_WINDOW to about 300K. Anthropic's Claude Code environment-variable documentation explicitly supports treating a 1M model as if it had a smaller window for auto-compaction. Anthropic's platform release notes confirm that 1M context is generally available on supported Claude models.

So when I say 300K–400K feels like the sweet spot, I am not claiming that models cannot use more. I am describing an operating preference: enough room for a serious coding session, but early enough compaction that the session does not become a landfill of stale decisions, old errors, and irrelevant tool output.

Maximum Context Is Not Effective Context

Long context solves a real problem. I do not want an agent to forget the architecture discussion from twenty minutes ago, reread the same files, or lose the acceptance criteria halfway through a change.

But a larger window also creates three different failure modes that people often collapse into “context rot”:

Failure mode	What goes wrong	What it feels like
Lost in the middle	Relevant information receives less reliable attention because of where it sits	The model remembers the opening rule and the latest message but misses a decision buried in between
Context rot	Performance degrades as more material is added, even when the answer remains somewhere in the prompt	The session knows more facts but makes worse choices
Context drag	Every turn carries more history, tool output, and conflicting state through the product	Higher latency, higher usage, and more opportunities to follow stale instructions

Lost in the Middle showed that long-context performance can change significantly based on where the relevant information appears. Chroma's Context Rot research tests the broader problem of performance degrading as input grows. These do not prove that 300K is a universal optimum. They do prove that “it fits” and “the model uses it well” are separate claims.

Even Artificial Analysis's long-context component, AA-LCR, tests reasoning across roughly 100K tokens per question. That is useful evidence about long-document reasoning. It is not evidence that a model remains equally reliable through every position of a million-token coding session full of edits, failed commands, and changing intent.

Why My Weekly Limit Can Disappear Faster

I cannot tell from my desk whether Sol is slow because of launch demand, backend load, model behavior, or the shape of my own tasks. OpenAI does not expose the telemetry I would need to separate those causes, so “a lot of people are using it” stays a hypothesis.

The usage side is easier to explain. OpenAI's Codex pricing documentation says usage depends on model choice, context, reasoning, tool use, retrieval, and caching; similar-looking tasks can consume different amounts. It also says additional weekly limits may apply.

A bigger retained context can therefore make an agent session more expensive even if the final answer is short. The model may process more history, reason longer, issue more tool calls, and revisit more evidence. Cached input can make repeated prefixes cheaper, but it does not turn a long agent loop into free work.

This is why I do not find “token-efficient model” and “my allowance vanished faster” contradictory. One is a model-level claim about how much output or reasoning the model needs for a benchmark. The other is a product-level outcome across my whole session.

The metric I want is not tokens per answer. It is allowance consumed per accepted result.

If Sol completes a difficult refactor correctly in one run, higher usage may be worth it. If it spends more, asks more questions, moves more slowly, and still needs the same review, the benchmark win has not reached my workflow.

Max And Ultra Are Different Products

OpenAI's Codex model guide makes a distinction that explains part of my reaction:

Control	What it changes	Best fit	Main risk
Extra High / xhigh	Raises the selected model's reasoning effort	Difficult work with multiple steps, sources, or trade-offs	Latency and usage rise without guaranteed improvement
Max	Gives the selected model more time to reason about one task	The hardest single problems, when depth matters more than speed or usage	A long single-agent run can still take the wrong path
Ultra	Uses subagents to work on separate parts in parallel	Work that divides cleanly into meaningful independent streams	More activity, coordination overhead, conflicting outputs, and less obvious control

The same documentation says there is no exact mapping from GPT-5.5 reasoning efforts to GPT-5.6. Max also changes more than the standard reasoning-level selector. That means my GPT-5.5 xhigh versus GPT-5.6 Max comparison is a valid experience report, but not a controlled model experiment.

The extra questions are similar. I noticed them. I cannot conclude from a few days that GPT-5.6 always asks more questions. Prompt shape, product instructions, safety checks, and ambiguity can all change that behavior. What I can say is that question frequency belongs in my evaluation because every unnecessary clarification interrupts flow.

A leaderboard row compresses a workload, harness, inference setting, grader, and aggregation rule into one number.

What A Benchmark Score Is Made Of

Before reading any leaderboard, I now look for five layers:

Layer	Question	Why it changes the result
Workload	What tasks are actually being attempted?	A physics problem, bank-support workflow, and repo patch measure different capabilities
Harness	What tools, prompts, sandbox, memory, and turn limits does the model receive?	The same model can perform differently in Claude Code, Codex, Cursor, or a neutral harness
Inference setting	Which model snapshot, reasoning effort, temperature, and token budget are used?	“GPT-5.6 Sol” at high and Max are not the same test condition
Grader	Who or what decides whether the result is correct?	Unit tests, database state, exact match, rubrics, and LLM judges have different failure modes
Aggregation	How do task scores become the headline number?	A weighted average can hide a major weakness behind strengths elsewhere

This is what the “trust me bro” benchmark feeling usually points at. The benchmark may be legitimate. The problem is that the headline chart hides the measurement contract.

Every Current Artificial Analysis LLM Index, In Practical Popularity Order

Artificial Analysis does not publish traffic for each leaderboard, so there is no honest way to claim a precise popularity ranking. The order below is my practical one as of July 2026: broad indexes with the most prominent placement and widest model-selection use come first; specialized cross-cutting indexes follow; professional indexes then use the order shown in Artificial Analysis's capability navigation. This covers the 12 current indexes for text models and coding agents—not its separate image, video, speech, music, or hardware leaderboards.

1–6: The Indexes Most People Should Understand First

Order	Index and recipe	Concrete scenario	What a high score does not prove
1	Intelligence Index v4.1 34% agents, 24% coding, 24% scientific reasoning, 18% general capability	Create a spreadsheet and memo, resolve a tool-using banking case, complete a terminal task, answer a research-level science question, and find evidence across long documents	That the model leads every component, feels fast, uses little allowance, or works well in my product
2	Coding Index 50% Terminal-Bench v2.1, 50% SciCode	Repair a broken environment through a terminal, or turn a scientific algorithm description into Python that passes tests	That a complete coding product understands my repository, reviews its patch, or is easy to interrupt
3	Agentic Index 50% GDPval-AA v2, 50% τ³-Banking	Use a sandbox and web tools to produce professional deliverables, or search policy and execute the correct multi-step account workflow	That the product uses a swarm, delegates well, or validates subagent output
4	Coding Agent Index v1.1 Equal average of DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA	Read a repository, implement a long-horizon change, operate the terminal, and answer architecture questions about the codebase	That the underlying model alone earned the result; agent harness, model, and settings are part of the tested variant
5	Multilingual Index Global-MMLU-Lite across 16 languages	Answer equivalent general-knowledge and reasoning questions in Chinese, Hindi, Arabic, Yoruba, or Burmese—not only English	High-quality translation, local tone, long-form writing, culturally safe advice, or multilingual tool use
6	Openness Index 0–100 score for model access and licensing, plus training-data and methodology transparency	Decide whether I can download the weights, use them commercially, inspect the disclosed data, and reproduce or audit the training approach	Intelligence, safety, factual accuracy, or that “open weights” means fully open data and training code

7–12: Professional Indexes

These industry indexes reuse many of the same underlying benchmarks, then weight capabilities using an O*NET-style map of how often those abilities appear in the field. They are useful routing signals, not professional certification.

Order	Index and weights	Concrete scenario	What a high score does not prove
7	Finance & Accounting 30% business knowledge, 30% agentic work, 20% reasoning, 10% customer interaction, 5% long context, 5% non-hallucination	Read filings, build a valuation and sensitivity model, reconcile figures, and produce an investment or management memo	That every formula is audit-ready, the market data is current, or the work complies with my jurisdiction and controls
8	Strategy & Ops 30% business knowledge, 30% agentic work, 30% customer interaction, 5% instruction following, 5% long context	Turn operating data, policy documents, and stakeholder requests into a staffing plan, process change, and completed office workflow	Good organizational judgment, political feasibility, adoption by employees, or reliable execution inside my systems
9	Legal 35% legal knowledge, 25% agentic work, 15% reasoning, 10% long context, 10% non-hallucination, 5% customer interaction	Review a contract and case-law packet, identify conflicting authorities, draft a memo, and avoid inventing a citation	Current law in my jurisdiction, privilege protection, citation validity, or work that is safe to file without a lawyer
10	Healthcare & Medical 35% medical knowledge, 25% agentic work, 15% non-hallucination, 15% reasoning, 10% customer interaction	Combine symptoms, history, medications, and guidelines into a clinical reasoning summary, then coordinate an EHR or pharmacy workflow	A diagnosis, safe patient-specific treatment, regulatory approval, or replacement for clinical review
11	Engineering 35% engineering knowledge, 35% reasoning, 25% agentic work, 5% terminal use	Size a wind-turbine support structure against fatigue and extreme loads, justify safety margins, and automate calculations	That the design was simulated, independently checked, code-compliant, manufacturable, or ready to sign off
12	Economics 35% economics knowledge, 35% reasoning, 15% agentic work, 15% long context	Estimate a tariff's incidence and elasticity effects, quantify welfare trade-offs, and synthesize conflicting research	That the data, causal assumptions, forecast, or resulting policy recommendation is valid for the real economy

One historical wrinkle is worth documenting. The Artificial Analysis Data API page still mentions a Math Index field, but the current Intelligence methodology says MATH-500 and AIME 2025 are retired from active reporting. I would treat Math as a legacy/API field, not a thirteenth current public index, unless Artificial Analysis restores a live methodology and leaderboard.

Artificial Analysis Intelligence Index: A Portfolio, Not One Test

As of July 2026, the Artificial Analysis Intelligence Index v4.1 combines nine evaluations. The category weights are Agents 34%, Coding 24%, Scientific Reasoning 24%, and General 18%.

So an Intelligence Index score is not “deep reasoning on one hard problem,” and it is not “swarm orchestration.” It is a portfolio:

Category	What is inside	Plain-English scenario
Agents — 34%	GDPval-AA v2 and τ³-Banking	Create professional files from supplied material, or resolve a banking case by finding policy and making the correct tool-mediated account changes
Coding — 24%	Terminal-Bench v2.1 and SciCode	Complete a multi-step terminal task, or write scientific Python that passes tests
Scientific Reasoning — 24%	Humanity's Last Exam, GPQA Diamond, and CritPt	Solve difficult expert-level questions in science and research physics
General — 18%	AA-LCR and AA-Omniscience	Reason across long documents, answer factual questions, and know when not to guess

This is much more useful than a single academic multiple-choice score. It is also an editorial choice. Artificial Analysis increased the agentic weighting in v4.1. A model can improve on the headline index because it became better at tool-using work, even if its improvement on the hardest single scientific question is smaller.

What The Underlying Benchmarks Feel Like

Benchmark	Human translation	How success is checked
GDPval-AA v2	Do a realistic professional task across one of 44 occupations and submit usable files	Blind pairwise judging, converted into an Elo rating anchored to human expert work
τ³-Banking	Search a large banking policy base, talk to a simulated user, and execute the right account workflow	The final backend database state, averaged across repeated attempts
Terminal-Bench v2.1	Operate a real terminal to complete software, system administration, data, training, or security work	Task-specific tests, pass or fail
SciCode	Turn a scientific problem description into working Python	Code execution and unit tests
AA-LCR	Find and combine evidence spread across about 100K tokens of long documents	An answer-equivalence grader
AA-Omniscience	Answer across 42 knowledge topics, but abstain instead of fabricating when unsure	Accuracy plus a separate non-hallucination component
Humanity's Last Exam	Answer difficult expert-written questions across science, math, and the humanities	Answer equality, pass@1
GPQA Diamond	Solve graduate-level biology, physics, and chemistry questions that non-experts usually miss	Four-option multiple choice, pass@1
CritPt	Solve unpublished research-level physics challenges	Numerical, symbolic, or Python-function grading through an official server

Now the top-line Intelligence Index has a shape. If my job is contract review, customer support, codebase maintenance, or scientific research, I should not care about every component equally.

Coding Index Is Not Coding Agent Index

This naming is where normal people have every right to get confused.

The Artificial Analysis Coding Index is a model capability index. It equally combines Terminal-Bench v2.1 and SciCode. In plain language: can the model operate in a terminal, and can it generate correct scientific code?

The separate Artificial Analysis Coding Agent Index evaluates coding-agent variants across DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA. Its scenarios include long-horizon repository changes, terminal execution, and questions that require understanding a codebase. Artificial Analysis even shows harness comparisons that hold the model constant while comparing products such as Claude Code, Cursor CLI, and OpenCode.

If I want to know…	Look at…
Can the raw model solve terminal and scientific programming problems?	Coding Index and its two components
Can a configured coding agent make patches, use a terminal, and understand a repository?	Coding Agent Index and the per-benchmark breakdown
Will Codex feel fast, controlled, and cheap on my subscription?	Neither index by itself; I need product and workload measurements

The Coding Agent Index is closer to my daily experience, but it still does not measure whether I can interrupt a swarm cleanly, whether the UI explains what each worker is doing, or how quickly I hit a weekly limit.

Agentic Index Does Not Mean Swarm Index

The Artificial Analysis Agentic Index is an equal-weighted combination of GDPval-AA v2 and τ³-Banking. It measures planning, tool use, autonomy, professional deliverables, knowledge retrieval, customer interaction, and final task completion.

Those are agentic skills. They do not require a swarm.

Artificial Analysis runs GDPval-AA through its open-source Stirrup harness. One model receives tools and a sandbox, works through the task, and submits deliverables. The benchmark can tell me that a model-plus-harness is better at completing this kind of work. It does not tell me that the model is a strong multi-agent orchestrator.

Delegation creates outputs. Verification creates evidence that those outputs deserve trust.

A Swarm Is Only As Good As Its Control Loop

My answer to the orchestration question is yes: worker-model selection matters. But it is only one term in the system.

Stage	What can fail	What good control looks like
Decomposition	The orchestrator splits the task along the wrong boundaries	Independent scopes, explicit inputs, explicit acceptance criteria
Worker execution	A worker misunderstands its part or lacks capability	Task-appropriate model selection and bounded tools
Aggregation	The parent combines incompatible answers without noticing	Conflict detection, provenance, and structured handoffs
Verification	The system treats confident worker output as evidence	Tests, schemas, independent review, source checks, and reproducible commands
Recovery	One failed branch poisons the final answer or causes endless retries	Visible failure states, retry limits, cancellation, and rollback
Human control	The user cannot see cost, progress, or why the swarm chose a direction	Budgets, checkpoints, pause/stop controls, and reviewable artifacts

A parent agent summarizing five worker responses is not validation. A parent checking those responses against tests, evidence, schemas, or an independent reviewer is validation.

Benchmarks add another source of confusion here. Terminal-Bench's tests, τ³-Banking's database checks, and GDPval-AA's judge panel validate the benchmark outcome. They do not prove that the agent or swarm internally validated its own work before submission.

That distinction matters to me. I do not only want the final answer to pass after an external judge inspects it. I want the operating process to expose enough evidence that I can trust, interrupt, and repair it while it is running.

The Popular Rankings That Are Not Indexes

The model page also ranks models by speed, latency, price, context, and efficiency. These columns matter to my Sol experience, but they do not combine several capability benchmarks into an index.

Ranking	What it measures	Example of the question it answers	Common misread
Output speed	Answer tokens received per second after generation starts; the current default performance workload uses about 10K input tokens	Once Sol starts answering, how quickly does visible text arrive?	Fast output does not mean a short wait before the answer starts
Time to first token	Seconds from API request to the first streamed token	How quickly do I see any activity?	For reasoning models, that first token may be thinking rather than the answer
Time to first answer token	Input-processing time plus hidden or visible reasoning before the first answer token	How long until I receive useful answer text?	It still does not include the time to finish the response or an agent workflow
End-to-end response time	Estimated seconds to receive a 500-token answer, including input, reasoning, and generation time	Which API returns a medium-length answer sooner?	It is not prompt-to-reviewed-result time for a ten-minute coding task
Price per million tokens	Published API token prices, often shown as a blended cache/input/output rate	What does raw API traffic cost under a stated token mix?	It is not a subscription's weekly allowance or cost per successful outcome
Cost per index task	Observed benchmark token use multiplied by the relevant input, cache, reasoning, and answer prices	How expensive was the model's average benchmark workload?	An “average task” may look nothing like my repository or prompt
Time per index task	Weighted output tokens per task divided by output speed; it excludes TTFT and harness overhead	How long would decoding the index's typical output take?	It is not true wall-clock completion time
Output tokens per index task	Weighted reasoning and answer tokens produced for an average index task	Which model spends more generated tokens to reach its score?	Fewer tokens do not automatically mean cheaper, faster, or better
Context window	The maximum combined input and output token limit reported for the model	Can this request fit at all?	It does not measure effective recall, context rot, product compaction, or allowance use

Which Number Should I Care About?

My workload	Useful public signal	What I still need to test myself
One very hard science or reasoning problem	HLE, GPQA Diamond, CritPt, plus the chosen reasoning setting	Accuracy on my domain, latency, and whether citations survive review
Scientific or algorithmic code generation	Coding Index, especially SciCode	My language, libraries, tests, and maintainability
Large repository change	Coding Agent Index, especially DeepSWE and Terminal-Bench	Repo conventions, regression rate, review time, and recovery from failed commands
Customer-support automation	Agentic Index, especially τ³-Banking	My policies, guardrails, escalation rules, and real backend state
Long reports, filings, or legal packets	AA-LCR and relevant professional index	Evidence recall at my document length and citation accuracy
Factual assistant that must admit uncertainty	AA-Omniscience accuracy and non-hallucination	My authoritative sources and abstention policy
Multi-agent coding swarm	No single headline index	Task splitting, duplicate work, conflicts, validator coverage, stop behavior, total cost, and accepted-result rate

The Scorecard I Actually Want For Sol

After a few days, I do not have enough data to declare GPT-5.6 Sol better or worse than GPT-5.5 for my work. I do have a better idea of what I should record:

• Accepted-result rate: how often I accept the work without a major repair.
• End-to-end time: from prompt to reviewed result, not output tokens per second.
• Allowance per accepted result: how much of the five-hour and weekly budgets the task consumed.
• Intervention burden: clarifying questions, permission pauses, corrections, and restarts.
• Context survival: whether early constraints and decisions still influence late-stage work.
• Control: whether I can see progress, cap scope, stop a branch, and understand what happened.
• Verification: whether the system produced tests, evidence, and reviewable artifacts instead of confident summaries.

That scorecard might disagree with a public leaderboard. That is fine. The leaderboard is answering its question. I am answering mine.

Where I Think LLMs Are Going

I do not think the direction is merely “make the single model smarter.” Three things are evolving at once:

1. The single model gets better at reasoning, coding, tool use, and deciding what matters.
2. The runtime gets better at context management, caching, compaction, permissions, and recovery.
3. The product gets better at orchestration: subagents, specialists, parallel work, judges, and validators.

Benchmarks are moving in the same direction. Artificial Analysis v4.1 gives agentic work the largest category weight in its Intelligence Index. Coding-agent leaderboards increasingly compare harnesses, not only model names. That makes sense because the useful unit is becoming the system.

But the industry is still too eager to compress the system back into one number.

My current opinion on context is still that 300K–400K is a comfortable operating range for a long coding session. My current opinion on Ultra is that I do not want more agents until I get a clearer control loop. And my current opinion on GPT-5.6 Sol is mixed: I can see the capability direction, but speed, limits, questions, and controllability are part of capability too.

The benchmark is not lying when it says a model is stronger. It is answering a narrower question than the one most people think they asked.

Originally published at markhuang.ai