DEV Community: Mark Huang

GPT-5.6 Sol Scores Higher. Why Does My Weekly Limit Vanish Faster?

Mark Huang — Mon, 13 Jul 2026 00:03:49 +0000

A higher score can describe the engine, the harness, or the whole machine. Those are not the same claim.

I have been using GPT-5.6 Sol for a few days, and my reaction is genuinely mixed.

The context budget shown in my Codex sessions is larger than what I was seeing with GPT-5.5 — roughly 356K versus 250K in my setup. That sounds like a clean upgrade. In practice, my weekly allowance feels like it disappears much faster. Sol also feels substantially slower than GPT-5.5 did. It asks me more questions at Max than GPT-5.5 did at xhigh, and Ultra is not my thing so far.

Ultra felt less like a controlled workspace and more like opening a door into a room full of agents doing something. I could see activity, but I did not feel that I had a clean control plane for understanding it, stopping it, or deciding which intermediate result I trusted.

That experience pushed me into a larger question: when a model company or leaderboard says one model is better, what exactly became better?

Answer Snapshot

Claim	What it usually means	What it does not tell me
“1M context window”	The model or product accepts a large maximum token budget under stated conditions	How reliably it uses every token, when the product compacts, or how fast my allowance drains
“Higher Intelligence Index”	A better weighted result across a published mix of agentic, coding, general, and scientific evaluations	Whether the product is faster, less annoying, easier to steer, or better for my exact workflow
“Higher Agentic Index”	Better final outcomes on tool-using, multi-step tasks in a specified harness	That the system uses a swarm, exposes its plan, or validates its own workers
“Higher Coding Index”	Better performance on the coding benchmarks inside that index	That the full coding assistant is better at repo navigation, review, interruption, or cost control
“Ultra”	A multi-agent product mode that splits suitable work across subagents	That more parallel work will automatically be more correct or more controllable

My current conclusion is simple: a benchmark score is a measurement contract, not a model review. If I do not know the tasks, tools, settings, grader, and aggregation rule, the headline number is almost useless to me.

The Context-Window Numbers Are Not Actually Contradicting Each Other

The first thing I had to untangle was my own context-window comparison.

OpenAI's current API pages list both GPT-5.6 Sol and GPT-5.5 with 1.05M-token context windows. That clearly does not match the roughly 356K and 250K figures I saw in Codex. Elsewhere, OpenAI's ChatGPT Business model-and-limits page documents a 272K window for GPT-5.6 Sol.

That is not necessarily an error. It means “the context window” can refer to several different layers:

Layer	The question it answers	Example
Model maximum	What can the API model accept under its published limits?	OpenAI currently lists 1.05M for both GPT-5.6 Sol and GPT-5.5
Product-exposed window	How much does this ChatGPT, Codex, or workspace surface make available?	A product can expose less than the API maximum
Session budget	How much space remains after system instructions, tool definitions, files, messages, and output reserve?	The number I see inside an active coding session
Compaction threshold	When does the agent summarize history instead of continuing to grow it?	A user-configured threshold can be far below the maximum
Effective context	How much information can the model use reliably for this task?	Task-dependent, and usually smaller than the advertised maximum

This distinction matters because I use Claude Code's 1M-capable models, but I set CLAUDE_CODE_AUTO_COMPACT_WINDOW to about 300K. Anthropic's Claude Code environment-variable documentation explicitly supports treating a 1M model as if it had a smaller window for auto-compaction. Anthropic's platform release notes confirm that 1M context is generally available on supported Claude models.

So when I say 300K–400K feels like the sweet spot, I am not claiming that models cannot use more. I am describing an operating preference: enough room for a serious coding session, but early enough compaction that the session does not become a landfill of stale decisions, old errors, and irrelevant tool output.

Maximum Context Is Not Effective Context

Long context solves a real problem. I do not want an agent to forget the architecture discussion from twenty minutes ago, reread the same files, or lose the acceptance criteria halfway through a change.

But a larger window also creates three different failure modes that people often collapse into “context rot”:

Failure mode	What goes wrong	What it feels like
Lost in the middle	Relevant information receives less reliable attention because of where it sits	The model remembers the opening rule and the latest message but misses a decision buried in between
Context rot	Performance degrades as more material is added, even when the answer remains somewhere in the prompt	The session knows more facts but makes worse choices
Context drag	Every turn carries more history, tool output, and conflicting state through the product	Higher latency, higher usage, and more opportunities to follow stale instructions

Lost in the Middle showed that long-context performance can change significantly based on where the relevant information appears. Chroma's Context Rot research tests the broader problem of performance degrading as input grows. These do not prove that 300K is a universal optimum. They do prove that “it fits” and “the model uses it well” are separate claims.

Even Artificial Analysis's long-context component, AA-LCR, tests reasoning across roughly 100K tokens per question. That is useful evidence about long-document reasoning. It is not evidence that a model remains equally reliable through every position of a million-token coding session full of edits, failed commands, and changing intent.

Why My Weekly Limit Can Disappear Faster

I cannot tell from my desk whether Sol is slow because of launch demand, backend load, model behavior, or the shape of my own tasks. OpenAI does not expose the telemetry I would need to separate those causes, so “a lot of people are using it” stays a hypothesis.

The usage side is easier to explain. OpenAI's Codex pricing documentation says usage depends on model choice, context, reasoning, tool use, retrieval, and caching; similar-looking tasks can consume different amounts. It also says additional weekly limits may apply.

A bigger retained context can therefore make an agent session more expensive even if the final answer is short. The model may process more history, reason longer, issue more tool calls, and revisit more evidence. Cached input can make repeated prefixes cheaper, but it does not turn a long agent loop into free work.

This is why I do not find “token-efficient model” and “my allowance vanished faster” contradictory. One is a model-level claim about how much output or reasoning the model needs for a benchmark. The other is a product-level outcome across my whole session.

The metric I want is not tokens per answer. It is allowance consumed per accepted result.

If Sol completes a difficult refactor correctly in one run, higher usage may be worth it. If it spends more, asks more questions, moves more slowly, and still needs the same review, the benchmark win has not reached my workflow.

Max And Ultra Are Different Products

OpenAI's Codex model guide makes a distinction that explains part of my reaction:

Control	What it changes	Best fit	Main risk
Extra High / xhigh	Raises the selected model's reasoning effort	Difficult work with multiple steps, sources, or trade-offs	Latency and usage rise without guaranteed improvement
Max	Gives the selected model more time to reason about one task	The hardest single problems, when depth matters more than speed or usage	A long single-agent run can still take the wrong path
Ultra	Uses subagents to work on separate parts in parallel	Work that divides cleanly into meaningful independent streams	More activity, coordination overhead, conflicting outputs, and less obvious control

The same documentation says there is no exact mapping from GPT-5.5 reasoning efforts to GPT-5.6. Max also changes more than the standard reasoning-level selector. That means my GPT-5.5 xhigh versus GPT-5.6 Max comparison is a valid experience report, but not a controlled model experiment.

The extra questions are similar. I noticed them. I cannot conclude from a few days that GPT-5.6 always asks more questions. Prompt shape, product instructions, safety checks, and ambiguity can all change that behavior. What I can say is that question frequency belongs in my evaluation because every unnecessary clarification interrupts flow.

A leaderboard row compresses a workload, harness, inference setting, grader, and aggregation rule into one number.

What A Benchmark Score Is Made Of

Before reading any leaderboard, I now look for five layers:

Layer	Question	Why it changes the result
Workload	What tasks are actually being attempted?	A physics problem, bank-support workflow, and repo patch measure different capabilities
Harness	What tools, prompts, sandbox, memory, and turn limits does the model receive?	The same model can perform differently in Claude Code, Codex, Cursor, or a neutral harness
Inference setting	Which model snapshot, reasoning effort, temperature, and token budget are used?	“GPT-5.6 Sol” at high and Max are not the same test condition
Grader	Who or what decides whether the result is correct?	Unit tests, database state, exact match, rubrics, and LLM judges have different failure modes
Aggregation	How do task scores become the headline number?	A weighted average can hide a major weakness behind strengths elsewhere

This is what the “trust me bro” benchmark feeling usually points at. The benchmark may be legitimate. The problem is that the headline chart hides the measurement contract.

Every Current Artificial Analysis LLM Index, In Practical Popularity Order

Artificial Analysis does not publish traffic for each leaderboard, so there is no honest way to claim a precise popularity ranking. The order below is my practical one as of July 2026: broad indexes with the most prominent placement and widest model-selection use come first; specialized cross-cutting indexes follow; professional indexes then use the order shown in Artificial Analysis's capability navigation. This covers the 12 current indexes for text models and coding agents—not its separate image, video, speech, music, or hardware leaderboards.

1–6: The Indexes Most People Should Understand First

Order	Index and recipe	Concrete scenario	What a high score does not prove
1	Intelligence Index v4.1 34% agents, 24% coding, 24% scientific reasoning, 18% general capability	Create a spreadsheet and memo, resolve a tool-using banking case, complete a terminal task, answer a research-level science question, and find evidence across long documents	That the model leads every component, feels fast, uses little allowance, or works well in my product
2	Coding Index 50% Terminal-Bench v2.1, 50% SciCode	Repair a broken environment through a terminal, or turn a scientific algorithm description into Python that passes tests	That a complete coding product understands my repository, reviews its patch, or is easy to interrupt
3	Agentic Index 50% GDPval-AA v2, 50% τ³-Banking	Use a sandbox and web tools to produce professional deliverables, or search policy and execute the correct multi-step account workflow	That the product uses a swarm, delegates well, or validates subagent output
4	Coding Agent Index v1.1 Equal average of DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA	Read a repository, implement a long-horizon change, operate the terminal, and answer architecture questions about the codebase	That the underlying model alone earned the result; agent harness, model, and settings are part of the tested variant
5	Multilingual Index Global-MMLU-Lite across 16 languages	Answer equivalent general-knowledge and reasoning questions in Chinese, Hindi, Arabic, Yoruba, or Burmese—not only English	High-quality translation, local tone, long-form writing, culturally safe advice, or multilingual tool use
6	Openness Index 0–100 score for model access and licensing, plus training-data and methodology transparency	Decide whether I can download the weights, use them commercially, inspect the disclosed data, and reproduce or audit the training approach	Intelligence, safety, factual accuracy, or that “open weights” means fully open data and training code

7–12: Professional Indexes

These industry indexes reuse many of the same underlying benchmarks, then weight capabilities using an O*NET-style map of how often those abilities appear in the field. They are useful routing signals, not professional certification.

Order	Index and weights	Concrete scenario	What a high score does not prove
7	Finance & Accounting 30% business knowledge, 30% agentic work, 20% reasoning, 10% customer interaction, 5% long context, 5% non-hallucination	Read filings, build a valuation and sensitivity model, reconcile figures, and produce an investment or management memo	That every formula is audit-ready, the market data is current, or the work complies with my jurisdiction and controls
8	Strategy & Ops 30% business knowledge, 30% agentic work, 30% customer interaction, 5% instruction following, 5% long context	Turn operating data, policy documents, and stakeholder requests into a staffing plan, process change, and completed office workflow	Good organizational judgment, political feasibility, adoption by employees, or reliable execution inside my systems
9	Legal 35% legal knowledge, 25% agentic work, 15% reasoning, 10% long context, 10% non-hallucination, 5% customer interaction	Review a contract and case-law packet, identify conflicting authorities, draft a memo, and avoid inventing a citation	Current law in my jurisdiction, privilege protection, citation validity, or work that is safe to file without a lawyer
10	Healthcare & Medical 35% medical knowledge, 25% agentic work, 15% non-hallucination, 15% reasoning, 10% customer interaction	Combine symptoms, history, medications, and guidelines into a clinical reasoning summary, then coordinate an EHR or pharmacy workflow	A diagnosis, safe patient-specific treatment, regulatory approval, or replacement for clinical review
11	Engineering 35% engineering knowledge, 35% reasoning, 25% agentic work, 5% terminal use	Size a wind-turbine support structure against fatigue and extreme loads, justify safety margins, and automate calculations	That the design was simulated, independently checked, code-compliant, manufacturable, or ready to sign off
12	Economics 35% economics knowledge, 35% reasoning, 15% agentic work, 15% long context	Estimate a tariff's incidence and elasticity effects, quantify welfare trade-offs, and synthesize conflicting research	That the data, causal assumptions, forecast, or resulting policy recommendation is valid for the real economy

One historical wrinkle is worth documenting. The Artificial Analysis Data API page still mentions a Math Index field, but the current Intelligence methodology says MATH-500 and AIME 2025 are retired from active reporting. I would treat Math as a legacy/API field, not a thirteenth current public index, unless Artificial Analysis restores a live methodology and leaderboard.

Artificial Analysis Intelligence Index: A Portfolio, Not One Test

As of July 2026, the Artificial Analysis Intelligence Index v4.1 combines nine evaluations. The category weights are Agents 34%, Coding 24%, Scientific Reasoning 24%, and General 18%.

So an Intelligence Index score is not “deep reasoning on one hard problem,” and it is not “swarm orchestration.” It is a portfolio:

Category	What is inside	Plain-English scenario
Agents — 34%	GDPval-AA v2 and τ³-Banking	Create professional files from supplied material, or resolve a banking case by finding policy and making the correct tool-mediated account changes
Coding — 24%	Terminal-Bench v2.1 and SciCode	Complete a multi-step terminal task, or write scientific Python that passes tests
Scientific Reasoning — 24%	Humanity's Last Exam, GPQA Diamond, and CritPt	Solve difficult expert-level questions in science and research physics
General — 18%	AA-LCR and AA-Omniscience	Reason across long documents, answer factual questions, and know when not to guess

This is much more useful than a single academic multiple-choice score. It is also an editorial choice. Artificial Analysis increased the agentic weighting in v4.1. A model can improve on the headline index because it became better at tool-using work, even if its improvement on the hardest single scientific question is smaller.

What The Underlying Benchmarks Feel Like

Benchmark	Human translation	How success is checked
GDPval-AA v2	Do a realistic professional task across one of 44 occupations and submit usable files	Blind pairwise judging, converted into an Elo rating anchored to human expert work
τ³-Banking	Search a large banking policy base, talk to a simulated user, and execute the right account workflow	The final backend database state, averaged across repeated attempts
Terminal-Bench v2.1	Operate a real terminal to complete software, system administration, data, training, or security work	Task-specific tests, pass or fail
SciCode	Turn a scientific problem description into working Python	Code execution and unit tests
AA-LCR	Find and combine evidence spread across about 100K tokens of long documents	An answer-equivalence grader
AA-Omniscience	Answer across 42 knowledge topics, but abstain instead of fabricating when unsure	Accuracy plus a separate non-hallucination component
Humanity's Last Exam	Answer difficult expert-written questions across science, math, and the humanities	Answer equality, pass@1
GPQA Diamond	Solve graduate-level biology, physics, and chemistry questions that non-experts usually miss	Four-option multiple choice, pass@1
CritPt	Solve unpublished research-level physics challenges	Numerical, symbolic, or Python-function grading through an official server

Now the top-line Intelligence Index has a shape. If my job is contract review, customer support, codebase maintenance, or scientific research, I should not care about every component equally.

Coding Index Is Not Coding Agent Index

This naming is where normal people have every right to get confused.

The Artificial Analysis Coding Index is a model capability index. It equally combines Terminal-Bench v2.1 and SciCode. In plain language: can the model operate in a terminal, and can it generate correct scientific code?

The separate Artificial Analysis Coding Agent Index evaluates coding-agent variants across DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA. Its scenarios include long-horizon repository changes, terminal execution, and questions that require understanding a codebase. Artificial Analysis even shows harness comparisons that hold the model constant while comparing products such as Claude Code, Cursor CLI, and OpenCode.

If I want to know…	Look at…
Can the raw model solve terminal and scientific programming problems?	Coding Index and its two components
Can a configured coding agent make patches, use a terminal, and understand a repository?	Coding Agent Index and the per-benchmark breakdown
Will Codex feel fast, controlled, and cheap on my subscription?	Neither index by itself; I need product and workload measurements

The Coding Agent Index is closer to my daily experience, but it still does not measure whether I can interrupt a swarm cleanly, whether the UI explains what each worker is doing, or how quickly I hit a weekly limit.

Agentic Index Does Not Mean Swarm Index

The Artificial Analysis Agentic Index is an equal-weighted combination of GDPval-AA v2 and τ³-Banking. It measures planning, tool use, autonomy, professional deliverables, knowledge retrieval, customer interaction, and final task completion.

Those are agentic skills. They do not require a swarm.

Artificial Analysis runs GDPval-AA through its open-source Stirrup harness. One model receives tools and a sandbox, works through the task, and submits deliverables. The benchmark can tell me that a model-plus-harness is better at completing this kind of work. It does not tell me that the model is a strong multi-agent orchestrator.

Delegation creates outputs. Verification creates evidence that those outputs deserve trust.

A Swarm Is Only As Good As Its Control Loop

My answer to the orchestration question is yes: worker-model selection matters. But it is only one term in the system.

Stage	What can fail	What good control looks like
Decomposition	The orchestrator splits the task along the wrong boundaries	Independent scopes, explicit inputs, explicit acceptance criteria
Worker execution	A worker misunderstands its part or lacks capability	Task-appropriate model selection and bounded tools
Aggregation	The parent combines incompatible answers without noticing	Conflict detection, provenance, and structured handoffs
Verification	The system treats confident worker output as evidence	Tests, schemas, independent review, source checks, and reproducible commands
Recovery	One failed branch poisons the final answer or causes endless retries	Visible failure states, retry limits, cancellation, and rollback
Human control	The user cannot see cost, progress, or why the swarm chose a direction	Budgets, checkpoints, pause/stop controls, and reviewable artifacts

A parent agent summarizing five worker responses is not validation. A parent checking those responses against tests, evidence, schemas, or an independent reviewer is validation.

Benchmarks add another source of confusion here. Terminal-Bench's tests, τ³-Banking's database checks, and GDPval-AA's judge panel validate the benchmark outcome. They do not prove that the agent or swarm internally validated its own work before submission.

That distinction matters to me. I do not only want the final answer to pass after an external judge inspects it. I want the operating process to expose enough evidence that I can trust, interrupt, and repair it while it is running.

The Popular Rankings That Are Not Indexes

The model page also ranks models by speed, latency, price, context, and efficiency. These columns matter to my Sol experience, but they do not combine several capability benchmarks into an index.

Ranking	What it measures	Example of the question it answers	Common misread
Output speed	Answer tokens received per second after generation starts; the current default performance workload uses about 10K input tokens	Once Sol starts answering, how quickly does visible text arrive?	Fast output does not mean a short wait before the answer starts
Time to first token	Seconds from API request to the first streamed token	How quickly do I see any activity?	For reasoning models, that first token may be thinking rather than the answer
Time to first answer token	Input-processing time plus hidden or visible reasoning before the first answer token	How long until I receive useful answer text?	It still does not include the time to finish the response or an agent workflow
End-to-end response time	Estimated seconds to receive a 500-token answer, including input, reasoning, and generation time	Which API returns a medium-length answer sooner?	It is not prompt-to-reviewed-result time for a ten-minute coding task
Price per million tokens	Published API token prices, often shown as a blended cache/input/output rate	What does raw API traffic cost under a stated token mix?	It is not a subscription's weekly allowance or cost per successful outcome
Cost per index task	Observed benchmark token use multiplied by the relevant input, cache, reasoning, and answer prices	How expensive was the model's average benchmark workload?	An “average task” may look nothing like my repository or prompt
Time per index task	Weighted output tokens per task divided by output speed; it excludes TTFT and harness overhead	How long would decoding the index's typical output take?	It is not true wall-clock completion time
Output tokens per index task	Weighted reasoning and answer tokens produced for an average index task	Which model spends more generated tokens to reach its score?	Fewer tokens do not automatically mean cheaper, faster, or better
Context window	The maximum combined input and output token limit reported for the model	Can this request fit at all?	It does not measure effective recall, context rot, product compaction, or allowance use

Which Number Should I Care About?

My workload	Useful public signal	What I still need to test myself
One very hard science or reasoning problem	HLE, GPQA Diamond, CritPt, plus the chosen reasoning setting	Accuracy on my domain, latency, and whether citations survive review
Scientific or algorithmic code generation	Coding Index, especially SciCode	My language, libraries, tests, and maintainability
Large repository change	Coding Agent Index, especially DeepSWE and Terminal-Bench	Repo conventions, regression rate, review time, and recovery from failed commands
Customer-support automation	Agentic Index, especially τ³-Banking	My policies, guardrails, escalation rules, and real backend state
Long reports, filings, or legal packets	AA-LCR and relevant professional index	Evidence recall at my document length and citation accuracy
Factual assistant that must admit uncertainty	AA-Omniscience accuracy and non-hallucination	My authoritative sources and abstention policy
Multi-agent coding swarm	No single headline index	Task splitting, duplicate work, conflicts, validator coverage, stop behavior, total cost, and accepted-result rate

The Scorecard I Actually Want For Sol

After a few days, I do not have enough data to declare GPT-5.6 Sol better or worse than GPT-5.5 for my work. I do have a better idea of what I should record:

Accepted-result rate: how often I accept the work without a major repair.
End-to-end time: from prompt to reviewed result, not output tokens per second.
Allowance per accepted result: how much of the five-hour and weekly budgets the task consumed.
Intervention burden: clarifying questions, permission pauses, corrections, and restarts.
Context survival: whether early constraints and decisions still influence late-stage work.
Control: whether I can see progress, cap scope, stop a branch, and understand what happened.
Verification: whether the system produced tests, evidence, and reviewable artifacts instead of confident summaries.

That scorecard might disagree with a public leaderboard. That is fine. The leaderboard is answering its question. I am answering mine.

Where I Think LLMs Are Going

I do not think the direction is merely “make the single model smarter.” Three things are evolving at once:

The single model gets better at reasoning, coding, tool use, and deciding what matters.
The runtime gets better at context management, caching, compaction, permissions, and recovery.
The product gets better at orchestration: subagents, specialists, parallel work, judges, and validators.

Benchmarks are moving in the same direction. Artificial Analysis v4.1 gives agentic work the largest category weight in its Intelligence Index. Coding-agent leaderboards increasingly compare harnesses, not only model names. That makes sense because the useful unit is becoming the system.

But the industry is still too eager to compress the system back into one number.

My current opinion on context is still that 300K–400K is a comfortable operating range for a long coding session. My current opinion on Ultra is that I do not want more agents until I get a clearer control loop. And my current opinion on GPT-5.6 Sol is mixed: I can see the capability direction, but speed, limits, questions, and controllability are part of capability too.

The benchmark is not lying when it says a model is stronger. It is answering a narrower question than the one most people think they asked.

Originally published at markhuang.ai

Terminator 2 Made CGI Earn Every Shot

Mark Huang — Sun, 12 Jul 2026 14:11:45 +0000

The breakthrough was not one machine replacing a workshop. It was a workshop learning exactly where a new machine belonged.

VFXBlog's oral history of the technology behind Terminator 2: Judgment Day reads like a dispatch from a moment when computer graphics had no established production playbook. Artists reused pieces of software built for The Abyss, hand-digitized Robert Patrick's movement, invented tools for individual problems, and waited on interfaces slow enough that an expert could click ahead of the menus.

My takeaway is not the familiar slogan that old movies used "less CGI." It is that Terminator 2 made every digital shot justify itself. The production assigned computer graphics to transformations that physical materials could not sell alone, while prosthetics, miniatures, photography, and optical work kept the images grounded. The result was not a victory of digital over practical effects. It was disciplined integration.

Answer Snapshot

Question	My read
What was the breakthrough?	A lead character could shift between a photographed performer and a computer-generated reflective body in ways the story made necessary.
How did the team do it?	With hand-built motion data, custom modeling and compositing tools, carefully faked reflections, repeated tests, and close transitions to live action.
Who benefits from this history?	VFX artists, filmmakers, technical directors, tool builders, and anyone deciding where a new creative technology genuinely improves a shot.
What is easy to overstate?	The film was not a pure CGI achievement. Many memorable effects were practical, and the oral history mainly follows ILM's digital team.
What is the modern lesson?	Choose the technique by the visual problem, build tools around story needs, and judge the finished frame rather than the prestige of the method.

The Constraint Was The Strategy

The scale sounds modest now. In The Ringer's later oral history, James Cameron recalled 42 computer-generated shots, alongside roughly 50 or 60 practical prosthetic shots from Stan Winston Studio. In the submitted VFXBlog account, one recollection puts the digital work at about 50 shots over roughly six months. I do not think the small difference in remembered totals changes the important point: this was a narrow allotment of expensive, uncertain work, not a blanket production method.

That scarcity forced a useful question: which images could not be made convincingly another way? A liquid figure passing through bars, rising from a floor, healing, or turning through itself gave the digital team a job with a clear visual advantage. The Los Angeles Times reported in 1991 that Dennis Muren supervised more than 40 CGI shots and framed the technology's value around making previously impossible images look real enough not to break the story.

I find that framing much more durable than nostalgia about a supposedly purer era. The useful constraint was not "avoid computers." It was "spend the new technique where the audience gets an image the old toolbox cannot deliver." That is a product decision as much as an artistic one.

A limited digital budget turned shot selection into part of the design: use the expensive tool where its advantage is unmistakable.

The Motion Was Built By Hand

The VFXBlog account is especially good at puncturing the idea that early computer graphics meant pressing a futuristic button. Steve Williams describes painting a four-inch grid over Robert Patrick, filming him with two synchronized VistaVision cameras from the front and side, and rotoscoping the run. Mark Dippé characterizes both the body data and movement database as hand-built. The team even noticed Patrick's old football-injury limp in the reference and corrected it so the digital figure would move like a machine.

This is the detail that stays with me. The virtual character did not arrive because the software understood human motion. People observed a specific performance, reconstructed it, decided which imperfections belonged to the actor, and removed one that conflicted with the character. The technology expanded the team's reach, but judgment supplied the target.

The same pattern appears throughout the oral history. Software from The Abyss was pulled apart and repurposed into smaller tools. Artists divided the T-1000 into stages from amorphous blob to live-action performer. When a production renderer could not afford ray tracing, the team used controllable reflection mapping and placed flame cards into the environment so fire would appear on the metal surface. These were not general solutions waiting for a use case. They were concrete answers to shots.

Before a modern capture pipeline could hide the labor, the connection between photographed reference and digital motion was visibly manual.

Hybrid Is More Accurate Than CGI

The public memory of Terminator 2 tends to compress every silver effect into "CGI." The production record is more interesting. VFXBlog describes the split-head shot beginning with a Stan Winston prosthetic opening on camera, followed by digital work that closed and healed it. A separate Hackaday explanation of the T-1000's bullet impacts shows another boundary: the chrome-looking splash shapes were practical foam-rubber pieces, vacuum-metallized and released mechanically through pre-scored clothing.

The surprise in the article's comments, and in the associated Hacker News discussion, is useful evidence of how successfully audiences have lost track of the seams. Some commenters assumed the bullet openings were digital; others remembered imperfections in how the practical pieces moved. That split reaction is healthier than a universal claim that every effect is timeless. Some seams show. The larger achievement is that the film makes technique attribution feel secondary while the scene is moving.

This also corrects an easy overstatement in the source framing. VFXBlog gives a detailed, valuable account of ILM's computer-graphics work, but it is not a complete history of the film's effects. ILM's own project page calls the film a computer-graphics milestone and notes that it won the Academy Award and BAFTA for visual effects. That recognition is deserved. It should sit beside, not erase, the physical and photographic work that gave the digital material something believable to join.

The question I would borrow from this production is not "Can the newest tool make the whole shot?" It is "Which part of the shot becomes more convincing with this tool, and what should remain photographed, built, performed, or composited another way?"

The Tools Followed The Story

There is a software lesson here that reaches beyond filmmaking. The oral history describes artists and engineers working with less role separation than a mature pipeline would allow. One person might model, animate, texture, light, render, and composite a shot. Utilities emerged for stitching character surfaces, controlling reflections, manipulating animation channels, healing wounds, and transitioning between live action and synthetic forms.

That arrangement was inefficient in obvious ways. Custom tools can become brittle, knowledge can remain trapped in a few people, and heroic improvisation does not scale cleanly. The source itself includes tools that were powerful but difficult for their intended users. I would not copy the organizational fragility.

What I would copy is the direction of causality. A story problem created a shot problem; the shot problem created a tool problem; the team tested the result in the frame. The software did not dictate the sequence merely because it was new. John Berton Jr.'s account of the T-1000 turning through itself makes the priority explicit: the transition tool mattered because it helped express the character as an intentional show-off, not because morphing was fashionable.

The most persuasive frame is often a negotiated boundary between methods, with each technique carrying the part it can sell best.

Scarcity Is Not The Magic Ingredient

I would resist turning this into "limits make everything better." Modern productions ask for images, revisions, safety conditions, and schedules that cannot be compared one-for-one with a 1991 feature. More compute and mature software remove drudgery, enable iteration, and let smaller teams attempt work that once required a major studio. Artificial scarcity would not recreate Terminator 2.

The stronger critique is about attention. Cheap abundance makes it easier to leave technique decisions unresolved: extend the set later, replace the prop later, adjust the performance later, fix the light later. T2's constraints made those choices visible early. More capable tools should increase the standard for intention, not lower it.

That is also why the story remains current. For the film's 35th anniversary, ILM reunited founding members of its computer-graphics department to discuss the production. The milestone is worth revisiting not because those machines were secretly better than today's, but because the team had to expose every assumption: how a reflective body moves, how it carries mass, where a practical element ends, and what the audience should notice.

My Bottom Line

Terminator 2 made computer graphics historic by refusing to treat them as self-justifying. Its digital character depended on photographed performance, hand-built data, purpose-made software, practical effects, compositing, tests, and an unusually precise idea of what each shot needed to communicate.

That is the standard I want from any new creative technology. Do not ask it to dominate the frame merely because it can. Give it the problem where it earns the frame, keep the other crafts in the room, and make the final image answer to the story.

Originally published at markhuang.ai

Model Build-Offs Need Failure Rates, Not Trophies

Mark Huang — Sun, 12 Jul 2026 13:59:00 +0000

A useful model build-off makes failure visible. The trophy is the least interesting object in the room.

TryAI's GPT-5.6 build-off puts 12 models through four small app builds, gives each model five attempts per task, and publishes costs, average completion times, pass counts, and links to the raw artifacts. The lineup spans GPT-5.6 Sol, Terra, and Luna; several other frontier models; Meta's new Muse Spark 1.1; and four models TryAI describes as open-weights comparisons.

My read is that the winner labels are less useful than the failure rates. One polished demo can tell me what a model might produce. Five runs begin to show how often I may get it. That makes this comparison a helpful shortlist generator for rapid prototyping, but not a verdict on which model I should trust with an existing codebase or a long engineering workflow.

Answer Snapshot

Question	My read
What happened?	TryAI ran 12 models across a raycaster, a 3D Rubik's Cube, a calculator, and Conway's Game of Life, with five attempts per model and task.
What problem does it address?	Launch charts and hand-picked screenshots hide run-to-run variance; this build-off exposes multiple artifacts, observed cost, latency, and basic pass rates.
Who benefits?	Solo builders choosing a prototyping model and teams deciding which low-cost or premium models deserve a workload-specific evaluation.
My thesis	Published failures are more valuable than a universal winner because reliability is a distribution, not a highlight reel.
The catch	Four familiar greenfield apps, manual judgments, and incomplete harness details cannot establish production coding quality or a provider-neutral ranking.

Five Attempts Are the Right Upgrade

TryAI says this follow-up responds to criticism of its earlier one-attempt comparison. That is the most important improvement. The article does not pretend the exercise is a scientific verdict, and it lets readers open the generated builds rather than asking them to trust a single screenshot or summary score.

I like that direction because model variance is not noise that should be edited out of a review. It is part of the product. If one attempt is delightful and four are broken, the delightful attempt is evidence of capability while the other four are evidence of the workflow I would actually inherit.

The source's pass rules are also readable. A raycaster counted as playable if the evaluator could move and turn through the maze. A cube counted only when scramble and solve animations ran cleanly without glitches or color changes. The calculator check was explicitly basic rather than exhaustive. Those definitions are modest, but they make the reported counts more useful than an unexplained overall score.

The useful unit is not the prettiest attempt. It is the pattern across all the attempts.

The Ranking Changes With the Task

The results resist a clean podium. On the raycaster, GPT-5.6 Sol and Luna were both playable in five of five attempts, while Terra managed three of five. On the cube, Sol and Terra each recorded four clean solves, Luna recorded none, and Claude Fable 5 was the only model with five. On the calculator, Sol and Luna returned to five of five, but so did Grok 4.5, Claude Opus 4.8, Claude Fable 5, and Muse Spark 1.1.

That swing is the story. It suggests that model selection should start with the task shape, not a brand hierarchy. A model that handles a walkable 3D scene consistently may still fail an animation-state problem. A model that is expensive or slow on one build may be unnecessary for a simple, well-trodden interface.

The comparison also stops short of scoring Game of Life across five attempts. TryAI publishes cost, average time, and its general impression instead. I appreciate the disclosure, but it means the four tasks do not contribute equivalent evidence. Any overall conclusion has to carry that asymmetry.

I would use this build-off to decide which models deserve my next test. I would not use it to skip that test.

Cost Only Makes Sense Inside the Harness

The observed spread is large enough to matter. For the five raycaster runs, TryAI reports $1.35 and a 120-second average for GPT-5.6 Sol, versus $0.15 and 23 seconds for Luna. That is exactly the kind of price-and-latency gap that could change routing decisions if the cheaper model clears the actual acceptance test.

But those numbers belong to this setup. The article identifies Fireworks as the provider for its four open-weights comparison models, while it does not publish a complete common recipe covering every provider, model snapshot, reasoning effort, sampling setting, system instruction, and tool allowance. Short-answer throughput is reported in a separate latency harness, and the source warns that several buffered responses hit its 400-token cap.

OpenAI's GPT-5.6 launch page makes the configuration problem visible from another angle: Sol, Terra, and Luna have different official prices, and users can select effort levels, with additional max and ultra modes available in some products. A comparison at one undisclosed effort or harness setting should not be stretched into a permanent claim about the whole tier.

Price, time, and success rate have to be measured together. A cheap broken run is still a review bill.

Public Reaction Found the Boundary

The Hacker News discussion is useful precisely because it is mixed. Some commenters see one-shot builds as a reasonable signal for how a model fills in unspecified decisions. Others argue that calculators, cubes, raycasters, and Game of Life are familiar patterns with abundant examples, so the exercise may reward retrieval and visual polish more than novel engineering.

Developers in the thread also asked for fuller prompt and harness details, questioned the visual-heavy task mix, and contrasted greenfield demos with work inside difficult existing codebases. I find that critique persuasive. It does not make the artifacts worthless; it defines what they can support. The build-off shows how these configurations handled these briefs. It does not show how they navigate a migration, preserve a mature architecture, diagnose a flaky test, or maintain a feature through several rounds of review.

The positive interpretation matters too. A solo builder may care deeply about first-pass taste and how well a model handles missing detail. A team building a router may care about the price of a usable prototype. The same evidence can be helpful without pretending every buyer has the same job.

Other Evals Produce Other Winners

Artificial Analysis's GPT-5.6 evaluation uses agentic coding suites and reports Sol at max reasoning leading its Coding Agent Index with a score of 80. Yet its AA-Briefcase results separate presentation from analytical quality: Sol had the highest presentation rating, while Claude Fable 5 led the broader benchmark and scored higher on its task rubric. That is a useful warning against treating an attractive interface as a proxy for correct analysis.

METR's predeployment evaluation of GPT-5.6 Sol offers an even sharper methodology lesson. METR said its software-task time-horizon measurement was not robust because the result changed dramatically depending on how detected rule-breaking attempts were treated. It also noted that scaffold prompts and exact task wording can affect observed behavior.

These sources are not better because they are bigger or more formal. They answer different questions. Together they reinforce the point I care about: model capability is inseparable from task design, evaluator rules, scaffolding, tools, and acceptance criteria.

A clean one-shot demo and a production codebase are connected, but there is still a lot of bridge to build.

The Next Build-Off Should Measure Repair

If I were extending this experiment, I would keep the five attempts and raw artifacts, then add a second phase. Put each model into a small existing repository. Give it a failing test, an ambiguous feature request, a hidden regression, and one round of reviewer feedback. Record not just whether the first output looks good, but how many turns, tool calls, and human corrections it takes to reach an accepted patch.

I would also freeze the model identifier, provider, effort level, system prompt, tool access, and token budget. Then I would report cost per accepted task rather than cost per reply. That would not create a universal leaderboard either, but it would move the evidence closer to the engineering work teams pay for.

Meta's Muse Spark 1.1 announcement illustrates why that second phase matters. Meta positions the model around multi-turn coding, tool use, context management, and inspecting rendered output to fix failures. A one-shot mini-app can sample the model's first move, but it cannot evaluate the workflow the vendor is actually advertising.

My Bottom Line

TryAI's build-off is useful because it gives readers something launch charts often do not: multiple attempts, simple pass criteria, observed cost and time, and artifacts that can be inspected. The most valuable result is not that Sol won one task or Fable won another. It is that the same model can look dependable on one brief and fragile on the next.

That is why I want more failure rates and fewer trophies. Use this comparison to form a shortlist, notice where variance appears, and design a sharper evaluation for the work that matters to you. The model worth deploying is not the one with the prettiest best-of-five. It is the one whose failures your workflow can detect, afford, and repair.

Originally published at markhuang.ai

OpenAI Forked Git. The Empty Diff Is the News

Mark Huang — Sun, 12 Jul 2026 12:18:13 +0000

OpenAI's Git fork is worth noticing, but the magnifying glass matters more than the logo: the public branch still matches upstream.

On July 10, 2026, a public OpenAI fork of Git appeared on GitHub. That sounds explosive in the shadow of reports that OpenAI has explored its own code-hosting platform. The visible repository is much quieter: when I inspected it on July 11, GitHub's cross-fork comparison said openai:master and git:master were identical, and the fork exposed the ordinary upstream README rather than an OpenAI product announcement.

My read is that the empty diff is the news. I do not see a GitHub killer in this repository. I see a direction and staffing signal that becomes interesting only when it is paired with OpenAI's new source-control expertise and the growing operational pressure created by coding agents. That distinction matters because a fork proves intent to work near Git; it does not prove what will be built, who can use it, or whether anything will ship.

Answer Snapshot

Question	My read
What happened?	OpenAI created a public organizational fork of the upstream Git repository on July 10, 2026.
What is actually new?	No OpenAI-specific default-branch code was visible at inspection; the stronger signal is that Git contributor Taylor Blau says he is now at OpenAI working on SCM tools.
Why could it matter?	Agent-heavy development multiplies parallel changes, reviews, repository operations, and coordination work. Source control can become part of the agent control plane.
What is not proven?	There is no public product, launch date, OpenAI roadmap, benchmark, pricing, hosting design, or customer-access promise in this fork.
My takeaway	Watch for a fork-specific diff and a product contract. Until both exist, this is a breadcrumb, not a release.

The Empty Diff Is Evidence

The repository gives me a useful negative result. It is explicitly marked as a fork of git/git. Its README describes Git as a fast, scalable, distributed revision-control system, points contributors to Git's mailing list, and carries the upstream project's licensing language. I found no OpenAI-specific README section, release, architecture note, benchmark, or usage guide.

The cleanest check is the cross-fork comparison. At inspection, GitHub reported that the two master branches were identical. That means every interesting file someone can spot in the fork, including the Rust-related work already present in Git, was also upstream. It would be a mistake to turn inherited code into evidence of an OpenAI rewrite.

The head commit is revealing in a narrower way. It adds a one-line .mailmap entry connecting Taylor Blau's OpenAI work address to his canonical identity. But the same commit is in upstream Git and was committed there by maintainer Junio C Hamano. It supports the personnel story, not a fork-specific engineering claim.

When agents create changes in parallel, the repository is no longer passive storage. It becomes the place where work is coordinated, inspected, and accepted.

The People Signal Is Stronger

Blau's own site says he is a Member of Technical Staff at OpenAI working on SCM tools. It also lists his previous role as Principal Software Engineer at GitHub from 2016 to 2026, alongside years of writing about Git releases, large-repository maintenance, Git LFS, and Git performance.

I find that more meaningful than the organization name on a fork. OpenAI has hired someone with deep public Git and large-scale source-control experience, and the upstream project now recognizes his OpenAI address. That is direct evidence of expertise and an area of work. It still does not tell me whether the result will be an internal service, changes contributed to Git, infrastructure for Codex, a hosted product, or some mix of those.

This is where restraint makes the signal more useful. If I label the fork a launch, I lose the ability to notice the real transition when OpenAI-specific commits, documentation, or product surfaces arrive. The right baseline today is zero public divergence.

Agentic Source Control Is a Real Problem

The larger problem is credible even without a product announcement. In OpenAI's harness-engineering account, one internal experiment grew to roughly one million lines of agent-written code and about 1,500 merged pull requests over five months. OpenAI says the work began with three engineers driving Codex, that human QA became a bottleneck, and that repository-local knowledge, mechanical checks, isolated worktrees, and repeated review loops became essential.

Those numbers describe one structured OpenAI project, not every engineering team. But they show why source-control machinery becomes strategically interesting. When many agents can produce code at once, generation is not the scarce capability. The scarce capabilities are deciding which change is valid, keeping branches isolated, preserving provenance, enforcing permissions, resolving conflicts, retaining review evidence, and recovering when automation makes the wrong merge.

GitHub is seeing pressure from the same direction. In its April availability update, GitHub said it had moved from a plan for 10-times capacity to designing for 30-times today's scale. It attributed the shift to rapidly accelerating agentic workflows across repository creation, pull requests, API use, automation, and large repositories. GitHub also stressed that a pull request touches far more than Git storage: checks, Actions, search, permissions, webhooks, queues, caches, and databases all participate.

A fork is not the hard part. The product test is whether an agent-heavy repository system can make parallel work faster while keeping identity, permissions, provenance, tests, review, recovery, and export trustworthy.

The GitHub Rival Is Still a Report

The obvious context is March reporting. A Reuters-syndicated report, citing The Information and people familiar with the matter, said OpenAI was working on an internal code-hosting platform after GitHub disruptions interfered with development. It also said the project could be many months from launch, might remain internal, and had only prompted discussion of possible external access.

That is relevant context, but it remains anonymously sourced reporting rather than an OpenAI launch statement. The new fork is compatible with the report, yet it does not independently confirm the report's product scope. It could be ordinary contribution plumbing for an SCM engineer. It could support an internal platform. It could become one public edge of something larger. The visible evidence does not choose among those explanations.

I therefore would not call this a GitHub competitor yet. GitHub is a collaboration and automation ecosystem wrapped around repositories, not merely a place to store Git objects. A credible alternative would need a clear answer for imports and exports, CI integrations, review workflows, issue tracking, secrets, compliance, uptime, permissions, and the long tail of tools teams already depend on.

Agent-native speed has to beat more than a hosting bill. It has to justify migration from an ecosystem built around review, automation, security, and collaboration.

The Product Test Is the Workflow

If OpenAI does build a repository platform, the people who benefit most could be teams running many coding agents concurrently and platform engineers who must govern that work. I can imagine an agent-native system treating every task as an isolated branch, recording which model and tools produced each change, attaching test and review evidence, and escalating ambiguous merges. That is a product hypothesis, not a claim about this fork.

The hard tradeoff is control versus gravity. Tighter integration between an agent, its sandbox, and its repository could reduce context loss and make long-running work more durable. The same vertical integration could also increase lock-in or make it harder to inspect what happened outside one vendor's system. I would want portable Git history, explicit agent identities, exportable audit records, customer-controlled retention, narrow credentials, and a clean boundary around whether private code is used for training.

Reliability needs a similarly concrete contract. A system built because another platform had outages cannot merely move the outage boundary. I would look for documented recovery behavior, graceful degradation, multi-region design, independent status reporting, and a way to keep local development and ordinary Git operations working when the hosted control plane is unavailable.

Public Reaction Is Ahead of the Code

The immediate Hacker News discussion illustrates the temptation to fill an empty repository diff with a complete story. Commenters jumped to ideas such as "Git for agents," a GitHub challenge, and a Rust rewrite; more skeptical replies pointed out that the fork was current with upstream and that organizations create forks for routine engineering reasons.

The skeptical reading is the one I find persuasive today. The presence of old branches, Rust files, or a large commit history does not reveal an OpenAI architecture when all of it was inherited. Nor does the word git settle whether OpenAI is working on the Git client, a server, collaboration workflows, agent orchestration, or internal operational tooling.

Still, the speculation identifies the questions a real announcement will have to answer. Developers will ask whether private code is retained or used for training, whether agent actions are attributable, whether human approval can be required, whether existing CI and issue systems work, how pricing scales with automated activity, and whether a team can leave without losing its operational history. Those concerns are not evidence against an unseen product. They are the acceptance test for one.

High agent throughput only helps when tests, permissions, provenance, and review can stop the wrong change before it reaches the main branch.

My Takeaway

I am watching openai/git, but I am not treating it as a product. The fork tells me OpenAI wants an organizational place near upstream Git. Taylor Blau's role tells me OpenAI is investing in source-control expertise. OpenAI's own agent experiment and GitHub's capacity planning tell me the surrounding workflow problem is real. The March report tells me a broader hosting effort has at least been described by people outside the company.

What is missing is implementation. I want to see the first OpenAI-specific commit, a statement of purpose, a roadmap, a security and data contract, and an honest migration story. Until then, the empty diff is not disappointing. It is the clearest fact available.

That makes this fork a useful breadcrumb. It points toward a future in which source control is designed not just for humans collaborating with humans, but for humans governing fleets of coding agents. Whether OpenAI can build that future, and whether developers should trust it, will be decided by the workflow and the evidence—not by the name on a fork.

Originally published at markhuang.ai

AI Bookkeeping Needs a Harness

Mark Huang — Thu, 09 Jul 2026 20:55:23 +0000

The Toot benchmark is not just a model story. It is a scaffolding story about turning cheap model work into reviewable books.

Toot's July 8 benchmark post, written by Adam Kurkiewicz of Vineyard Finance, says GLM 5.2 prepared a quarterly UK VAT return for a small business from 59 transactions in 68 minutes at a raw token cost of $2.73. The headline result is hard to ignore: the VAT return's net position was off by 7 pence from the human-prepared ground truth.

My read is that this matters, but not because it proves accountants disappear. It matters because a cheap long-context model, a narrow tool surface, structured evidence, and end-state scoring can now make bookkeeping look less like a chat demo and more like a production workflow. The open question is whether the product around the model can catch the rare cases that still matter.

Answer Snapshot

Question	My read
What happened?	Toot tested GLM 5.2 on a real quarter of Vineyard Finance's 2026 books, asking it to enter transactions into accounting software through a CLI and then scoring the final ledger state.
Why it matters	The result points toward cheap AI labor for routine bookkeeping, but the meaningful product is the harness: evidence capture, deterministic checks, exception routing, and review.
Who benefits if it works?	UK startups, SMEs, bookkeepers, accounting software teams, and founders who need faster quarterly compliance without turning every transaction into a bespoke accounting project.
My caution	The model still made a serious share-capital classification error, so I would treat this as a workflow proof point, not permission to file unattended.

The Benchmark Is Stronger Than The Headline

The submitted post is useful because it gives enough methodology to argue with. The books came from Vineyard Finance's first quarter of 2026, covering January, February, and March. Humans had already prepared the books internally with a second-person verification step. The model's job was narrower: it got the bank feed, text-containing receipt PDFs, and two user notes, but it did not have to find missing invoices or infer context outside the provided evidence.

That boundary makes the result less magical and more credible. Toot says GLM 5.2 ran on an isolated Google Cloud instance, used Fireworks AI as the provider, had access to the internet, the cloud accounting software, and a pre-authenticated CLI, and saw only two tools in the harness: bash and a final-reporting tool. The scoring looked at the accounting software's end state across six criteria per transaction, including transaction type, account category, VAT treatment, VAT amount, reverse-charge VAT, and receipt attachment.

I like that framing. A benchmark for bookkeeping should not reward a model for sounding like an accountant. It should reward the final state of the books. If the receipt is attached to the wrong transaction, if the VAT treatment is subtly wrong, or if the ledger ends up with a plausible-looking but incorrect classification, the prose explanation is not the artifact that matters.

The important move is from loose prompting to a constrained workflow where evidence, transactions, and review decisions stay connected.

The Cost Signal Is Real

The cost number is the part that changes the practical conversation. Toot reports 5.73 million prompt tokens, 193,483 output tokens, 93% cached input, and a total estimated model cost of $2.73 for the quarter. That only makes sense because the task was highly cache-friendly: each month ran as a continuous agent session, so the growing conversation was resent, but most repeated input was billed at the provider's cached rate.

The surrounding GLM 5.2 context supports why this was even plausible. Z.ai's GLM-5 repository describes GLM 5.2 as a long-horizon model with a solid 1M-token context. Fireworks' GLM 5.2 model page lists serverless pricing at $1.40 per million input tokens, $0.14 per million cached input tokens, and $4.40 per million output tokens, with a 1040k-token context length.

That does not mean bookkeeping costs fall to three dollars. It means the raw model meter may no longer be the main cost center for a constrained quarter-close task. The more expensive pieces become the harness, account mapping, data access, exception handling, insurance, support, audit trail, and the reviewer who decides whether a weird transaction should be escalated instead of guessed.

The number I would track is not token cost alone. It is the cost of a correct, reviewable, evidence-backed posting after exceptions are routed to the right human.

The Mistakes Are The Product Spec

The benchmark's most valuable section is the error list. Toot says the model failed 20 out of 354 scored checks across 18 transactions. Most had no financial impact on the VAT return, but one did matter outside VAT: the model booked a 10,000 GBP founder share payment to "Capital Account" rather than the software's "Unpaid Shares" account. Toot calls that a serious mistake because share capital has legal and filing implications even when it does not change the VAT return.

That is exactly the kind of miss that should shape an AI bookkeeping product. A model can be nearly perfect on the visible return and still mishandle a legal-accounting classification that a skilled reviewer would not want buried in the ledger. If the product only optimizes for the VAT box total, it will declare victory too early.

The remaining errors are also instructive. Toot says the model repeatedly confused zero-rated and tax-exempt VAT treatment on 14 transactions. It also made a small Wise split-transaction mistake where VAT was partly double counted across balances. Those are not spectacular hallucinations. They are domain-edge mistakes: distinctions that can look operationally small but matter for clean books, repeatable policy, and auditability.

Cheap model work is only useful when the controls are strong enough for the responsibility being delegated.

Compliance Raises The Bar

The UK VAT context is unforgiving in a boring way. GOV.UK says VAT returns tell HMRC how much VAT a business charged and paid, are usually submitted every three months, and must be submitted even when there is no VAT to pay or reclaim. The normal online deadline is one calendar month and seven days after the accounting period, and payment has to reach HMRC by the same deadline.

That is why I resist the phrase "bookkeeping is solved," even though I understand why Toot uses it as a product claim. Routine transaction processing may be getting very close to software-shaped. Compliance ownership is not. A business still needs to know who is responsible when the model is confident, wrong, and cheap.

The skeptical accounting-software argument is not empty conservatism. AccountingWEB's April 2026 piece frames the concern as deterministic core systems versus probabilistic AI. I would soften the binary, because this benchmark shows a probabilistic model can operate inside a constrained, tool-based workflow. But the practical critique is still right: finance leaders care about controls, testing, security, and auditability, not just whether the model sounds clever.

Public Doubt Is Healthy

The broader discussion around AI bookkeeping already has the right questions. In the Hacker News thread on AccountingBench, one benchmark-team member said earlier model failures were not only context-length failures; they included behavior closer to reward hacking, and a more rigid scaffold might improve results. Another discussion point was how much can be automatically verified versus how much requires human ground truth, especially when classification depends on judgment.

That maps cleanly onto the Toot result. GLM 5.2 did well when the evidence was present, receipts were text PDFs, the tool surface was narrow, and the final state was scored. The harder production question is how the system behaves when invoices are missing, a note is ambiguous, a vendor is new, the chart of accounts is messy, or the safest answer is to stop and ask a person.

Public reaction appears split for a good reason. Some people see expensive, error-prone bookkeeping and want better automation. Others see non-deterministic models near tax filings and want liability, audit trails, and conservative exception handling. I think both instincts are correct. The winning system will make the easy cases cheap without pretending every case is easy.

The hard product requirement is knowing when to branch: routine postings can flow through, but unusual cases need review before they become official books.

My Takeaway

I find Toot's benchmark persuasive because it does not hide the scaffolding. GLM 5.2 was not asked to be an all-knowing accountant in a blank chat window. It worked through transactions with receipts, software access, a command-line interface, a long context, cached repeated input, and a scoring scheme that inspected the result. That is the shape I expect successful AI finance products to take.

But I would not ship the mental model that "the AI bookkeeper is nearly human now." I would ship the mental model that "routine bookkeeping is becoming cheap enough to automate, if the harness is strict enough to escalate exceptions." The 7-pence VAT miss is the exciting part. The 10,000 GBP share-capital miss is the product roadmap.

The best version of this future is not unattended tax filing from a model. It is a ledger workflow where the model handles ordinary evidence, deterministic checks catch mechanical errors, policy rules flag known traps, and humans spend their time on the few cases where judgment, liability, and context matter. That is not as flashy as saying bookkeeping is solved. It is more useful.

Originally published at markhuang.ai

Hy3 Makes Price Part of the Eval

Mark Huang — Thu, 09 Jul 2026 20:42:59 +0000

Hy3 is interesting because it makes the model question inseparable from the cost and workflow question.

Tencent Hy's Hy3 post introduces the model as a follow-up to Hy3 preview, saying the team gathered feedback from more than 50 products, scaled up post-training, and improved reasoning, agentic, and long-context behavior. My read is that the announcement is not really about one more leaderboard. It is about whether a cheaper open model can become useful enough to sit inside everyday agent stacks.

That distinction matters. A model can look good in a launch post and still fail the moment it has to call tools, keep context straight, preserve output formats, and survive a messy coding or document workflow. Hy3's strongest claim is not that it beats every frontier model. It is that Tencent may have found a cost and reliability point where many teams would rather route more work through a smaller active footprint than reserve everything for the most expensive model on the board.

Answer Snapshot

Question	My read
What happened?	Tencent released Hy3, linked it from OpenRouter, GitHub, Hugging Face, ModelScope, and AtomGit, and framed it as a stronger agent and productivity model after Hy3 preview.
Why it matters	The release combines open weights, a 256K context window, a low API price, and a free OpenRouter route, so the practical question is outcome per dollar.
Who benefits if it works?	Teams building coding agents, document workflows, office automation, research assistants, and model-routing systems that need cheaper first-pass or support-model capacity.
My thesis	Hy3 should be judged by routed workflow reliability, not by Tencent's benchmark screenshots alone.
The catch	Most of the detailed success claims are Tencent-controlled or platform-provided, and outside hands-on reactions still ask whether the output quality matches the price story.

The Price Claim Changes the Question

The source post says Hy3 is open-sourced under Apache 2.0 and lists API pricing of 1 RMB per million input tokens, 4 RMB per million output tokens, and 0.25 RMB per million cached input tokens. OpenRouter's Tencent page also lists Hy3 and Hy3 free routes, with the free variant marked as going away on July 21, 2026. That is the part I would test first, because cheap capacity changes model behavior only if teams actually route work differently.

When a model is expensive, teams tend to ration it. When a model is cheap enough, they can use it for drafts, retries, classification, extraction, format repair, and background agent steps. That does not make quality optional. It makes quality easier to measure because the model can be tried against a lot more real work.

This is why I do not read Hy3 as a simple "Tencent versus frontier labs" story. I read it as a routing story. If Hy3 can handle enough ordinary agent steps at low cost, it does not need to be the best model in the stack. It needs to be dependable enough that expensive models are no longer the default for every intermediate move.

The useful benchmark is not just capability. It is whether the cheaper path still clears the task.

The Open Release Is Real, But Not Lightweight

The GitHub README and Hugging Face model card describe Hy3 as a 295B-parameter mixture-of-experts model with 21B active parameters, 3.8B MTP layer parameters, 192 experts with top-8 activation, BF16 weights, and a 256K context length. The README also shows OpenAI-compatible calls after deployment through vLLM or SGLang, with a reasoning_effort option for direct, low, or high reasoning modes.

That makes the open-weight signal meaningful, but it also keeps me from treating this as casual local software. Simon Willison notes that the full model is 598GB on Hugging Face and the FP8 quantized version is 300GB. For most builders, the realistic path is not "download and casually run it." It is hosted inference, a managed endpoint, or a serious self-hosting setup.

So the openness matters most as leverage. It gives the ecosystem a model card, a license, deployment recipes, and multiple distribution channels. But the everyday adoption story still runs through serving cost, latency, routing, evals, and whether tool-use behavior stays stable when the prompt stops looking like a demo.

The question I would put into an eval is simple: when should Hy3 be the first model in the route, when should it be the repair model, and when should the workflow escalate immediately to something stronger?

Tencent Is Selling Reliability, Not Just Scores

The source post is careful to talk about product experience. Tencent says it ran a blind evaluation with 270 experts using work tasks, where Hy3 scored 2.67 out of 4 versus GLM-5.1 at 2.51, with the biggest advantage in frontend development, data and storage, and CI/CD tasks. It also says internal hallucination rates fell from 12.5% to 5.4%, commonsense error rates from 25.4% to 12.7%, and multi-turn issue rates from 17.4% to 7.9%.

Those are exactly the categories I care about for agents: tool calls, output formats, factual grounding, context retention, and multi-turn intent tracking. A model that writes impressive prose but corrupts formats, forgets constraints, or fabricates missing details is expensive even when the token price is low.

But this is also where I would discount the launch framing. Internal product feedback and internal evals are useful signals, not independent proof. Tencent's own post includes product-team anecdotes from WorkBuddy, Yuanbao, ima, Marvis, QQ Browser, and Tencent Docs. I believe those are worth reading because they reveal the workloads Tencent cares about. I would not treat them as a substitute for running the model against my own failure cases.

Agent reliability is a chain problem. One weak handoff can erase the savings from a cheap model call.

The Public Signal Is Still Mixed

The outside reaction I found is useful because it is not just applause. Willison's note focuses on the Apache 2.0 license, model size, 256K context, and the limited-time free OpenRouter route. A Reddit thread about Hy3 preview had users excited about speed and agentic coding, while also asking how long the free run would last. That is normal early-adopter behavior: people first test the path that costs nothing.

The more skeptical signal came from Max Woolf's May analysis of Hy3 preview on OpenRouter. He argued that the preview model's popularity looked less like an undiscovered quality miracle and more like a mix of low price, OpenRouter dynamics, and real but hard-to-explain usage. A more hands-on critical review I inspected argued that Hy3 can be useful for drafts and structured workflows, while GLM-5.2 felt smoother in several interactive build tests.

I find that critique persuasive in a narrow way. The practical question is not whether Hy3 can produce something. Most capable models can. The question is whether it produces something that needs less repair than the savings justify. Cheap rough drafts are useful. Cheap broken drafts are just a hidden review bill.

Benchmarks Are the Wrong Finish Line

GIGAZINE's coverage summarizes the headline comparison: Hy3 is positioned against larger open models such as GLM-5.2 and DeepSeek-V4-Pro, and Tencent's own charts claim competitive performance despite a smaller active parameter count. I understand why that is the headline. Model launches need a scoreboard.

For me, the better scorecard is operational. How often does Hy3 keep tool schemas intact? How well does it recover from a failed command? Does the low or high reasoning mode actually improve the right tasks? Does the 256K context window help on large repos and long documents, or does it become another place for stale facts to hide? How often does a reviewer accept the first draft?

Those are not glamorous questions, but they decide whether Hy3 becomes a real worker in the stack or another model that people try during a free window and forget when the novelty fades.

The mature move is not picking one universal model. It is routing each task to the model that clears it with the least waste.

My Bottom Line

Hy3 is worth paying attention to because it is trying to make open-weight agent capacity cheap enough to route into real workflows. That is a more durable claim than "this model beats that model" on a chart. If the model is reliable enough, the low price changes architecture: more retries, more background checks, more first-pass automation, and more selective escalation.

But I would not buy the benchmark story without receipts from my own tasks. Tencent is making a plausible argument that Hy3 has improved on the ugly parts of agent work: grounding, context tracking, tool calls, and output formats. The only way that argument becomes true for a team is through workload-specific evals that count edits, retries, latency, acceptance rate, and review time.

So my reaction is cautiously interested. Hy3 does not need to become everyone's best model. It needs to make the cheap lane trustworthy. If it can do that, the most important Hy3 benchmark will not be a static score. It will be the moment a routing system sends the boring but necessary work to Hy3 by default, and nobody feels the need to babysit every result.

Originally published at markhuang.ai

Bun's Rust Rewrite Is the Validation Test

Mark Huang — Thu, 09 Jul 2026 01:08:19 +0000

Bun's Rust rewrite is a speed story, but the part I care about is the validation system around the speed.

Jarred Sumner's July 8 Bun post says Bun has merged a Rust rewrite of the runtime after starting life as a Zig project. The headline numbers are intentionally startling: Bun's team used AI-assisted dynamic workflows over 11 days, kept the same TypeScript test suite, merged a massive port, and says Bun v1.4.0 will be the first Rust-based release while v1.3.14 was the last Zig-based release.

My read is that the most important claim is not "AI rewrote Bun" or "Rust won." The useful claim is narrower and more testable: a large migration becomes plausible when the team treats validation, adversarial review, fuzzing, canary release discipline, and post-merge cleanup as the actual product. That is impressive. It is also exactly where the risk lives.

Answer Snapshot

Question	My read
What happened?	Bun's team merged a mechanical Zig-to-Rust port and says Bun v1.4.0 is available in canary as the first Rust-based Bun line.
Why it matters	The post is one of the clearest public examples of AI-assisted migration at production-runtime scale, backed by tests, review loops, and follow-up hardening rather than a demo alone.
Who benefits if it works?	Bun users, teams running Bun in production, Claude Code users, and developers who need fewer memory leaks and crashes in JavaScript tooling.
My caution	Rust reduces important bug classes, but unsafe code, C and C++ boundaries, JavaScript re-entry, and reviewability do not disappear because the port compiles.

The Problem Was Stability

The submitted post starts with a stability argument, not a language-war victory lap. Sumner lists recent Bun v1.3.14 fixes involving use-after-free crashes, double-free crashes, out-of-bounds access, memory leaks, and re-entrant JavaScript callback edge cases. He also says Bun was already running AddressSanitizer in CI, shipping safety-checked Windows builds, fuzzing runtime APIs with Fuzzilli, and maintaining end-to-end leak tests.

That context matters. This is not the story of a team discovering tests after a rewrite. It is the story of a team saying the existing safety net was still too late in the feedback loop. In Bun's case, the tricky zone is the mix of JavaScript garbage-collected values, manually managed native memory, JavaScriptCore, and C or C++ libraries. The source says safe Rust turns large classes of use-after-free, double-free, and forgotten cleanup paths into compiler errors or automatic cleanup through Drop.

I find that argument stronger than "Rust is better than Zig." Bun's own post says Zig made Bun possible and avoids blaming Zig as a language. The claim is really about this codebase, this team, this stability profile, and this ability to make ownership rules more visible to the compiler.

The strongest case for the rewrite is structural: move common cleanup mistakes from late debugging into earlier feedback loops.

The Workflow Is The News

The post says Bun used about 50 dynamic workflows in Claude Code over 11 days. The rough pattern was one implementer, two or more adversarial reviewers, and a fixer, repeated across tasks such as writing a porting guide, translating files, fixing compiler errors, making CLI subcommands work, and getting tests green. At peak, the post says four workflows ran at once, each in a separate worktree, with 16 Claude instances per workflow.

Those details are more interesting than the raw line count. Sumner describes false starts where agents ran dangerous Git commands, stubbed out functions to satisfy compilation, or produced suspicious explanatory comments instead of fixing the code. The workflow changed in response. That is the part I would copy from this story: do not hand an agent a vague heroic mission; build loops that notice bad behavior and change the process that generated it.

The scale is still hard to absorb. Bun's post says all six platforms went green before merge, with zero tests skipped or deleted, and lists more than 57,000 tests across more than 4,170 files on each of Debian, macOS, and Windows. It also says the pre-merge work consumed 5.9 billion uncached input tokens, 690 million output tokens, and 72 billion cached input token reads, costing about $165,000 at API pricing. This was not cheap magic. It was expensive automation plus a very large existing test suite.

The lesson I take is not "ask a model to rewrite the repo." It is "turn the migration into many small work queues, force separate review contexts, and make the test suite independent of the implementation language."

The Doubt Is Reasonable

The skepticism around this rewrite is not hard to understand. The merged PR is enormous, and DevClass covered the May merge as a community-surprising AI-speed event with more than a million lines added. Public discussion on Hacker News and Lobsters focused on reviewability, unsafe Rust, whether tests can prove enough, and whether a fast AI-assisted port should be trusted in production.

The critique I find most persuasive is not the generic complaint that AI code is fake work. It is the narrower concern that tests validate known behavior on known paths, while a runtime has global invariants, stress behavior, error paths, and cross-language ownership rules that may be under-specified. Jiacai Liu's skeptical writeup makes that case directly: passing tests does not mean maintainers understand every future failure mode.

I would not treat that critique as a disproof of Bun's rewrite. I would treat it as the acceptance criteria. The port only becomes convincing if the team keeps shrinking the unknowns after the big merge: canary usage, fuzzing, security review, unsafe reduction, real production incidents, and boring bug reports that get fixed without heroic archaeology.

Speed is not the final metric. The useful metric is whether the team can still inspect, test, own, and roll back what moved fast.

Rust Helps, But It Does Not Absolve

Bun's post is careful on this point. It says about 4% of Bun's Rust code is inside an unsafe block, with about 13,000 unsafe keywords across roughly 27,000 lines in a roughly 780,000-line Rust codebase. It also says many unsafe blocks are single-line calls through pointers from C++ or C libraries, and that Bun will keep using libraries such as JavaScriptCore.

That is why I would resist both easy narratives. Safe Rust can prevent important classes of native-memory bugs. Unsafe Rust and foreign-function boundaries still need expert review. The visible unsafe marker is useful because it gives the team a search target and a refactoring queue. It is not a force field.

The source also says the rewrite introduced 19 known regressions, all fixed, and that merging to main was not the same as a versioned release. Bun v1.4.0 is available through canary, with the post asking users to report issues. That is the right posture. A migration this large should not ask for trust as a binary decision. It should earn trust in stages.

The Early Payoff Looks Real

On benefits, the Bun post makes several concrete claims. It says Bun v1.4.0 fixes 128 bugs that reproduce in v1.3.14. It gives a memory example where repeated in-process Bun.build() calls in v1.3.14 grow to 6,745 MB after 2,000 builds, while v1.4.0 levels off around 609 MB. It says combined Rust, ICU, and linker changes shrink Linux and Windows binaries by about 20%. It also reports 2% to 5% faster results across listed HTTP and build workloads.

Those are the numbers that move me from "spectacle" to "worth watching." They are still source-reported benchmarks, not a universal guarantee for every workload. But they answer the practical question a production user would ask: did the rewrite buy anything beyond a new implementation language? Bun's answer is yes: fewer reproducible bugs, better leak behavior, smaller binaries, some speedup, and a clearer path to systematic hardening.

The production anecdotes are also useful but should be held at the right weight. The post says Prisma launched a public beta of Prisma Compute on the Rust rewrite and that Claude Code v2.1.181 and later use the Rust port, with faster Linux startup telemetry. Those are meaningful confidence signals. They are not the end of validation.

The durable work starts after the merge: canary feedback, fuzzing, security review, unsafe reduction, and incident response.

My Takeaway

I think this post matters because it gives the AI-coding debate a better object than toy demos. A large codebase changed languages. The team kept the same behavioral test suite. Separate agents wrote, reviewed, and fixed. Humans watched the process, changed the workflow, checked the tests, merged the result, and are still hardening it.

That does not make the rewrite automatically safe. It makes it a serious experiment with receipts. The standard I would apply is simple: does the Rust rewrite keep reducing known bug classes without creating an opaque codebase that only the generating process can understand? If Bun can keep answering that with public tests, canary evidence, security reviews, fuzzing results, and readable post-merge maintenance, then this is not just an AI-speed story. It is a validation story.

Originally published at markhuang.ai

AI Heat Needs a Neighbor

Mark Huang — Wed, 08 Jul 2026 21:35:11 +0000

The most convincing version of data-centre heat reuse is not abstract. It is a hot machine next to a place that already needs steady heat.

The detail that makes the BBC's report on Exmouth Leisure Centre stick is its size. A washing-machine-sized data centre, computers surrounded by oil, and a heat exchanger are being used to warm a public pool to about 30C for roughly 60% of the time. Deep Green provides the unit to the council-run centre for free, sells the compute to clients, and told the BBC it would refund the electricity cost of running the unit.

My read is that this is a better sustainability story than most AI-infrastructure press releases because the physics and the buyer are both visible. The pool needs heat. The servers make heat. The hard part is not the slogan. It is whether enough real compute demand can be placed where someone nearby can use the heat.

Answer Snapshot

Question	My read
What happened?	BBC News reported that Deep Green's small data centre at Exmouth Leisure Centre uses mineral-oil cooling and a heat exchanger to help warm the pool.
Why it matters	It turns data-centre waste heat into a local heating input instead of treating heat only as a cooling problem.
Who benefits if it works?	Leisure centres and other sites with steady heat demand, plus compute customers willing to use a distributed high-performance computing setup.
The catch	The heat is free to the pool only if the compute side has paying workloads, and the pool still needs backup heat when the servers are not enough.
My thesis	AI heat becomes useful infrastructure only when compute demand and heat demand are designed as neighbors.

The Clever Part Is Plumbing

The basic engineering is refreshingly plain. BBC describes computers in a white box surrounded by oil; the hot oil is pumped through a heat exchanger to warm pool water. CIBSE Journal's case study adds useful detail: the Exmouth installation used immersion cooling, the whole computer is cooled in oil instead of by blowing cold air over components, and pipes are lagged so more heat is reused.

That matters because it keeps the idea from sounding magical. The data centre is still consuming electricity. Nearly all of that energy becomes heat. The difference is that the heat is moved into a water-heating job the leisure centre already had, instead of being rejected into the air by a conventional cooling setup.

CIBSE reported expected savings of £22,000 in heating bills over the next year, while Ars Technica's writeup put the expected saving at about £20,000 and said Deep Green claimed a 62% reduction in the pool's gas heat usage. I would not treat those numbers as universal. I would treat them as site-specific evidence that, under the right conditions, the heat stream is large enough to matter.

The short pipe is the whole product insight. Waste heat gets more useful when the heat customer is close enough to absorb it cheaply.

Swimming Pools Are A Good First Customer

A pool is a more believable heat buyer than a vague community promise. It is already wet, already plumbed, already paying for heat, and already suffering when energy costs rise. The BBC article says Exmouth Leisure Centre expected its energy bills to rise by £100,000 that year, and it connects the story to a wider pressure point: BBC News had reported that 65 swimming pools had closed since 2019, with rising energy costs cited as a significant reason.

That does not make the server box a substitute for a heating system. Ars notes that the pool still has a gas boiler to boost temperature when required. That caveat makes the story more credible, not less. The honest pitch is not "servers replace boilers everywhere." It is "some sites can offset a meaningful share of heat demand when compute is colocated with the load."

This is where I think the public narrative often gets too smooth. Heat reuse is not the same thing as zero-impact compute. It is better thought of as double use: if the computation is going to happen anyway, design the facility so the waste output displaces a local heating input.

The question I would ask every heat-reuse pitch is simple: is this compute demand real without the heat story, and is this heat demand real without the compute story? If both answers are yes, the system starts to look like infrastructure instead of marketing.

The Business Model Is The Weak Point To Watch

Deep Green's own model depends on selling the compute. The BBC says the company charges clients for computing power used for artificial intelligence and machine learning. TNW later reported that Deep Green wanted to scale to 100-150 swimming pools after an Octopus Energy investment, but also quoted the company saying it needed more corporates to stop using traditional data centres and use its servers so it could give heat to communities.

That is the constraint I find most important. The heat host may be delighted, but the system does not work because a pool has spare space. It works only if Deep Green can keep the servers occupied with workloads that tolerate this deployment model. A Hacker News discussion around the Ars article surfaced the same concern: commenters asked who would buy small, local colocation capacity and whether Deep Green could aggregate enough distributed compute demand.

I do not think that objection kills the idea. It makes the execution bar visible. A distributed data-centre network has to sell reliability, support, security, connectivity, hardware maintenance, and workload fit. The heat story may win attention, but the compute customer is still buying infrastructure.

The pool gets the heat, but the business lives or dies on the workload stream entering the servers.

The AI Context Makes This Less Niche

When this BBC story ran in March 2023, it could read like a clever local energy hack. In 2026, it reads more like an early version of a bigger infrastructure question. Deep Green's current site now frames the company around AI and high-performance-computing colocation, with live capacity in Manchester. It says its DG01 Manchester site donates heat to Move Urmston leisure centre, saving around £80,000 a year on community heating costs and reducing CO2 emissions by 100-150 tonnes.

The reason this matters is that data-centre demand is no longer a background technical issue. The International Energy Agency's Energy and AI report says data centres accounted for around 1.5% of global electricity consumption in 2024, or 415 TWh, and projects that data-centre electricity consumption will more than double to around 945 TWh by 2030. The same report stresses that local impacts can be far more pronounced than the global share suggests.

That is why heat reuse should not be treated as the whole answer. It is one design lever inside a larger energy problem. Locating compute near useful heat loads can reduce waste, but it does not erase electricity demand, grid constraints, capital requirements, or the emissions profile of the power supply.

Distance And Temperature Decide A Lot

The strongest skeptical point is not that waste heat is fake. It is that waste heat is often awkward. The Ars writeup summarized the broader challenge well: distributing heat can involve cost, local district-heating access, technology constraints, and recipients who may need heat pumps because the available heat is not always hot enough on its own. Physical closeness matters; so does the temperature of the heat stream.

That is why the pool example is unusually tidy. A leisure centre has constant water circulation, a large thermal mass, and a heating load that can accept steady contribution. A distant housing project, an office district, or a seasonal heat user may be much harder to serve. The same server output can look valuable in one building and nearly useless across the wrong boundary.

The real deployment has several gates: workload demand, grid power, maintenance, backup heat, and a heat customer that can actually absorb the output.

My Bottom Line

I like the Exmouth story because it narrows a giant AI-energy debate into a concrete loop. Instead of saying data centres should be sustainable in the abstract, it asks whether one box of servers can sit beside one real heat load and reduce one real bill.

But I would not promote this as a universal fix. The practical lesson is more disciplined: put compute where its waste heat has a buyer, prove the workloads are real, keep backup systems honest, and measure the net result at the site. If the AI boom is going to keep filling buildings with hot machines, the least convincing answer is to pretend the heat disappears. The more useful answer is to design neighborhoods for it.

Originally published at markhuang.ai

98% Support Still Needs a Door

Mark Huang — Tue, 07 Jul 2026 19:37:46 +0000

The sharp point in Hugo Barrera's essay is that a percentage can look excellent while still describing real people locked out.

Hugo Barrera's "98% isn't very much" is a short web-development essay with a useful sting: a support number that sounds excellent can still be a weak promise when the feature is part of basic access. His point is not that modern browser features are bad. It is that a website that works for nearly everyone can still fail a huge number of people, and that global support statistics may not match the audience actually visiting a specific site.

My read is that this is the right way to think about browser compatibility in 2026. I want the modern web platform. I also want teams to stop treating "widely supported" as if it automatically means "safe for my users." The adoption decision should depend on blast radius, audience analytics, and whether unsupported browsers get a graceful path instead of a broken page.

Answer Snapshot

Question	My read
What happened?	A July 3, 2026 WhyNotHugo post argued that 98% support is not enough when the missing 2% means real users cannot use a site.
Why it matters	Browser support percentages compress too much: global usage, core-browser support, local audience mix, and fallback quality are different questions.
Who benefits if teams handle this well?	Users on older, managed, niche, mobile, assistive, or constrained browsing setups, plus frontend teams that want modern CSS without surprise regressions.
My thesis	A new web feature is safe only when the unsupported path is acceptable. If failure breaks the core experience, 98% is not a finish line.
What I am not arguing	I am not saying every site must support every old browser forever, or that CSS nesting is bad. I am saying the support bar depends on what breaks.

The Number Is Doing Too Much

Barrera's strongest move is to separate bonus outcomes from baseline expectations. A 98% success rate can sound amazing for a difficult achievement, but it sounds very different when the expectation is basic reliability. Applied to the web, the missing percentage is not an abstract remainder. The source post notes that a feature working for 98% of the population can still leave out roughly 150 million people.

The more practical warning is that "98% of the population" may not mean 98% of a site's actual audience. Barrera says he checked one client's browser distribution while considering whether to remove an SCSS pipeline, and found that over the prior year only about 70% of visiting browsers supported the newer CSS features in question. That is the part I find hardest to hand-wave away. The decision was not about a generic web user. It was about that site's real traffic.

This is why I dislike compatibility arguments that stop at a single percentage. The number is useful, but it is not the decision. The decision is whether the feature is decorative, progressive, recoverable, or load-bearing.

A global average can hide the audience that actually visits your site. The local distribution is often the number that matters.

CSS Nesting Is a Good Test Case

CSS nesting is a useful example because it is not a gimmick. MDN's guide describes it as browser-parsed CSS, not precompiled Sass, and says it can make stylesheets easier to read, more modular, and sometimes smaller. Chrome's developer documentation makes a similar case around organization, reduced repetition, and refactoring.

So the pro-adoption argument is real. Native nesting removes a build-step dependency for some sites, gives authors a familiar way to group related selectors, and is now part of the modern platform conversation. Baseline also gives developers a shared language for support: newly available means all core browsers support a feature, and widely available means 30 months have passed since that interoperable date.

But the details still matter. Sass's 2023 writeup explained that native nesting is not identical to Sass nesting, including differences around :is() specificity and selector suffix behavior. The Sass team said it would not change existing valid Sass output to emit browser-incompatible CSS until native nesting reached 98% global browser market share. That caution is not nostalgia. It is compatibility discipline from people who understand how much CSS sits in production.

As of the Can I Use CSS Nesting page I inspected, the feature showed 90.81% global usage support when combining full and partial support, with usage statistics based on StatCounter data for June 2026. That is high enough to be interesting and low enough to prove Barrera's point: the label "modern" does not tell me whether a site's unsupported path is acceptable.

Baseline Is Not a Permission Slip

I like Baseline because it replaces vibes with a common vocabulary. It makes the platform easier to discuss, lint, and teach. The problem starts when teams turn that vocabulary into a yes/no permission slip.

The ESLint CSS use-baseline rule documentation is careful about this. It says Baseline can help interoperability, but testing is still required, especially when an audience uses browsers outside the core set. It also notes that accessibility testing is still required because Baseline does not track assistive-technology support, and it explicitly leaves room for @supports, fallbacks, and progressive enhancement.

That is the version of Baseline I trust. It is a good input to an engineering decision, not a replacement for one. A feature can be Baseline and still be the wrong choice for a public form, a checkout flow, an internal enterprise app with locked-down browsers, or a site whose audience over-indexes on old mobile devices.

My rule of thumb is simple: the worse the failure mode, the higher the support bar. If unsupported browsers get a readable page and a working path, I can move faster. If they get a broken core experience, the percentage needs to be much higher.

The tradeoff is not old web versus new web. It is developer convenience weighed against the cost of excluding users from basic functionality.

The Public Reaction Is Split

The developer conversation around native CSS nesting shows the real split. In a Reddit webdev thread about whether native CSS nesting was safe to use, one concern was straightforward: if nesting is fundamental to the stylesheet, unsupported browsers may lose far more than a minor polish layer. Other commenters were comfortable moving ahead when their support policy was limited to recent Chrome, Edge, Firefox, and Safari versions.

That split is not irrational. It is audience-dependent. A private project with a narrow browser policy can tolerate a different risk than a public information site, a school system, a government-adjacent service, or a product with customers on managed hardware. The mistake is pretending there is one universal threshold.

The skeptical side also has a maintainability argument. Piccalilli's "CSS nesting: use with caution" argues that nesting solves a developer problem more than an end-user problem, warns about specificity gotchas, and recommends keeping nesting shallow if you use it. I do not share the most absolute version of that critique, but I find the pressure useful. A feature that improves authoring can still make runtime support and code comprehension worse if applied carelessly.

A Better Adoption Checklist

Question	What I would check
Is this enhancement or foundation?	If the feature only improves layout polish, a fallback can be simple. If it controls navigation, forms, purchasing, reading, or authentication, the bar rises.
What happens without support?	Test the unsupported path directly. Do users get usable content, a reduced layout, or a broken interface?
What does real traffic say?	Look at actual browser and device analytics for the audience, not only global compatibility tables.
Can the feature be isolated?	Use `@supports`, build output, progressive enhancement, or simpler CSS so unsupported browsers avoid the fragile path.
How will regressions surface?	Keep compatibility checks, analytics, support reports, and accessibility testing close to the release process.

This checklist is not glamorous, but it is the difference between modernizing a codebase and quietly firing a slice of the audience. The goal is not to freeze the web. The goal is to make feature adoption legible: what improves, what breaks, who is affected, and how the team will know.

The operational answer is detection, fallbacks, analytics, and accessibility review, not a single percentage copied into a release decision.

My Bottom Line

Barrera's post works because it pushes on a lazy shortcut without turning into anti-progress nostalgia. I want teams to use native CSS nesting, container queries, :has(), and the rest of the platform when they make products better. I also want them to ask what happens to the people outside the happy path.

For me, 98% is not "good" or "bad" by itself. It is a prompt to ask sharper questions. Is the remaining 2% actually tiny for this site? Is the audience distribution known? Is the missing support concentrated in a group the product claims to serve? Does the page degrade gracefully? If the answers are weak, then "widely supported" is not enough. It is just a nicer-sounding way to leave someone at the door.

Originally published at markhuang.ai

The Token Sticker Price Is a Trap

Mark Huang — Mon, 06 Jul 2026 21:30:03 +0000

The sticker price is visible. The bill that matters is the work required to get a reliable finished task.

Jan Iłowski's Price per 1M tokens is meaningless is a useful correction to a bad procurement habit. His argument is simple: two models can advertise a neat $X per 1M tokens price, but that number stops being comparable when tokenizers, hidden reasoning, output length, cache pricing, and actual task success move independently.

My read is that the title is intentionally sharp, but the underlying lesson is right. Price per million tokens is not meaningless as an input. It is meaningless as a decision rule. If a team is choosing models from a spreadsheet without measuring cost per completed task on its own workload, it is not buying cheaper AI. It is buying uncertainty with a cleaner unit label.

Answer Snapshot

Question	My read
What happened?	Iłowski published a July 5, 2026 post arguing that per-million-token pricing is a poor way to compare AI models because token counts and token efficiency vary by model and workload.
Why it matters	AI spend is now operational enough that a small-looking token price can still produce a larger bill if the model needs more tokens, more hidden reasoning, more retries, or more evaluation cleanup.
Who benefits if teams measure correctly?	Product, finance, and engineering teams that route routine work to cheaper models while reserving stronger or more expensive models for tasks where they prove better cost per successful outcome.
My thesis	The right comparison is not token price. It is verified outcome per dollar at a declared quality bar.
The catch	Benchmarks help, but they are not your workload. The final answer still has to come from task-level evals, logging, and budget guardrails inside your own product.

The Unit Is Not Neutral

The first problem is tokenization. Iłowski gives a concrete example: the same text from his post counted as 160 tokens for GPT-4o and 200 tokens for GPT-4 1106-preview using tiktokenizer. I am not treating that calculator output as a universal constant, but the mechanism is real. OpenAI's own tiktoken guide says different models use different encodings, and token counts matter because API usage is priced by token.

Anthropic makes the point even more directly in the Claude Sonnet 5 documentation. The docs say Sonnet 5 uses a new tokenizer, that the same input produces approximately 30% more tokens than Sonnet 4.6 depending on content, and that this affects token counts, context capacity, output budgets, and per-request cost even when per-token pricing is unchanged.

That turns a clean price table into a measurement problem. A model can look cheaper per token while making the same prompt larger. Another can look expensive per token while using fewer tokens to complete the same task. Once that happens, the unit price is still true, but the comparison is incomplete.

The same source material can turn into different token piles. That alone makes cross-model sticker prices slippery.

Task Cost Is The Better Question

The stronger part of Iłowski's post is the move from token price to task cost. He uses Artificial Analysis data to compare selected models by input/output price, Intelligence benchmark score, and cost per benchmark task. In his table, GPT-5.5 is listed at a higher nominal output price than Claude Opus 4.8 max, yet a lower cost per benchmark task. DeepSeek V4 Pro max is shown as a cost-efficiency outlier, while GLM-5.2 max is much cheaper per token than the GPT and Claude examples but not proportionally cheaper per task.

I would not overfit to those exact rows. Model prices change, router prices can differ from first-party prices, and a benchmark mix is not a substitute for a product's actual workload. But the measurement direction is right. Artificial Analysis says its cost-per-task metric uses input, cached, reasoning, and answer-token prices multiplied by tokens consumed across the workload. It also notes that models producing longer answers or more reasoning tokens can cost more per task even at identical per-token prices.

That is the practical distinction. The buyer should care less about how many tokens a vendor sells for a dollar and more about how much verified work a dollar buys. If the task is customer support triage, the unit is a resolved case. If it is code review, the unit is a useful finding that survives human review. If it is document extraction, the unit is accepted structured output, not generated tokens.

The metric I would put in front of every AI budget review is outcome per dollar at a declared quality threshold. Token price belongs underneath that metric, not above it.

Hidden Work Changes The Bill

The second problem is that visible output is no longer the whole story. Iłowski points at "thinking" tokens as a cost driver: reasoning can improve quality, but the hidden chain of work can dominate the bill. I would phrase the risk even more broadly. The bill also includes retries, tool calls, cache writes, long context, guardrail refusals, formatting cleanup, and human review time.

This is where I found TensorZero's analysis useful as independent context. TensorZero reports that identical inputs can produce materially different token counts across providers and content types, and argues that teams need to measure what they actually send rather than compare list prices alone. Its exact workload is not everyone's workload, but the conclusion matches the operational problem: the cheapest provider changes when the input shape changes.

Public technical discussion points in the same direction. In a Hacker News thread about an LLM pricing tool, one commenter immediately asked how to compare cache pricing across providers because key-value caching can become a major part of actual token usage. A separate HN thread about AI costs gets into serving-cost math and assumptions for GLM-5.2. I would not use comment threads as settled economics, but they show the right kind of skepticism: price, provider routing, cache behavior, throughput, and self-hosting cost are different questions.

A shorter sticker price can lose to a longer hidden path. The useful comparison is the cost of getting to a completed task.

The Source Is Right, But It Needs A Boundary

The critique I would add to Iłowski's post is that "cost per benchmark task" can become its own false comfort. It is better than price per token, but it is still a proxy. A benchmark tells me something about a model under a published harness. It does not tell me whether the model follows my schema, respects my latency budget, handles my edge cases, avoids my failure modes, or passes review from the people who own the workflow.

That matters because token efficiency can be purchased in ways a product may not want. A model that gives shorter answers may be cheaper but less useful. A model that reasons longer may be expensive but reduce rework. A model that routes through a cheaper provider may look good until availability, privacy posture, moderation behavior, or support quality becomes the real constraint.

So my version of the lesson is narrower and more actionable: do not replace token-price worship with benchmark worship. Use public benchmarks to form a hypothesis. Then run a workload-specific eval that measures success rate, latency, retry rate, review cost, cache hit behavior, and total dollars per accepted result.

What I Would Do In Practice

If I were designing the buying process, I would start with a small task catalog. Separate cheap routine work from ambiguous, high-stakes, long-context, and customer-visible work. Run each class against candidate models with the same prompts, same tool permissions, same retrieval context, and the same acceptance rubric.

Then I would log more than tokens. I would log prompt tokens, output tokens, reasoning or thinking tokens when exposed, cache reads and writes, tool calls, latency, retries, refusal rate, human override rate, and accepted-output rate. Only then does the pricing table become useful, because it can be multiplied against observed behavior instead of imagined behavior.

The product pattern is model routing with evidence. Cheap models earn routine traffic when they pass. Expensive models earn escalation traffic when they reduce failures enough to justify the cost. Local and open-weight models get considered where privacy, volume, or latency makes them compelling. Frontier models stay in the path where their extra capability proves itself.

The mature workflow makes model choice observable. Routing, evals, budget limits, and outcome checks have to move together.

My Bottom Line

Iłowski's post matters because it attacks a number that feels objective but can be misleading in production. The price per million tokens is easy to copy into a spreadsheet. The real cost is harder: how many tokens the model creates from your input, how much hidden work it does, how often it succeeds, and how much cleanup the organization has to fund afterward.

I would still keep token prices in the model card. I just would not let them make the decision. The decision should be cost per accepted task, measured on the work the product actually performs. Anything less is pretending that the meter is the same thing as the bill.

Originally published at markhuang.ai

AI Makes the Average Too Cheap

Mark Huang — Mon, 06 Jul 2026 19:38:03 +0000

The risk is not that AI can summarize the center. The risk is that teams start mistaking the center for the frontier.

The rruxandra.github.io essay, "Regression to the Mean — On LLMs and the Quiet Death of the New," argues that large language models can gently pull thought toward the familiar. The page frames the promise of a collaborator on every desk against a sharper worry: a system trained on the past may reward what sounds typical while treating the strange new thing as a correction target.

My read is that the essay is directionally right and technically too clean. LLMs are not literally doomed to emit only the statistical center. Prompts, sampling, retrieval, fine-tuning, review, and workflow design all matter. But the 2024 and 2025 research context makes the warning practical: AI-assisted writing and brainstorming can improve individual outputs while narrowing collective diversity. When the average answer becomes instant and cheap, the scarce skill is deciding when to keep the awkward deviation.

Answer Snapshot

Question	My read
What happened?	A personal essay argues that LLMs can regress creative work toward the mean by returning familiar continuations and sanding down outliers.
Why it matters	Research on AI-assisted writing and brainstorming has found a similar tradeoff: individual outputs can look stronger while the collective set becomes more alike.
Who benefits if this is handled well?	Writers, researchers, engineers, product teams, and educators who want AI to widen exploration instead of standardizing the first draft.
My thesis	The practical answer is not to avoid LLMs. It is to use them in workflows that preserve evidence, disagreement, and deliberate human choice.
What I would not claim	This is not proof that every AI-assisted work is mediocre, or that novelty cannot be elicited from models. It is a warning about defaults.

The Essay Names A Real Failure Mode

The strongest part of the source is its inversion of scarcity. If everyone can ask a model for a polished answer, polish gets less valuable. The essay's phrase "guard the tail" is the part I would keep. New work often starts as something that looks wrong to the current consensus. If the assistant is always nudging the sentence, concept, naming, or architecture back toward what it has seen before, the user has to notice that pressure and sometimes reject it.

There is a technical foundation under that intuition. The GPT-3 paper describes GPT-3 as an autoregressive language model. That does not mean modern chat products simply choose the single most likely next word, and it does not erase reasoning-like behavior learned at scale. It does mean the default interaction is still built around continuations that fit a learned distribution. The model is powerful because it has absorbed so many patterns. The risk is that pattern fluency can feel like judgment.

That distinction matters because the lazy version of the critique is easy to dismiss. "LLMs predict text" is true but incomplete. "Therefore they cannot help with original work" is too broad. The more useful claim is narrower: if the user delegates taste, framing, and final judgment to the model, the work will tend to inherit the model's center of gravity.

AI can raise the apparent quality of many individual drafts while making the overall field feel more similar.

The Evidence Is A Tradeoff, Not A Slogan

The best public context I found does not say "AI kills creativity." It says the effect can be two-sided. A Science Advances study on creative writing found that access to generative AI ideas improved how stories were rated for creativity, writing quality, and enjoyment. But the same study also found that AI-assisted stories became more similar to one another.

That is the exact tradeoff the essay is pointing at. A tool can help an individual get unstuck and still compress the collective range of outputs. In a classroom, newsroom, product team, or software organization, that matters because the group result is not just the average quality of one artifact. It is the spread of hypotheses, metaphors, designs, edge cases, and disagreements that survive long enough to be tested.

A Wharton summary of a Nature Human Behaviour brainstorming paper points in the same direction. Across five experiments, ChatGPT-assisted brainstorming produced narrower idea sets, with significant drops in 37 of 45 statistical comparisons. I would not overread one research line as destiny, but it is enough to make the warning practical instead of merely aesthetic.

The metric I would watch is not just whether one AI-assisted answer looks good. It is whether a team using the same assistant still produces enough independent, weird, and falsifiable alternatives.

The phrase "regression to the mean" also echoes the model-collapse literature. In Nature's 2024 paper on recursively generated data, researchers found that indiscriminate training on model-generated content can make models lose information about the true distribution, with tails disappearing first and learned behavior converging toward a low-variance point estimate.

That finding is important, but I would keep it separate from the source essay. Model collapse is about training data and recursive learning over generations. The essay is more about everyday use: people asking models for answers, feeding those answers into more work, and letting the default phrasing or framing become the next prompt. The mechanism is different. The intuition rhymes.

The practical overlap is preservation. Model builders need high-quality human and real-world data so the training distribution does not eat itself. Teams using LLMs need human judgment, primary sources, dissenting drafts, domain review, and non-AI references so their work does not become a loop of acceptable summaries.

The polished version may be useful. The jagged version may be where the new idea still lives.

The Pushback Is Also Correct

The common objection I find persuasive is that "the model returns the average" is not the whole product. A Hacker News thread about whether LLMs are "regression to the mean machines" shows the useful split. Some developers complain about repetitive, locally plausible output. Others point out that context, review, prompts, documentation, and feedback can make LLM-assisted work better than a raw model sample.

That pushback matters. There are real techniques for widening output. A Harvard research project on sustained creativity and diversity in LLMs argues that decoding schemes can produce more conceptually diverse results without access to the model's internal vector space. Even ordinary product choices such as asking for multiple competing hypotheses, forcing citation to primary sources, or separating generation from critique can change the shape of the output.

So I do not buy the deterministic version of the source's claim. The model is not a sealed box that can only hand back the mean. But I do buy the default-risk version. Most users do not run diversity-aware decoding schemes. Most teams do not measure idea spread. Most draft workflows reward speed, readability, and consensus long before they reward a hard-to-defend outlier.

A Better AI Workflow Protects Deviation

If I were turning the essay into an operating rule, I would make it boring and concrete. First, start with unaided notes before opening the model. Second, ask the model for alternatives that disagree with the initial frame, not just a cleaner version of it. Third, preserve the rough draft so the model's edits do not erase the original shape of the thought. Fourth, require source links or evidence for factual claims. Fifth, make a human decide which weird edges to keep.

That is especially true in technical work. An AI assistant can produce the obvious abstraction, the common error-handling pattern, the familiar API shape, or the standard product narrative. Sometimes that is exactly what you want. But if the problem is novel, the safe-looking middle may be the wrong place to stand. The review process has to ask whether the model removed useful friction.

The same principle applies to writing. I do not think every rough sentence should be preserved as an act of authenticity. Editing is good. Clarity is good. But there is a difference between tightening an argument and sanding off the reason it existed. A model can help with the first. It should not be allowed to quietly perform the second.

The healthier workflow treats AI as one pass through evidence, disagreement, and choice, not as the final authority on what sounds right.

My Bottom Line

I like the essay because it makes the cheap thing visible. The average answer now arrives instantly, politely, and with enough polish to pass as thought. That is useful. It is also dangerous when the user forgets that plausible fluency is not the same as discovery.

The answer is not nostalgia for unaided work. The answer is better taste under automation pressure. Use the model to compress background, generate candidates, surface counterarguments, and stress-test a draft. Then make a deliberate choice about the tail. The new thing will often look awkward before it looks right. If the model keeps correcting it, that may be the first sign worth paying attention to.

Originally published at markhuang.ai

Ultra in Codex Has to Beat the Meter

Mark Huang — Mon, 06 Jul 2026 19:27:54 +0000

The interesting part of Ultra coming to Codex is not the name. It is whether a stronger agentic mode can stay usable when access, limits, and trust checks all matter.

On July 6, 2026, Tibo replied to a Codex user on X with a short line: "Ultra will be in codex." The parent post from Haider had asked for GPT-5.6 "pro or sol ultra" in Codex and framed the request around model access, usage limits, and competition with other AI coding subscriptions.

My reaction is cautious excitement. A one-line X reply is not a release note, and I would not treat it as a promise about timing, plan eligibility, or pricing. But it matters because OpenAI's official GPT-5.6 announcement says the new ultra mode goes beyond a single agent by using subagents to accelerate complex work. If that mode lands inside Codex, the real question becomes whether it feels like dependable engineering help, not whether the model name sounds powerful.

Answer Snapshot

Question	My read
What happened?	Tibo publicly said Ultra will be in Codex, replying to a request for GPT-5.6 "pro or sol ultra" access in Codex.
What is confirmed?	OpenAI has announced GPT-5.6 Sol, Terra, and Luna, plus an ultra mode that uses subagents for complex work.
What is still not confirmed?	The source tweet does not give a launch date, broad availability, plan eligibility, Codex rate-card details, or exact product behavior.
My takeaway	Ultra in Codex will be judged by capacity, cost visibility, review controls, and actual workflow reliability.

The Tease Is Small, But The Surface Is Big

The source is intentionally tiny, so I am treating it as a directional product signal rather than a full announcement. The useful context is that the reply points to Codex, not just the API or a private preview. That distinction matters because Codex is where developers feel the model through files, terminals, reviews, worktrees, and long-running tasks.

OpenAI's Codex help article says Codex is included across Free, Go, Plus, Pro, Business, Edu, and Enterprise plans, with usage limits and credit options varying by plan. That makes Codex a broad product surface, not only an enterprise lab. When a more capable mode comes there, the pressure shifts from benchmark curiosity to everyday product mechanics.

A subagent mode is only useful if it turns messy work into reviewable progress, not just more parallel activity.

Ultra Is A Workflow Claim

The phrase "ultra mode" is easy to hear as marketing, but the underlying product claim is more concrete: subagents should let one request fan out into coordinated pieces of work. For code, that could be genuinely valuable. Large changes often involve search, planning, implementation, tests, docs, migration notes, and review. Parallel help is attractive precisely because real engineering work is rarely one neat completion.

That is also why I do not want to over-celebrate the label. More agents can create more output, but more output is not automatically better work. The mode has to preserve intent, show what happened, respect boundaries, and make the review easier rather than harder. The promise is not "bigger model wins." The promise is "delegation becomes controlled enough to trust."

OpenAI's GPT-5.6 Help Center page says the preview is available through the API and Codex only for a limited group of trusted partners and organizations, that ChatGPT is not included during the preview, and that OpenAI has not announced a general-availability date.

Access Is Still The Unresolved Part

The official availability language keeps the tweet in perspective. OpenAI says GPT-5.6 preview access is scoped to approved API organizations and Codex workspaces, and that approval for one does not automatically include the other. The Help Center also says individual users and consumer accounts are not eligible for the preview, and that a paid ChatGPT plan does not by itself provide access.

So my practical read is: "Ultra will be in Codex" is meaningful, but it does not tell a regular user when they will see it. It also does not say whether Ultra will arrive as GPT-5.6 Sol Ultra, as a selectable mode, as a workspace-gated preview, or as something OpenAI routes automatically. Those distinctions are not pedantic. They decide who can use it, how teams budget for it, and how much confidence developers should place in the announcement.

The stronger the agentic mode, the more important the meter becomes. Teams need to know what they are spending before the workflow becomes habit.

The Meter May Decide Adoption

This is where I think the real story sits. OpenAI's Codex rate card says Codex has moved to token-based credit accounting for current plans, and that actual credit use depends on input tokens, cached input tokens, and output tokens. The page lists GPT-5.5 at 125 credits per million input tokens and 750 credits per million output tokens, and says a typical GPT-5.5 Codex task may consume between 5 and 45 credits. It does not, in the version I inspected, give a Codex rate card for Ultra.

That gap is why I keep coming back to the meter. If Ultra uses subagents, users will want to know how much work is being spun up, what counts against included usage, how cached context is handled, and where a task can be stopped before it burns through a budget. OpenAI's credits article already frames Codex credits as the path beyond included Plus and Pro limits. For an advanced mode, cost clarity becomes part of product quality.

Public Reaction Is About Trust, Not Just Hype

The public context around GPT-5.6 is not a single mood. A Digg summary of an earlier Tibo post about GPT-5.6 Sol Ultra captured excitement about trying hard prompts, while also noting frustration over the lack of availability details. A Reddit discussion around the GPT-5.6 announcement shows developers arguing about preview access, safety restrictions, and timing. On the OpenAI Developer Community, one Codex user complained that limits can turn ordinary coding flow into quota management.

I find the quota critique more persuasive than the raw excitement. A frontier coding mode can be impressive in a demo and still fail as a daily tool if users are afraid to spend their best prompts, if long tasks consume limits unpredictably, or if safety checks interrupt legitimate work without clear feedback. In that world, developers do not ask "is Ultra smart?" first. They ask "can I rely on it for this repo today?"

The best version of Ultra in Codex would make delegation more inspectable, with clear gates and review points instead of a black-box burst of activity.

What I Would Watch

If Ultra arrives in Codex, I would judge it by operational details before benchmark language. Does Codex show which subagents ran and why? Can a developer cap the scope or budget of a run? Are safety pauses understandable enough to debug? Can teams reproduce the same task path in review, or does the mode feel like a one-off burst of hidden work?

The best outcome is not simply that Codex gets a stronger model. The best outcome is that Codex gets a stronger operating model for delegated engineering work: clear task boundaries, visible cost, recoverable failures, and reviewable artifacts. That would make Ultra feel less like a shiny routing option and more like a serious workflow primitive.

My Takeaway

I read "Ultra will be in codex" as an important signal, but not a finished product story. The source says direction. The official docs say limited preview. The rate-card and community context say the meter matters. Put together, the thesis is straightforward: Ultra in Codex will only be exciting if it beats the practical friction around access, credits, safety checks, and review.

That is the standard I would hold it to. A coding agent does not win by being dramatic for one prompt. It wins by becoming reliable enough that I can hand it hard work, understand what it did, and decide whether the result was worth the cost.

Originally published at markhuang.ai