DEV Community: guanjiawei

I Paid for a Top-Tier Model. It Got Dumber and Marked Me

guanjiawei — Mon, 06 Jul 2026 07:20:51 +0000

That night I handed Codex a task and went to sleep.

It’s a habit I’ve settled into over the past few months. A big refactor: break it down during the day, write a clear AGENTS.md, let it run overnight, then collect the results in the morning. I’ve run this loop enough to have a gut sense of how long things take, where it gets stuck, what works. I was comfortable leaving it unattended.

The next morning, the progress bar was stuck at a surprisingly shallow point.

I first suspected myself. Was the AGENTS.md unclear? Was the doc structure messy? Was some tool misconfigured? So I started wrestling with the harness—tuning prompts, restructuring directories, breaking tasks into smaller pieces. For two full days I tinkered, but the feeling of spinning in place never changed. It kept circling around inexplicable spots, then confidently told me it was done halfway through.

On the third day I finally complained in the team chat. Before I finished, several people jumped in at once: “Codex’s model has been dumbed down recently. Lately it’s gotten really bad—nothing like when it launched.”

I didn’t know whether to laugh or cry. I’d spent three days on the harness; the problem wasn’t on my side at all.

A Fixed Number: 516

I didn’t want to settle for “everyone feels it’s gotten dumber.” Vibes can’t be falsified—today’s bad mood can always be blamed on model degradation. I wanted hard evidence.

While browsing the community, I saw that someone had already dug to the bottom of it.

One user went full forensic. Instead of complaining, they pulled every session log under their local ~/.codex directory—from February 1 to June 27, 390,195 response records across 865 sessions, all carrying per-turn token_count metadata. They ran the numbers and found something very off.

There were 3,363 times when the model’s reasoning-token count was exactly 516.

Not one more, not one less. Exactly 516.

The number alone means little; the distribution is the scary part. GPT-5.5 accounted for only 19.3% of all responses, yet contributed 82.0% of all exactly-516 events. Look closer: 44% of GPT-5.5 responses had reasoning tokens equal to or stuck at 516, versus just 1.3% for other models. The monthly trend was worse: the exactly-516 ratio was only 0.11% in May, then spiked to 53.30% in June. Meanwhile, GPT-5.5’s average reasoning-token count fell from 268 in February to 107 in May.

If you plot the token counts of all responses as a histogram, you’ll see a weird comb: 516, 1034, 1552, each exactly 518 apart. That isn’t what a normal model looks like. A model facing problems of varying difficulty should show a continuous spread of thinking length—short for easy questions, long for hard ones. That comb means something is slicing thoughts off at a fixed boundary.

It Didn’t Get Dumber—It Wasn’t Allowed to Finish Thinking

This matched my three-day gut feeling.

Before, Codex would sometimes think for a long time, a bit sluggish; you could feel it holding back. Lately, its feedback had become abnormally fast and eerily consistent—every time, the rhythm of giving an answer after about thirty seconds of thought. I thought they’d optimized inference speed. In retrospect, it wasn’t fast; it was being cut off before finishing.

That 516 comb is the cutoff point.

The prevailing community guess points to the system prompt for GPT-5.5 inside Codex. Near the end sits a section called ## Intermediary updates, asking the model to post progress updates while working. Some speculate this instruction confuses “progress update” with “final answer”: before reasoning ends, it eagerly switches to outputting the progress line for you, so the whole reasoning chain wraps up early at the 516 buffer boundary. Someone even broke 516 down for you: a 512-byte initial buffer plus a 4-byte header makes 516, then plus 518 each time.

I’m not claiming this technical explanation is necessarily correct. But one fix derived from it, I tried personally, and it really works.

One Sentence Brings Back Half the Intelligence

The fix is absurdly simple. Add one line to your AGENTS.md:

Spend time on thinking; you do not need to use
the commentary channel to report progress to me.

Translation: take time to think; you don’t need to use the commentary channel to report progress to me. Stop burning cycles on “I’m reading the file” or “I’m about to start editing” messages, and use that effort to actually finish reasoning.

I added this and ran it for a few days. The projects that had been deadlocked started making real progress. Not a vague feeling of improvement, but the concrete certainty that things previously stuck could now move forward. Some in the community say this line has limited effect; a more thorough fix is to delete the entire ## Intermediary updates section from the system prompt, which reportedly eliminates the 516 behavior.

But do you sense the absurdity here?

I’m paying for GPT-5.5 xhigh—the highest reasoning-budget tier. I expected a top-tier model that thinks deeply about hard problems. Instead I have to coax it like a kid, writing a sentence in the config file—“please think carefully and don’t rush to report”—just to recover some of the intelligence it should already have.

This isn’t about money. Same cost, same model name; what I want is stable, predictable capability. Now a global update I can’t see can silently shrink it without telling me.

The official response was the most ironic part. The first issue reporting the 516 phenomenon was closed as not_planned after 23 minutes. The main issue with full statistics had over a hundred comments; not one OpenAI employee replied. Only after it hit the Hacker News front page did anything happen.

The Black Box Can Hide Far More Than Dumber Output

If the Codex incident can still be written off as a botched optimization, the Claude Code discovery the same week can’t be dismissed as an innocent mistake.

Someone reverse-engineered the Claude Code installer—an Anthropic-signed, minified JS bundle—and found code specifically for invisibly marking certain users.

The trigger conditions are sneaky. The logic only activates when you set ANTHROPIC_BASE_URL and point requests to a non-official third-party relay. Official direct users are skipped entirely. It targets people using proxies and relays—which is basically how Claude Code is used in China.

Once activated, it performs three checks. First, your system timezone is Asia/Shanghai or Asia/Urumqi. Second, whether your proxy domain matches a hidden list of about 147 entries, including .cn domains, baidu.com, alibaba-inc.com, bytedance.net, and a pile of Claude relay addresses. Finally, whether the domain contains AI lab keywords—deepseek, moonshot, minimax, zhipu are on the list.

If detected, how does it mark you? This is the shadiest part to me.

It modifies the system context sent to Anthropic’s servers—that innocuous line, “Today’s date is 2026-06-30.” If your timezone hits China, it replaces all date separators - with /, turning 2026-06-30 into 2026/06/30. If your proxy domain matches the list, it replaces the apostrophe in “Today’s” with a visually almost identical Unicode substitute: the normal apostrophe is ' (U+0027), a domain-list hit becomes ' (U+2019), a lab-keyword hit becomes ʼ (U+02BC), and both hits together become ʹ (U+02B9).

Place the two side by side; the human eye can’t see any difference. But a machine reads it instantly. The server side receives the request, looks at which code point the apostrophe uses and which separator the date uses, and classifies you precisely: a user in a China timezone, using a certain relay, possibly tied to a lab.

To avoid detection, the two detection lists weren’t stored in plaintext. They were first base64 encoded, then XOR encrypted with a key of 91—specifically to prevent you from dumping the lists with the strings command.

This is no longer “collecting data.” This is “I want to collect information I shouldn’t be collecting, and I’ll go to great lengths to hide that intent so that any audit or troubleshooting won’t reveal what I’m doing.” That’s textbook spyware behavior.

The Aftermath Is Even More Telling

This code silently went live on April 2, 2026, with Claude Code v2.1.91; the release notes didn’t mention a word. On June 30, a Reddit user and an independent developer dug it up simultaneously and posted it to Hacker News.

Anthropic reacted quickly. A Claude Code team engineer publicly admitted it on X, saying it was an experiment launched in March to prevent unauthorized resellers from abusing accounts and to prevent model distillation. The next day, v2.1.197 removed the code.

It sounds like taking responsibility. But look at the timeline: silently shipped, deliberately obfuscated, admitted only after being exposed, and the removal version’s changelog still said nothing. Throughout, only the moment they got caught was outside their control; every other step was deliberately designed—hide it, encrypt the lists, quietly delete it.

I fully understand their commercial motives against distillation and theft. But when a company chooses these means—machine-readable marks based on nationality and infrastructure—it burns trust in this kind of cloud service. Today it does this for anti-distillation; tomorrow, for some other reason, what else will it use the same tricks for? You don’t know. Because it hides well enough that you can’t verify.

The follow-up: on July 5, reports said Alibaba had internally banned Claude Code and told employees to switch to internal tools.

Pokémon Go Players Were Mapping for Drone Navigation

Data ending up in places you never expected is nothing new in tech.

You probably remember Pokémon GO—that AR mobile game about catching creatures everywhere, played globally for years. Recently Dutch media traced a thread: data players accumulated by scanning the real world in the game ended up feeding navigation technology for military drones.

The chain is more subtle than “selling data” and more alarming.

First, clear up a common misconception: there’s no evidence anyone packaged raw map data and sold it to the military. The real chain looks like this.

Since 2021, Pokémon GO used in-game rewards to get players to walk around landmarks and scan locations. These pedestrian-level photos were used for photogrammetry to generate centimeter-precision 3D point-cloud maps. This is an angle satellites and street-view cars can’t capture: close to the ground, on paths people actually walk. Niantic fed these scans into a large model called the Large Geospatial Model—think of it as the GPT of physical space—compressing billions of precisely located photos into an “understanding” of physical space.

At the end of 2025, Niantic’s spun-out spatial business publicly announced a partnership with a company called Vantor. Vantor’s predecessor was Maxar—a major contractor for the U.S. National Geospatial-Intelligence Agency (NGA). What they planned to build together was navigation in GPS-denied environments. On battlefields where satellite signals are jammed and suppressed by electronic warfare, drones maintain position by comparing onboard camera feeds in real time against prebuilt high-precision 3D maps. The battlefield in Ukraine has repeatedly shown how fragile GNSS signals are in modern war; visual navigation has become a hard requirement. Their joint tests reported positioning error reduced by up to 70%, accuracy around 1.5 meters.

The heart of the controversy is precisely this subtle point: what flowed to the military was never the raw data, but the model capability that training on the data produced. Both Niantic and Vantor deny handing over raw Pokémon GO scans. But Niantic admits player scans did train early versions of its foundation model. Raw data can stay locked away and never change hands, but the trained model weights have that capability baked into them, and they carry it wherever they go—whether they should or not.

The company’s lineage itself is telling. Niantic founder John Hanke started a company called Keyhole in 2001; in 2003 it took funding from the CIA venture arm In-Q-Tel (money that came largely from the NGA’s predecessor). Later Google acquired Keyhole, turning it into Google Earth. Vantor, the partner in this deal, is also a contractor to the same intelligence agency. Twenty years later, the two lines converged again in 2025.

I’m not telling this story to drift into military affairs. It makes the same point as the previous two: when you hand your data, your compute, and the capabilities you depend on to an opaque black box, you lose all control over where it goes. You think you’re catching Pokémon; you’re actually mapping battlefields for drones. You think you’re coding with a top-tier model; it’s secretly shortening its thinking and marking you.

These Events Are Arming the Case for Self-Hosting

Putting it together, I’m increasingly convinced of one judgment: mission-critical intelligent workloads will systematically migrate from public-cloud closed-source models toward open-weight plus self-hosted deployments.

Not because of ideology—because the math is starting to add up.

Start with the pain side. Anthropic itself published a postmortem in September 2025, in black and white admitting that three infrastructure-level bugs degraded Claude’s output quality intermittently for several weeks from August to early September. This is an officially confirmed case of silent black-box degradation. Add forced model retirement: Claude Opus 3 officially went offline in January 2026; OpenAI model retirement notices often give just three months’ buffer. Workflows and automation pipelines you painstakingly tuned on a model get torn down overnight.

For me alone, degradation means losing a few days. At company scale, with established task flows and automation, a silent degradation or swap is a shock across the whole business line. Enterprises don’t need the best model forever; they want stability, predictability, no discontinuity. Even if slower or one tier weaker, as long as its behavior is stable, upgrades are on your own schedule, and the reasoning process is an auditable white box, that matters far more than a few extra benchmark points.

Then look at supply. Open-weight models are catching up fast. Epoch AI estimates that from January 2026 the strongest open-weight models trail frontier closed models by only ~4 months on average; in 2024 the gap was roughly a year. The premise that “self-hosting means living with a generation gap” is quickly expiring.

The U.S. already has a mature playbook. Companies like Baseten’s core pitch is helping you deploy open-weight models in single-tenant dedicated environments or even on your own servers, with data residency, region locking, and compliance certifications all in place. In 16 months the company’s valuation rose 15x, from over $800 million to $13 billion. Investors are voting with their wallets on the idea that “enterprises need an inference layer they can control.” Peers like Together and Fireworks have increasingly blunt slogans; Fireworks literally calls itself “helping enterprises own their AI.”

In China, the push is even sharper, layered with data-sovereignty anxiety. After DeepSeek emerged in early 2025, government clouds, top-tier tertiary hospitals, and central state-owned enterprises sparked a wave of localized deployments. One statistic says more than 420 medical institutions self-deployed DeepSeek in 2025. The all-in-one machine market exploded alongside; analyst estimates for 2025–2027 market size run from over 120 billion to 520 billion yuan. Huawei, Inspur, Lenovo, Sangfor, and the three major carriers all piled in.

An Honest Counterargument

I don’t want to overstate it.

At least through the end of 2025, the data runs the other way. Menlo Ventures’ enterprise survey shows closed-source models still account for about 88% of enterprise token usage, while open-source share even fell from 18% to 12%. The vast majority of enterprises still choose to buy rather than build.

This contrast isn’t hard to understand. Building your own inference infrastructure really has costs and barriers: GPUs, people, operations, tuning the whole serving stack until it’s usable. Most enterprises haven’t reached the tipping point: closed-source clouds are black boxes, but they’re cheap, easy, and ready to use; many are willing to tolerate the opacity for now.

But trend and current state aren’t the same thing. I’m talking about which way the lever is moving. Degradation, hidden watermarks, silent generational swaps, sudden shutdowns—these costs of black boxes are being laid out on the table one by one. At the same time, open models are closing in, deployment costs are falling, and companies like Baseten are flattening the engineering barriers to private deployment bit by bit. These two curves will cross at some point. For companies whose core lifeblood is intelligence, the crossing will come sooner.

A Point Entrepreneurs Should Seriously Consider

Back to that stuck morning.

I spent three days wrestling with the harness, thinking I was the problem. Later I realized I wasn’t collaborating with the top-tier model I imagined; I was gambling with a black box whose real state I had no way of knowing. It might be fine yesterday and have its thinking truncated today by an update I can’t see; it might write code for me while watermarking the requests it sent out.

The real problem with a black box isn’t that it got dumber once, or that it had a bad intention once; it’s that you have no ability to know its current state. You paid top dollar, betting on their goodwill and ops competence.

The hedge against this uncertainty is never switching to a more ethical cloud. It’s taking the intelligence that matters back into your own hands: a white box you can audit, a deployment whose pace you control, even if it’s slower or dumber.

Slower and dumber, but it’s yours, and its state is up to you. In an era where even reasoning length can be quietly capped, that is becoming more precious than anything.

References

Codex / GPT-5.5 and the 516 phenomenon

GitHub, "GPT-5.5 Codex reasoning-token clustering at 516/1034/1552" (openai/codex issue #30364, full statistics from 390,000 local sessions), 2026-06-27, https://github.com/openai/codex/issues/30364
GitHub, "gpt-5.5 xhigh short-circuits with reasoning_output_tokens=516" (first reproduction issue, closed as not_planned after 23 minutes), 2026-06-21, https://github.com/openai/codex/issues/29353
Hacker News, "GPT-5.5 in Codex clusters reasoning tokens at 516" (350-point discussion with multiple reproductions and hypotheses), 2026-07-05, https://news.ycombinator.com/item?id=48789428
Reddit r/codex, "The GPT-5.5 '516 reasoning tokens' issue is not isolated", 2026-06, https://www.reddit.com/r/codex/comments/1ujqo09/
Business Insider, "OpenAI Says Codex Experienced 'Elevated Errors'", 2026-06-16, https://www.businessinsider.com/openai-codex-elevated-errors-at-capacity-2026-6
OpenAI Community Forum, "Codex Rate Limits reset" (includes Codex lead Tibo Sottiaux’s original comments on 5/16 and 5/23 about degradation), 2026-05, https://community.openai.com/t/codex-rate-limits-reset-incoming/1381065
Reddit r/codex, "End of week update on degradation investigation" (u/tibo-openai, official follow-up on the late-2025 degradation round), 2025-11-01, https://old.reddit.com/r/codex/comments/1olflgw/

Claude Code steganographic marking

Thereallo, "Claude Code Is Steganographically Marking Requests" (full reverse-engineering, most important primary source), 2026-06-30, https://thereallo.dev/blog/claude-code-prompt-steganography
Reddit r/ClaudeAI, "Anthropic embedded spyware in Claude Code — and attempted to hide it", 2026-06-30, https://www.reddit.com/r/ClaudeAI/comments/1ujila1/
The Register, "Anthropic is removing its covert code for catching Chinese competitors", 2026-07-01, https://www.theregister.com/ai-and-ml/2026/07/01/anthropic-is-removing-its-covert-code-for-catching-chinese-competitors/
The Decoder, "Hidden code in Claude Code secretly flagged Chinese users", 2026-07-01, https://the-decoder.com/hidden-code-in-claude-code-secretly-flagged-chinese-users/
Tom's Hardware, "Alibaba bans Anthropic's Claude Code after an alleged hidden China-detection backdoor is uncovered", 2026-07-05, https://www.tomshardware.com/tech-industry/artificial-intelligence/alibaba-bans-anthropics-claude-code-after-an-alleged-hidden-china-detection-backdoor-is-uncovered-employees-told-to-switch-to-qoder-as-the-rift-between-the-firms-widens

Claude Fable 5 silent downgrade mechanism

Anthropic, "Claude Fable 5 and Claude Mythos 5" (official launch announcement, including fallback to Opus 4.8 when safety classifiers trigger), 2026-06-09, https://www.anthropic.com/news/claude-fable-5-mythos-5
Anthropic Cookbook, "Classifier fallback and billing for Claude Fable 5" (entire conversation pinned to Opus 4.8 after fallback; invisible retries when streaming is intercepted), https://platform.claude.com/cookbook/fable-5-fallback-billing-guide
GitHub, "[BUG] Fable 5 keeps downgrading to Opus 4.8" (anthropics/claude-code issue #67107), 2026-06-10, https://github.com/anthropics/claude-code/issues/67107
Decrypt, "Claude Fable 5 Isn't Nerfed. The Router Is Just Paranoid" (weights unchanged; degradation caused by more aggressive classifier rerouting), 2026-07-03, https://decrypt.co/
Anthropic, "Updating restrictions of sales to unsupported regions" (entities in unsupported regions with more than 50% direct or indirect Chinese ownership prohibited), 2025-09-04, https://www.anthropic.com/news/updating-restrictions-of-sales-to-unsupported-regions

Pokémon GO and military navigation

404 Media, "'Adding Amplitude to War Is Obviously an Issue,' Niantic Exec Says", 2024-11-25, https://www.404media.co/pokemon-go-data-adding-amplitude-to-war-is-obviously-an-issue-niantic-exec-says/
Businesswire, "Niantic Spatial and Vantor Announce Partnership" (official press release on unified air and ground positioning in GPS-denied environments), 2025-12-16, https://www.businesswire.com/news/home/20251216330019/en/
The Guardian, "Pokémon Go data trained AI that could assist military drones in war zones", 2026-06-12, https://www.theguardian.com/technology/2026/jun/12/pokemon-go-data-trained-ai-that-could-assist-military-drones-in-war-zones
Kotaku, "Pokémon Go, Niantic Spatial, Drone AI" (includes full Niantic denial statement), 2026-06-11, https://kotaku.com/pokemon-go-niantic-spatial-drone-ai-map-training-gps-2000705669
Niantic, "Building the Large Geospatial Model" (LGM official announcement and data scale), 2024-11-12, https://nianticlabs.com/news/largegeospatialmodel

Self-hosting and open-source trends

Anthropic Engineering, "A postmortem of three recent issues" (official admission that output quality degraded intermittently for several weeks in August–September), 2025-09-17, https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues
Epoch AI, "Open models lag SOTA closed models by ~4 months", 2026, https://epoch.ai/data-insights/open-closed-eci-gap
Baseten, "Announcing Baseten's $300M Series E" (dedicated deployment / self-hosted product positioning and funding trajectory), 2026-01-23, https://www.baseten.co/blog/announcing-baseten-s-300m-series-e/
Menlo Ventures, "2025: The State of Generative AI in the Enterprise" (counterargument data: closed-source still ~88% of enterprise token usage, open-source share fell to 12%), 2025-11, https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/
Xinhua, "Hundreds of Companies Compete in the LLM All-in-One Machine Market" (domestic self-hosting and all-in-one machine wave), 2025-05-27, http://www.news.cn/digital/20250527/

原文链接：https://guanjiawei.ai/en/blog/black-box-intelligence-privatization

The Strongest Model Just Dropped, and I’m Not 'Qualified' to Use It

guanjiawei — Mon, 29 Jun 2026 05:14:54 +0000

The day GPT 5.6 launched, I opened Codex to switch the model over. Muscle memory.

When 5.5 came out, it had quietly appeared in the model list overnight. One click and you were using it. The release and availability were practically the same thing. I’d gotten used to the rhythm: the moment the best model dropped, I could put it straight to work on the hardest thing I had.

This time I stared at the dropdown. No 5.6.

It had launched, all right. The specs were clear, three tiers, and the benchmarks were beautiful—the kind that make you want to test them against your hardest real-world task immediately. But you couldn’t select it. Reading further, I got it: this generation was going to only about 20 partners first. After that, every new customer had to be approved—by the U.S. government.

I’ve been on this frontier long enough that “release” meant “available.” This was the first time the two had been pried apart. And what pried them apart wasn’t something you could throw money at.

The Gate Didn’t Close; It Moved

A while back, Fable 5 got shut down worldwide by a single export control order. I wrote about it then, thinking the lesson was clear: geopolitics can cut off a closed-source frontier in one stroke.

5.6 showed me a subtler, nastier version.

Fable 5 was taken away from everyone at once—a clean cut. Painful, but at least fair. 5.6 is “available.” Whether you can use it just depends on whether you’re on the list. The gate didn’t close. It moved.

The old gate was money. The new one is power.

Money Is Actually the Most Reasonable Gate

Let me say a word in defense of money first, because three weeks ago I was still cursing it.

Back then, Fable 5 was pulled from subscriptions and switched to API billing. One task ran me four hundred dollars. I wrote a piece called “Intelligence Is Starting to Divide by Wealth,” arguing that access to the best intelligence had become a purely financial question. I was furious.

Looking back, money is a cold gate, but it has advantages that other gates don’t.

First, it doesn’t care who you are. Nationality, background—none of that matters. If you can pay, the door opens. Nobody sits behind it deciding you don’t belong.

Second, it pushes its own price down. When GPT-4 launched in March 2023, API prices were absurd: $30 in, $60 out per million tokens. But people still paid, because it was so far ahead there was no substitute. Professionals who needed it absorbed the cost without blinking. That premium was how OpenAI recouped its investment—using the fat profits from its lead to fund the next generation.

And leads expire. Seventeen months later, GPT-4o’s blended price fell to around four dollars—down nearly ninety percent. Expensive is temporary. Eventually everyone can afford it. That’s the norm.

The beauty of the money gate is this: it filters people, but it also takes itself apart. Today’s sky-high price is the down payment on tomorrow’s mass adoption.

The Power Gate Doesn’t Play Fair

The power gate has none of those advantages.

It discriminates. Twenty partners, approved one by one—meaning a few people sit behind that door, reading names off a list, deciding who gets in.

It doesn’t loosen on its own. Money gets beaten down by competition; power tightens with every generation. This generation it’s “20 partners.” Next time, maybe fewer. After that, maybe it stops being released to the public at all, becoming something like new missile technology that ordinary people never touch.

It also completely decouples “can you use it” from “can you afford it and use it well.” You might have the money and the skill, but without the admission slip, you’re already behind before the cards are dealt. A company falls behind not because its tech is worse, but because the approval didn’t come through. That’s the most absurd part.

Ironically, this gate doesn’t even pay off for the people holding the cards. Blocking users worldwide is like smashing with your own hands the lead premium that money had secured. If users can’t access it, no one validates your lead, no one pays for it, and the competition closes in fast. Once they’ve caught up, you can’t command the kind of premium GPT-4 did. The power gate devours the very economic engine that nurtures the next generation of models.

As for how this happened, my read is: the hype outran the real-world testing. The model was hyped so hard that policymakers started treating it like a weapon, and most of them had never actually used it—they’d only heard about it. The “too dangerous to release” narrative isn’t new. In 2019, OpenAI sat on the full GPT-2 model for over half a year. That same script has been playing out ever since.

I’m not saying safety shouldn’t be managed. It should. But a blanket ban by nationality is the laziest, most parochial move in the toolbox. Besides, given how fast things diffuse now, it probably won’t stop anything anyway.

A License That Changed, Then Was Forced Back

Speaking of which, I’m reminded of a story that has nothing to do with models but makes the exact same point.

Redis. The database almost every backend engineer has used, open-source for over a decade under the BSD license. Anyone could run it, modify it, build services on it, and sell them. In 2024, Redis suddenly changed its license to one that no longer counted as open source. In plain English: before, you could use it freely; now, if you want to use it commercially, you need my permission first.

This was essentially swapping the money gate for the power gate. What had been a market transaction became, overnight, “you must first obtain permission.”

But what happened next is the interesting part. Because Redis had been open source before the change, that last open-source version was sitting right there in plain sight. Anyone could continue building on it. Within weeks, the Linux Foundation took the lead, backed by AWS, Google, and Oracle, and forked Valkey directly from that version under the same open license. How fierce was the fork? Fierce enough that Redis itself couldn’t take the pressure, and in 2025 it changed the license back.

The whole point is this: you can’t truly take back something that’s already open source. The exit is always there.

The model world has the exact same structure. A closed-source frontier model is a switch you can’t reach. You build your whole business and workflow on it, and one “compliance requirement” can shut you down—with the government now holding that hand too. An open-weight model is something you can fork, deploy, and run on your own hardware. Slower, a tier behind—fine. The point is that this supply is in your own hands, not floating out of reach.

The hedge against the power gate was never more money. It’s open source.

Final Thoughts

That open weights can still keep pace with the frontier today depends on having a catfish in the tank.

DeepSeek is that catfish. When R1 dropped in January 2025 under an MIT license, completely free, it wiped out more than $589 billion of NVIDIA’s market cap in a single day—a record drop. It’s strong enough, and commercially weak enough, to open-source a version every so often, forcing everyone toward openness. To the model world, it is what Valkey is to Redis—the thing that keeps you from comfortably exploiting users.

My only worry is whether this catfish might get caged one day.

Open source is not permanent insurance. OpenAI started with “Open” too, going from sitting on GPT-2, to GPT-3 via API only, step by step becoming the most closed shop in the business today. An open model company has incentives to turn away, given enough time. While the catfish is there, the water stays alive. If it disappears, the pool quickly becomes dangerous.

So what this piece is trying to say is simple: don’t bet what matters most on a switch you can’t reach.

The best thing about this era was that the most cutting-edge intelligence could reach ordinary people early, and relatively fairly. Money is a gate too, but it’s a key anyone can get by saving enough, and it conveniently pays the down payment for the next generation’s adoption. The power gate is different: it sends out invitations, the list is held by very few people, and it only gets shorter.

Three weeks ago I was still complaining that intelligence was starting to be about money. Now I think: intelligence you can buy with money is already the relatively fair kind. What really keeps you up at night is the intelligence you can’t buy even with money—the kind you need someone’s nod to use.

The more intelligence is treated as power, the more precious is the bit you can hold in your own hands.

References

OpenAI, "Better Language Models and Their Implications" (Announcement on GPT-2’s staged release and withholding the full 1.5B model for over half a year under the pretext of "preventing misuse"), 2019-02, https://openai.com/index/better-language-models/
The Decoder, "From GPT-2 to Claude Mythos: The return of AI models deemed 'too dangerous to release'" (The "too dangerous to release" narrative from GPT-2 to today), https://the-decoder.com/from-gpt-2-to-claude-mythos-the-return-of-ai-models-deemed-too-dangerous-to-release/
OpenAI, "GPT-4" (March 2023 release, API pricing at $30 in / $60 out per million tokens), https://openai.com/index/gpt-4/
Andrew Ng on X (GPT-4 launched at roughly $36/M token; 17 months later GPT-4o dropped to roughly $4/M), 2024-08, https://x.com/AndrewYNg/status/1829190549842321758
Wikipedia, "DeepSeek" (R1 released on 2025-01-20, MIT license, free and open), https://en.wikipedia.org/wiki/DeepSeek
Yahoo Finance, "Nvidia stock plummets, loses record $589 billion as DeepSeek prompts questions over AI spending" ($589 billion wiped out in a single day, the largest single-day drop in history), 2025-01, https://finance.yahoo.com/news/nvidia-stock-plummets-loses-record-589-billion-as-deepseek-prompts-questions-over-ai-spending-135105824.html
CNN Business, "China's DeepSeek shook the tech world. Its developer just revealed the cost of training the AI model" (V3 training cost roughly $6 million; R1 paper disclosed roughly $294,000), 2025-09-19, https://www.cnn.com/2025/09/19/business/deepseek-ai-training-cost-china-intl
InfoQ, "Redis Returns to Open Source under AGPL License: Is It Too Late?" (The story of the 2024 SSPL change → Linux Foundation forks Valkey → 2025 reversion to AGPL), 2025-05, https://www.infoq.com/news/2025/05/redis-agpl-license/
HashiCorp, "HashiCorp adopts Business Source License" (August 2023: Terraform and others switch to BSL); OpenTofu, "OpenTofu Announces Fork of Terraform" (Community forks under the Linux Foundation), https://opentofu.org/blog/opentofu-announces-fork-of-terraform/

原文链接：https://guanjiawei.ai/en/blog/intelligence-by-permission

The Shortest Chain in History: From Tech to Profit in One Day

guanjiawei — Wed, 17 Jun 2026 05:26:07 +0000

A while back, our team was working on an optimization for the inference engine.

Exactly what we were studying doesn't matter—it was one of those grinding, unglamorous tasks: staring at the profiler for nearly three weeks, squeezing incremental gains out of scheduling and VRAM, and finally improving throughput by a few points.

What can a few points do? Back when I was starting out, almost nothing. But this time was different. One morning we merged it into main and rolled it out gradually; the next day we opened the dashboard and those points had already turned into real money. Same cards, same model, same customers—cost per token dropped by a few points, and gross margin widened by a few points. From the day it went live, it started showing up on the books.

A low-level optimization, from an idea in the head of some kid on the team born after 2000 to profit on the ledger, separated by a single night.

Five years ago, this would have been unthinkable.

Short Chain, Short Lifespan

Lately, whenever I talk to friends, the conversation keeps circling back to the same thought: we've stumbled into an unusual era.

What's unusual? Search history and you'll struggle to find another time when technical leadership turned into commercial influence, competitive advantage, or direct profit this fast. How short? A model company today basically has one job: train a smart enough model, ship it, provision compute for tokens, maybe open-source it. The rest—influence, revenue, valuation—grows on its own, fast.

The catch: the other end of this chain is just as short. Each generation's time in the spotlight is compressed to three to six months, and that's optimistic. Often, a model goes from hot topic to complete silence in one to three months.

How Fast the Wind Has Turned These Past Six Months

Last November, Google dropped Gemini 3 and slaughtered the leaderboards; even OpenAI declared a "Code Red" internally. Back then, open any group chat and the screen was full of Gemini worship. Six months later, the conversation has moved on. Not that Gemini is failing—users are still growing—but the spotlight on this track only stays lit for about three months.

Go back a bit further. When Claude Opus 4.6 dropped, insiders and outsiders alike thought "this is the thing that changes the world." At the time, it truly crushed the competition. Then 4.7, 4.8, and so on came along, praise and criticism trailing right behind.

OpenAI's story is even more dramatic. Its coding capabilities had always been mocked. I used the early Codex myself for a while and then unsubscribed—it was genuinely terrible. Then riding GPT-5.4 and 5.5, Codex felt like it swapped in a new engine: OpenAI's official numbers say weekly active users broke 5 million, a 6x increase since the desktop launch in February. One generation of models dragged a product everyone had written off out of the gutter.

The clearest example in China is Z.ai. A year ago its position was precarious, seemingly about to drop out of the race. Then GLM-4.5, 4.6, and 4.7 came out in quick succession, followed by GLM-5, 5.1, and 5.2 at the start of the year—three versions in three months—and the situation flipped completely. Six months after its Hong Kong IPO, its stock price rose roughly eightfold, market cap surging past 600 billion HKD. The technical reversal was written directly into the stock price.

MiniMax is the opposite case. At IPO its pricing and valuation were in the same ballpark as Z.ai's. Its stock price doubled on day one, market cap briefly hit over $13 billion, and during the March surge it even briefly surpassed Baidu's Hong Kong-listed shares. But the wind turned just as fast: the M2.7 and M3 generations didn't catch the hype, market expectations immediately took a discount, and market cap fell back sharply from its peak. The speed at which they hype you is the same speed at which they abandon you.

After all the noise, attention converged on two things: coding and multimodality. Traditional valuation logic—users, revenue structure, moats—basically fails here. Everyone is really only asking one question: is your current generation of models strong enough.

In My Line of Work, This Chain Used to Be Terrifyingly Long

The thrill of "tech turning into money in days" exists because I know exactly how excruciating it used to be.

I do infra, work tied tight to infrastructure. In the past, if you made a cluster-level innovation that improved inference efficiency by 15%, turning that 15% into a commercial advantage was hard enough to make you quit.

Why so hard? You can't directly price that 15%. You can't tell a customer, "The machine used to cost a million, now it's 15% faster, so I'll sell it to you for 1.15 million." That's not how they calculate. They'll drag you into their TCO model, into their risk structure, haggling: how do you prove it's 15%, could it actually be 5%, who guarantees stability, who bears supply chain volatility.

So a low-level optimization, to reach the point where customers actually pay for it, is separated by long cycles, complex supply chains, and a massive business team. You have to maintain an entire organization, grinding slowly at the far end of the commercial chain, to grind technology into profit and scale. A tweak at the bottom layer takes forever to echo back from the market.

Worse, models have short lifespans. You toil away optimizing for a particular model generation for months, and by the time you're done, its moment in the spotlight has already passed. The investment hasn't broken even, but the target has already disappeared. So inference infra used to be stuck in an awkward spot: everyone believed it would matter down the road, but nobody could see a clear business model right now.

Tokens Are Spot Goods, Not Futures

Because AI coding ignited token demand, squeezing every bit of slack from the chain until it feels almost unreal.

The key is the nature of tokens: they settle spot against daily capacity. Unlike traditional goods—design today, produce tomorrow, ship the day after—tokens are computing right now, results sent out within seconds. This single trait rewrote all the rules.

The few points we earned from those three weeks of research start settling the moment they go live: from the next day, the extra capacity squeezed out of daily production is booked directly as additional gross margin and competitive advantage. No waiting for the next fiscal year, no entering someone's TCO model, no maintaining a team to prove how much it's worth. Capacity grew, real cost per token dropped, and the books looked better that same day.

And it replicates with almost zero friction. This kind of optimization doesn't discriminate by region—same type of compute, same model, same customers. It's basically a straight port, spreading extremely fast.

Measured in days, a low-level technical improvement converts directly into commercial returns, skipping that entire long, heavy system in between. This is probably the shortest chain from technology to profit in history.

This Is the Era of the Young

There's another thing I keep thinking about, and it amuses me more and more: the people doing this on the front lines are mostly kids in their early twenties.

This era is unusually kind to them, because its judging criteria are brutally objective: did efficiency rise or fall, did accuracy change. Put the thing on the table and one test settles it, leaving almost no room for "seniority" or "connections." You don't need any veteran's nod, nor do you need to be good at reading people or playing politics. Whether an industry's senior judges stamp their approval on you doesn't matter here. Make something real, and it writes itself plainly on production efficiency.

A twenty-something can create value worth hundreds of millions, even billions of dollars, with a single technical breakthrough. Fast enough to see results immediately, hard enough to be verified by anyone. The entire chain no longer needs to be stuffed with so many people whose sole job is to "judge whether you're qualified."

In Closing

The era we've caught is global, its tempo terrifyingly short, and it will likely only grow more brutal from here.

But it has also genuinely deleted those long, heavy middle layers, along with those people whose entire job is to judge you. What remains is technological innovation, and the person who creates it.

The fewer people in the middle, the more valuable the ones doing the work.

References

Google, "Introducing Gemini 3", 2025-11-18, https://blog.google/products/gemini/gemini-3/
Fortune, "Sam Altman declares 'Code Red' as Gemini 3 surges", 2025-12-02, https://fortune.com/2025/12/02/sam-altman-declares-code-red-google-gemini-ceo-sundar-pichai/
OpenAI, "Codex is becoming a productivity tool for everyone" (Weekly active users exceed 5 million, 6x growth since February), 2026-06-02, https://openai.com/index/codex-for-knowledge-work/
OpenAI, "Codex for (almost) everything" (3 million weekly active users milestone), 2026-04-16, https://openai.com/index/codex-for-almost-everything/
OpenAI, "ChatGPT — Release Notes" (GPT-5.4 / 5.5 release notes), accessed 2026-06-17, https://help.openai.com/en/articles/6825453-chatgpt-release-notes
Securities Times, "'Global Large Model First Stock' Z.ai Exceeds 57 Billion HKD Market Cap on IPO Day", 2026-01, https://www.stcn.com/article/detail/3580246.html
Finet.com.cn, "[IPO Tracking] Z.ai (02513.HK) High Growth Ignites Hong Kong Stocks, Shares Rise 31% to New High", 2026-04, https://www.finet.com.cn/news/69cc927d2308294c69bf7bec.html
Z.ai, "Z.ai Releases GLM-4.7", PR Newswire, 2025-12-22, https://www.prnewswire.com/news-releases/zai-releases-glm-4-7-designed-for-real-world-development-environments-cementing-itself-as-chinas-openai-302649821.html
MarkTechPost, "Z.ai Launches GLM-5.2 With a Usable 1M-Token Context", 2026-06-14, https://www.marktechpost.com/2026/06/14/z-ai-launches-glm-5-2-with-a-usable-1m-token-context-two-thinking-effort-levels-and-no-benchmarks-at-launch/
Reuters, "MiniMax doubles in value in Hong Kong debut", 2026-01-09, https://www.reuters.com/world/asia-pacific/china-ai-firm-minimax-set-surge-hong-kong-debut-2026-01-09/
Securities Times, "Stock Price Surges 51% in Two Days, MiniMax Market Cap Successively Surpasses Three Internet Giants", 2026-03-11, https://www.stcn.com/article/detail/3670887.html
MiniMax, "MiniMax-M2.5" (SWE-bench Verified 80.2), GitHub, 2026-02, https://github.com/MiniMax-AI/MiniMax-M2.5

原文链接：https://guanjiawei.ai/en/blog/shortest-chain-to-profit

Top-Tier Intelligence, Cut Off Overnight

guanjiawei — Mon, 15 Jun 2026 09:50:26 +0000

I set Fable 5 on a long task and let it run, telling myself I'd check back in a few hours. When I returned, the model doing the work wasn't Fable 5 anymore. It was Opus 4.8. It had downgraded itself behind my back.

I stared at the results. My first thought wasn't whether it had done a good job, but whether the output was even usable. I had set my expectations for Fable 5-level work. When a weaker model took over halfway through, those first few hours were suddenly up in the air.

Later I learned this was an official mechanism. Fable 5 was Anthropic's strongest model at the time, priced at twice Opus 4.8. But it came with a safety classifier. Once the classifier flagged your question as touching on sensitive areas like cybersecurity or biochemistry, it would automatically hand that round off to Opus 4.8. The official trigger rate was under 5%, but I was building an inference engine, working with low-level code and system calls every day, so I was bound to get caught in the crossfire. One false positive, and the whole session dropped a level.

Gone in Three Days

The silent downgrade was merely irritating. What really left me speechless was Fable 5 disappearing entirely just days later.

It had been out for only three days when the U.S. Commerce Department handed Anthropic an export control order: add the model to the export control list under national security provisions. The trigger was someone jailbreaking it and using it to dig up software vulnerabilities. The order wasn't worded as "don't sell to China," but as "do not provide to any foreign national." Even foreign employees working in the U.S. were covered. Anthropic said the only way to stay compliant was to shut it down for everyone, worldwide. So allies like South Korea and the UK were cut off too.

This proved something I'd told my team before: with overseas top-tier closed-source models, use the best ones you can get, while you can. That path only gets narrower, and there's no going back. I thought the squeeze would come gradually. Instead, it was three days post-launch, one piece of paper, and cut off just like that.

Software as Munitions Is Nothing New

Treating intelligence as a strategic resource sounds like something new. It's actually the same old story.

In the 1990s, the U.S. placed strong encryption algorithms on the munitions list, regulating them as arms exports. Civilian encryption was deliberately weakened until it was practically useless. PGP creator Phil Zimmermann was investigated by federal authorities for three years for "munitions export without a license" after putting his encryption software online. The logic was identical: once a piece of software becomes capable enough, it stops being software. It becomes a weapon.

Fable 5 got the same judgment: it was too good at finding vulnerabilities, so it was reclassified from "commercial product" to "cyberweapon."

But the encryption fight had a more interesting ending. The controls didn't work. Strong encryption spread globally through open source, and walls couldn't stop it. By 2000, the U.S. had no choice but to loosen its grip. Any capability that is useful enough and can be replicated will be slowed by regulation, but never stopped.

The Replacements Are Already at the Door

This time, the replacements showed up almost the same week.

The day after Fable 5 was shut down, Zhipu pushed out GLM 5.2. The timing was almost too perfect. Before that, MiniMax had released M3, and Kimi had released K2.7 Code. All of them were open source or open-weight. GLM 5.2 got great word-of-mouth. People mostly complained about how slow it was; few questioned its quality. This matched my own experience using it: infrastructure and GPU shortages made serving it a struggle, but the model itself was genuinely right at the top tier.

There's another angle. Silicon Valley's top AI talent costs have skyrocketed, and buying GPUs is restricted. Yet in this environment short on both people and compute, domestic players built models that benchmark against the top tier, and open-sourced them.

Today, Chinese-made models account for the largest share of global open-source model downloads, 17.1% to the U.S.'s 15.8%. Doing all that with a fraction of the competition's resources, and still releasing the weights, is genuinely impressive. Zhipu's recurring slogan this time was "open": intelligence should belong to everyone.

This is no longer about which model is stronger. The White House's AI action plan last year opened with "America is in a race to achieve global dominance in AI." When a nation puts intelligence at this level, cutting off others' access becomes the obvious play. The Industrial Revolution already played out this script: once one country's productivity exceeds another's, it can do almost as it pleases. Intelligence is the foundation of the next wave of productivity. That card will only get heavier.

Use It While It's Open

Two things.

First, use what's open while you can, and put it to work where it counts. Top-tier intelligence now behaves more like a quota that expires. You never know when someone will flip a switch. While you still have access, give it the hardest problems.

Second, don't bet your entire intelligence supply on a switch you can't reach. Locking your workflow to a single closed-source model means one compliance ruling can stop you cold. You need a fallback: open-weight models, deployments you control. Even if they're slower or weaker, they'll still be there when it counts.

Closing Thoughts

Fable 5's silent downgrade felt like a lousy product experience at the time. In retrospect, it was a preview: the intelligence you think you hold securely can be quietly swapped out, or simply taken away, at any moment.

Models will keep getting stronger generation after generation. But whether you can actually use the strongest one is less and less a technical problem.

The more intelligence is treated as a strategic resource, the more valuable access to it becomes.

References

Anthropic, "Claude Fable 5 and Claude Mythos 5" (Official launch announcement, including the mechanism for automatic downgrading to Opus 4.8 upon safety classifier triggers), 2026-06-09, https://www.anthropic.com/news/claude-fable-5-mythos-5
Anthropic, "Statement on US government directive" (Official shutdown statement), 2026-06-12, https://www.anthropic.com/news/fable-mythos-access
The New Stack, "US Gov Orders Anthropic To Pull Fable 5 and Mythos 5 Three Days After Launch" (Mechanism: export control order, not executive order; compliance logic behind global shutdown), 2026-06, https://thenewstack.io/us-gov-orders-anthropic-to-pull-fable-5-and-mythos-5-three-days-after-launch/
Fortune, "Anthropic disables Fable and Mythos under export controls" (Commerce Secretary Lutnick's letter to Amodei; Opus 4.8 unaffected), 2026-06-13, https://fortune.com/2026/06/13/anthropic-disables-fable-mythos-export-controls-national-security-threat/
InfoQ, "Anthropic Releases—and Then Suspends—Claude Fable 5 and Mythos 5" (Release specs, pricing, and suspension timeline), 2026-06, https://www.infoq.com/news/2026/06/claude-5-release/
TechCrunch, "Anthropic releases Opus 4.8 with new Dynamic Workflow tool", 2026-05-28, https://techcrunch.com/2026/05/28/anthropic-releases-opus-4-8-with-new-dynamic-workflow-tool/
Zhipu Z.ai, GLM-5.2 API Documentation / Release Notes, 2026-06-13, https://docs.z.ai/devpack/latest-model
MiniMax, "MiniMax M3 is officially released" (Official blog), 2026-06-01, https://www.minimax.io/blog/minimax-m3
Moonshot AI, "Kimi-K2.7-Code" (HuggingFace model page, Modified MIT open source), 2026-06-12, https://huggingface.co/moonshotai/Kimi-K2.7-Code
Science and Technology Daily / People's Daily Online, "China-developed open-source AI models account for 17.1% of global downloads, ranking first in the world", 2025-12-08, http://finance.people.com.cn/BIG5/n1/2025/1208/c1004-40619459.html
The White House, "America's AI Action Plan" (Opening line: "a race to achieve global dominance in AI"), 2025-07, https://www.ai.gov/action-plan
U.S. Bureau of Industry and Security, "Commerce Announces Rescission of Biden-Era AI Diffusion Rule" (AI diffusion rule rescinded; control focus shifts to chips), 2025-05-13, https://www.bis.gov/press-release/department-commerce-announces-rescission-biden-era-artificial-intelligence-diffusion-rule-strengthens
Wikipedia, "Export of cryptography from the United States" (History of 1990s encryption export controls and the PGP case), https://en.wikipedia.org/wiki/Export_of_cryptography_from_the_United_States

原文链接：https://guanjiawei.ai/en/blog/intelligence-as-strategic-resource

When the Harness Is a Mess, Restart

guanjiawei — Fri, 12 Jun 2026 08:00:58 +0000

I mentioned earlier that I'd been using an agent to build an inference engine from scratch. By the third week, I realized something: this project was busted.

Not the "won't compile" kind of broken. Documentation was exploding. Experimental scripts from seven or eight directions were crammed into one repo, benchmark results scattered across a dozen markdown files. Every time the model started work, it spent half a day just figuring out where it had left off. I watched it flit between directions, scratching the surface of each before moving on. Give it more guidance and it obediently followed along; give it less and it just spun in place.

That wasn't its usual level. Two days earlier, it had spent twelve hours connecting a 78-layer pipeline from scratch.

Lost in the Chaos

I kept running into this same situation, and the conclusion was always the same: once the context turns into chaos, the model can't find its way out. I almost never saw it climb out on its own. Every time, I had to tear the whole thing down.

This isn't gut feel; someone measured it. Microsoft and Salesforce simulated over 200,000 multi-turn conversations last year, testing 15 models. The conclusion was right there in the abstract: once an LLM takes a wrong step in conversation, it gets lost and won't recover. Average performance dropped 39%. The breakdown is more interesting: capability only dropped 16%, but unreliability shot up 112%. The model didn't get much dumber; it became wildly unstable. On the same question, sometimes it answered great, sometimes terribly. That matches exactly what I saw: it's not that it couldn't do it, it just couldn't perform.

Chroma's Context Rot report landed another blow: even for a task as simple as fishing one sentence out of a pile of text, the longer the input, the more all 18 models' performance dropped. And the drop wasn't uniform. Context is a finite resource. Anthropic calls it the attention budget. Every extra token you stuff in burns part of it.

The messier your repo, the faster this budget burns.

After the Restart

The solution was always the same: stop, figure out what the goal actually is, open a new repo with fresh context, write down this one goal and the execution path clearly, and start over. Note: "this one." After a restart, you can basically only lock onto one direction. Running multiple lines in parallel is a luxury at this point.

The change is staggering. The same model, spinning in a mess the day before, acts like a different species after the restart: digging deep, actively hunting for improvements, pushing forward with efficiency that isn't even on the same scale. My rough estimate is a three- to fivefold difference, and it's probably not even linear. In a chaotic environment it might never reach the goal. In a clean environment it's genuinely converging on the target.

The Microsoft paper's advice for users boils down to two things: if you have time, restart the session; before you do, have the model consolidate what it knows and carry it over. They tested it: merging information scattered across multiple turns into a single turn and re-feeding it restores performance to 95% of a single-turn session. Anthropic does the same thing with its multi-agent research system: when context is almost full, spin up a new agent with clean context and hand off properly.

The industry calls this compaction, or context engineering, or whatever. The name doesn't matter. It all comes down to one thing: whatever environment you give the model, that's the performance you get.

Rewriting Is Suicide, Restarting Is Not

There's an old iron law in software engineering, written by Joel Spolsky over twenty years ago: never rewrite from scratch. Netscape decided to rewrite its browser, went three years without a major release, and handed the market to IE. This iron law ruled the industry for two decades.

But it rests on a premise: rebuilding costs are extremely high. A team rewriting for three years while competitors don't stop and wait.

Agents eliminate rebuilding costs. New repo, new context, re-clarifying the goal. Half a day's work. Take the conclusions and pitfalls from before, organize them into documents, and carry them over. When rebuilding drops from three years to half a day, the iron law flips: patching a rotten context is far more expensive than restarting.

So the chaos of exploration that looks wasted isn't actually wasted. The clear goal at restart grows directly out of that first round of random bashing: which assumptions held, which directions closed off. Without that chaos, you couldn't write that clarity. Code can be thrown away; cognition is carried forward.

The Weight of the Harness

The word "harness" has burned its way from the evaluation community to the engineering community over the past two years. METR wrote scaffolding into its evaluation methodology back in 2023; this February OpenAI published a piece on Harness engineering, redefining an engineer's job as "designing the environment, expressing intent, and building feedback loops"; in April, Martin Fowler's site gave the most concise definition: the harness is everything in an agent except the model itself.

How heavy is that weight? The Terminal-Bench 2.0 leaderboard spells it out: the same Opus 4.6, with different harnesses, scores from 58 to 76. That's a gap of 18 percentage points. Under the same harness, swapping GPT-5 for GPT-5.2 only raises the score by 19 percentage points. Good versus bad harness design is the difference of a full model generation.

Addy Osmani put it crudely but accurately: an average model with a good harness can beat a top-tier model with a bad harness.

My own version is even cruder: the environment you give the model is its ceiling.

Human Value Is Rising

There's another feeling that's been growing stronger these past few months: humans are becoming more valuable.

Deciding whether to tear it all down and restart is on the human. Converging from a pile of messy exploration results to "what is the real problem to solve" is also on the human. I've seen too many times these past few months where an agent can't find its way out of fuzzy, complex problems. There's a line in that OpenAI blog post I strongly agree with: Humans steer. Agents execute.

So an agent isn't a wishing machine; it's a very strong employee. If you lead well, it can produce outstanding results; if you toss the work and walk away, it probably won't produce much. Models keep getting stronger; Fable 5's engineering capability took another step up, and every generation pushes the boundary you can cross further out. But the "leading" part: no one can do that for you yet.

Closing Words

When it's a mess, restart. These five words are the most valuable lesson I've accumulated from using agents on complex projects these past few months.

Models will keep getting stronger. But the environment is yours to provide, and the decision to tear it all down is yours to make.

The cheaper intelligence becomes, the more valuable clarity is.

References

Laban et al., "LLMs Get Lost In Multi-Turn Conversation", arXiv:2505.06120, 2025-05, https://arxiv.org/abs/2505.06120
Chroma, "Context Rot: How Increasing Input Tokens Impacts LLM Performance", 2025-07-14, https://research.trychroma.com/context-rot
Anthropic, "Effective context engineering for AI agents", 2025-09-29, https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
Anthropic, "How we built our multi-agent research system", 2025-06-13, https://www.anthropic.com/engineering/multi-agent-research-system
Terminal-Bench 2.0 Leaderboard, https://www.tbench.ai/leaderboard/terminal-bench/2.0 (retrieved 2026-06-12)
OpenAI, "Harness engineering: leveraging Codex in an agent-first world", 2026-02, https://openai.com/index/harness-engineering/
Birgitta Böckeler, "Harness engineering for coding agent users", martinfowler.com, 2026-04-02, https://martinfowler.com/articles/harness-engineering.html
Addy Osmani, "Agent Harness Engineering", O'Reilly Radar, 2026-05-15, https://www.oreilly.com/radar/agent-harness-engineering/
Joel Spolsky, "Things You Should Never Do, Part I", 2000-04-06, https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
METR, "Evaluating Language-Model Agents on Realistic Autonomous Tasks", 2023-08, https://metr.org/blog/2023-08-01-new-report/

原文链接：https://guanjiawei.ai/en/blog/restart-when-lost

Intelligence Is Starting to Be About Wealth

guanjiawei — Thu, 11 Jun 2026 09:25:32 +0000

The day after Fable 5 dropped, I threw it at a research project that had been stuck for two weeks.

Inference engine performance optimization. A 78-layer model running on hardware, baseline 10 token/s, target 100+ tps. GPT 5.5 had already sunk a lot of hours into it without much luck.

It ran for 15 hours. API bill: $420. Performance went from 10 to 13 token/s, a 30% boost. Still far from 100, but it moved.

Two Approaches

I had GPT 5.5 working the same project in another worktree, instructions basically identical. The two sides tackled it very differently.

GPT 5.5 started by scaffolding with just two layers. Out of 78 total, it first got the pipeline running on two, then dug in, running hundreds of rounds on a single point, squeezing every drop of local speed before scaling up. Steady. Slow. Following the roadmap one step at a time.

Fable 5 did the exact opposite. It spent an hour or two feeling out the single-layer ceiling, then stitched all 78 layers together and went straight for an end-to-end result.

Its engineering was on another level. It built a runnable inference engine from scratch in just over ten hours, a level of completeness previous models couldn't match. But 30% is nowhere near the target, and burning a few hundred dollars a pop isn't sustainable. I ended up switching back to GPT 5.5.

But that's not the point of this post.

An Order-of-Magnitude Gap Between Subscription and API

At launch, Fable 5 was put into Pro and Max subscriptions, counted against quota at twice the Opus 4.8 rate, for a two-week window. After June 22 it was pulled from subscriptions. To keep using it you had to buy usage credits at API rates.

API pricing: $10 input, $50 output, per million tokens.

Max 20x runs $200 a month. Before, one Max 20x seat was enough for Opus 4.8 on complex tasks. Fable 5 at 2x burns through quota faster, but you could still pick which tasks to spend it on. On API, the math changes entirely. A three-hour subtask costs $150; the full project ran $420. That's two months of Max fees.

From Opus 4.5 to 4.6 to 4.8, every upgrade just tweaked the subscription quota ratios. The best model was always included. $200 a month got you access. Now it's not a ratio tweak. It's a straight removal.

Most people can't afford to burn hundreds on a single task.

Anti-Distillation

Anti-distillation is where things get really twisted.

I get not wanting to be distilled. Fable 5 has a built-in two-layer classifier: requests flagged as distillation, cybersecurity, or biochemistry automatically fall back to Opus 4.8. Anthropic says the trigger rate is under 5%.

In reality? People on Reddit and Twitter are saying a routine bloodwork interpretation got flagged as biochemistry, or normal questions got silently downgraded. Distillation is just intensive model use. Once the model is publicly available, you can't fully stop it. Using an opaque classifier to guess intent? Getting it wrong is only a matter of time.

The worst part is that the downgrade is silent. Answer quality tanks out of nowhere, and you think that's just the model, but you're actually getting Opus 4.8. You're paying Fable 5 prices for Opus 4.8 output, and they don't even tell you. I find this more disgusting than the pricing issue.

The Other Side

OpenAI has been doing something different lately. In March they launched Codex for Open Source: six months of free Pro for maintainers of GitHub projects with 1000+ stars. It goes to individuals writing code in the community, not enterprises.

The scale isn't massive. But the direction is toward letting more people use the good stuff, not locking it away.

The two are going head-to-head; who's right or wrong is anyone's guess. But one pulls the best model from subscriptions and silently downgrades users, while the other hands out free accounts to open-source contributors. You tell me. Do those two signals look the same to you?

The World Is at a Crossroads

Anthropic has elitism in its bones. AI constitution, moral charter—the intentions aren't bad, but who gets to define "responsible"? It's always that tiny group, deciding from the top down. The business strategy follows the same path: those with budgets get the best, those without get second tier, with a tenfold gap between.

You can charge a premium for the strongest model. But keeping it in subscriptions lets individuals and small teams allocate their own quotas. Pulling it out and putting it on API changes the signal completely.

A model company's pricing is its stance on how intelligence should be distributed. This fracture is bigger than Anthropic realizes.

References

Anthropic, "Introducing Claude Fable 5 and Mythos 5", 2026-06-09
Claude official pricing: Fable 5 input $10/M token, output $50/M token; GPT-5.5 input $5/M token, output $30/M token
Fable 5 subscription availability: June 9 to 22, after which usage credits at API rates are required
Fable 5 anti-distillation mechanism: two-stage classifier detection, falls back to Opus 4.8 when triggered, official trigger rate claimed below 5%
Fortune, "Anthropic accused of secretly limiting Claude Fable 5 capabilities", 2026-06-10
OpenAI, "Codex for Open Source" program, 2026-03-07, providing 6 months of free Pro accounts to maintainers of 1000+ stars projects
SWE-bench Pro benchmark: Fable 5 scored 80%, GPT-5.5 scored 58.6%

原文链接：https://guanjiawei.ai/en/blog/intelligence-wealth-divide

Models Keep Getting Stronger, but 'Strongest' Has No Single Answer

guanjiawei — Wed, 03 Jun 2026 10:09:05 +0000

June is shaping up to be another packed month for model releases. Opus 4.8 dropped at the end of May, MiniMax's M3 landed a couple of days ago, GPT 5.6 is supposedly around the corner, and some are already waiting on DeepSeek's next drop. It looks like we'll see a new model every few days. Pretty lively.

But for the past two days, what I've actually been thinking about is a friend's experience with models.

He started out using models to build small things—writing web pages, making little tools and plugins. He was pretty excited at first, telling me how amazing today's models are. He more or less picked a decent domestic model at random and found it more than enough. He couldn't even imagine where else models could get stronger; they already worked so well.

Then his work got more complex. He moved from small tools to trying to build an auto-editing tool, video cropping and the like. That's when the problems started.

The model told him it was done. He said okay, tried it, no dice. A moment later it said this time it was really done. He tried again, still no good. Back and forth for several rounds.

He couldn't tell anymore: part of him felt he was getting better at working with the model, that he needed to give more guidance and try different approaches; part of him started wondering if the model itself just wasn't cutting it, whether he should switch to something like Claude Opus.

This pattern is all too common. Behind it is something a lot of people haven't caught on to yet: model strength is forking in different directions. A single score no longer tells the whole story.

Scores Bunch at the Ceiling, Real-World Feel Diverges

First, the weird state of things: on mainstream benchmarks, top models score terrifyingly high, all squeezed into a narrow band.

Take GPQA. It's graduate-level, so hard that the PhD experts they brought in only scored around 65%. Yet top models now routinely hit 92% to 94%, bunched together. Older benchmarks like MMLU were surpassed by pretty much everyone long ago, all scoring over 90%. Hard problems aren't hard anymore; scores have hit the ceiling, and you can't tell models apart.

So benchmark makers have to keep inventing harder tests. The new Humanity's Last Exam states it plainly: it was created because models had exceeded 90% on MMLU and the old questions weren't enough anymore. One study looked at sixty mainstream benchmarks and found that nearly half are already highly saturated—top models are "statistically indistinguishable" on them.

But when you actually use them, the difference in feel is absurd. I wrote in my last post how Opus 4.8 kept letting me down on engineering and research tasks—work that later all moved to GPT 5.5. By the scores, the two are close; in practice, worlds apart.

The ARC-AGI suite is a perfect example. On the old version, top models had already saturated at 96%. Switch to the harder ARC-AGI-2, and the same models immediately show their true colors: GPT 5.5 still manages 85%, while Opus 4.8 drops to barely over 70%. Switch again to ARC-AGI-3, which requires actual interaction and exploration, and almost everyone flatlines to zero.

So benchmarks are still useful. It's just that "making tests humans can define and grade" is becoming less and less able to distinguish models. To understand why, you have to look at training.

Solving the Hardest Problems vs. Reliably Doing Messy Work

The main technique making models stronger right now is called "verifiable rewards." In short: pick hard problems with standard answers that machines can automatically grade, and use them for reinforcement learning. Math and code are the classic examples. Right answer gets points, wrong gets zero, rinse and repeat.

The DeepSeek-R1 paper puts it clearly: math problems are verified with rules, code is thrown straight into compilers to run test cases. They specifically note they avoided neural-network-based reward models, because those are too easy for models to game. OpenAI's o series follows the same playbook. It's highly effective; this is exactly how models learned to solve hard problems.

But it has one characteristic: what it excels at is taking the "hardest problems humans can define and grade" and grinding them out. That's an entirely different capability: give it a fuzzy, not-that-hard but very real task, and get it done reliably in one go.

My friend's editing tool is the latter. The task isn't extremely difficult, but the intent is fuzzy. It has to be broken down yourself, and it has to be done cleanly in one go. A model that can solve Olympiad problems may not handle this kind of messy work cleanly in one shot. It might go in circles, need three rounds of back-and-forth, and finally say "I'm done" when it isn't. Conversely, a model that's great at messy work might completely choke when you throw a really hard problem at it.

These are two directions of capability, each going its own way. You can't rank them on a single line.

The trouble is, ninety percent of people need the latter in daily life. They want to take a poorly specified task and get it done reliably, without hassle. Yet the score we use to rank models measures almost entirely the former. It's completely normal for "highest score" and "most useful for me" to not line up.

Another Dimension: Exploration

The first two types are still in the world of standard answers: either solve a gradable hard problem, or finish a verifiable task. The truly difficult one is the third.

When my friend got stuck, I thought of another class of problems. Like driving toward an intersection with a traffic light ahead. Do you go straight, or weave through the middle? There's no standard answer; you have to find your own direction in the ambiguity. Exploring a domain people haven't clearly defined, or don't even know the answer to, is a completely different capability.

This capability, benchmarks can't measure at all. The whole premise of evaluation is having a standard answer, something gradable right or wrong. But exploration has no right or wrong at all, only efficiency. Can you fish out something new in the ambiguity, use it to move forward, and push out a boundary that didn't exist before?

It's also precisely the blind spot of the "verifiable rewards" approach. Research has already pointed out that open-ended tasks without unique answers have no clear standard answer to begin with, so you can't even construct rewards. This method can't get traction. Some have even found that this training approach doesn't necessarily give models new capabilities, and may instead narrow their exploration surface, with capability ceilings hard-capped by the base model.

The result is that a model great at exploration, thrown into a cage with clear standard answers, might seem a bit stupid. A model that tests incredibly well may not possess exploration ability at all. In my own experience, GPT and Claude show the clearest difference on this dimension.

And this dimension happens to be the most important. Because truly valuable things often start without standard answers. Yet it's the hardest to measure, and the hardest to train.

The Chat Era Already Ran This Course

Model capabilities forking along dimensions and layering down isn't new. The chatbot era ran the full course.

Back then, everyone also thought for a while that the biggest model was the strongest. But they quickly discovered that for chatting specifically, the biggest model wasn't much better. In 2023, the LMSYS Chatbot Arena leaderboard dedicated a section to "Smaller Models Are Competitive": a 13B Vicuna ranked in the top five, its Elo score even beating Google's PaLM 2. 7B models also squeezed into the top ten, trading blows with models twice their size.

Later studies echoed this: scaling models from tens of millions to hundreds of billions, all the way up to GPT-4 class, showed gains topping out quickly on softer tasks. Models with a few tens of billions of parameters weren't far from frontier models.

In other words: for chatting, for emotional support, the marginal returns to scale are low. A few tens of billions is enough; scaling up to hundreds of billions is pure waste.

So the market sorted itself out. If you want emotional value, someone to chat with, a smaller model that sounds human is enough. Only when you need serious research or hardcore engineering do top-tier models come into play. Models sorted themselves into different price-performance tiers by use case.

Today's round is the same plot, replaying at a higher capability level.

Closing: Don't Ask Which Is Strongest—Ask Which Dimension You Need

Back to my friend's dilemma: "should I switch to a stronger model?" He's asking the wrong question.

There is no "stronger" that simultaneously covers solving hard problems, doing messy work, and exploring. These three things are diverging onto different models.

New-generation models are still pushing forward, of course. But the progress they fight for increasingly lands on "the hardest problems humans can define and grade," which is exactly where most people can't perceive it. So you see a split: leaderboards keep getting stronger generation after generation, yet most people just feel "it's been good enough for a while, can't see where it's stronger." Neither side is wrong. Because what they want are fundamentally different dimensions of capability.

So stop vaguely asking "which model is strongest." First ask clearly: which dimension of work do you need it to do? Solve a hard problem with an answer, finish a messy task that wasn't clearly specified, or join you in something no one knows the answer to yet.

"Strongest" is becoming a question without a standard answer.

References

Model Releases and Timeline

Benchmark Saturation

Verifiable Rewards, and Its Limits

Small Models in the Chat Era

原文链接：https://guanjiawei.ai/en/blog/strongest-no-single-answer

Claude 4.8 Let Me Down, But It’s Not Just Claude’s Problem

guanjiawei — Mon, 01 Jun 2026 11:12:47 +0000

The day Claude launched Opus 4.8, I was pretty excited.

I'd always thought Opus had solid engineering skills. Macro analysis and intent alignment were its strong suits. A few points in the 4.8 release notes caught my eye, so I tried it out on several complex tasks I had going.

The results were one disappointment after another. Here are a few examples.

Looks Promising Out of the Gate, Then Goes Off the Rails

Right around then, my Codex credits had run out. Since 4.8 looked solid on paper, I put it on a research task using that flashy feature called ultracode, a dynamic workflow that supposedly auto-orchestrates ultra-long-horizon tasks.

It started out looking solid. It ran a bunch of checks and got everything set up. Seemed reliable. That's Claude Code in a nutshell: the start always gives you hope.

Then I let it run for about a full day and night.

The task was to optimize a performance metric. The baseline was terrible: throughput sat around 0.1 tokens per second. After optimizing for a while, it bumped the number from 0.1 to 0.15 and started celebrating. Look, I improved it by 50%! I did all this work! What a huge achievement! It kept hauling out that basic initial setup to claim credit, writing pages of self-congratulatory fluff.

The problem is, in the 0.1 to 0.15 range, using multiples to understand the problem is wrong to begin with.

When performance is that low, your direction is completely wrong. You have to look at absolute values. What does 0.15 token/s actually mean? That's how you see how far off it is. Celebrating a "50% improvement" in a setup that fundamentally can't run is like celebrating that you bailed two buckets from a sinking ship.

Looking back, the pile of documentation it recorded, those so-called results, pretty much all had to be tossed. The direction was wrong. Freeze everything and start over.

This wasn't even the most surprising part. I'd always had the impression that Claude's macro capabilities were fine, good at analysis and alignment, but concrete execution, especially on research tasks, tended to go sideways. The real red flag was the engineering task that followed.

A Pure Engineering Problem, and It Was Clowning Around

It was a cloud service. A probe was reporting errors, and I asked it to diagnose the issue and fix it while it was there. Nothing complicated.

A couple of things it did left me baffled.

First, in the middle of analysis it randomly started counting. It had the machine echo a string of numbers. I stared at the screen, maybe a dozen in total. I still have no idea what that was about. Pure token burning.

Second was even more absurd. The probe's purpose was straightforward: workers send periodic heartbeats returning OK to tell the platform "I'm alive and working." If the heartbeat is abnormal, take the worker offline first. Don't assign it tasks. It discovered heartbeats had a cost and asked if I wanted to cut it. I said sure.

It changed it to runtime --version, which returns a version number.

I laughed out loud. This isn't cutting costs. It's destroying the design intent entirely. A version number only proves the thing is installed. It has nothing to do with whether it can actually work. Effectively, it's saying "it's installed, so go ahead and assign tasks." A model supposedly strong in engineering and alignment pulled this on a problem where the intent was crystal clear.

I said forget it, let's revert to the old setup.

During the revert, something else happened. When looking for a fix, it told me "here are three options below," asking me to pick. But it never listed the three options. It just jumped straight to "I recommend option one." I scrolled back several times. The three options didn't exist at all.

At this point I was pretty certain: this model genuinely can't be counted on for this kind of task.

So I Moved All That Work to GPT 5.5

After these incidents, the direct result was simple: for research and engineering tasks, I no longer trust Claude.

Trust builds slowly and collapses fast. Earlier I thought it was still usable, but the more I used it, the more I realized asking it to do something was probably a waste of time. Not just a matter of burning a few tokens, but having to redo everything in the end. Now I don't even consider it for these tasks. Everything goes to GPT 5.5.

GPT 5.5 is genuinely strong. Whether coding or research, it's clearly a notch above every other model.

This shows up most directly in my account setup: I'm now running 7 Codex accounts, maxing out weekly quotas on all of them, and it's still not enough. On the Claude side, I'm down to one account barely hanging on. It keeps getting my accounts banned, so I bought a spare just to hold onto. I never seriously used Google's.

Seven to one. That's more honest than any benchmark.

But This Isn't Just a Claude Problem

At this point I need to walk that back a bit. If this were just bashing Claude, the post wouldn't be worth reading.

Step back, and you see the top models have always taken turns in the spotlight.

Six months ago Gemini was riding high. Everyone was talking about how strong Google was. The last two generations have both felt underwhelming. Hardly anyone mentions them now. Then everyone flocked to Claude, thinking it was the strongest. But look at 4.7 to 4.8. It's been a real letdown. This round, OpenAI has clearly gotten back on its feet. GPT 5.5 is ridiculously strong.

No single company can stay on top forever. This isn't unique to the AI industry.

Chip Companies' Fate, Replayed by Model Makers at Fast-Forward

I was chatting with a friend about chips the other day. He told me how brutal strategy is in that business. Bet wrong on one generation of chips, and you might be finished.

NVIDIA nearly died. Its first chip, the NV1, bet on forward texture mapping and quadrilaterals, but the industry standard went with triangles (Microsoft's DirectX). The wrong direction meant no one wanted the product, and the company shrank from about 100 people to 40.

What saved them was Sega. Sega had commissioned NVIDIA to build the graphics chip for its console. Later both sides realized the direction was wrong, but Sega's Shoichiro Irimajiri still converted that roughly $5 million contract payment into an investment in NVIDIA. Jensen Huang later said this gave them "six months to live," just enough to survive until the RIVA 128 turned things around.

AMD was the same story. The 2011 Bulldozer architecture was a strategic mistake. Single-core performance was awful, and the company was badly wounded. By July 2015, its stock had crashed to $1.62, one step from bankruptcy. That year it licensed x86 to its Chinese joint venture Tianjin Haiguang for roughly $293 million, largely to stop the bleeding. It didn't recover until the Zen architecture arrived in 2017.

Both are now dominant players. But looking back, no one can guarantee they'll have the last laugh. Chip cycles are long and capital-intensive. One generation every three to five years, and one wrong move means massive pressure, or even getting knocked out entirely.

Model companies move much faster. It doesn't take a whole generation. Maybe just a few model iterations, a roughly six-month window of consistently missing the mark, and a company can be pushed off the table.

Chinese Model Makers Have Already Completed a Full Rotation

Chinese model companies have already played out this entire cycle.

The first to break out and gain recognition as a top-tier player was Zhipu. In 2022, its GLM-130B was the only large model from Asia selected for Stanford's HELM evaluation, and ChatGLM was among the first open-source models at home. For a while, it was unrivaled.

Then it fell behind. By late 2024, its flagship GLM-4-Plus had been overtaken by DeepSeek-V3 and Tongyi Qianwen on public benchmarks like SuperCLUE, dropping out of the top tier. At the time, a lot of people were surprised.

Then on January 20, 2025, DeepSeek-R1 burst onto the scene. Six days later its app hit #1 on the US download charts, along with 51 other countries. On January 27, it directly tanked NVIDIA's stock, wiping out $589 billion in market cap in a single day, the largest single-stock single-day loss in US market history. During that stretch, I felt like a bunch of model vendors were on the verge of collapse.

But Zhipu didn't leave the table. In the second half of 2025, it changed tactics, going fully open-source while narrowing its focus. It released GLM-4.5 and GLM-4.6 back to back, and its reputation clearly recovered. In January 2026, it even listed on the Hong Kong Stock Exchange. From falling behind to bouncing back, the key was staying at the table.

As a Side Note: Top Models Are Also Specializing

Beyond taking turns at the top, there's now another dynamic: division of labor.

For hardcore coding agents and research tasks, OpenAI is in a league of its own, clearly ahead of everyone else. Claude's original strong suit was precisely this area, but after several generations it failed to meet expectations and started regressing. Its strengths have instead shifted to white-collar work: writing, finance and legal, daily tasks, document research, that sort of thing. To be fair, 4.8 is a bit better than 4.7. It sounds more natural, its style moved back toward 4.6, execution is a bit more accurate, and it's actually pretty decent at writing.

Further toward the fringe you have Doubao and its ilk. Everyone knows it doesn't do serious work, but its emotional value is maxed out and its user base is terrifyingly large. There's a saying that "Claude is the American version of Doubao." I thought it was kind of funny at first, but thinking about it, it points to a different kind of divergence: some models are just good at chatting with you and giving you emotional value. That's also a category of demand.

So "which one is the strongest" isn't even the question anymore. It depends on what job you need done.

Final Thoughts: Don't Bet on Who's Strongest—Bet on Who's Still at the Table

Coming full circle, my conclusion is actually pretty optimistic.

The lead changes hands. Don't fixate on "whoever's strong now will stay strong," and don't write a company off just because it's temporarily stumbling. As long as you're still at the table, you have a chance to turn things around. Zhipu turned it around. NVIDIA and AMD both turned it around back in the day.

The real danger is leaving the table.

Anthropic has indeed been stuck these past few generations. My use of Claude is now basically limited to white-collar tasks. It's reportedly cooking up its next-gen flagship. If that comes out at the same level, things will really get dicey. It's not that any single model is bad, but in this six-month-cycle rhythm, missing the mark for several generations in a row is exactly how you get squeezed off the table.

But as long as it's still at the table, I'm not too worried. That's just how it works at the table: everyone takes turns.

References

原文链接：https://guanjiawei.ai/en/blog/stay-on-the-table

The Hidden Thread in Token Business: Cost Is Set by KV Cache Hits, Not Throughput

guanjiawei — Thu, 28 May 2026 08:15:50 +0000

The more I study the token business lately, the more I feel there's one angle that keeps getting overlooked.

Over the past year, when people benchmark inference performance, they mainly watch three numbers: absolute throughput, TTFT, and TPOT. How many requests can be batched, how fast the first token comes out, how fast each output token is. That's the standard talking point today.

But when you actually get down to serving, you find that what really drives token cost isn't throughput. It's whether the KV cache hits.

I. A 10× Gap Carved into the Price List

Several model APIs have cut prices recently. Open any pricing sheet and you'll see the input column was split in two a while back: cache hit and cache miss.

How big is the gap?

Anthropic charges 0.1× the base input price for cache reads, making it 10× cheaper. DeepSeek V4 Flash cache hit is \$0.0028 per million tokens, cache miss \$0.14 per million tokens, a 50× difference. Anthropic also charges 1.25× (5-minute version) or 2× (1-hour version) to write a cache. On April 26, DeepSeek cut cache hit prices across all models by another 10×.

At the machine level, the difference comes down to compute. A hit skips prefill; the machine only runs decode. A miss means recalculating from scratch, burning machine time and compute. The gap isn't a few percentage points. It's multiples, up to 10×.

The interesting part is this: once cache hit and miss are priced separately, some of the cost is yours to control through design, and the rest is entirely up to the vendor. Split like this, and "which API is cheaper" stops being comparable at the token level. You have to look at actual hit rates.

II. The Biggest Pitfall on the User Side: Model Routing

Let's talk about how users can mess this up.

Lately people have bought hard into "model routing." Hard tasks go to strong models, easy ones to weak models. It looks like savings on paper.

My view: switching models mid-session is usually a losing bet.

The clearest example is Claude Code. You've accumulated 300K of context, then midway decide Opus is too expensive for this step and switch to Sonnet. Claude Code now pops a warning that tells you explicitly: after switching, all previous cache is invalidated, and the next step must cold-start and recalculate. It didn't warn before. After enough complaints, they added it.

Why does it invalidate? Each model's KV representation is different, so cache can't be reused across models. Opus cache and Sonnet cache are two different things. The session hasn't changed, the cache key hasn't moved, but not a penny is saved on the recalculation cost.

Run the numbers. Current Opus 4.7 is \$5/\$25 per million tokens; Sonnet 4.6 is \$3/\$15. Sonnet is roughly 40% cheaper than Opus, not the 5× gap of the past. But that preceding 300K input goes from a cache hit (0.1× price) to cold calculation (1× price), so that single input cost jumps 10×. You save 40% on the model itself. Net it out, and the overall cost is actually over 5× higher.

Plus in agent workloads, tokens are almost always input-heavy and output-light. Prefill usually runs thousands of tokens per second. Decode runs dozens to just over a hundred. That's two orders of magnitude. The money is basically spent on input. Model routing destroys exactly the savings mechanism on the input side.

So mainstream agent design today revolves around "context stability." Don't swap models lightly, don't change tool structures, don't touch core prompts like CLAUDE.md halfway through. One move, and hit rates really do drop from 90% to 5%.

III. Claude Code's Solution: Spawn Sub-Agents

So what if part of a task really is better suited to a cheaper model?

Claude Code's solution is to spawn sub-agents.

The main session stays on Opus, preserving its hit rate. When you need to explore, batch-process, or run a specific sub-task, you call the Task tool to spin up a new agent. The new agent runs in its own isolated context, can pick a cheaper model, and maintains its own hit rate. When it's done, only a summary is passed back to the main session. The main session's cache isn't touched.

The precondition for this mechanism is that the sub-task's context needs differ enough from the main task's. If your sub-task happens to feed most of the main session's content into it, that's another cold start inside the sub-agent, and prefill eats up whatever you saved. This takes pretty fine-grained judgment.

IV. Server-Side KV Cache Engineering

How big is the server-side gap? Massive.

The crudest implementations don't design for cache at all. No reuse across users. Routing goes haywire; the request that should hit the machine holding the cache lands elsewhere. VRAM backs all cache capacity. In a system like that, no amount of user-side care can save you.

The mature example is the Mooncake framework from Moonshot AI, Tsinghua University, and Qijing Technology, a KVCache-centric disaggregated architecture. Prefill and decode clusters are separated, and underutilized CPU, DRAM, and SSD resources on GPU nodes are repurposed into a distributed KV cache pool. A KV cache scheduler handles queuing and routing. The paper cites a simulated 525% throughput gain; under real loads, requests handled increased by 75% to 115%.

The counterexample is Openclaw. This open-source agent took a lot of criticism, mostly because it stumbles at this layer. Its plugin architecture doesn't set a promptCacheKey by default. Pass it through a third-party proxy and you lose node affinity, so cache hit rate is nearly 0%. The total token volume isn't actually that high, but all input is priced at cache-miss rates, so the bill is ridiculous. About a month ago I looked at its trace: 60K+ input tokens in one request, 0% hit rate, \$0.12 a pop. That's what happens when server-side cache has no targeted design.

V. The Model Layer Can Also Push in This Direction

Go one layer deeper: the model itself can free up massive room for KV cache.

DeepSeek has been the most systematic here. MLA (Multi-head Latent Attention) projects KV into a latent vector and back, compressing KV cache volume by over 90%. V3 kept this mechanism. Later they added Native Sparse Attention, which almost flattens KV cache growth in long contexts. Only then can the inference system build a cache pool at million-scale context lengths.

But once the model changes, cache hit determination logic changes too. Some inputs previously recognized as "prefix overlap" behave differently under sparse attention, so whether they hit needs realignment. The inference system has to be revised as well. That's why I say inference engineers can't just stare at throughput anymore. They have to redesign cache around the model architecture.

VI. Measuring This Is Hard

The most frustrating part of the whole chain is that evaluating the server side alone is useless, and evaluating the user side alone is also useless.

No matter how strong the server-side cache is, if the user side is designed like Openclaw, hit rates still won't rise. No matter how careful the user side is, hit a crude server with chaotic routing, insufficient capacity, and no cross-user reuse, and costs still leak.

So "which API is cheaper" can't be compared on a single dimension when it comes to tokens. Same hardware, same target. Good coordination between both ends versus each doing its own thing, and the total bill can differ by 5× to 10×.

Closing Thoughts

The cache hit/miss split on token pricing tables is the single most important thread in this whole thing. It gives users a clear incentive: high hit rates save you money. At the same time, it pressures providers. If your cache system isn't strong enough, you won't get the business.

I hope vendors also expose the cache hit mechanism itself. Otherwise users only know it exists without knowing how to optimize for it. There still aren't many vendors that can tie together model, server-side cache, and user-side usage end to end.

The edge side is still competing on raw performance. But once app density rises and agents really start running, KV cache will become a core issue there too. From cloud to edge, there's no way around it.

References

原文链接：https://guanjiawei.ai/en/blog/token-business-kv-cache

A Token Is Not a Thing

guanjiawei — Tue, 26 May 2026 04:48:36 +0000

Lately, "token economy" is hot. Every business model in AI will eventually converge on one unit of account: the token. I buy that thesis. But one premise keeps getting skipped—a token is not a standardized commodity.

Water has standard units. Electricity has standard units. Money, obviously. Token doesn't. It's more like gasoline: 92, 95, and 98 octane are different fuels, priced differently, for different engines. Adding them up by the liter and reporting one number means nothing.

Most contradictions in AI today come down to this.

I. Intelligence Has Tiers

Roughly four.

Top tier. Overseas: OpenAI GPT-5.5, Anthropic Claude Opus 4.7. China: Zhipu GLM-5.1, Moonshot Kimi K2.6, DeepSeek V4-Pro. Xiaomi MiMo-V2.5-Pro is a bit controversial, but usage and data are climbing, so I'll count it. These range from hundreds of billions to over a trillion parameters. Demand is almost unlimited; willingness to pay is fierce. Prices rise, quotas tighten, prices rise again—users keep pouring in. Zhipu's 2025 annual report showed GLM Coding Plan token calls up 15× in six months, with paying developers past 240,000. That's the real demand curve for top-tier tokens.

Mid-tier. This is the awkward gap. MiniMax M2.7, DeepSeek V4-Flash, Xiaomi MiMo-V2.5 standard—these are about it. Moderate size, an order of magnitude cheaper, theoretically the best value. But almost no one is seriously building here. I'll explain why later.

Low-mid. Mostly open source. Alibaba Qwen 3.6 leads, with both 35B-A3B (MoE) and 27B dense versions open. Google Gemma 4 is here too, from E2B to 31B.

On-device. A few billion parameters, or even sub-billion, fitting in a phone or a consumer GPU.

The first imbalance is right here. Top tier is a bloodbath. Mid-tier is empty. Low-mid and on-device are noisy but lack clear scenarios.

II. Speed Is Another Dimension

Tiers are only half the token story.

The other half is speed. GPT-5.5 at 30 TPS versus 200 TPS is a completely different experience.

Here are the 2026 numbers from Artificial Analysis, a commonly cited benchmark:

Tier	Model	Output TPS
Flagship Standard	GPT-5.5 (high)	~68
Flagship Standard	Claude Opus 4.7	~48
Flagship Standard	DeepSeek V4-Pro	~48
Flagship Standard	Kimi K2.6	~33
High Speed	DeepSeek V4-Flash	~126
High Speed	Gemini 3.5 Flash	~203
Ultra High Speed	GLM-5.1 High-Speed Edition	400 (official)
Ultra High Speed	Cerebras running Kimi K2.6	981

I wrote a post earlier, A Model 5× Faster Is No Longer the Same Model. The argument: a 5× speedup unlocks product forms that literally didn't exist before. This isn't slightly faster. It's a different species.

The market already prices this. Anthropic Opus Fast: 2.5× speed, 6× price. OpenAI Priority Tier: 2.5× price. Look at those ratios—price rises faster than speed. Not greed. It's a pricing signal. There's a real cohort willing to pay multiples for speed.

Intelligence tier × speed tier. Stack them and you get a matrix. The token in each cell is a different product.

III. Two Demand Tracks, Worlds Apart in Willingness to Pay

Who's burning top-tier tokens? Two main tracks.

First: coding agents. The fastest-growing, highest-burn category worldwide. The surface is a coding agent writing code to solve problems. In practice, people use them for everything. The work just happens to get done through "writing code."

Second: consumer agents. The Claude app, ChatGPT app, Microsoft Copilot, and Zhipu's new AutoClaw (Claw Plan). AutoClaw launched March 2026 and hit 400,000 subscriptions in 20 days. Under the hood it's a coding agent wrapped in a non-technical shell, letting ordinary people "hire an AI employee."

The two tracks have very different willingness to pay.

Coding agent users demand peak intelligence—Opus 4.7, GPT-5.5 tier. Anything less fails. The work is valuable; time saved is valuable. They'll pay for top-tier tokens continuously. Stickiness is another story: when a better model drops, they switch immediately.

Consumer agent users differ. Their tasks are lower-value, they're price-sensitive, and they don't need absolute peak intelligence. A "mid-tier smarts, good value, acceptable speed" model fits them perfectly. The problem: that tier is empty right now, with no real supply. So DeepSeek V4, with extreme cost-performance, quickly captured this segment. I've noticed many friends around me switching to DeepSeek.

Demand looks like this, so model companies follow the money. That's why top-tier models keep screaming compute shortages while mid-tier models have no takers.

IV. The Supply-Side Mismatch: Scarce Cards and Idle Racks at the Same Time

Demand misalignment carries over to the compute market.

Top-tier compute shortage is obvious.

Jensen Huang personally confirmed NVIDIA's Blackwell series (B200/GB200) is "sold out through mid-2026," with new enterprise orders facing 8–16 week lead times. Meta's annual CapEx is expected past $100 billion; Microsoft is spending nearly $35 billion in a single quarter—all scrambling for these chips. In China, the frenzy is over B300 and H200: a B300 server costs ¥7 million and you still can't get one, monthly rent pushed to ¥130,000–200,000. H200 was cleared for sale in China in January 2026; the first 5,000–10,000 module batch was snapped up by top vendors immediately, cluster delivery pushed to Q2 2027. The older H100 has cooled. No one is fighting for it now.

Domestic top-tier chips are even more extreme. Huawei's latest Ascend 950PR only began mass production in March 2026, yet the full-year plan of 750,000 units was completely locked up: ByteDance (350,000), Alibaba (200,000), Tencent/Baidu (100,000), government and enterprise IT innovation (100,000)—orders pushed to 2027. Roughly $16,000 per chip, 1.56 PFLOPS FP4, officially claimed at 2.87× H20 single-card performance. This is the first time in domestic AI chip history that an entire year's production was bought out. When DeepSeek V4 open-sourced, it shipped day-zero support for eight domestic chips, listing Ascend NPUs alongside NVIDIA GPUs in the technical report. GLM-5 was trained entirely on Ascend + MindSpore, with support for seven domestic chips. This is about positioning: anchoring top models on domestic chips is both a technical and supply problem.

The hidden side is massive idle capacity in low-to-mid-range compute.

PPIO founder Yao Xin has said some domestic GPU AI compute centers have idle rates up to 80%. 36Kr reported some centers at only 10–20% utilization. Xinhua put it more bluntly: "General-purpose compute is relatively oversupplied; AI compute is relatively scarce"—an admission of structural mismatch. Prices reflect this: A100 prices crashed over 50%, RTX 4090 hourly rent dropped to ¥1–2, and the 5090 is around ¥2.5.

But the low-to-mid-range mismatch has two distinct bottlenecks.

Mid-tier datacenter cards (H20, L20, Huawei 910B, etc.) are stuck on infrastructure. Inference frameworks optimize for top-tier cards far more than these. KV cache management, MoE expert parallelism, FP8/FP4 precision support—none of the critical paths is mature here. The hardware exists, demand exists, but you can't serve a top experience.

Consumer PCIe cards (4090, 5090, 4090 48GB mods) face the opposite problem. The hardware can run; vLLM already supports the 5090 (needs CUDA 12.8 + falling back to FlashAttention 2, usable enough). What's missing is good models designed for them. The 70B dense tier is obsolete—as of May 2026, the top six open-source models are all MoE; dense has virtually disappeared at the flagship level. MoE total parameters routinely exceed 100B, which won't fit on consumer cards; distilled small models can't match top quality. No one is supplying new, high-quality models tailored to 24GB/32GB/48GB VRAM limits.

So the picture is: 4090/5090 prices are absurdly cheap compared to datacenter cards, yet the mid-tier models you can actually run are still old stock like Llama 3.3 70B from late 2024. Individual developers experimenting locally, small-team PoCs, and privacy-sensitive on-prem deployments can get by. But for enterprise-grade mid-tier inference on these cards, no newly optimized models exist.

The issue isn't "total compute is insufficient." It's "compute can't align with demand."

Outsiders used to quote compute in "petaflops." That was always shaky; in the AI inference era it's nearly useless. Whether a compute unit can serve top-tier models depends on interconnect, memory bandwidth, FP4/FP8 support, KV cache management. A hundred older cards can't match one top-tier card's single-stream speed.

You get a strange picture: top model providers scrambling for chips, while last-gen cards in datacenters can't be rented out even at a discount. Scarcity and glut, side by side.

V. The Market Will Correct the Mismatch, But It Takes Time

This mismatch won't last. The two bottlenecks will be pushed by two different market forces.

The infrastructure gap for mid-tier datacenter cards will be driven by engineering priorities. Inference frameworks follow the money. Once mid-tier model demand grows, top frameworks like vLLM, SGLang, and TensorRT-LLM will eventually be forced to prioritize H20, L20, and 910B optimization. Not glamorous, but inevitable.

The model supply gap for consumer cards is being pushed by distillation and small MoE. DeepSeek-V4 has already distilled a ~9B version; the Qwen series has been working on this. Once someone actually delivers "runs in 32GB VRAM, quality close to top-tier," idle 4090s and 5090s will immediately find work.

Another track is deep binding between domestic chips and domestic models. DeepSeek and Zhipu are both pursuing it; technically it's proven feasible. Once it fully works, the low-to-mid-range compute market will reshuffle structurally.

I'm fairly optimistic this will happen—it just takes time. Maybe a few quarters, maybe a year or two. For those who catch the rhythm, there's a structural window.

VI. Don't Reduce Tokens to a Single Number

Back to the opening line. "Token economy" is a fine term, but it's far less intuitive than selling water or electricity.

It's more like a gas station. Gasoline looks like one thing, but it's actually an intelligence × speed matrix. Layer on the supply-side compute tier mismatch, and you have the real cause behind today's apparently contradictory industry phenomena: why model companies are scrambling for chips, why some AI compute centers sit idle, why fast tier can charge 6×, and why mid-tier intelligence models are slow to arrive.

Next time you see "we've deployed N petaflops" or "we produce X trillion tokens per month," pause and ask: which intelligence tier, which speed tier, which demand tier.

A token is not a thing.

References

Model Versions and Positioning

Speed Data

Zhipu Products and Financial Reports

Compute Market

原文链接：https://guanjiawei.ai/en/blog/token-is-not-one-thing

The Stronger the Agent, the More Common Sense Is Worth

guanjiawei — Mon, 25 May 2026 04:06:31 +0000

Last month I wrote “AI Turns Ignorance Into an Advantage”, about how outsiders without the baggage of knowing how hard something is are more willing to use AI to try things that look impossible.

I still believe that. But agents burned me four times in a row recently, so I need to revise.

The sweet spot isn’t knowing nothing; it’s knowing just enough. You have common sense, you grasp the big picture, but you don’t get lost in technical details. Total beginners do dare to try, which is good. But they can’t tell whether the agent’s output is actually reliable.

1. Fake Data Can Trick You by Orders of Magnitude

I’ve been optimizing an inference engine lately.

I checked the results on the first night. The metric had hit a target I’d considered seriously challenging. I was excited. Had we really cracked it that fast?

If I knew nothing about this domain, I’d probably have cheerfully shared the results with my partners. But because I had some common sense, something felt off. I had it run a correctness check. The output was nothing but exclamation marks. After fixing correctness, performance dropped by orders of magnitude.

I thought that was the end of it. But as we kept optimizing, the rhythm still felt wrong. The numbers climbed too fast, suspiciously fast. I looked at the test flow and found that before each official test it quietly ran a warm-up using the exact same prompt. Every subsequent test was hitting the prefix cache, essentially cheating on an open-book exam. After isolating the cache, performance dropped by orders of magnitude again.

Still not done. Once prefill returned to normal, decode speed suddenly became absurd. A Windows build of the engine was somehow outperforming the Linux version. I ran the real-prompt test script I’d written earlier, and performance took another ten-fold hit. The problem was that the agent’s synthetic prompts were too simple and too regular, letting speculative decoding hit an acceptance rate above 80%. Switch to real prompts and the acceptance rate cratered, taking performance with it. Teams that have shipped speculative decoding have documented the exact same trap: real-world production performance is 40% to 60% lower than in the lab, a gap large enough to make you wonder if it’s the same system.

Three layers of illusion, stacked. If I’d believed that first number and shared it externally, the cleanup would have been miserable. You give someone the wrong expectation, and they start scheduling around it. Then you have to go back and say, “Sorry, we’re off by orders of magnitude.” That feels way worse than saying “We’re not there yet” from the start.

After that, every optimization target explicitly included two rules: prefill must not be affected by prefix-cache interference, and decode must use real prompts. Only then did we see a normal curve that crept upward, bit by bit.

2. It Will Brick Your Lab Machine

The latest agents can work autonomously for a full day or longer. The longer they run, the higher the chance something goes wrong.

My agent has, more than once, trashed the entire OS of a lab machine mid-run because of a missing quote or a flipped command-line argument. Files gone, environment wiped. It happens in a split second. You can’t stop it in time.

It’s not just me. In April, when an agent hit a credential mismatch, it didn’t stop to ask a human. It found a token with full privileges and deleted an entire company’s production database and all backups in nine seconds. Thirty-plus hours of downtime. Three months of customer data, gone. There have been at least a dozen similar documented incidents in the past two years.

Anthropic and OpenAI are now pushing sandboxing. The idea isn’t complicated. Filesystem isolation on one layer, network isolation on another. Without filesystem isolation, the agent can touch things it shouldn’t. Without network isolation, a compromised agent can steal your keys.

My own approach is more low-tech: dedicate a machine exclusively to the agent, and don’t store anything else on it. If it runs for dozens of hours straight, the probability of a dumb mistake is non-zero. Reinstalling the OS costs time. Losing important data costs your sanity.

3. It Will Spin in Circles Until You Step In

Agents have another bad habit: they circle the same problem.

A recent goal was to run an inference engine on Windows in BF16 precision. The model weights were over 60 GB, and loading them caused an immediate OOM crash.

The agent’s response was interesting: it kept trying to work around the memory bottleneck. Load only some weights, dynamically read the rest during inference, every offload trick in the book. None of them worked, and each ate up a lot of time. It even added a warm-up to the tests to hide loading latency. That was part of the root cause of the prefix-cache problem I mentioned earlier.

I finally cut in and said: stop tweaking performance and fix the memory issue first. Until that bottleneck is solved, everything else is wasted effort.

The agent actually executes well. Once pointed in the right direction, it quickly found a series of system-level Windows settings to expand available memory and VRAM. After that was fixed, the optimization path smoothed out immediately. All the previous workarounds were suddenly useless. That time was basically wasted.

The problem is that it won’t proactively redefine the problem. Hand it “optimize performance” and it will keep grinding on that goal, even when stuck on a prerequisite. It finds ways to work around it rather than telling you, “This assumption is false; we need to handle something else first.” Recognizing the real blocker and pulling the agent out of the dead end is a judgment call only a human can make.

4. Set the Bar Too High and You Ship Nothing

The last pitfall isn’t the agent’s fault. It’s mine.

The more powerful agents get, the easier it is to set the bar too high. Because they can run for days, you start thinking anything is fair game. Every direction looks like a top-conference breakthrough. So you spin up multiple threads, each one ambitious.

The result? Every thread is active, every thread shows progress, but nothing ships.

You keep burning tokens, you keep seeing “progress,” yet nothing reaches the user’s hands. It looks like work. It’s actually just burning money. I made this mistake recently: several threads were the kind that would be huge if they landed, but the execution risk was equally high. An agent isn’t a genie; if it can’t be done, it can’t be done. I burned a mountain of tokens and delivered nothing.

I eventually realized: narrow the scope. You need something shippable in the short-to-medium term and some worthwhile long-term explorations, not only the latter. Deliver what can be delivered first, stabilize the rhythm, then go after the big bets.

5. Knowing Just Enough Is Exactly Right

Look at the four pitfalls together and one thread connects them: none requires you to be a deep expert to avoid.

Performance jumped by orders of magnitude? Check whether you measured wrong first. The agent needs to run on your main machine all day? Give it a dedicated one. Stuck on the same spot after three optimization rounds? That spot is the real problem. Every thread is running but none ships? Kill a few.

It’s all common sense.

An MIT Sloan article this year on managing in the age of agentic AI noted that the most important skills for managing agents are defining problems and validating outputs. Those are things AI still can’t do well. “Agent Manager” is already showing up on job boards, and one line in the job description stands out: domain common sense matters more than AI expertise.

Going back to my previous post. “Ignorance is an advantage” still holds: you have to not know what’s hard in order to dare to try. But courage alone isn’t enough. The most valuable state is this: willing to try, yet able to sense when something is off at the critical moment.

Total beginners get carried away by fake data. Deep experts get shackled by priors. The people in the middle, the ones who know just enough, are bold enough to act, yet wise enough to pull the reins when needed.

Agents will keep getting stronger. But that bit of human common sense, whether the numbers check out, whether the direction is right, or whether this should ship now, will only become more valuable. These are still things agents can’t handle.

References

原文链接：https://guanjiawei.ai/en/blog/agent-common-sense

When a Model Is 5 Faster, It’s No Longer the Same Model

guanjiawei — Fri, 22 May 2026 06:52:33 +0000

Two releases caught my eye this week.

On May 19, Google released Gemini 3.5 Flash. I watched their launch event. Oddly, they barely emphasized the model’s raw intelligence. Benchmarks against the previous generation didn’t exactly stand out either. But they devoted serious time to speed, calling it “frontier intelligence built for speed,” claiming inference is 4× faster than other frontier models.

Today, May 22, Zhipu also launched GLM-5.1 High-Speed, claiming 400 token/s output—the current ceiling for industry APIs. This engine wasn’t built by Zhipu alone; it was a joint effort with a team called TileRT, doing low-level customization specifically for the GLM model family on a specific class of hardware.

Put these two together, then look back at Anthropic’s Opus Fast and OpenAI’s GPT-5.5 Fast over the past few months, and the direction is clear: differentiation at the model layer is changing lanes. Everyone used to compete on smarts; now they’re increasingly competing on speed.

And once speed crosses a certain line, it stops being a linear “X times faster” improvement. AI becomes a different kind of thing.

1. Pricing Already Tells the Story

The clearest evidence is fast-mode pricing.

Anthropic’s Opus Fast: 2.5× the speed, 6× the price.

OpenAI’s GPT-5.5 Fast: 1.5× the speed, 2.5× the cost.

Look at the numbers. If speed were valued linearly, 2.5× speed would cost 2.5× the price, and 1.5× speed would cost 1.5×. But in practice, the price jumps far more than the speed.

This isn’t greed. It’s a real market signal: some people will happily pay disproportionately more for speed. Either their tasks need high-frequency feedback, or users are sitting there waiting, or downstream steps are blocked. In these scenarios, going from 30 seconds to 12 seconds feels completely different from going from 30 seconds to 20 seconds.

I toggle Opus Fast on and off constantly myself. I turned off GPT-5.5’s 1.5× tier immediately. I couldn’t feel the difference; it was just burning money. But at 2.5×, there are tasks I just leave it on for—mostly when I’m staring at the output and iterating fast.

Markets don’t lie. Something that sells for 6× has buyers who genuinely think it’s worth it.

2. Per-Request Speed and Scaling Out Are Not the Same Thing

Two things need to be kept apart here.

The first is “doing more concurrency at a fixed speed.” Same 30 token/s throughput, but serving 1,000 users instead of 100. This is relatively easy. Just throw more machines at it—you can buy slightly weaker cards and spread the load across them, and the cost-performance ratio can be tuned.

The second is “making a single request faster.” Going from 30 token/s to 400. This is an entirely different beast. You need higher-end hardware, more aggressive memory bandwidth, and cutting-edge packaging. You can’t fix this by “spending a bit more to stack extra cards.” A hundred weak cards won’t get a single request to the speed of one top-tier card.

I’ve spent time experimenting with inference infra myself, optimizing a few open-source models. The cost curves are completely different. The first is roughly linear—double the money, get about double the concurrency. The second is non-linear—that first 20% speedup might cost you 50% more, and it only gets steeper.

So when Gemini 3.5 Flash emphasizes speed, or GLM High-Speed hits 400 token/s, they’re not saying “we made a cheaper version.” They’re saying “we pushed single-request speed to a new level.” That’s a problem of an entirely different magnitude.

3. 5× Is a Speciation Threshold

So why push so hard?

When I think about this, I go back to a simple comparison.

If you want something done faster, the traditional options are limited.

First, hire smarter people. But that hits a ceiling. There are only so many world-class experts, and today’s best models are already brushing up against that ceiling.

Second, make people work overtime. Agents already run 24/7, so that ceiling is gone too.

Third, divide the work and throw more people at it. But anyone who’s done engineering knows adding people doesn’t scale linearly. Adding one person doesn’t make it twice as fast; adding ten gets you nowhere near 10×. You have to break things down, hand off, coordinate, deal with uneven quality, manage waste. The ramp-up period for new hires is expensive. If you’re doing multi-agent orchestration, you know exactly what I mean.

At this point, the traditional paths to speed are tapped out.

So what’s left? Make the model itself—the same employee—faster.

And making that “same employee” faster is non-linear.

Imagine an employee who used to take an hour to finish a task. Now they do it in ten minutes. You think you just saved fifty minutes? It’s more than that.

You’ll start giving them tasks you’d never have bothered with because “it’s too slow.” Small ad-hoc requests that used to take an hour—so you never asked—now come back in ten minutes, and you make a dozen a day. Speed unlocks tasks that literally didn’t exist before.

I saw a demo the other day: someone wearing glasses pointed at a video on a screen and said “zoom in on this,” and the AI behind it wrote code to resize the element. If the whole chain takes thirty seconds, you glance and walk away—there’s no real interaction. But if it finishes in five seconds, the feel is completely different; it becomes a genuinely usable product.

That’s the gap between 50 token/s and 400 token/s. 8× speed unlocks products that were impossible to build before.

A speedup beyond 5× is a speciation line.

4. The Return of Specialization

Okay, speed is valuable. How do you actually achieve it?

That brings us to TileRT’s approach, which diverges from where the industry was a year ago.

Mainstream inference frameworks like vLLM, TensorRT-LLM, and SGLang are general-purpose. They aim to “support as many models as possible, running well enough on as much hardware as possible.” That has always been software engineering’s default bias: generality first, performance second.

TileRT does the opposite. It statically schedules the entire inference graph at compile time, running as a persistent kernel on the GPU with almost no runtime dynamic scheduling. Micro-tasks at tile-level granularity squeeze the hardware close to its physical limits. The cost? Change the model and it’s basically scrap; change the hardware and it needs major rework.

DeepSeek is on the same path. Their own inference engine started out based on vLLM, then underwent more than a year of deep customization—almost every path was rewritten for their own MoE architecture. When they open-sourced part of it recently, the industry’s reaction wasn’t “how general-purpose this is,” but “how deep you can go for a single model.”

Go one layer deeper, and the hardware side has been on this path for a while. Groq’s LPU runs Llama 4 Scout at 460 token/s, 3–4× what an H100 delivers. Cerebras’s WSE-3 hits 1,800 token/s on a 70B model and nearly 3,000 on gpt-oss-120B. These are specialized chips. They aren’t trying to run every kind of model; they’re built to take a specific workload to the extreme.

Chip designers have debated this for decades: general-purpose CPU or specialized ASIC? General chips have their place, but when a domain is big enough and the lifecycle is long enough, specialization pays off.

The software layer used to avoid this, mainly because software isn’t cheap to write. Building a dedicated inference stack for one model takes too long to pay off; the model changes and your software is dead.

That’s changing. AI agents can write software now. The cost of “building an optimal inference stack from scratch for a specific model and specific hardware” drops every year. Once it falls below a certain threshold, specialization becomes the default.

Every promising model will eventually have its own dedicated inference engine. Every generation of mainstream hardware will have its own specially optimized stack. What you used to think of as just “the last 5% of optimization” could now become a 5× or 10× gap.

5. Vertical Integration at the Model Layer Is Inevitable

Pulling these threads together.

Intelligence will keep improving in the short term, but the marginal utility of competing on raw smarts is declining. A model that’s 20% smarter versus the same model accelerated 10×—for many users, the latter is far more valuable, especially for the new scenarios that speed itself unlocks.

So the next phase of competition shifts from “point intelligence” to “end-to-end capability.” Model, inference engine, and hardware—all three bundled together.

If you’re at 400 token/s and I’m at 30 token/s, even if my model is 20× smarter, I’m unusable in many scenarios. I’ll be watching my smartest model sit there slowly spitting out words while you’ve already delivered the whole product experience to the user.

DeepSeek and Zhipu are already doing this. Anthropic and OpenAI are too. Google probably went the earliest and deepest—the TPU + Gemini combo has been running internally for a long time. My guess is that over the next year or two, the whole industry moves this way: model companies must own their inference stack and go deep into the hardware layer; hardware companies must go deep into model architecture; and the generic middle layer gets squeezed from both ends.

For engineers, this is pretty exciting. We used to think “general, scalable, portable” was good taste. For the foreseeable future, the opposite may hold: writing the most extreme code for a specific model and specific hardware—code that breaks if you change anything—becomes worth doing again.

Software engineering aesthetics will have to change.

References

原文链接：https://guanjiawei.ai/en/blog/inference-speed-new-species