DEV Community: David Aronchick

Painted Seams

David Aronchick — Fri, 10 Jul 2026 18:27:12 +0000

Apple raised the price of a Mac and an iPad this month, somewhere between 15 and 25% depending on the configuration, and the machines did not get better. Sadly, the thing that made it more expensive doesn't even get to be part of a keynote: the memory. Framework jacked up its DDR5 upgrade prices by half. And they aren't the only ones: Dell warned of hardware increases measured in hundreds of dollars; and back in February, Micron quietly retired Crucial, the consumer memory brand a whole generation of people who built their own PCs grew up on, so it could point every wafer it makes at enterprise AI. If you have shopped for a laptop lately and felt like you were being mugged, you were not imagining it, and it has almost nothing to do with the laptop.

The mechanism is cleaner and crueler than a shortage.

One bit up top, three bits gone below

One thing to understand is the topology of a machine (or cluster) that supports ML is very different than even the God Box you may have sitting out under your desk. The AI buildout wants high-bandwidth memory, HBM, the exotic stacked stuff that feeds a GPU, and there are (for now anyway) exactly three companies on the planet that can make it: Micron, SK Hynix, and Samsung. The catch is that HBM and the plain DRAM in your phone come off the same fabs and the same finite pile of wafers. When Micron commits a wafer to an HBM stack, it forgoes roughly three bits of the conventional memory it could have sold to everyone else. As a result, HBM has quietly grown to claim around 23% of all DRAM wafer output, up from 19% a year earlier, and every point of that came out of the supply that used to go into cheap laptops, mid-range phones, game consoles, and the SSD in your camera.

So the price did what prices do when a giant new buyer corners a fixed supply. Consumer DRAM ran up as much as 90 to 95% quarter over quarter in the first three months of 2026 alone. Contract DRAM prices were up more than 170% year over year heading into the year, and enterprise SSDs doubled. Gartner is telling buyers to brace for a 130% memory cost surge, Bloomberg has been calling it a genuine crisis since February, and IDC does not expect real relief until new fabs come online in 2027 or 2028. Intel's Lip-Bu Tan put it more bluntly: no relief until 2028. Two full years in which the memory inside a device that has nothing to do with AI costs more because of AI.

And the allocation is already locked. Micron's entire 2026 HBM output sold out under binding contracts before the year even started, some of it under multi-year deals that lock in roughly $100 billion in minimum contracted revenue and $22 billion in upfront customer cash. What you have is a perfect storm of supply chain constraints; the consumer is not being outbid in a live auction, and so has no say in the price. Instead, they are last in a line that was already full when the doors opened.

The stockings went to the parachutes

There is a rhyme here, and it is not from the chip industry.

In 1942, nylon and silk stopped showing up in American stores, and the reason was not that DuPont had forgotten how to make them. The War Production Board requisitioned the material for parachutes, glider tow ropes, and powder bags. Silk stockings, the small everyday luxury of an entire generation of women, simply vanished, and the vanishing had nothing to do with anyone's feelings about stockings. It was a straightforward consequence of a bigger buyer with a bigger priority taking the whole supply. The famous part is what people did about it; they drew seams up the backs of their bare legs with eyeliner to fake the look of a stocking that no longer existed. Painted seams; a cosmetic workaround for a supply chain that had been pointed somewhere else.

That is where the consumer memory market sits in 2026, minus the war and minus the ration book that at least made the trade honest. In 1942 the government stood up and said out loud that the material was going to the front, and it handed you a coupon so you understood the deal. In 2026 there is no declaration and no coupon. There is just a price, and a laptop that costs 25% more for reasons the person at the counter cannot explain, and a memory maker retiring its consumer brand rather than say in plain words that you are no longer the customer.

The shape of the demand is the problem

It is true that memory has always been cyclical, gluts and shortages are the heartbeat of that industry, and the fabs really are coming in 2027. But the cyclicality is showing up elsewhere, because right now the ENTIRE STACK is cyclical. The electric bill and the model that just got eaten from the bottom are showing the exact same characteristics. Turns out when you concentrate an enormous demand into one synchronized spike aimed at one finite shared resource, the shock does not stay where you put it. It radiates until it finds the person who never placed an order and hands them the bill, whether the resource is wafers or watts.

A single, centralized, all-at-once draw is the thing that breaks a shared pool. It was true of the grid in one Virginia county and it is true of three fabs in Asia. Spread the demand across time and place and the same total consumption stops being a crisis and goes back to being a line item. That is the physics of shared resources, and we keep relearning it the expensive way, one requisition at a time.

Your kid's laptop went to war this year. Nobody told you, because this time there was no coupon to hand out. Just eyeliner, and a longer wait for the seams to come back.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!

Originally published at Painted Seams.

The Cheapest Connection You Never Build

David Aronchick — Tue, 07 Jul 2026 18:36:39 +0000

On June 18, FERC stopped studying the data center power problem and started issuing orders about it. No notice of proposed rulemaking, no request for comment, none of the usual multi-year administrative slow walk. Instead, six show cause orders under Section 206 of the Federal Power Act, aimed at the six organized markets that keep the lights on for roughly 200 million Americans across more than thirty states: PJM, MISO, SPP, the California ISO, ISO New England, and the New York ISO. Each operator gets 30 days to account for its spare capacity and 60 to defend or rewrite its rules, with the goal being to get big loads onto the grid fast, or let them co-locate with their own generation, and either way, stop sticking ordinary ratepayers with the bill.

There's a great reason for this decision; the interconnection queue in this country is genuinely broken. Getting a large load connected can take years, and by Bloomberg's reporting PJM now projects it will be six gigawatts short of its own reliability requirement by 2027, while wholesale electricity has climbed as much as 267% against where it sat five years ago. Data center demand, for its part, is on track to nearly triple through 2035. Back in April I wrote about eleven gigawatts of announced capacity sitting frozen because the grid physically could not deliver the power. So FERC moved. This is good.

And the ratepayer protection is also good! Ish. If you want priority access, you pay for it. The data center pays for its own interconnection: the wire, the substation, the transformer, the steel. This kills a real and ugly abuse, where a utility quietly smears a hyperscaler's hookup costs across every residential meter in the region and calls it the cost of doing business. Fixing that is worth doing, and FERC deserves credit for doing it under threat of a federal enforcement deadline instead of a strongly worded letter.

But the interconnection cost is the part you can itemize, and the part you can itemize is the cheap part. The expensive part has no line item, and it cannot be assigned to anyone, because a grid is a shared pool and the price clears at the margin. When you bolt six gigawatts of new demand onto a system that is already six gigawatts short, the marginal price moves for every single person drawing off that pool. The data center can write a check for its on-ramp, but it cannot write a check for the price of the electricity it just bid up, because that cost never arrives as an invoice. It arrives as everyone's rate. That 267% did not happen because some grandmother in Toledo had her interconnection mispriced. It happened because demand outran supply in a market that prints exactly one clearing price for all of us. In other words, FERC can ring-fence the wire, but physics does not recognize the fence around the scarcity.

If we needed evidence of this, six days later, the same PJM agreed to bolt a new capacity advisory onto its emergency playbook, which is basically a way to warn its 67 million customers that supply can run short now even on ordinary days, without the heat waves that used to be the only thing that rationed power. That is the operator conceding, in its own paperwork, the part the order cannot itemize. The scarcity is already here, it is shared, and it does not read the invoice.

I like to think of this as a "new stadium" problem, basically. You can make a new stadium for a city pay for its own parking structure and its own freeway on-ramp and then stand at the ribbon-cutting and announce that on game day there will be no troubles getting here to enjoy your $73 beer, hotdog, and soft serve out of a plastic helmet. But every road for ten miles received no upgrades whatsoever, and was certainly not the budget, received no zoning variance, and, generally, will just degrade much more quickly. And the people who will eat it are the ones six blocks away who were sitting at home PROBABLY watching the game on tv (which is what they were doing before the new stadium went in anyway). We have spent a century learning that the parking lot is never the part of the development that costs the neighborhood something. The road is.

The whole order reveals - starkly - that the bigger assumption is that the demand should be measured by the shared grid, not the interconnects. I really like the colocation option which says fine, go sit next to your own generation (in the stadium scenario, this would be the equivalent of adding some a high rise hotels where you could just walk to the stadium since it's right next door). The cheapest interconnection in the world is the one you never have to build, because the load already lives where the power is. Compute that sits next to its own power doesn't bid up grandma's rate, because it isn't standing in grandma's line. That is not a regulatory trick. It's just where the physics has been pointing the entire time, and it is the opposite of hauling a gigawatt of demand three states over to a substation that was already maxed out.

NVIDIA, for its part, published a blog the same week calling the FERC orders a win for affordability. The orders may well be. The only question worth asking is affordability for whom, and the invoice, conveniently, doesn't say.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Cheapest Connection You Never Build.

Eaten From the Bottom

David Aronchick — Tue, 30 Jun 2026 18:31:03 +0000

On June 17, a Beijing company most people in American boardrooms still cannot pronounce released the best open model on Earth and barely made the front page. Z.ai, formerly Zhipu, shipped GLM-5.2, a roughly 750-billion-parameter model with a million-token context window, under an MIT license, which means you can download the weights, run them on your own hardware, modify them, and ship a product on top of them without asking anyone's permission or paying anyone a toll. The independent benchmarker Artificial Analysis put it at number one among open-weight models and number four overall, behind only the closed Western frontier, and it does that at roughly one-sixth the price of the model just above it. Chinese open models now hold most of the top slots on the open leaderboards and supply a majority of the world's open-model tokens.

Six weeks ago I wrote The Frontier Became a Club, about Anthropic's Mythos preview going to eleven named organizations with a hundred million dollars in credits attached and to nobody else. That post was about the top of the market sealing itself off, and it was correct. The genuinely hardest reasoning still lives behind the closed labs, the index still has a Western model at the summit, and four points of separation on a capability benchmark is four real points. The club is right that the very top still matters.

While everyone watched the top, the floor moved. And the floor is where these things always get decided.

The rebar nobody wanted

In 1969, a company called Nucor built a steel mill in Darlington, South Carolina, that did something the giants of American steel found mildly amusing. It melted scrap in an electric furnace and rolled it into rebar, the cheap reinforcing bar that gets buried in concrete where nobody can see it and nobody checks the metallurgy. It was the garbage tier of the steel business. Low margin, low status, low everything. US Steel and Bethlehem were happy to let it go, because they owned the high end, the structural beams and the sheet steel that went into car doors and appliances, the stuff that actually required good steelmaking. Ceding rebar to the upstarts was the obvious call. Why fight over the worst product in your catalog?

So the mini-mills took rebar. Then, with the rebar money, they got a little better and took angle iron and merchant bar. The integrated mills retreated up the ladder again, and again it was the rational move, because each tier they gave up was lower margin than the tier they kept. Then in 1989 Nucor opened a plant in Crawfordsville, Indiana, using thin-slab casting to make flat-rolled sheet, the crown jewel, the product the giants had told themselves the upstarts could never touch. By 2001 Bethlehem Steel was in bankruptcy. The integrated mills were right about quality at every single step of the retreat. Their steel really was better at each tier, right up until the moment "good enough and a sixth the price" climbed all the way up the ladder and there was no higher rung to retreat to. This is the most thoroughly documented pattern in business history, and it still fools the incumbent every time, because every individual decision to abandon the low end looks smart in isolation.

Open weights are rebar. Four points behind the frontier on the index, free to download, a sixth of the cost, and they run in a building you control. For the overwhelming majority of what enterprises actually do with these models, which is not frontier mathematics but classification, extraction, summarization, routing, and the ten-thousand boring tasks that make up real production work, "good enough at a sixth the cost and I can run it on my own machines" is not a compromise. It is the winning bid. The closed labs keep the genuinely hardest tier, and they are right that they have it, and they are watching the price of everything below it get set by a company in Beijing that licenses its weights for the cost of agreeing to an MIT license.

Where the moat went

Here is the part the leaderboard does not measure, and it is the whole game. The word that matters in "open-weight model" is not "model." It is "open." A closed frontier model is a dependency. You rent it, you live on its pricing, its release schedule, its content policies, and its jurisdiction, which is the exact bind I described when Apple wired Siri to a competitor's Gemini and could not attest its way back out of renting the part that thinks. An open-weight model you run yourself is the structural opposite of that. Nobody can reprice it on you, nobody can deprecate it out from under you, and nobody can change what it will and will not say after you have built on it.

Which means that when the model itself becomes free, open, and good enough, the leverage stops living in the model. It moves to the two things the leaderboard will never score: where you run the thing, and what data you feed it. If the weights are a commodity you can put anywhere, then the entire competitive question becomes whether you can put them next to your data instead of shipping your data to them. The moat drains out of the model and pools in the data plane, in locality, in the boring infrastructure that decides whether your near-free intelligence runs against a local cache or racks up egress fees round-tripping to a central cluster. Beijing just did the industry the favor of making the model the cheap part. The expensive part is the part nobody is benchmarking.

The integrated mills kept making the best steel in America right up until the day the best steel stopped being the thing that decided who survived. Figure out where your leverage actually sits before the commodity tier figures it out for you.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Eaten From the Bottom.

We Rented the Mainframe Back

David Aronchick — Fri, 26 Jun 2026 18:29:52 +0000

On Wednesday, June 10, Google's Gemini stopped answering, world-wide. It threw Error 1076 and Error 1099 at users in at least nine countries for roughly seven hours, from 3:26 in the morning Pacific until 10:30, when Google called it resolved and pointed at a backend database. The next afternoon, Microsoft's Copilot went dark for thousands of people, DownDetector reports spiking past twelve thousand, the company eventually tracing it to a botched software update it had to roll back. Two assistants, two companies, about thirty-six hours apart.

In neither case did the model break; the thing that broke was the wire.

Platforms go down! And, as a former employee of both companies, I can tell you that many thousands of employees are working INCREDIBLY hard to prevent this. But even so, a whole bunch of people (those tasked with choosing an AI model/company) who have never thought about tail latency and number of 9s of uptime are suddenly having to become aware of the basics of service availability. And, sadly, we've done them no service since we have spent two years arguing about whether these systems can reason, whether they're conscious, whether they'll take everyone's job. Last I checked, a team of humans fairly rarely disappear for hours on end. You can have the smartest model ever built and it is worth exactly nothing to the person staring at a spinner because the token issuer two hops upstream just fell on its face.

So who cares if a chatbot takes an afternoon off? Well, in our new world, the chatbot has become load-bearing. Copilot is wired into Windows, into Edge, into the guts of Microsoft 365, doing code completion and drafting and the actual minute-to-minute of how a lot of people get work done. When it goes quiet, those people don't fall back to doing it the old way, because for a lot of them there is no old way anymore. And, as they become more load bearing, they are also facing growing pains. Network monitors logged a 30 percent jump in public-cloud outage events that same week backed up by Forrester who has been saying out loud for months that the AI build-out will trigger two multi-day hyperscaler outages this year. This is not a fluke; it is the shape of the thing.

NOW WE GET TO THE STUPID THING THAT MAKES ME SHAKE MY HEAD. We spent forty years walking away from this exact architecture, and last week we walked right back into it. The entire arc of computing from about 1980 to 2010 was decentralization. The PC pulled compute off the mainframe and put it on your desk, and the reason that mattered wasn't speed, it was blast radius. If your machine died, the company kept running. Then the cloud quietly recentralized all of it, which was a perfectly good trade when the cloud was mostly where your files lived and your email got sorted. But the AI assistant is a different animal. It isn't something that generally you can route around, or build a caching layer for that hides any intermittent outages. It's become the core of the engine that makes these local rich apps work, and welcome to timesharing on a PARC-MAXC in 1981. (AS AN ASIDE: If you have not watched Halt and Catch Fire, PLEASE go do so. It is both an exceptional story about really interesting characters and a love letter to the entire computing industry of that time).

This in NO WAY is saying that Google and Microsoft are bad at this! They are about as good at running infrastructure as anyone who has ever lived, and it happened anyway, because at this level of concentration it is supposed to happen. When one backend database sits in the path of every Gemini query on Earth, that database is not a database. It's a fuse. The only open question is when it blows, and the status page will say everything is fine right up until the smoke clears. What we - the industry - need to do is built a multi-layer inference strategy, as we have been doing for other services for 20+ years, and enable some/all of that inference to live near each other and survive each other. An assistant baked into your editor ought to degrade to something small and local when the mothership is unreachable, not transform into a loading animation. Interestingly, part of Gemini DID stay up during the outage: Flash Lite, the smallest, cheapest tier, kept partially answering. The "dumb" little model that ran closer to the edge survived because it wasn't routed through the expensive part that fell over.

A few weeks ago I wrote that Apple had subcontracted Siri's brain to Gemini. Two days after that post went up, Gemini spent seven hours returning error codes to half the planet. There's zero schadenfreude here, it's a super annoying problem that no amount of engineering can prevent. What I hope happens is figuring out how we augment the existing choices in architecture. "The Cloud" is already the number-two line item on a lot of IT budgets, right behind payroll, and InfoWorld has gone ahead and called 2026 the year we stop trusting any single cloud. We solved this problem in 1995 and then we just un-solved it, because renting was easier than owning. The bill for that decision doesn't come due as a price. It comes due as a Thursday when nobody can work.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at We Rented the Mainframe Back.

The Token Got Cheaper. Your Bill Didn't.

David Aronchick — Tue, 23 Jun 2026 18:35:22 +0000

An enterprise client of an AI consultant SUPPOSEDLY accidentally spent half a billion dollars on Claude in a single calendar month (I am going to leave whether or not this is true as an exercise to the reader, because it LIKELY will happen... let's call it historical fiction?) Apparently, they had failed to set per-employee usage limits on the licenses, the agentic workflows their employees were running compounded against each other until the bill hit the comma where it did, and the consultant told Axios about it in late May. And while it is being called a "cautionary tale," the reality is that the cost structure of enterprise AI in 2026 is mismatched against the way it is being priced, sold, and budgeted for, by enough that one missing license control compounded to nine figures inside thirty days.

This is one of my BIGGEST pet peeves in the industry right now... per-token pricing for end-users.

Gartner's latest forecast says inference on a trillion-parameter LLM will cost more than 90% less in 2030 than it did in 2025. Epoch AI's tracker puts the year-over-year drop at roughly 10x for equivalent capability. Equivalent-to-GPT-4 performance, which cost more than $400 per million tokens in 2023, now sits at $0.40. That is, by every reasonable benchmark, the single largest deflation in price-per-unit performance any computing platform has ever produced.

And yet.

The companies actually deploying this technology in production are watching their monthly AI bills go up by roughly 320% year-over-year, against unit prices that fell something like 280x. Uber's CTO admitted (claimed?) in April the company had already burned through its entire 2026 Claude Code budget. We have a structural mismatch between what the industry is pricing and what the industry is actually buying. This is not going to last.

The vendors know it, and the ones closest to the cost structure are repricing first. In the first week of June, the three tools that own agentic coding all stopped pretending the flat seat could survive contact with the actual cost of inference. GitHub Copilot moved to usage-based billing on June 1, a monthly credit allotment and metered tokens after that, and developers running long agent sessions opened their first invoice to jumps of 10x to 50x. Within forty-eight hours Cursor had carved its team plans into tiers with separate usage pools, and Cognition had relaunched Windsurf as a metered Devin. Three competitors who would happily watch each other go bankrupt made the identical unpopular move inside one week, which is as good a leading indicator as anything. The all-you-can-eat seat was a venture subsidy against a bill that has now come due, and a subsidy is the most expensive thing in the world to be a customer of right up until the moment it ends.

The arithmetic of the loop

The thing the per-token price chart does not tell you is how many tokens a single user request actually generates. In 2023, the typical "AI feature" inside an application was a single model call. The user typed a question, the model returned an answer, the bill was one round trip. The unit economics were simple enough: price per token times tokens per response times number of responses per day.

In 2026, however, a modern agentic workflow, the kind every enterprise vendor is selling and every Fortune 500 is buying, calls the model somewhere between 10 and 20 times per user task. There is a planner call, a retrieval call, a verifier call, a tool-use call, a critique call, a refinement call, possibly a second retrieval informed by the critique, and a final answer-formatting call. Each of those calls is cheaper than the one call it replaced. The product of all of them, against the same user task, is more expensive than the original was.

The RAG pipelines that are now mandatory in any enterprise deployment make this worse, not better. Every retrieval-augmented call inflates the context window with retrieved documents, which means the input token count for the model balloons by a factor of three to five. The cost of an input token is lower than it has ever been, and the number of input tokens being shoved into every call is higher than it has ever been, and the two trends are not converging. They are diverging, and the divergence is the bill.

Always-on monitoring agents, the ones every cybersecurity vendor and every observability platform is now shipping with a default-on toggle, are the third factor. A monitoring agent that runs continuously against a production data feed does not generate a single request per user. It generates a continuous request per data point. The unit cost of that request is trivial, but the product of unit cost and request rate, over a month, is not trivial. It is the largest line item the buyer did not budget for. The unnamed half-a-billion-dollar customer is what happens when you stack all three of those factors on top of each other, give the result a default-on toggle, and then go home for a long weekend.

Containers got cheap. The shipping business didn't.

The cleanest analogy here is the shipping container, and I am going to use it because the parallel is exact, not because it is fashionable.

Containerization, which arrived as a serious industrial standard in the late 1960s, reduced the cost of moving a ton of goods across an ocean by roughly an order of magnitude in fifteen years. The container itself became a commodity and the price of a single trans-Pacific shipment plummeted. By every measurable unit, the cost of moving cargo went down. YET the result was not that shipping got cheaper as a category. The result was that the volume of cargo being shipped exploded, because the cost reduction made entire product categories economic that previously were not. Cheap electronics. Fast fashion. Perishable food on long-haul routes. Just-in-time global manufacturing. None of it existed at meaningful scale before the container. All of it exists now.

The visible cost is the container price, which fell. The invisible cost is what the cheap container made possible: warehousing networks the size of small countries, the inventory-financing operations needed to keep them stocked, the customs and compliance infrastructure that absorbs the friction, and the consumer behaviors that assume a six-day delivery window from anywhere on Earth. The container did not save the world money. It moved the money from the moving of goods to the storing, financing, choreographing, and consuming of them. The bill went up. The container got cheaper. Both can be true.

A token is a container. The model call is the box. The thing you actually pay for in a 2026 production AI deployment is not the boxes. It is the warehouse: the data plane, the retrieval substrate, the orchestration layer, the eval harness, the safety review, the monitoring system that runs against your monitoring system. The token is what the vendor quotes you on. The warehouse is what you actually built.

The bill you have not seen yet

The dominant cost of a 2026 enterprise AI deployment is not the LLM bill. It is the data movement that feeds the LLM, where every RAG retrieval pulls data from somewhere, and every agent invocation reads context from a database, a vector store, a cached document, a tool call, an upstream system. The bytes moved per useful answer have gone up by orders of magnitude. The price of moving a byte across a public cloud has not gone down. In some regions, against some egress paths, it has gone up.

This is the place where the entire architecture conversation should be happening, and it isn't. The vendors are competing on price-per-token because that is the metric the customer is measuring. The customer is measuring price-per-token because that is the metric the vendor is publishing. Both sides agree to compete on the part of the bill that is collapsing, and quietly ignore the part of the bill that is growing. The result is a market in which the headline cost is falling 10x per year and the actual cost is going up, and nobody is willing to put both numbers on the same slide.

There is a version of enterprise AI architecture that handles this correctly, and it is the version where the compute moves to the data rather than the data moving to the compute. If the retrieval substrate sits next to the model, you stop paying egress fees. If the agent loop runs against a local cache of the relevant context, you stop paying for the redundant retrieval round-trips. If the monitoring agents run at the edge against the data they are monitoring, you stop paying to ship that data into a central inference cluster and back out again. The unit-cost-of-token chart says nothing about this, because it is not measuring it. The total bill does.

Akamai and Comcast ran a benchmark on this in March where they had a voice small language model on four NVIDIA RTX PRO 6000 GPUs, single centralized cluster versus an AI Grid distributed across four sites, under burst traffic. The distributed deployment ran 52.8% cheaper at baseline and 76.1% cheaper during bursts, with sub-500ms latency at P99 and an 80.9% throughput gain at peak. That is what the architecture conversation looks like when you measure the right thing. It is not a per-token comparison. It is a total-cost-of-delivering-the-answer comparison, and the centralized model loses.

Stop watching the wrong number

If you are signing a contract for AI infrastructure this quarter, stop optimizing for the per-token price. The price will keep falling, on a timescale that makes any contract you sign for it irrelevant inside of a year. The vendors competing on it are competing on the visibly cheap part of a cost structure that is shifting somewhere else.

Optimize for where the data sits, what it costs to move, and which calls have to round-trip through your central inference path. The bill you have not yet seen is in the egress line item, the vector store retrieval costs, and the monitoring spend that compounds while you sleep. The bill on the model is the easy one. It is also, increasingly, not the bill.

The half-a-billion-dollar customer set their license limits wrong. That was a control failure. The control failure is interesting because the thing it failed to control got large enough in a single month to make the news. Two years ago that same control failure would have produced a five-figure bill, the CFO would have noticed at the next quarterly review, and nobody would have written about it. The control failures are getting expensive faster than the controls are getting better. That gap is the part of the cost curve nobody has put on a chart.

The token got cheaper. Your bill didn't. Both of those things are true at the same time, and the gap between them is where the next decade of enterprise AI architecture is going to be decided.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at The Token Got Cheaper. Your Bill Didn't..

Six Hundred Ways Not to Connect a Hose

David Aronchick — Sat, 20 Jun 2026 18:24:44 +0000

On the morning of February 7, 1904, fire crews from Washington, Philadelphia, and as far off as New York loaded their pumpers onto railroad flatcars and raced to Baltimore, which was on fire. When they got there, they ran their hoses to Baltimore's hydrants, and the couplings didn't fit. So a good number of those men stood in the freezing street and watched the city burn, because the threads on their hose wouldn't bite the threads on Baltimore's plugs.

The fire took thirty hours and leveled more than 1,500 buildings across seventy blocks of downtown, some 140 acres. Thirty-five thousand people lost their jobs in a weekend. Adjusted forward, the loss runs north of five billion dollars. It's still the third-worst urban conflagration in American history, behind only the Great Chicago Fire and the San Francisco quake. And the craziest thing is that the water was there, the pumps were there, and the firefighters were there. What stopped them were several hundred thin sheets of metal that made up the threading on fire hydrants. By 1903 the United States had more than six hundred different sizes and variations of fire-hose coupling.

Six. Hundred.

Ask yourself how a country ends up with six hundred incompatible ways to attach a hose to a water source, and you get to the real lesson, which has nothing to do with fire. The incompatibility wasn't an accident or an oversight; manufacturers patented their own couplings and guarded them. This made every city forced to sink real money into whatever system it already owned. The standards efforts that had been kicking around since the 1870s went exactly nowhere, because the people who would have had to adopt a standard were doing fine without one and the people selling the equipment made more money when you couldn't take your business across the street. (W3C and IETF folks, I KNOW you are shaking your heads right now). It took an enormous disaster to expose a market structure where proprietary couplings were more important than safety.

I'm now supposed to say "and isn't that just like the cloud," and the irritating truth is that it is, in almost perfect detail. Your data sits in one provider's object store, in their preferred table format, and the day you decide to move it to a competitor you discover the coupling: the egress fee on the way out, the proprietary format that needs translating, the API that is almost but not quite the one next door. It's the hose that won't thread onto the other guy's hydrant, and like the 1904 version, it is engineered to not fit. Brussels finally lost patience with it, and the EU Data Act bans cloud switching fees outright as of January 12, 2027, caps the notice period for leaving at two months, and forces providers to publish a full list of the data categories you're allowed to take with you when you go. That is, almost word for word, a fire-hose coupling standard, about 122 years after the lesson learned by the National Fire Protection Association.

But the Baltimore story does not end with "and then they fixed it." After 1904 the NFPA did the obvious thing and published a national standard coupling. A century later, a NIST study went and checked, and found that only 18 of the 48 largest American cities had actually adopted it. Thirty cities, a hundred years on, were still running their own thread. The standard has been put in place! It's been around FOR A CENTURY. But inertia is still winning today. The 1991 Oakland Hills firestorm burned hotter and longer in part because Oakland's hydrants used a three-inch coupling while the mutual-aid crews showed up with the two-and-a-half-inch national standard. Twenty-five people died in a fire made worse by a thread mismatch, eighty-seven years after Baltimore made that exact lesson free for anyone willing to read it.

So while you CAN say "standardize everything," but that's just not enough. A world where every system speaks one format and routes through one provider is a world where a single bad config push knocks everyone flat at the same instant, which happens ALL the time (I'm not providing links, because i don't want to shame people, but search yourself; no matter what date you are reading this - today or ten years from now - I will bet dollars to donuts that there's a "down time" notice from a major provider in the past week). The real lesson is narrower and more annoying than "pick a standard." It's that the failure is almost never inside the box. It's in the coupling between boxes, the seam everyone treats as a tedious detail because it's boring and it's somebody else's job. Baltimore had no shortage of water. The cloud has no shortage of compute. What neither one reliably had was a connection that worked when it actually mattered, owned by someone whose business depended on it working rather than on it quietly not.

We are very good at building magnificent boxes. We ship the seams as an afterthought, or worse, as a moat. Mayor McLane stood in the ashes in 1904 and told the cities offering help that "Baltimore will take care of its own, thank you." It's a great line. It's also what every cloud contract says to you, in much smaller type, on the way in.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at Six Hundred Ways Not to Connect a Hose.

Apple Just Subcontracted the Voice

David Aronchick — Sat, 20 Jun 2026 00:23:07 +0000

On June 8, the keynote that opened WWDC unveiled "Siri AI", the rebuilt assistant Apple has been promising and delaying since 2024. The demo was really good! And it did all the things that we would EXPECT an AI should do in 2026. I thought it was particularly interesting that the new Siri runs on Google's Gemini. Apple licensed a custom Gemini build of around 1.2 trillion parameters, is reportedly paying something close to a billion dollars a year for it, and quietly retired the ChatGPT hand-off that was the showpiece of the 2024 launch. The most tightly controlled hardware in consumer technology now does its hardest thinking on a competitor's model.

I want to be fair to the engineering, because it IS very good, and it is not the cartoon version where Apple ships your diary to Mountain View. Apple built a three-tier stack: simple requests stay on the device, moderately hard ones go to Apple's Private Cloud Compute, and only the heaviest reasoning routes out to Google Cloud, where the custom Gemini runs on what is probably some combination of TPUs and NVIDIA accelerators. Queries that leave the phone are anonymized and tokenized so that, by Apple's account, neither Apple nor Google can tie a request back to a person. If you are going to rent a brain, this is close to the most careful way to wire it, and the byte-level privacy story mostly survives the announcement. That is not the part of the announcement that is interesting.

What the architecture used to say

In June 2024, Apple staked Apple Intelligence on a specific architectural claim. The premium property of an Apple model was that it ran on the device, your data never left, and the rare query that exceeded on-device capacity went to Private Cloud Compute, Apple's own hardware in Apple-controlled enclaves with cryptographic attestation. Third-party models were a fallback, available when you explicitly chose them. ChatGPT was the named partner; Gemini was discussed but not shipped. The hierarchy was on-device first, Apple's cloud second, somebody else's model last and only by choice.

The bet that Apple was making was their silicon team and their model team, running the same roadmap, would close the gap to frontier capability inside two years. The need for an outside frontier model was supposed to be temporary.

And in many ways it was! But the external world kept going even faster.

Why it widened

The on-device model Apple shipped in late 2024 was not the one the original pitch implied. Its capable cousin, the internal frontier model, slipped twice and landed in restructured form after the WWDC 2025 reorganization. Apple's foundation-model group lost senior people to Meta's superintelligence group and to Anthropic over the same stretch. Google, meanwhile, shipped Gemini 2.5, then 3.0, then 3.1 Pro on roughly a six-month clock, each one clearing a bar the last one missed. By early 2026 Apple's choices on the assistant had narrowed to two: ship a Siri that worked, or ship a Siri whose architecture matched the 2024 marketing. Monday told you which one Apple picked.

What actually changed

The thing that changed on Monday is not where your bytes go, because Apple engineered that fairly well. The thing that changed is who supplies the intelligence. For a decade Apple's entire argument, the one that justified designing its own chips and writing its own frameworks and refusing the easy integration, was that owning every layer of the stack was the only way to keep the promises it made about the device. On Monday Apple kept the assistant promise by renting the most important layer from the one company it competes with most directly across phones, ads, browsers, and now models. Their "we own the whole stack" became "we own the stack except the part that does the thinking," and you cannot attest your way out of that sentence.

Lots of folks are calling this a blow to "sovereign AI", and in the small and specific sense that matters to anyone who builds systems, it kind of is. Apple's most strategic consumer feature now carries a hard dependency on a competitor's model, a competitor's pricing, and a competitor's release schedule, and for the heaviest queries it runs inside a jurisdiction Apple does not control. Most users will never notice and most queries will never matter.

The biggest thing that changed here is the strategy that caused Apple's position movement, not any individual query. They admitted that the industry (and customer expectations) are moving too fast for them to keep up.

Right in physics, wrong in calendar

The on-device thesis was the architecturally correct answer to the question Apple was asking, where privacy by construction beats privacy by contract, and on-device latency beats a data-center round trip. Apple's silicon division spent ten years building the substrate that should have made on-device frontier intelligence a category.

However, the calendar call, and the rest of the world, missed. Apple bet its model team could reach the frontier as fast as its silicon team and product team could ship, and the frontier moved faster than any single company's roadmap. By the time an on-device path would have reached parity, Google had three more model generations out, OpenAI had four, and Anthropic had the tier jump that produced Mythos. Right on the physics, wrong on the calendar, and in product the calendar wins every time.

There is a pattern here that is going to define the next couple of years. The vertically integrated "own every layer" architecture is the correct answer to the long-horizon question about control. However, for a while anyway, it will lose to the federated "compose across whoever is best this quarter" architecture on the short-horizon question of what ships now.

The part to watch starts about eighteen months out. It might show up as Google's Gemini roadmap shipping on a clock that is inconvenient for Apple's launch calendar, or the billion-a-year tenancy gets renegotiated in a direction that pinches the Services margin Apple has spent years defending, or a Google policy change moves what Siri will and will not say, on a timeline that is not Apple's. None of that has happened yet, but it could, and it would cause a huge chasm. It's certainly uncharted waters (or at least uncharted for many years) for a company that previously prided itself on owning everything down to the silicon, where now it carries possibly huge decisions on a schedule Apple does not fully set.

Apple spent a decade telling you that owning the whole stack was the only way to keep a promise. On Monday it kept the promise by leasing the part that thinks.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!

Originally published at Apple Just Subcontracted the Voice.

Apple Just Subcontracted the Voice

David Aronchick — Tue, 16 Jun 2026 19:08:09 +0000

On June 8, the keynote that opened WWDC unveiled "Siri AI", the rebuilt assistant Apple has been promising and delaying since 2024. The demo was really good! And it did all the things that we would EXPECT an AI should do in 2026. thought it was particularly interesting that the new Siri runs on Google's Gemini. Apple licensed a custom Gemini build of around 1.2 trillion parameters, is reportedly paying something close to a billion dollars a year for it, and quietly retired the ChatGPT hand-off that was the showpiece of the 2024 launch. The most tightly controlled hardware in consumer technology now does its hardest thinking on a competitor's model.

I want to be fair to the engineering, because it IS very good, and it is not the cartoon version where Apple ships your diary to Mountain View. Apple built a three-tier stack: simple requests stay on the device, moderately hard ones go to Apple's Private Cloud Compute, and only the heaviest reasoning routes out to Google Cloud, where the custom Gemini runs on E_SECRET_HARDWARE_BUT_PROBABLY_SOME_COMBINATION_OF_TPU_AND_NVIDIA. Queries that leave the phone are anonymized and tokenized so that, by Apple's account, neither Apple nor Google can tie a request back to a person. If you are going to rent a brain, this is close to the most careful way to wire it, and the byte-level privacy story mostly survives the announcement. That is not the part of the announcement that is interesting.

What the architecture used to say

And in many ways it was! But the external world kept going even faster.

Why it widened

What actually changed

Lots of folks are calling this a blow to "soverign AI"and, in the small and specific sense that matters to anyone who builds systems, it kind of is. Apple's most strategic consumer feature now carries a hard dependency on a competitor's model, a competitor's pricing, and a competitor's release schedule, and for the heaviest queries it runs inside a jurisdiction Apple does not control. Most users will never notice and most queries will never matter.

Right in physics, wrong in calendar

The part to watch starts about eighteen months out. It might show up as Google's Gemini roadmap shipping on a clock that is inconvenient for Apple's launch calendar, or the billion-a-year tenancy gets renegotiated in a direction that pinches the Services margin team at Apple spent his tenure defending, or a Google policy change moves what Siri will and will not say, on a timeline that is not Apple's. None of that has happened yet, but it could, and it would cause a huge chasm. It's certainly uncharted waters (or at least uncharted for many years) for a company that previous prided it self on owning everything down to the silicon, wher now they have possibly huge decisions on a schedule Apple does not fully set.

Apple spent a decade telling you that owning the whole stack was the only way to keep a promise. On Monday it kept the promise by leasing the part that thinks.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Apple Just Subcontracted the Voice.

The Leopard's Head

David Aronchick — Tue, 09 Jun 2026 18:38:25 +0000

On May 19, somebody logged into a single npm account and, over the next twenty-two minutes, published 637 malicious versions across 317 software packages. I wish the attack had been a least a little bit interesting, but it wasn't.

They logged in with valid credentials, the registry said welcome back, and an automated script did the rest. The poisoned packages included echarts-for-react, a charting wrapper that pulls well over a million downloads a week, along with a pile of the @antv data-visualization libraries that sit quietly underneath dashboards at companies that have never once heard the name AntV. The payload was a 498-kilobyte obfuscated script that went looking for everything worth stealing: AWS keys, Kubernetes service-account tokens, GitHub tokens, npm tokens, SSH keys, and the local vaults of 1Password and Bitwarden. If your project carried "echarts-for-react": "^3.0.6" in its package.json, that innocent little caret resolved you to the malicious 3.2.7 on the next clean install. You did not have to do anything wrong; you only had to have done everything normal.

This one is called Mini Shai-Hulud, and it is the small, fast cousin of the Shai-Hulud worm that tore through npm last September, the self-replicating one that used each maintainer's stolen token to poison the next maintainer's packages and backdoored hundreds of them before anyone could react. The mechanism never changes; it's always one account with all the trust, and the account belongs to a person.

Everybody in software has seen the xkcd. All of modern digital infrastructure drawn as a teetering tower of blocks, the whole thing balanced on one tiny load-bearing piece labeled "a project some random person in Nebraska has been thanklessly maintaining since 2003." We laughed because it was true, but it stopped being funny the moment somebody noticed that the person in Nebraska also has an npm token, that the token is the actual load-bearing piece, where if you can just phish the human holding it, you win. We built a trillion-dollar industry on a trust model that reduces, when you say it plainly, to "the package is fine because Dave uploaded it and Dave seems nice."

There are two popular responses to this. The first says the answer is memory-safe languages, rewrite the world in Rust, and a lot of that is genuinely good engineering, but it is repairing the wrong floor of the building. No amount of memory safety protects you from a process that had the password. The second response is more sophisticated and much closer to right: provenance, signing, software bills of materials, attestation, the whole supply-chain-security apparatus. Verify what you install instead of trusting where it came from. On paper this seems great!

The problem comes from who holds the stamp. Most of these schemes end up living as a feature of the same registry that earns its numbers by making publishing as frictionless as possible, which means the body certifying the package and the body that profits from a flood of packages are the same body. That is not verification; that is self-attestation with extra steps.

The year 1300

A silver spoon has exactly the same trust problem as an npm package. You cannot tell by looking whether it is sterling or whether the maker quietly cut the silver with something cheaper, and by the time you find out, the maker is three towns away and so is your money. The medieval answer was not "trust the silversmith." It was also, and this is the part we keep skipping, not "make all the silver in one royal workshop." In 1300, a statute of Edward I required that every article of silver meet the sterling standard, 92.5 percent, and be tested by independent guardians of the craft who struck it with a leopard's head if it passed. In 1363 they added the maker's mark, so the object carried the identity of who made it, permanently, stamped into the metal. By 1478 the testing was consolidated at Goldsmiths' Hall in London, which is where the word hallmark comes from. The mark struck at the hall.

In a lot of ways, this is the same problem (and solution) to what we have today. The object carries its own provenance, struck into it, so the proof travels with the thing and not with some database you have to phone at install time. The assayer is independent of both the maker and the seller, and is paid to be right rather than to move volume. And the standard is a published number, 92.5, not a vibe about whether the silversmith seems trustworthy. Seven hundred years ago, a guild of fiercely competing London metalworkers agreed to submit to an outside examiner with a stamp, because every honest maker understood that a market where buyers cannot verify quality is a market that eventually charges everyone the fraud discount. Daniel Stenberg, who has maintained curl for more than twenty-five years and has watched more of this go wrong than almost anyone alive, said it this month: the industry has to move from trust to verification. He is describing the leopard's head, and we have just not struck it yet.

In our case, the code lives in a million repositories on a thousand machines and is totally decentralized. But the trust lives in one account protected by one password belonging to one tired volunteer, which is about as centralized as a thing can get. We spent a decade congratulating ourselves on the first fact and ignoring the second, and Mini Shai-Hulud is what the second fact looks like when somebody finally reads it back to us at machine speed.

I wrote last week about cloud egress and the company store, about the man behind the counter who shrugs when the price of flour goes up because nobody in the conversation is allowed to be responsible for it. The package registry is the same counter. When the poisoned version lands in your build, you call your vendor, who points at the dependency, which points at the maintainer, who points at the phishing email, and everyone is technically blameless while your AWS keys are already in somebody else's terminal. Until the software commons has a leopard's head, struck by somebody who does not get paid by the package, "supply chain security" is going to keep being a man behind a counter, shrugging.

The Goldsmiths' Company figured this out before England had a central bank. Maybe we send them a résumé.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Leopard's Head.

A Pledge Is Not a Repair Ship

David Aronchick — Fri, 05 Jun 2026 18:34:00 +0000

On May 30, on the sidelines of the Shangri-La Dialogue in Singapore, seventeen countries launched a framework called GUIDE, the Guiding Principles for Underwater Infrastructure Defence Exchanges. The signatories are Singapore, Britain, France, Italy, the Netherlands, Sweden, Finland, Estonia, Latvia, Lithuania, Australia, New Zealand, the Philippines, Malaysia, Brunei, Thailand, and Qatar. The framework is, in the explicit language of the people who wrote it, voluntary, non-legally binding, and non-financially binding. It exists to share information and best practices and to improve crisis response should the need arise. It is, functionally, a very serious group chat.

I want to be fair to it, because the diagnosis underneath it is exactly right, and it took the world an embarrassingly long time to articulate. There are more than 550 submarine cables on the seabed, running over a million kilometers, and they carry something like ninety-nine percent of the data that moves between continents. That's ninety-nine percent of everything: interbank settlement, military traffic, the API call your phone makes before you have finished lifting it off the nightstand. Since 2022, around ten cables have been cut in the Baltic alone, seven of them in a single stretch between November 2024 and January 2025, by anchors that happened to drag for dozens of miles across exactly the wrong piece of ground. China operates a purpose-built cable-cutting vessel it is not especially shy about. Taiwan loses cables in the Strait nearly on a schedule. The cables are the most important infrastructure almost nobody owns, and seventeen governments finally noticing that out loud is a good thing.

HOWEVER.

A voluntary, unfunded framework that does not include the United States or China is not a defense of anything. It is a description of a problem, co-signed. The two countries with navies that could actually escort a cable ship or shadow a loitering trawler are the two countries not in the room, and the United States in particular has its own Critical Undersea Infrastructure Resilience Initiative Act sitting in committee, which is its own species of pledge. Everyone agrees the ocean matters yet, in a thing which is far too common for our modern times, nobody has agreed to pay for the ocean.

And paying is the whole game, because the binding constraint on undersea resilience is not awareness and it is not principles. It is hulls. The entire planet is served by a repair fleet of sixty-two vessels, and fewer than twenty of them are dedicated to repair rather than to laying new line. A lot of the recent additions are secondhand oil-and-gas construction ships pressed into a job they were never designed for. By 2040 roughly half the fleet reaches the end of its service life, and TeleGeography puts the bill to modernize it at around three billion dollars that no one has volunteered to cover. You can sign all the principles you like, but when the cable parts off the coast of Taiwan, the thing that fixes it is a thirty-year-old boat and a crew of splicers, and there are not enough of either.

What kills me is that we already solved this once, on purpose, with worse technology and a clearer head.

On October 31, 1902, Britain completed the All-Red Line, the round-the-world telegraph network that ran only through territory the empire controlled. The design goal was not cost and it was not even speed; the goal was survivability under attack. The cables were routed so a message from London could reach Australia going west through Canada and the Pacific, or east through the Mediterranean and India, and the landing stations sat on soil the Royal Navy could defend. By 1911 the Committee of Imperial Defence ran the arithmetic and concluded that an enemy would have to cut forty-nine separate cables to isolate Britain from her empire. That's some resilience... way before the days of ARPANET. They engineered that much redundancy into a copper network in the age of coal, because they understood in their bones that a wire on the ocean floor is a wire somebody can cut.

They understood the other half of it too. On the first morning of the First World War, Britain sent a ship out to dredge up and sever Germany's transatlantic cables, forcing German traffic onto wires the British could read. They had pre-positioned the means to do it years before they needed it. The Victorians did not write a framework about the importance of cables. They built route diversity, put the landing points on their own ground, and kept a boat ready to go. The whole apparatus was the opposite of a pledge; it was capital, topology, and a navy.

What we did instead, over the last twenty years, was let the map collapse. We abstracted the ocean into a blue rectangle behind the word "cloud," and we let cables follow the cheapest dredging route, which is precisely why so much of the world's traffic now funnels through a handful of chokepoints, the Red Sea, the Luzon Strait, the Strait of Malacca, where one well-placed anchor takes out three systems at once. A dozen cables stacked through the same trench is not redundancy. It is a dozen cables in one big trenchcoat. Redundancy is a property of topology, not of quantity, and we optimized the topology away because the spreadsheet that approves a cable route has a column for cost per kilometer and no column for what happens when somebody who hates you owns the seabed it crosses.

The fix is not a treaty you sign after the cut; it is the same fix it was in 1902. Spread the routes. Own, or at least diversify, the landing points. Pay for the boats before you need them. That is true of a cable map and it is true of every architecture decision you stack on top of one. If your business stops the day one trench floods, you're going to wish you had built something other than a single point of failure and a whole lot of hope.

The Victorians were imperialists with a telegraph monopoly and a great deal to answer for. They also understood, in a way a 2026 procurement process does not, that the cheapest path and the survivable path are almost never the same line on the map. Seventeen countries just signed a piece of paper agreeing that the ocean matters. The Royal Navy would have asked where the boats were.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at A Pledge Is Not a Repair Ship.

Sharp Bones

David Aronchick — Tue, 02 Jun 2026 19:06:20 +0000

Three weeks ago, SoftBank announced it is converting a former Sharp LCD factory in Sakai, Osaka into a 140-megawatt AI data center, then bolting a gigawatt-hour-scale battery manufacturing plant onto the same site. And whether or not you call this a growth strategy or a a new business line, it is a fascinating admission about where compute can and cannot be built in 2026.

The factory in question is the old Sharp Display Products plant, the one that briefly made Japan the center of the global LCD industry in the late 2000s and then, after the panel war was lost to Korean and Chinese competitors, sat as a kind of monument to a defeat nobody wanted to commemorate. SoftBank is putting the AI compute infrastructure inside the same shell. 110 ExaFLOPS of capacity, drawn from the same grid connection that once fed cleanrooms producing television panels for the Trinitron generation that didn't quite make it. The battery factory, a separate facility on the same site, is being built to manufacture grid-scale storage for both the AI workload itself and, eventually, the broader Japanese power market. Production starts in fiscal 2027 and reaches gigawatt-hour scale by fiscal 2028.

What is actually scarce

While the headline shortages in AI infrastructure right now read like a list of components. H100s, then Blackwells, then HBM, then the substrates the HBM stacks sit on. Every one of those component shortages has resolved itself, usually within eighteen months of the panic peaking. The shortages that have not resolved are the ones nobody can manufacture: permissions, easements, transmission interconnects, water rights, and the local political will to host a hundred-megawatt load.

PJM, the largest grid operator in the United States, has admitted in writing that it has years, not decades, to figure out how to absorb the AI load growth its territory is already committed to delivering. The chair of the federal regulator has gone on the record calling PJM "too big to function". American Electric Power, one of PJM's largest member utilities, is openly considering leaving the operator entirely. The moratorium count keeps climbing: 78 jurisdictions have now paused or banned new data center construction, against eight the same time last year. None of that is a chip problem. It is a problem about the prior generation of infrastructure choices, made by people who had no idea what we would later ask the grid to do, locking in the topology that determines what we can build now.

The cheapest gigawatt of AI capacity you can buy in 2026 is one that already exists. A substation, a transmission corridor, a parcel zoned heavy industrial, and a workforce that already has the security clearances, the union agreements, and the muscle memory of running clean-room shift schedules. SoftBank is buying all of that in Sakai. Meta, in different ways, is doing it on the Holly Ridge site in Louisiana, the same way it earlier did in Prineville. The hyperscale build that gets press is the one with a hyperscaler logo on the fence. The hyperscale build that gets done is the one with a grid interconnect already approved.

The factory inherits the politics, too

There is a second piece that the SoftBank story that I found particularly interesting. When you reuse an industrial site, you inherit the political relationship along with the physical infrastructure. Sakai City already knows what a Sharp factory is, the local government has decades of practice negotiating with a heavy industrial employer that consumes the kind of power a small city consumes, and the water utility has the supply curves. And, I think most importantly, the trained workforce is already accepted and well integrated into the surrounding community as the people who go to work at the factory.

Compare this to Loudoun County, which I wrote about last month. Loudoun was the most permissive jurisdiction for hyperscale data center construction in the United States, until eighteen months of accumulated local frustration about substation noise, transmission visual impact, and groundwater drawdown converted it into the most adversarial. The same logic that produced the boom produced the backlash, and the backlash compounded faster than the boom did, because the residents had stopped recognizing the buildings going up around them.

A reused industrial site does not have that problem. The neighbors have already lived next to the factory, in some cases for two generations. The political contracts are renewed, not negotiated. That is a non-financial asset which, in the current environment, is worth more than any chip allocation a hyperscaler can extract from a foundry roadmap. SoftBank just bought it for the price of the building.

The factory is available because somebody lost

The Sharp factory in Sakai exists because, in the 2000s, Japan decided it was going to win the panel war, and it built the factory specifically to do that. The facility was the largest of its kind in the world when it opened. Sharp poured the capital expenditure of a small national defense budget into the building, the cleanrooms, the supply contracts, and the workforce. Japan lost the panel war anyway, to a combination of state-subsidized Korean champions and aggressive Chinese late-entry capacity, and the factory operated below its design capacity for a decade before Sharp's panel division was effectively absorbed by Foxconn.

That loss is the precondition for the SoftBank deal. The factory is available because the industry that built it failed. The grid interconnect is available because the original consumer of the gigawatt is gone. Industrial reuse, at this scale, is a story about what remains after a generation of national champion strategy ends in defeat, and what the next generation builds on top of the bones.

The United States has a different version of the same problem coming. The Inflation Reduction Act and the CHIPS Act produced an industrial buildout that has been extraordinarily efficient on a dollar-per-gigawatt-built basis. Yet, some of those facilities will not run at the capacity they were designed for, perhaps at margins that do not justify the federal subsidy used to construct them. In ten years, somebody will be looking at the SK Hynix fab in West Lafayette, or one of the TSMC complexes in Phoenix, and asking whether the building can be repurposed for the next workload nobody has named yet, and most of them will be. The depreciation curve on industrial real estate, in periods of structural overcapacity, runs through reuse, not abandonment.

What this implies for the build

If the cheapest path to a megawatt of AI capacity in 2026 is one that already exists, the strategic mistake many AI infrastructure investors are making is treating the problem as a build problem. It is a discovery problem. The capital is available. The chips are available. The site, the interconnect, the water rights, the political permission, and the trained workforce are the binding constraints, and they are non-fungible. You cannot manufacture a substation faster than the queue allows. You cannot manufacture a community that already accepts a factory. You can only find the ones that exist, and rebuild what is inside them.

That is not the architecture diagram the slide decks have been showing. It is what the build is going to look like anyway.

Originally published at Sharp Bones.

The Pope and the Dynamo

David Aronchick — Sat, 30 May 2026 18:17:53 +0000

On Monday, Pope Leo XIV released a 42,300-word document about artificial intelligence. The English text runs ninety pages, named it Magnifica Humanitas. He picked the date of signature for May 15, the 135th anniversary of Rerum Novarum, which is the 1891 encyclical on labor, capital, and the industrial revolution.

I find it INCREDIBLY interesting that AI has now reached a point where religious figures feel the need to weigh in. The Pope is worried about AI in warfare, and he is, and the final chapter is direct enough about it that the just-war tradition gets explicitly retired. But I read the document (so you don't have to? but you should?) and that is a small fraction of what I took away.

Specifically, the 1891 problem, for lack of a better term, is back, the Catholic Church has had a hundred and thirty-five years to think about that problem, and the answer it landed on then is the same shape as the answer it would land on now.

What 1891 was actually about

Rerum Novarum was published into a world where electrical power and steam-driven manufacturing had centralized the means of production in a way that was new in human history. A worker who used to own his tools and the seasonal value of his labor was now showing up to a factory floor where someone else owned the boilers, the looms, the dynamo, and the building that contained all three. The 1891 question was not "is technology good" since the concept of technology barely existed - it was all just "stuff" that let you "work faster." The 1891 question was who gets to own the dynamo, and what the rest of society owes to the people whose labor is now mediated by an asset they will never personally afford.

Leo XIII's answer was specific and, for the time, contrarian. He defended private property against the socialists, defended workers' associations against the laissez-faire crowd, and insisted the state had a role to play that neither side wanted to admit. He also insisted that the family and the parish and the local trade group all had functions that should not be absorbed upward into the corporation or downward into the atomized individual. That tradition is called subsidiarity. The idea is that decisions get made at the smallest competent unit, and authority only moves upward when the smaller unit cannot do the job.

Not to turn everything into computer science, but subsidiarity is, accidentally, a load-balancing principle, that is core to how a working distributed system gets built. The decision should happen where the data is, where the context is, and where the people affected by the decision actually live. Authority that gets bumped up a layer when it should have stayed local tends to ossify into something nobody asked for, and once it is up there, you cannot get it back down without breaking something.

I want to be careful here, because it is easy to make a Pope into a mascot for whatever position you already held. The encyclical is not a Distributed Thoughts blog post; it is a moral and theological document with an institutional purpose, written for 1.3 billion Catholics, and the parts of it that talk about transhumanism and embryonic dignity are not the parts I am qualified to summarize.

The part I am qualified to summarize is the part that maps to the architecture conversation, and the encyclical itself invites that mapping. Pope Leo XIV does not write about "agents." He does write about subsidiarity in the algorithmic age. He does not use the words "data sovereignty." He does spend about ten thousand words arguing that the asymmetry between the people who own AI infrastructure and the people whose labor is increasingly mediated by that infrastructure produces the same structural problem Leo XIII identified in 1891. He concludes that it does, and goes on for a while about why.

Who showed up to the launch

The presentation on May 25 was attended by, among other people, Chris Olah, who runs interpretability research at Anthropic, is one of the company's co-founders, and gave remarks at the press conference. The remarks said, more or less, that the labs operate inside incentives and constraints that can conflict with doing the right thing, and that people outside those incentives need to pay close attention and be willing to be honest critics. He thanked the Pope for being one of those critics. He used the word "unsettling" about what his own team is finding inside frontier models.

Sit with that for a second. Anthropic flew its head of interpretability to Rome to stand next to the Pope and say "we need outside oversight because we cannot reliably oversee ourselves." Whatever else you think about that, it is the most theologically literate move any of the labs has made in three years. The other labs noticed. The Washington Post coverage framed Anthropic's appearance as a deliberate alignment away from the White House and toward the Vatican, which, regardless of intent, is now the cleanest description of where the moral high ground of this debate actually lives.

The 1891 prescription, in 2026

The labs are very willing to talk about safety. They are very unwilling to talk about who owns the dynamo. The Pope just wrote ninety pages saying the second question is the question, and that the first question, on its own, gets you nothing useful. You can build the safest possible model and still hand it to eleven counterparties under a Glasswing-class contract the rest of the market cannot sign, and you will not have addressed any of the questions a serious moral framework would have asked you to address. You will have addressed about half of one of them.

I do not agree with everything in Magnifica Humanitas. I do not need to. The point is that the institutional response to a wave of centralized infrastructure is forming, the framework that is going to do the most coherent intellectual work over the next decade was just published by a 70-year-old Augustinian, and the people responsible for the centralization have, with one notable exception, not read it.

They should. The diagnosis is good. The diagnosis was already good in 1891. The prescription is the same as it was then, which is that authority not held locally tends to ossify into something nobody asked for and nobody can leave. The labs have built infrastructure of exactly that shape. The grid bills are going up. The data center moratoriums are spreading. The people whose work is increasingly mediated by the model cannot vote on it, cannot leave it, and increasingly cannot afford the electricity it draws.

Push the decisions down. Push the compute down. Keep the dynamo close enough that the parish can see it.

That is what 1891 actually figured out. The Pope is the one who remembered.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at The Pope and the Dynamo.