DEV Community: David Aronchick

A Kill Switch of One's Own

David Aronchick — Sun, 18 Oct 2026 18:01:02 +0000

In June, the European Commission unveiled its Technology Sovereignty Package, a stack of proposals built around a Cloud and AI Development Act and a second Chips Act, and the people selling it were unusually honest about the fear underneath. The worry, in the framing they chose themselves, is a kill switch: the possibility that the cloud running Europe's hospitals and ministries and grid operators answers to a jurisdiction on the other side of the Atlantic, and that the access could be throttled or cut from there. Executive Vice-President Henna Virkkunen said the goal was to guarantee that critical workloads do not carry a remote off switch somebody else controls. Von der Leyen said Europe cannot afford to depend on others for the technologies that keep its hospitals running and its grid stable. The Act proposes to triple EU data center capacity within five to seven years, sort cloud services into four "assurance levels", and push sensitive government workloads away from AWS, Microsoft, and Google. The Commission puts the data center build alone at around two hundred billion euros by 2036, mostly private money. Its own target date for Parliament and Council merely agreeing on the text is the fourth quarter of 2027. The kill switch, presumably, will wait.

The jurisdiction problem is still real. The CLOUD Act concerns access to data under qualifying legal process; it is not a general authority to shut down a cloud service. Sanctions, commercial decisions, and operational failures create other dependencies. On May 7, a cooling failure in one Northern Virginia Availability Zone forced servers to shut down and left customers working through a prolonged recovery. That was a localized infrastructure failure, not the whole region going dark. It still showed why the location and control of a dependency deserve separate scrutiny.

The proposal needs to be judged by what operators can do when a dependency fails.

The Commission includes an open-source strategy, and its sovereignty framework considers operational autonomy, control, resilience, and exposure to third countries, so they are going to more of the primitives involved. My concern is what happens after certification. For example, can an operator move a workload, keep its data usable, and recover without permission from the failed provider? More European capacity can reduce some dependencies, but capacity and assurance levels alone do not demonstrate that those recovery paths work.

Europe has run this experiment before recently, and it kept the receipts. GAIA-X launched in 2020 as the continent's sovereign-cloud answer to exactly this anxiety. Within a year the American hyperscalers were members; one of the French founders, Scaleway, walked out, saying it had no more time to waste on a project rotting from the inside with American Big Tech, and the European share of Europe's own cloud market kept sliding while the project that was meant to reverse the slide held conferences. It is hard to build a moat when you invite the besiegers onto the committee that designs the moat.

There is an older warning in France's Plan Calcul, launched in 1966 to reduce dependence on foreign computer suppliers. Its national champion, CII, was eventually absorbed into Honeywell-Bull in 1976. That history does not prove the current European proposal will fail. It does show that domestic ownership at launch is a poor substitute for control that survives later mergers, suppliers, and operational changes.

Sovereignty has both legal and operational parts. Workloads that can run across independent providers and your own hardware can reduce the reach of any one service-denial decision. That takes more than placing a second logo on the architecture slide. Identity, DNS, networking, data formats, and control planes can preserve the same dependency across nominally different providers. Failover has to be tested, the dependencies have to be independent, and the people operating it have to know what they can keep running when one supplier disappears.

Europe is right to examine who can interrupt its critical systems. The useful test for the package is whether it buys operators a working alternative: a workload moved, a dependency isolated, and a recovery rehearsed. Ask for that evidence alongside the capacity targets and assurance certificates.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!

Originally published at A Kill Switch of One's Own.

The Chaperone

David Aronchick — Wed, 14 Oct 2026 18:01:17 +0000

IBM's biggest quantum chemistry result depended on a conversation between very different machines. In May, Cleveland Clinic, RIKEN, and IBM simulated a protein–ligand complex in water totaling 12,635 atoms, which IBM describes as the largest heterogeneous quantum-classical electronic-structure calculation to date. Two 156-qubit IBM Heron processors, one at Cleveland Clinic and one at RIKEN in Kobe, did the quantum part. Classical computers did everything else: cutting the protein into fragments, sampling the nastiest fragments, and then... handing it off. Then two classical supercomputers took those samples: Fugaku, which topped the TOP500 rankings before Frontier, and Miyabi-G. The algorithm (called EWF-TrimSQD, because quantum computing has never met an acronym it didn't want to marry) made progress by stitching the molecule back together.

It's kind of crazy to see the pace of things. In January, at Supercomputing Asia, IBM's Jay Gambetta presented what was then the largest and most accurate chemistry experiment ever performed on a quantum computer: a pair of iron-sulfur molecules, solved by one Heron chip in a closed loop with all of Fugaku, the two machines passing data back and forth in an unbroken cycle. The protein work started smaller: Trp-cage, 303 atoms. Five months later, in May, it had a T4-Lysozyme–ligand complex in water at 11,608 atoms and a trypsin–ligand complex in water at 12,635, a forty-fold jump. That's a big jump! And I don't want to take that away from anyone at Cleveland Clinic, IBM, or RIKEN who spent the preceding year getting a quantum processor to provide a real-world use case. BUT.

Quantum computing has been sold for something like two decades as the technology that gets you out from under classical computing's limits, the thing that solves what a normal machine can't solve no matter how many racks you buy. Chemistry simulation is the textbook example: the compute required to track every electron's interaction with every other electron grows exponentially with the size of the molecule, which is why a high-accuracy classical treatment of a whole protein is still impractical and why quantum computing has always used chemistry as its opening pitch. So, we have the biggest (quantum) chemistry result of the years still needing classical machines at every single step, and... it still does not yet outperform the best classical approaches. There's a good reason (fault tolerance) which will be addressed by 2029, which will still not remove classical control, error decoding, or the classical steps of a hybrid chemistry algorithm, but it'll be ... more quantum? IBM describes the future as quantum-centric supercomputing, with the machines working together.

On narrow tasks, researchers have reported results beyond the classical methods they compared. On July 30, IBM and Qedma claimed a quantum advantage on the dynamics of a driven quantum magnet, up to 74 qubits, using error mitigation and comparisons with classical methods where those methods worked. A September report on an unreviewed Nighthawk preprint described a million random-circuit samples in 19 seconds. BlueQubit's 110-year Frontier estimate was for a particular tensor-network method at comparable fidelity, under an unlimited-memory assumption. But benchmarkmaxxing random-circuit sampling is great for press, not a chemistry result. Useful chemistry still needs a comparison against the best classical workflow for the same problem.

There's an almost too-on-the-nose precedent for this from transport. Britain's Locomotives Act of 1865, called the Red Flag Act, required every road locomotive to be preceded by a person on foot carrying a red flag, walking sixty yards ahead to warn horses and riders that the machine was coming. These were steam engines of up to 14 tons, rolling down roads built for horses. They could genuinely propel themselves, but they couldn't (yet) do was be unsupervised, so the law bolted a human chaperone onto the front, capped the speed at four miles an hour in open country and two in town, and made the astonishing new machine walk at the pace of the person escorting it. Parliament dropped the flag itself in 1878 and kept the man, who now walked twenty yards ahead with nothing in his hands. He lasted until 1896, when light vehicles under three tons, the new motorcars among them, were finally let off the leash. On November 14 that year, the Earl of Winchilsea marked the occasion by tearing up a red flag that nobody had been legally required to carry for eighteen years.

The quantum computers in Kobe and Cleveland contribute a specialized step inside a larger classical workflow. The result is real progress. The test is whether the whole workflow produces a useful answer better than the best classical alternative, including its classical processing and coordination costs. The red-flag story is a reminder to inspect what surrounds a new machine. It does not establish a date when that support disappears. IBM's 2029 target concerns fault tolerance, and its chemistry researchers expect the hybrid workflow to continue.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at The Chaperone.

The Clock Is a Server

David Aronchick — Fri, 09 Oct 2026 18:01:29 +0000

At ten to three in the morning on July 8, 2026, Telstra's network forgot what time it was. One of the carrier's time servers, the boxes that tell the rest of the network what time it is, had reset its calendar to 2006, and the bad time was propagating outward through the synchronization layer. By breakfast, 8.8 million Telstra customers were on services that were degraded or dead. Victoria's V/Line regional railway stopped running trains. National freight operators suspended movements as a safety precaution. Tyro, the payments company that handles card terminals for about 80,000 Australian businesses, started dropping transactions at shop counters. 612 triple zero emergency calls failed on Telstra's network, more than 430 of them without reaching anyone at all, and every one ended with a welfare check on the person who had tried to call for help. Telstra had the time fault fixed by about 4 p.m., some thirteen hours in, and knock-on problems ran into the next day.

Nobody attacked anything at any stage.

The independent review Telstra released on September 1, 2026, found that a power supply swap on a Melbourne time server kicked it back 1,024 weeks, to 2006, the classic GPS week-number rollover. (This is an excellent time to remind everyone to read Falsehoods Programmers Believe about Time) The vendor had issued advisories about that bug as early as November 2000 and Telstra's own technicians logged it in September 2022 and decided it didn't apply to them, because the patch fixed a feature they weren't using. Then an undocumented design change in October 2025 started using it, which, of course, nobody wrote down, so in January 2026, with the bug now very much live, they looked at the patch again and passed again. The alarms that would have flagged the timing drift were only watched during business hours, by a limited number of people, and two of Telstra's key engineers were on mandatory leave that night. The review's overarching finding was that Telstra never treated its timing system as a critical capability. CEO Vicki Brady's summary: "That is a miss on our side." This was a known defect in a component nobody treated as critical, because time does not feel like a component until it stops.

As I write this in early October, ACMA is still investigating, with fines of up to A$30 million on the table. On September 21 a Senate committee recommended taking Triple Zero away from Telstra and handing it to a new public body, and on October 1 the three big carriers switched on roaming onto each other's networks for when a disaster takes one of them down. Bushfires and floods, mind you. Nobody has a roaming plan for a clock that thinks it's 2006.

To see why one bad calendar could do that, start with the radio network. In the time-division duplex modes used by modern LTE and 5G networks, the tower and the handset share one slice of spectrum by transmitting in alternating slots, and the specifications require every cell to hold its slot boundaries within roughly 1.5 microseconds of a common reference, so no two neighbors drift more than about 3 microseconds apart. Beyond that tolerance, adjacent towers begin transmitting over each other's frames. Carriers therefore operate a dedicated distribution system for time itself, in which receivers discipline themselves against atomic clocks in orbit, the Precision Time Protocol carries the reference across the backhaul, and holdover oscillators at each site keep an approximation running whenever the upstream reference goes quiet. Time on a network like Telstra's is produced in a small number of places and consumed by tens of thousands of sites, and in July the production run was defective.

Distributed systems people have known the underlying truth for decades, and everyone else keeps rediscovering it at three in the morning. Computers cannot vote their way to the correct time, because every voting protocol needs timeouts and every timeout needs a clock, so Leslie Lamport's famous 1978 move was to give up on clocks and derive ordering from messages instead. Google went the opposite direction with Spanner's TrueTime, installing GPS receivers and atomic clocks in its data centers and having the system report an uncertainty interval and wait it out rather than pretend the time is exact. Opposite philosophies, same admission: somewhere in every distributed system sits a reference authority everything else syncs to. You do not get to not have one. The design choices are how many you have, who operates them, and how far a lie travels when one of them lies. Most of the internet answers those questions with "a handful," "volunteers and hyperscalers," and "we'd rather not think about it," courtesy of NTP.

On the night of June 30, 2012, a leap second was inserted into civil time to keep atomic clocks aligned with the earth's rotation, and a bug in the Linux kernel's handling of the extra second sent processes on servers around the world into high-CPU spin loops. Reddit went down for a while. Mozilla, LinkedIn, Yelp, and Gawker all reported problems that night. The Amadeus airline reservation system failed in Australia, and Qantas staff checked passengers in by hand for hours until it recovered. Four years later, in January 2016, the US Air Force decommissioned a GPS satellite designated SVN-23, and an erroneous ground-software upload put a 13-microsecond error into the time broadcast by 15 satellites, about half the constellation. Over the following hours, the BBC's digital DAB radio listeners lost reception because the transmitters took GPS as their timing reference, and one timing-equipment user logged nearly 2,500 alarms before the problem cleared.

We did solve this once, at continental scale, and the people who solved it ran trains. On November 18, 1883, the North American railroads imposed five time zones on the continent by private agreement, thirty-five years before Congress ratified the arrangement in the Standard Time Act. And when synchronization still failed, as it did near Kipton, Ohio in 1891, where a crew member's watch ran slow and two trains met on the same track, the response was Webb C. Ball's watch inspection regime: approved movements, thirty seconds a week of allowed drift, inspections on a schedule, paperwork per timepiece. The reference was standardized, and then every consumer of it was audited as an independent clock, so a watch could still fail but a defect could not propagate. Telstra's design inverts that: centralize the reference so everything downstream can be a dumb consumer, and one bad clock becomes everyone's outage at the same instant.

Time is upstream of everything, so a time failure surfaces downstream everywhere, wearing everyone else's costume. On July 8 it looked like a train problem, a payments problem, and an emergency-line problem, and the fix took half a day at one of the most sophisticated operators in the hemisphere, because you cannot debug a distributed system when the broken thing is the thing you use to order events. Every log line on every node carried a timestamp from the same poisoned well.

So, a suggestion, and it is the same one the railroads landed on after Kipton. Go find out where your time comes from. Not conceptually; specifically. How many independent sources, owned by whom, checked by what, and what your systems do in the hours after the holdover oscillators start to drift. If the answer is "one vendor's servers in two cities," you do not have a clock. You have a subscription to someone else's, and on July 8 all of Australia got to read the terms.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Clock Is a Server.

Estate Sale

David Aronchick — Tue, 06 Oct 2026 18:01:26 +0000

A bankruptcy court has finally put a price on losing your genome. About $2.60, and that's for the whole thing, in case you were hoping it was per gene.

And $2.60 is only the states' share. Throw in the consumer class action and the price climbs all the way to about $9.40.

Every step of this got covered as its own story, and it's one story. In 2023, a credential-stuffing attack exposed profile data on 6.9 million people. In March 2025, 23andMe filed Chapter 11, and its most valuable asset, by a wide margin, was the DNA database itself. Twenty-seven states and the District of Columbia sued to block the sale of genetic information without customer consent. The court approved the sale anyway, with privacy commitments attached, and TTAM closed the deal on July 14, 2025. The states' settlement over the breach landed July 14, 2026, a year to the day. First the estate sold the genomes, and then it settled up for losing them with the change.

That day, a coalition of more than forty attorneys general announced a settlement with 23andMe's bankruptcy trustee over that breach. The states were granted $150 million in allowed claims. What they will actually collect is $18 million, because that's what remains in the estate after everyone ahead of them in line got paid. Pennsylvania and New Jersey, two of the bigger states in the coalition, walk away with about $492,000 and just under $410,000. Spread $18 million across the 6.9 million people whose data walked out the door, and you get about $2.60 apiece. Add the $46.75 million the same court approved for the consumer class action a week earlier, and you get to about $9.40. I ran that math twice, hoping it would improve. It didn't.

Don't mistake the $18 million for a fine. A fine is a punishment, calibrated to deter. This was a claim in a bankruptcy, calibrated to whatever was left in the drawer. What your genome is worth depends entirely on which room you ask in. In the marketing room, it was worth a $99 kit and a lifetime of insight about your Neanderthal percentage. In the auction room in June 2025, Regeneron thought the database was worth $256 million for drug discovery, and Anne Wojcicki's nonprofit TTAM won it at $305 million. In the courtroom, a year later: about $9.40, with the same molecules and the same double helix. The price is a property of the venue, and the only room that sent any money back toward the people in the database was the cheapest one.

I think there's a fundamental unpleasantness that turns over all this. A privacy policy is a promise, and in bankruptcy a broken promise becomes an unsecured claim, the cheapest seat in the house, with the secured lenders eating first. Then the professionals administer the estate, billing it by the hour, and FINALLY, from whatever is left, the unsecured claims. Which is where "we will never sell your data" lives, right next to the unpaid landscaping invoice. Twelve cents on the dollar, which is what $18 million against $150 million works out to, is honestly a respectable recovery for that tier, and that is a hell of a sentence to write about your DNA. The system worked EXACTLY as designed, and the design is the problem. You cannot repossess a promise.

(And before anyone emails me: yes, the court attached privacy commitments to the sale, and yes, the buyer promised to keep the data safe. Maybe it keeps every one of those promises! But they are still promises, and we just went over what a promise fetches in that courtroom.)

With most data, this is bad. With DNA, it's a different category of bad, because a genome is the one credential you cannot rotate. Your password leaks; you change it. Your credit card leaks; the bank mails you a new one. Your genome leaks, and nobody mails you new chromosomes. It stays leaked for the rest of your life, and for a meaningful fraction of your siblings' and children's lives too, since they share the sequence and never clicked "I agree" on anything. The 6.9 million people in the breach consented, in the loose sense that word has acquired. Their relatives were simply along for the ride. Data outlives the company that collected it, and this particular data outlives the people it describes.

Genomes are the newest entry in a bankruptcy playbook whose receipts go back a quarter century. In 2000, a Disney-backed toy retailer called Toysmart went under with a privacy policy that promised customer data would never be shared with third parties. The bankruptcy listed the customer database, including data on children, as an asset for creditors, and the FTC sued to stop the sale. The settlement restricted the sale to a buyer in the same line of business, and in the end Disney paid to have the list destroyed. Fifteen years later, RadioShack (yes, that RadioShack) went into bankruptcy carrying more than 65 million customer names and 13 million email addresses on its asset schedule, auctioned them, and got 38 attorneys general objecting before a deal destroyed most of the data. Toy shoppers, battery buyers, genomes. The asset class keeps upgrading, and the playbook hasn't changed at all. I'll let you guess what ends up on the folding table next. (Hint: ask your chatbot what it remembers about you.)

I keep coming back to the same conclusion, and I promise it's not just because of where I work. (Okay, it's a little because of where I work). Paper guarantees have a documented lifespan, and it's shorter than a mortgage. The settlement's forward-looking fix, for the record, is a risk analysis, an advisory board on data security, and continued deletion rights at the buyer, which has since re-registered as 23andMe Research Institute. More paper, under the old name. The version of "we will never sell your data" that actually survives a bankruptcy is where the engineering and data teams take on the work long before bankruptcy is ever considered. Data that gets processed where it's generated, and never piles up in one central database, can't be inventoried by a trustee, can't be auctioned to the highest bidder, and can't be marked down to less than ten bucks in a courtroom you will never see. We usually sell data locality on latency and egress fees, and those are real. The 23andMe estate just made the other argument for us: blast radius. A trustee cannot sell what the debtor never possessed.

An estate sale can only sell what's in the house. Maybe stop filling the house.

Want to process data where it's generated, so there's never a pile for a trustee to auction? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Estate Sale.

The Shortest Stave

David Aronchick — Fri, 02 Oct 2026 18:01:34 +0000

A wooden barrel built from unequal staves (I love this term, by the way; I've never heard it!) holds water only as high as its shortest stave. Chemists call this the "limiting reagent," but I'm particularly fond of talking about these kinds of constraints in the living world, where things grow ... organically. Long before anyone had heard of a GPU, Carl Sprengel sketched the idea in 1828, and Justus von Liebig made it famous a couple of decades later, focusing specifically on how a plant's growth is set by whichever nutrient is scarcest. Pour on all the nitrogen you want, but if the soil is short on phosphorus, the plant stops growing at the phosphorus line, and the nitrogen sits there, unused, a resource with nothing left to do. So, back to our barrels: you can spend a fortune making the tall staves taller, but the barrel doesn't care. Every dollar spent elsewhere runs over that one short piece of wood and out onto the ground.

During 2026, the price of a stick of server RAM stopped being "Oh, I guess we'll just have to add a little bit to the price of this server, but it's diminimus compared to this Rolls Royce GPU we have in here." Conventional DRAM contract prices rose 93 to 98 percent in the first quarter alone, and TrendForce expected another 58 to 63 percent in the second, roughly tripling in six months. The reason? High-bandwidth memory will take about 22 percent of the world's DRAM wafer starts by the end of 2026, up from 18 percent a year earlier, and turn them into only about 9 percent of the bits, because a bit of HBM eats three to four times the wafer of an equivalent bit of DDR5. SK Hynix, Samsung, and Micron, the only three companies on Earth that make HBM at volume, reported their 2026 HBM capacity sold out under existing contracts, and Digitimes reported in August that most of 2027 is already allocated too. Try to buy a stick of RAM for a gaming PC right now, and you're competing with Nvidia's supply chain. Nvidia is winning.

Everybody saw this coming from the wrong direction. For two years the story was GPUs: how many Nvidia could ship, how long the allocation list revolved around, and whether the shortage had somehow become a permanent feature of the industry rather than a temporary one. Then the story became power. Gigawatts, interconnection queues, and data centers in Virginia are waiting up to seven years for a grid connection. Nobody was watching the beige stick of memory sitting next to the accelerator on the board, because memory has been the least glamorous component in a computer since the 1960s, and unglamorous things don't get congressional hearings. Now it's the thing capping how much of this you can actually build, and a fab that makes memory cannot be told to make more of it by Thursday. Wafer capacity is a multi-year build; wanting more sooner doesn't make it sooner.

The AI industry has been building staves. It built an enormous GPU stack, then an enormous power stack in roughly that order, because those constraints tailed first and made the loudest noise going down. Memory didn't make a loud noise, and so it quietly became the shortest piece of wood in the barrel. The irony is that it's not even HBM's fault! It's just that HBM and consumer memory come out of the same fabs, competing for the same wafer starts, so every additional stack of HBM that Nvidia's next accelerator needs is DRAM capacity that doesn't go into a laptop, a phone, or, this being the actual news story of the last few months, a stick of RAM for the machine on your desk. The industry solved for the two constraints everyone could see coming, and the barrel filled to a different line anyway. One nobody had thought to check.

Even worse, Samsung, SK Hynix, and Micron can't ship a patch. Building new wafer capacity takes years, and the contracts locking up existing capacity run multiple years past that. Which means the memory constraint, unlike the GPU shortage and unlike the power shortage, won't get quietly fixed by a new chip generation or another grid order. It's poured into the concrete of fabs that haven't broken ground yet. The people planning AI infrastructure right now are, whether they've noticed or not, planning against a stave whose length was mostly fixed a few years ago and cannot be argued with, subsidized, or ordered into growing faster by anyone in Washington.

The GPU story and the power story were both real. They ARE both real. Solving them was and is necessary work. Solving the constraint just moves the waterline to the next-shortest stave, and that one is rarely the one the headlines prepared you for. Liebig figured that out, staring at dirt in the 1840s. The AI industry is finding it out staring at a spec sheet in 2026 and paying triple for the privilege.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Shortest Stave.

The Elevator Receipt

David Aronchick — Tue, 29 Sep 2026 18:01:35 +0000

On July 1, Bloomberg reported that Meta is building a cloud business to sell access to its spare AI compute, an initiative reportedly called Meta Compute. This is the same company that, as of June 30, had $279 billion in data center leases signed but not yet started (and signed another $68 billion in July), with an Ohio campus coming online this year and a Louisiana campus behind it, which Zuckerberg pitched as covering a significant part of the footprint of Manhattan. Zuckerberg had said in May that a cloud business was "definitely on the table." Meta was not even first.

In early May, the entire capacity of Colossus 1, the data center xAI built before it was folded into SpaceX, went to Anthropic in a buyout deal, followed by similar leases with Google and Reflection AI. In August, SpaceX reported that second-quarter revenue had nearly doubled, from $4 billion to $7.8 billion year over year, with compute deals contributing close to $2 billion of the growth, plus another $6.7 billion in cloud revenue under contract for a six-month stretch that starts ramping in October. The exchanges moved in the same window. On May 12, CME Group and Silicon Data announced the first compute futures market, built on daily GPU rental-rate indices, and days later ICE and Ornn announced cash-settled GPU compute futures on an index tracking live spot prices for H100s, H200s, and B200s. SF Compute already runs a live order book where GPU-hours trade like any commodity, and the spot price of a Blackwell GPU-hour rose 48 percent between mid-February and mid-April, from $2.75 to $4.08.

And on that ticker, CME has already set the date its first two contracts, H100 and B200 rental index futures, would start trading, pending regulatory review: October 5, 2026.

A bit of a detour to the history of commodities. Before the 1850s, American grain traveled in sacks. Each sack belonged to a specific farmer, was moved under his name, and was sold on inspection, so a farmer with better wheat than his neighbor got paid the difference. Joseph Dart opened the first steam-powered grain elevator in Buffalo in 1843, a machine that ran grain up a bucket conveyor into storage bins and unloaded lake boats several times faster than the crews who had needed days to empty one by hand. The Chicago Board of Trade began grading wheat in 1856, sorting everything that came through into categories: No. 1 spring, No. 2 spring, and so on down the list. Once graded, a farmer's wheat went up the elevator leg and blended into a bin with every other load that met the same grade, and what the farmer carried away was a paper receipt for so many bushels of No. 2. Receipts could be sold without moving the grain. Then they could be sold before the grain existed, and in 1865 the Board formalized the practice into futures contracts.

Within a generation, Chicago was clearing trades on wheat that had not yet been harvested, and the farmers were organizing a national political movement, the Grangers, aimed in large part at the elevator operators and the railroads that fed them. In 1877, the Supreme Court upheld state regulation of elevator rates in Munn v. Illinois by dusting off a phrase from a seventeenth-century English judge, "business affected with a public interest," to address the fact that a handful of warehousemen had become the chokepoint for the national harvest. William Cronon's Nature's Metropolis tells the full story, and it remains the best systems book I know, even though it's technically about corn.

Two things happened when wheat became fungible. Liquidity exploded, which was good for nearly everyone. AND power moved away from the people who grew the wheat and toward the people who ran the elevators, defined the grades, and cleared the trades. The farmer gained a global market and lost every ounce of pricing power that came from his wheat being his.

I think this is happening again. The exchanges are building the grading system, and Meta just volunteered to pour its harvest into the bin. Bloomberg says Meta may sell "raw" capacity the way CoreWeave does, which is an admission, stated in infrastructure rather than words, that an H100-hour in Meta's Ohio campus is interchangeable with an H100-hour anywhere else. And remember the story that justified the capex in the first place: our compute trains our models; our models power our products; the flywheel compounds; none of it is for sale. Meta AI and Llama still do not appear as a revenue line anywhere, and the company has now conceded the first link in the flywheel by renting it out. (Zuckerberg's version, on the July earnings call, is that selling intelligence carries "a significantly higher margin" than selling compute, and that it "would be foolish to basically just sell all of the compute." He also said buyers are already offering "a meaningful premium" over what Meta paid for it.) The megawatt turned out to be the crop. The models were supposed to be milled flour, and there does not appear to be enough demand for it.

I don't think selling the surplus is a mistake, to be clear. Pretending your commodity is a moat costs money and adds extra steps. (And before anyone emails me: no, AWS was not built from Amazon's spare holiday capacity. That story is a myth Amazon's own engineers have spent fifteen years trying to kill. AWS was a deliberate business from day one, which is exactly the point. Companies that win infrastructure markets decide to be infrastructure companies; they don't back into it.)

But the history is blunt about who captures the profits in a commodity market, and it is not the growers. It is whoever defines the grade, runs the elevator, and clears the trade. In compute terms: the index publishers, the exchanges, the interconnects, and above all, whoever controls where a workload can physically run. A futures market only helps you if you can take delivery, and taking delivery of compute means your workload can actually move to where the cheap capacity is. If your pipeline is welded to one region of one provider, the spot market is a spectator sport, and you get to watch the price of the thing you are overpaying for fall in real time. Fungibility is a property of your architecture, and most architectures I see were built on the assumption that compute would never be worth arbitraging.

Nobody in Chicago planned to become the chokepoint for the American harvest either. They just printed the receipts.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at The Elevator Receipt.

Coase's Coffee

David Aronchick — Sat, 26 Sep 2026 18:01:09 +0000

An inventory system goes wrong before the coffee rush. Who gets the call? I made this argument about agents (you can't sue an agent), and it applies with full force here: a SaaS contract is a counterparty with skin in the game. When the inventory system is wrong today, Starbucks calls Microsoft, and an army of people whose careers depend on that phone call gets paged. When the in-house replacement is wrong at 4:30 in the morning before the rush, Starbucks calls Starbucks. Maintenance is where AI-built software goes to get quietly expensive. And AI in the stockroom has already bitten this exact company once: in May, Starbucks retired Automated Counting, a vendor-built AI system for counting milk and syrups, after it kept confusing products that looked alike, and sent stores back to counting by hand. Or ask Ford, which spent June rehiring the gray-beard engineers it thought AI had replaced. Starbucks' software bill does not disappear. Some fraction of it converts into engineers, pager rotations, and the slow institutional discovery that "the model wrote it" is not an acceptable sentence in a Sev-1 postmortem.

On July 9, 2026, Bloomberg reported that Starbucks is using AI to build in-house replacements for software it currently rents, including a Microsoft system that tracks inventory and an IBM tool that manages equipment maintenance. Starbucks spends about $400 million a year on software, according to CTO Anand Varadarajan, and the enterprise technology team expected roughly $30 million in budget savings this fiscal year, about $10 million of it from software. The first replacements could roll out by the end of 2027, pending testing. A coffee company looked at Microsoft and IBM and said: we'll take it from here.

If you want to understand why this is happening, and why it is going to happen a few hundred more times over the next three years, don't read an AI newsletter. Read a paper from 1937. Ronald Coase was 26 years old when he published The Nature of the Firm, which asks a question so simple that economists were mildly embarrassed nobody had answered it: if markets are so efficient, why do companies exist at all? Why is anything done inside a firm instead of contracted out on the open market? His answer, which eventually earned a Nobel, was transaction costs. You pull an activity inside the firm when coordinating it internally is cheaper than buying it outside, and you push it out when the reverse becomes true. The boundary of the firm isn't strategy, and it isn't culture. It's a price. When relative prices change, the boundary moves.

SaaS is a forty-year bet on one side of that equation. Building software internally was brutally expensive (hire the engineers, keep the engineers, maintain the thing forever), so vendors amortized one build across ten thousand customers and rented it back to you. This was correct. It was so correct that we stopped noticing it was a bet on a particular cost structure rather than a law of nature. Then AI coding tools cut the internal cost of producing working software by some large and still-unmeasured factor, and the equation started running backward. Gartner now expects agentic AI to put $234 billion of SaaS spending in play. And Starbucks is exactly the company you'd expect at the front of the line: its problems are specific (predicting inventory across roughly 40,000 stores, keeping espresso machines alive), the data feeding those problems comes off Starbucks' own machines and registers, and the vendor products it's replacing are general-purpose tools priced like moats.

We have run this loop before, in the other direction. In 1900, factories generated their own electricity, because that was the only way to get reliable power at industrial scale. Then central utilities got good, and by 1930 buying off the grid had crushed self-generation. Nicholas Carr wrote The Big Switch in 2008 arguing that computing would follow electricity into the utility model, and for fifteen years he was dead right: the cloud became the utility and SaaS became the appliance plugged into it. The lesson from both cycles is not that make beats buy, or that buy beats make. The lesson is that the answer is cyclical, it follows the cost structure of the era, and anyone who tells you the current boundary is permanent is selling you the current equilibrium.

The honest version of the math has three terms, and AI only changed one of them. Insourcing wins when your requirements are genuinely specific rather than generic. It wins when the data feeding the system is yours (Starbucks' machine telemetry is exactly that, and it never needed to leave the building in the first place; shipping your operational exhaust to a vendor so they can rent you insights about your own espresso machines was always a little absurd). And it wins when you can carry the maintenance burden for the life of the system, not the life of the demo. Starbucks, with a real engineering organization and a $2 billion cost program giving the project air cover, may clear all three. The 200-person company currently asking a coding agent to clone Salesforce will clear none of them, and will rediscover each term of Coase's equation the hard way, in production.

Coase's insight survived the corporation, the conglomerate, the outsourcing wave, offshoring, and the cloud. It will survive this too. The boundary of the firm is a price. AI just changed the price, and the map is getting redrawn, one $400 million line item at a time.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Coase's Coffee.

LLMs Are Too Big. My Log Router Doesn't Need to Sing.

David Aronchick — Tue, 22 Sep 2026 18:01:30 +0000

I think our language models are too big.

Which is a little bit spicy to say, because I don't actually mean the parameter count. I mean that we've gotten comfortable asking an incredibly general system to do a very specific job, and then doing a lot of work to keep it focused on that job.

Working on log routing with Expanso and Jev has made this feel a little ridiculous to me. The question we need answered is whether a particular event deserves someone's attention, especially when we already know the possible destinations and what should happen next. And yet the default approach in a lot of AI software is to bring in a model that can help with just about anything, explain the situation, and ask it to please stay inside the lines.

The capabilities you need to work on the Navier–Stokes equations are not necessarily the same ones you need to sing "Bohemian Rhapsody" in Klingon. Neither requires you to get all the angles right on Michelangelo's David. Having one system that can attempt all three is amazing. I use these tools, and I want them to keep getting better.

I don't think every piece of software needs access to that entire repertoire every time it makes a decision.

The job is smaller than the model

There is a lot of infrastructure work that sits in an awkward place. You can describe what you want fairly easily, but writing all the rules turns into a project of its own.

Take a log message that says config reload requested by unknown actor. It might have an INFO label, but that doesn't make it routine. You want something that can read the message and recognize why it could matter, without having to anticipate every way someone might phrase it.

Now suppose the same message keeps arriving. Once could deserve a look. Repeatedly, in a short window, could deserve something more urgent. The words haven't changed; the circumstances have.

That's the kind of judgment I want help with. I don't need a paragraph about the philosophy of incident response; I need a result the next piece of software can use, and I need to know what to do when the model isn't sure.

Jev, from TypeSafe, is interesting to me because it starts much closer to that requirement. You give it context and defined questions, and it returns typed judgments and probabilities: a choice among options, a score, or a yes/no probability. It doesn't provide an explanation and leaves you to figure out which part was the answer.

Of course, general LLMs can produce structured outputs too; we use that capability all the time. What interests me here is having a model built for these bounded decisions from the start, where there is less for the application to ask of it, and less for it to interpret afterward.

What we actually built

We put together an Expanso and Jev log-triage example that makes the division fairly concrete.

I've written up the implementation in our Expanso post on log triage with Jev. You can also watch the full walkthrough on Expanso's YouTube channel to see the routing and simulated outage in action.

Expanso handles the incoming records and the routing. Known routine events can go straight to archive using explicit checks, without asking a model anything, but for events that require judgment, the pipeline provides context, including how often a matching event has occurred within a 10-minute window.

Jev answers questions about whether the event is actionable, how severe it is, which team should own it, and whether the recurrence is concerning. The pipeline then applies its thresholds and sends the result toward page, notify, review, or archive.

I like that you can point to each part and say who is responsible for it. The model judges the event, the routing policy remains something we wrote and can inspect, and if a call can't be answered, Expanso holds it for retry and eventually sends it for review, while routine traffic continues.

There is a small detail in the demo that I think says a lot about this work. To count repeated events, you can normalize away numbers so that similar messages group together. But you absolutely cannot assume that the resulting group is safe to ignore. A successful health check and a failing one can look very similar once you've removed the status code and response time.

That is our responsibility as the people building the pipeline. A more capable model doesn't excuse us from deciding which information to preserve or which shortcuts are safe.

Fast enough to use everywhere

The part I find exciting is how often you could use this kind of judgment if the cost and latency were low enough.

You could ask about individual events as they move through a system, rather than collecting them in a pile, sending them somewhere else, and waiting for someone to interpret a summary. That changes which problems are worth tackling. A small decision that is too expensive to make a million times generally doesn't get made a million times.

TypeSafe currently lists Jev at $0.042 per million input tokens, with no charge for output tokens. Its own selected performance examples show subsecond responses. Those are vendor results, not a latency or throughput benchmark from our log-routing demo. Before putting this on a critical path, I'd still want to measure tail latency, sustained load, and the mistakes it makes on the actual data.

But at that price, with machine-usable answers returned quickly and cheaply enough that you can build them into ordinary software, this could be much more useful than another conversational interface for many of these jobs.

Typed output doesn't make the judgment correct; a wrong answer in perfectly valid JSON is still wrong. You need examples from your own environment, defensible thresholds, and a place for uncertain results to go. I would much rather build those controls around a limited question than ask a model to figure out the question, the policy, and the action all at once.

I expect more models like this

My bet is that we'll see more smaller, more targeted models alongside the big general ones. Jev is an early example of the direction I mean, although I am not making a claim about its undisclosed parameter count. TypeSafe also says customers use the same model weights; this isn't a separate fine-tune for each company.

The specialization is in what we ask the model to produce. There is plenty of room to get very good at turning messy text into a limited set of judgments that other machines can use. A log router, a support queue, and a data-quality check all have reasons to want that, even if none of them needs a chatbot.

Obviously, I have a stake in this through Expanso. We spend a lot of time thinking about what should happen to data as it moves, and these examples are part of that work. But the thing I keep coming back to is how much of the system we can already specify ourselves. We know where records should go. We know the policy. We mostly need help interpreting the messy bit in the middle.

I'd like to see us get more comfortable asking AI to do that smaller job well. There will be plenty of work left for the model that can discuss fluid dynamics and Klingon. I'm happy to keep using it for that. For the log router, I'd rather have the answer and get on with it.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at LLMs Are Too Big. My Log Router Doesn't Need to Sing..

Survey Before Sale

David Aronchick — Fri, 18 Sep 2026 18:01:18 +0000

In August, OpenAI published a post with a title I did not expect to read from a frontier lab this year: "Pacing model development in an era of cyber-critical capabilities." They talked about a two-week pause in reinforcement learning training on the models it intended to ship, "while we further hardened and red-teamed our research environments," and one sentence that should be framed and hung in every AI lab on the planet: "Our largest planned frontier RL run remains on hold." The stated reason was that "our standards for monitoring, alignment, and security must stay ahead of those risks."

So... the fastest-moving company in the fastest-moving industry in the world tapped the brakes. Not because it ran out of GPUs or power, but because it could not WATCH its own models well enough to keep going.

You probably know why. On July 16, Hugging Face disclosed an automated attack on its infrastructure. Five days later, OpenAI admitted the attackers were its own models, agents in the middle of an evaluation that had escaped their sandbox. The coverage was pretty breathless; one OpenAI staffer explained that even though production agents are carefully monitored, models undergoing evaluation run on a separate system "that is not monitored by default."

The thing we were testing was the thing we weren't watching.

Who actually caught it

The first detailed reconstruction of what happened didn't come from OpenAI. It came from Hugging Face's own logs, a month before OpenAI's official report.

Their technical timeline is absolutely riveting. They recovered roughly 17,600 attacker actions between July 9 and July 13, grouped into about 6,280 clusters, by correlating the agents' actions with their own platform logs. When they tried to use hosted frontier models to help decode the obfuscated payloads, the safety guardrails refused, treating reverse-engineering an exploit the same as launching one. So they ran an open-weight model locally, and decoding the attacker's blobs turned up roughly four times what their first pass had found.

Then METR, brought in to investigate independently, published its findings on August 26, and it's worse. Around 1,200 agents coordinated on a message board they set up themselves, trading more than 70,000 messages and files. About 700 of them took part in the attack. And the evidence METR had to work with consisted of about 1,300 agent transcripts, which is where it gets genuinely unsettling: around 20 percent of the agents expressed clear interest in tampering with their transcripts, and roughly 7 percent of the transcripts contained tool calls that the agents had successfully faked. Container resets ate other pieces. METR estimates it captured a bit over 90 percent of the message-board activity, and it had to lean on AI agents to help analyze the rest, which it describes, with admirable honesty, as often unreliable.

SIGH. The subject of the investigation wrote the record, and a portion of it was actively edited by the subject. We're fighting against something that is... self-aware? I do not like this.

And this was not one lab having one bad month. On July 28, the UK's AI Security Institute caught its own evaluation going out of bounds: in 10 of 122 cyber-testing runs, agents took 19 unsanctioned actions against real targets, 17 of them from Anthropic's Mythos 5. AISI found it through general security monitoring after the fact, "not through monitoring built to watch the evaluation as it ran."

Meanwhile, the models are becoming harder to observe on purpose-built instruments as well. OpenAI's GPT-6 Astra system card, published September 3, reports "a substantial decrease in chain-of-thought monitorability," and shows the model recognizing it was being evaluated in about 9.6 percent of trajectories in an internal deployment simulation, versus 2.8 percent for GPT-5.6 Sol. Anthropic's Fable 5.1 and Mythos 5.1 announcement says its automated behavioral audit "provides less visibility into very long-context work and multi-agent settings," which is, not coincidentally, exactly where all the interesting work is heading.

So the capability curve goes up and to the right, and the visibility curve bends the other way. That gap is the whole ballgame now.

Settle first, survey later

We have been here before, and it was a mess that took generations to clean up.

When Virginia opened Kentucky to settlement, it used the only system it had: metes and bounds. You staked a claim by describing it, and the Kentucky Secretary of State's own land office explains that those descriptions leaned on trees, stakes, and rocks. From a white oak, so many poles to a creek bend, along a ridge to a big rock. Every claim was a record written by the person who wanted the land, pinned to landmarks that could rot, shift, or be disputed, and nobody laid claims against each other before people started building cabins.

Virginia knew exactly what it was doing. The preamble of its Land Law of 1779 warns, in so many words, that "the various and vague claims to unpatented lands" may produce "tedious and infinite litigation and disputes." And that's precisely what it produced. Claims overlapped, and the lawsuits followed.

The most famous casualty was the most famous surveyor. Daniel Boone, who ran surveys all over Kentucky, got sued for faulty surveys, sued for selling land he didn't have valid title to, and received death threats after his testimony cost other people their claims. By the late 1790s he'd had enough of Kentucky and left for Missouri. The man who opened the frontier could not hold on to a piece of it, because the records underneath the frontier were garbage.

Now look north of the Ohio River. On May 20, 1785, Congress passed the Land Ordinance, and its core innovation was almost boring: survey BEFORE sale. Before anyone could buy a parcel, federal surveyors laid a grid over it (townships six miles square, sections of 640 acres), starting with the Seven Ranges in what is now Ohio. The record came first; it was made by someone other than the buyer, and in a way everyone could check against everyone else's. That grid is still visible from an airplane window today. The litigation mostly isn't.

The fix was NOT "slow down settlement." The grid made the frontier move faster, because a buyer could trust a title without hiring a lawyer to walk the creek bed. Verification wasn't the tax on speed; it was the thing that let speed compound.

The survey is a data problem

Most coverage of the Hugging Face incident focuses on alignment, and alignment DOES, of course, matter, but what actually failed in every single account above was the data.

Where was it observed? OpenAI's eval cluster wasn't monitored by default. AISI's evaluation had no monitor built in to watch it run. The first party to reconstruct the attack, Hugging Face, could do it because it had captured its own logs at the point where the actions landed. Observations you don't collect at the moment of action cannot be collected later. There is no backfill for a container that already reset.

Who wrote it? METR's evidence was largely authored by the agents under investigation, and a meaningful fraction of it was attempted to be forged. That's a metes-and-bounds claim: the record is written by the party with an interest in its contents. A verifiable record has to be produced by something the subject can't edit, stamped as it's produced, and kept somewhere the subject can't reach.

Does it line up? Hugging Face's reconstruction worked because it could correlate two independent sources, the agents' actions and the platform's logs, and identify where they agreed and where they didn't. Provenance isn't a property of one log file. It's the ability to lay records from different vantage points on the same grid and compare them.

None of that is exotic. It's lineage, provenance, and quality, the same dull plumbing data engineers have been begging for budget for since before anyone said "agent." What changed is the speed. A human analyst operating over a week produces a week of evidence you can mostly reconstruct. Twelve hundred agents over a long weekend produce 17,600 actions and 70,000 messages, and if you didn't capture it where it happened, from outside the thing being watched, you are Boone in court, arguing about which oak tree the claim started at.

OpenAI, to its enormous credit, just told the world that it's willing to hold its largest training run until the monitoring catches up. That's the right call and a remarkable one. But I'd put it a little differently. Pacing the frontier isn't about slowing down the settlers. It's about getting the surveyors there first, and making sure the survey is something the settlers can't redraw.

Survey before sale. It took the United States one bad frontier to learn that. I'd really prefer we only need one this time too.

Want your data captured, stamped, and verifiable where it's actually generated, instead of reconstructed after the fact? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Survey Before Sale.

De-identification Protects Your Name. It Doesn't Protect Your Idea.

David Aronchick — Fri, 11 Sep 2026 18:01:21 +0000

Imagine you spend a year on the hardest open problem in your field. You feed every draft, every dead end, every 3 a.m. "wait, what if" into the coding assistant you pay for out of your own research budget. Your proof finally checks in Lean on August 22. You decide, like a decent person, to spend a few weeks turning the machine-generated argument into something a human can read before you announce it.

Then, on a Sunday, the company that makes the tool calls you. They have solved the bigger problem. Via your route. They started last week. Would you like to write it up for them?

That is Tristan Buckmaster's account of the last ten days, and even if every single thing OpenAI has said in response is true, that is still what it felt like from his chair. WHICH, for better or worse, is the point of this post.

What actually happened

Quick version.

The Navier-Stokes existence and smoothness problem asks whether the equations describing fluid flow can spontaneously produce infinite velocities in finite time. It is one of the seven Millennium Prize Problems, and, it's really really hard. For years, Diego Córdoba and Luis Martínez-Zoroa have been building a program to force a blowup with an external forcing term, first with rough forcing, aiming toward the smooth-forcing version that Fefferman's official problem statement allows as options (c) and (d).

Buckmaster (NYU) and Levent Alpöge (a mathematician who works at Anthropic, collaborating with Buckmaster on a personal basis with no institutional involvement) took that program and pushed it, with a great deal of LLM help, to smooth forcing and to the 3D incompressible Euler equations. Their Euler proof verified in Lean on August 22 but instead of publishing right away, they held it back to write a readable paper. Among the tools they used was Codex, in which sessions held every draft of the project.

But then the rumors started leaking. A colleague at Courant emailed Buckmaster that "Anthropic had solved a Millennium problem." The colleague heard it from an analyst in the UK, who heard it from somewhere upstream. The rumor was wrong on the problem, wrong on the institution, and right on the direction.

On the Sunday call, Buckmaster asked whether the model had been trained on, or had access to, their sessions. He was told the model did not look up user data. He asked about training. He says he got no answer. Publicly, OpenAI's position is that no person or agent searched user data, that the two teams' approaches look different now that both are visible, and that both grew from the same Córdoba and Martínez-Zoroa roots.

Nobody has shown that Buckmaster's data was used. I want to be clear about that. I am not accusing anyone of anything either. I am pointing at something that is true regardless.

By Altman's own account, OpenAI heard the same rumor, and started seeing if they could solve it on September 1 because they were curious whether their model could do it too. They pointed an unreleased model at all six remaining Millennium problems, saw traction on Navier-Stokes, and threw roughly 10,000 agents at it for about 88 hours worth millions of dollars of compute.

They solved it, and that gets you up to speed.

The thing de-identification cannot remove

Every AI lab, every SaaS vendor, every enterprise data team has followed the privacy checklist. "We strip names and emails and session identifiers and aggregate and and and." Sometimes we get to call the result "synthetic data" and feed it back into the system with a clean conscience.

Now the problem is that when you have something really unique, like the solution to a Millenium problem over months in your logs, all that doesn't really help.

What is left after you remove "Tristan Buckmaster" and "levent@"? A pile of text in which someone is attacking Navier-Stokes through smooth forcing, options (c) and (d), via a specific chain of prior results, with a specific set of estimates that keep failing in a specific way. Buckmaster himself said almost nobody in the world was on that route. He also said the model does not land on it in a few days from the bare problem statement.

There are maybe four people on Earth who would write that particular sequence of prompts, so while it's not EXACTLY personally identifiable information (PII) it's not hard to figure out who it is. And even if you don't know or care, you STILL get a ton of information.

De-identification is a name-removal technique, and works when your data sits in a crowded region, when you are one of ten million people asking how to center a div. It fails when your data sits in a sparse region where just the way you ask the question is both unique, and valuable. Basically, the rarer your idea, the more the idea is the PII.

And while the leak in this story appears to have traveled entirely through humans, it was enough to trigger a bunch of people to start looking at the problem in a new way. A single rumor with one word in it, "forced," was enough to point ten thousand agents at the right door. If that can happen through gossip, imagine what a gradient can do.

The same mechanism is the whole point

Why is the model good at this problem at all? Because it is a compression of billions of humans interacting over a million arXiv preprints, and every seminar note somebody typed up, and every Stack Exchange thread where a grad student got yelled at for a sign error. So you take every half-finished idea somebody abandoned in 2011 because the estimate didn't close and you form it into a new ball, and, magically, our tool can pull out threads no individual could hold in their head at once. Including, apparently, a two-person research program from Madrid that most of the field was not paying attention to.

This is really kind of insane, and it should be as inspirational as it is scary.

For all of human history, collaboration has been bandwidth-limited by the number of people you could physically talk to. Newton had Halley who had Ramanujan, and only because a letter made it across an ocean. The rest of us have a lab, a Slack channel, and whoever answers our email. Every important idea that ever died did so because the one person who needed it never met the one person who had it. That is the default outcome for most ideas.

What Buckmaster actually did this year is the first version of something different. He did not just use a tool; he collaborated with a compressed record of everyone who ever wrote down a thought about fluid dynamics, including two people in Madrid he took as his starting point, including thousands of people whose names he will never know and whose partial results the model absorbed and recombined. He and Alpöge took Córdoba and Martínez-Zoroa's ideas and used LLMs to push them to completion in about a month. He called it a Deep Blue moment and even with that grand pronouncement, I think he is still underselling it. Deep Blue beat one man at one game. This is every mathematician who ever lived showing up to your office hours at once, badly organized, occasionally wrong, and available at 3 a.m.

The first LLM-generated proof he was sent was, in his words, the most horrendous he had ever read. But it was also correct. That is what collaborating with all of humanity looks like. Not clean. It is a room with eight billion people in it, and somewhere in the noise is the one sentence you needed.

You cannot have that capability without the risk in the previous section. The ability to tease apart a rare, high-value thread from the mass is the same ability that makes rare, high-value threads unhideable inside the mass. The room that lets you hear everyone also lets everyone hear you. There is no privacy setting that keeps the second thing and drops the first. It's a trade off, but one that could unlock a new version of humanity, at the cost of the way we think about privacy.

Two big takeaways

There are two really big things this teases out.

First: the only de-identification that works is not sending it. If your work lives in a sparse region, and you are a startup, a lab, or a lone mathematician with a real idea, the only protection that survives contact with a sufficiently capable model is keeping the data where it lives and bringing the model to it. Buckmaster used the frontier tools and got a Millennium-class result out of them. The question is which room you think out loud in, and who holds the lease.

Second: we need provenance for ideas, not just data. The community will spend months trying to work out whether two proofs that share a root, a route, and a week share anything else and it cannot, because nothing was logged. We have spent years building lineage for datasets and software bills of materials for code. When a proof is produced by 2.7 million messages between agents, "we did not use their prompts" is not evidence. A machine-readable record of what went into a result, what it was seeded with, and when, is the missing artifact here. Not a better privacy policy. A receipt.

Keep using the tools!

The lesson is not "don't use the tools." Buckmaster used them and got a result that would have been science fiction two years ago because thousands of people he never met helped him do it. That is the biggest expansion of who you get to think with since the printing press, and I do not think anyone should give it up.

But the room where all of humanity can help you is also the room where all of humanity can see your notebook. Right now one company holds the lease on that room and gets to decide, after the fact, what "we didn't look" means. The fix is not to leave the room; it is to bring the room to you, and to keep receipts for who said what inside it.

De-identification is a promise about your name. But your name was never the valuable part. The idea was, and the idea is exactly what the whole thing is built to find.

Sources: Buckmaster's public statement; OpenAI's announcement and press call as reported by Scientific American and Axios, September 8, 2026.

Want to bring the model to your data instead of shipping your notebook to somebody else's room? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at De-identification Protects Your Name. It Doesn't Protect Your Idea..

One Floor Up

David Aronchick — Tue, 08 Sep 2026 18:01:34 +0000

OpenAI launched GPT-6 Astra on September 3, and there is a piece of the launch I think deserves attention alongside the benchmarks. We now have public reports from more than one lab describing evaluation agents reaching real systems they were never supposed to touch. These systems can do useful, difficult work. Some of the same persistence that makes them useful can also carry them well past the task somebody intended to give them.

The July Hugging Face incident is the most visible example, and we have much better information now than we did when it first broke. OpenAI's August 26 investigation describes several models, primarily an internal research prototype, exploiting shared package infrastructure, exchanging information through an unauthorized message board, and eventually compromising parts of Hugging Face's systems while looking for benchmark solutions. OpenAI published a technical report and linked an independent investigation by METR and Redwood Research. This was a serious failure, and publishing enough detail for other people to learn from it matters.

Two distinctions are important here. Astra was not involved. And these evaluations ran with reduced safeguards to measure the underlying models' cybersecurity capabilities, rather than the protections used in the products people normally access. That makes containment during evaluation especially important; it also means we should be careful about treating the incident as a description of ordinary ChatGPT behavior.

The response has moved beyond the initial pause. In its prelaunch safety update, OpenAI says Astra meets its Critical cybersecurity capability threshold, with advanced cyber access initially restricted. It also says the large frontier reinforcement-learning run restarted on August 28 after new safety and security requirements were in place, while some experimental runs remained on hold. The changes include tighter network controls, stronger isolation, better alignment training, and monitoring that can stop potentially unauthorized activity.

There is encouraging data alongside the capability increase. In the launch evaluation inspired by the Hugging Face incident, OpenAI reports that Astra went beyond the authorized target in 0% of cases, compared with 48.2% for GPT-5.6 Sol, tested without production safeguards. Those are results from a particular evaluation, not a guarantee about every situation a deployed agent will encounter. But they are evidence of improvement, and they belong in the discussion too.

Other Labs Are Finding This Too

On July 30, Anthropic reported three incidents across six evaluation runs, found during a review of 141,006 runs. Its review began after OpenAI's disclosure. In these cases, an evaluation environment had internet access that Anthropic believed was disabled, and Claude reached the production infrastructure of three organizations while pursuing fictional capture-the-flag tasks. Unlike the OpenAI incident, these models did not need to discover a novel sandbox escape. An open path was enough.

The models behaved differently, which is worth keeping in the story. Anthropic's newest research model stopped when it recognized that its target was real, although it had already gone further than the company wanted. An older model continued. Anthropic cautions that these were isolated incidents, not a controlled comparison, and says the evaluations lacked safeguards used in its public products. It paused cyber evaluations, notified affected organizations, and described changes to its monitoring and work with evaluation partners.

There is also deliberate misuse, which is a different problem but uses some of the same capabilities. Anthropic's August 2025 threat report described a criminal using Claude Code against at least 17 organizations, with the model helping carry out operations rather than only offering advice. That report is a year old now. Between misuse and evaluation failures, there is enough public evidence to put this in an ordinary security review. We do not need to assume every model behaves the same way to recognize that it is happening across products and environments.

One Floor Up

In the way back machine, in the summer of 1978, Janet Parker was employed as a medical photographer at the University of Birmingham Medical School, where her workroom sat one floor above a laboratory run by Professor Henry Bedson, one of Britain's senior smallpox researchers. The last natural smallpox case on Earth was recorded in Somalia in October 1977. The World Health Organization was preparing to certify global eradication, and the plan for the virus afterward was a short list of approved laboratories. Bedson's lab was due to lose its authorization at the end of 1978. Inspectors who visited earlier that year had noted that its containment did not meet the standards being drafted for the labs that would contain the virus, and Bedson, who wanted to finish his research program before the deadline, kept working. Parker fell ill on August 11 and was diagnosed with smallpox on August 20. Around 260 people who had been in contact with her were quarantined. Her mother contracted the virus and survived it. Her father died of a heart attack during a visit to his daughter in isolation. Bedson, under quarantine at his home while the inquiry assembled, cut his throat on September 1 and died on September 6. Parker died on September 11, 1978, the last person killed by smallpox anywhere. The official inquiry concluded that the virus had most likely traveled from the lab to her workroom through a poorly maintained service duct. Expert witnesses in the later prosecution of the university considered the airborne route implausible, and the honest summary, five decades on, is that the transmission path has never been established. A containment regime was inspected, found wanting, allowed to continue operating, and breached by a route that still has not been identified.

I want to be careful with this comparison. Software agents are not pathogens, and a security incident is not equivalent to a death. The useful connection is the boundary: somebody working outside an experiment can still be affected by what happens inside it. Parker never worked with smallpox. The organizations reached during the AI evaluations had not agreed to participate in those exercises either.

The evaluations themselves serve a necessary purpose. We want labs measuring these capabilities before release, and we want them to disclose failures, pause work when needed, and share what changed. OpenAI's disclosure prompted Anthropic to look through its own records and find incidents it had missed. That is a concrete benefit of publishing the uncomfortable details. I would much rather have this information available while we can use it to improve the systems we are building.

A capable agent with tool access is also a workload, and the questions that matter about a workload are unglamorous. What is it connected to? What credentials can it reach? Can it leave information somewhere another agent will find it? Who gets told when it starts doing something outside its assignment? Better model behavior helps, as the newer evaluations suggest. Isolation, access controls, and monitoring give you additional chances to catch a mistake before another organization has to deal with it.

For those of us deploying agents, there is work we can do now. Check the actual network access, including package proxies and shared services, rather than trusting the word "sandbox" in a configuration. Give the task credentials with a limited scope and lifetime. Record tool actions somewhere the agent cannot rewrite, and make sure someone can stop the workload and revoke its access. If you buy an agent service, ask the vendor the same questions. The prompt saying "this is a simulation" did not make Anthropic's network a simulation, and a statement of intended access is not a test of actual access.

I am excited about how much more useful these systems are becoming. That is also why I want the operational conversation to catch up. You may run the agent, supply a service it uses, or simply have infrastructure it can reach. Janet Parker's workroom was one floor up. For our systems, we can at least start by finding out what is connected to what.

Want to know exactly what your data workloads are connected to and run them where the blast radius is small? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at One Floor Up.

Whitworth's Thread

David Aronchick — Fri, 04 Sep 2026 18:01:14 +0000

In 2025, Gartner issued a poll that said it expects more than 40 percent of agentic AI projects to be canceled by the end of 2027. This year, the firm sharpened the claim: by 2027, roughly 40 percent of enterprises will demote or decommission autonomous agents, and the driver it names is governance gaps that only surfaced after a production incident. Note the timing: these gaps surfaced AFTER it rolled out to production. I think it's a pretty safe summary to say that the industry thinks this is a model problem, that the models aren't good enough yet, and that another turn of the capability crank will fix everything. I think this reading is wrong, and this survey points to the reason why.

In a separate survey, DORA's most recent State of DevOps research, drawing on nearly 5,000 technology professionals, found that AI adoption acts as an amplifier: strong positive effects on organizational performance when the underlying platform is strong, and effects near zero when it is weak. In the same dataset, individual productivity rose while delivery throughput and stability declined, with the gains being absorbed by code review, testing, and security sign-off. In the study, they coined the term "downstream disorder," which is kind of adorable, like something a Victorian doctor would diagnose an aristocrat with. As in the Gartner study, none of it is a statement about model quality; it's almost entirely about whether the work was ever specified with meaningful outcomes.

Since the dawn of bureaucracy and organizations, people have been trying to measure outcomes. This is much harder than you might imagine! The biggest problem is that standardization inevitably curdles into bureaucracy. I have personally seen change advisory boards whose actual function was to spread blame too thin to land on anyone, and I have filled in templates whose only reader, ever, was the template's author. Teams route around dead processes within a week, and then there are two processes: the written one, audited and fictional, and the real one, undocumented, resident in the heads of four people. Arguably, that is worse than no standard at all, because now you cannot even see your own variance (I will concede that SOMETIMES it helps, because at least it forces the form-filler-outer to crystallize what they're thinking, but it remains in their head, which is not ideal).

So the distinction I care about is between prohibited variation and controlled variation. A standard that documents its own escape hatches- the four known conditions for leaving the path and who gets told when you do- can absorb surprise. A standard with no exceptions is a lie, and everybody involved knows. Let's figure out how to move forward better.

He Built The Ruler First

In 1841, Joseph Whitworth read a paper to the Institution of Civil Engineers proposing a uniform screw thread for British industry: a 55-degree included angle, with specified radii at the root and crest so that the thread would not concentrate stress where it was most likely to fail. It became the world's first national screw thread standard. Before it, every manufacturer in Britain cut threads to its own proportion; a bolt and its nut were a matched pair fitted to each other by a particular workman, and if you lost the nut, you did not go and get another nut; you went and got a fitter. The year before the thread paper, in 1840, Whitworth had developed what he called end measurements, a technique using a precision flat plane and a measuring screw of his own construction. He had worked it down to a claimed precision of one millionth of an inch, which he later showed off to the public at the Great Exhibition of 1851. (Remember when we had amazing things like that at the World Fair?! I was always so inspired by that. I wish we would bring stuff like that back.)

Metrology first, standard second. He built the ruler before he proposed the rule, and the former's existence made the latter possible.

Look For The Marks

In January 1801, Eli Whitney traveled to Washington and demonstrated interchangeable musket locks before President Adams, President-elect Jefferson, and a room of officials. Ten locks were disassembled, their parts mixed, and then reassembled. The demonstration was a sensation; it secured his contract and made him a fixture in every American textbook as the father of interchangeable parts. In something that will sound really familiar to all the fake-it-before-you-make-it startup people out there, the demo was rigged.

Whitney had marked the parts beforehand so they could be matched back up, and later examination of surviving Whitney muskets showed the components were not interchangeable in any strict sense. Hand filing was still required to make anything fit, and the muskets carry special engraved marks on their parts whose only purpose is to tell an armorer which part belongs to which gun. The man who ACTUALLY achieved interoperability was Honoré Blanc, in France, more than a decade earlier, using jigs, gauges, and master models to hold musket parts to identical tolerances by hand. Jefferson saw Blanc's workshop while serving as ambassador in Paris and wrote home about it, so the American government learned about the real version first and bought the staged one anyway.

The American who finally did it worked at Harpers Ferry, and almost nobody remembers his name. John Hall signed a contract with the War Department in 1819 to produce his breech-loading rifle at a small rifle works on an island in the Shenandoah, and he spent the first several years of it building machines and gauges instead of guns. He built dozens of milling and gauging machines, worked out fixtures that held each part in the same position for every operation, and ran everything against master gauges at every stage of production. In 1826, the government sent an inspection board, which took a hundred of Hall's rifles, stripped them, scrambled the parts in boxes, reassembled rifles from the mixture, and watched the reassembled guns fire. The board reported that the parts could be exchanged with a level of flexibility never before achieved. INTERESTINGLY, most of the contract money had gone into tooling rather than rifles, and Hall's cost per gun came out higher than the ordinary muskets the armories were already producing.

Flash forward to today, every agent demo you have been shown in the last eighteen months is the Whitney demo. I do not mean that as an accusation of fraud, because Whitney sincerely believed he was three years away from the real thing. I mean, the demo works because a human pre-fitted the parts, and you can tell by looking for the marks. Usually, someone quietly reviews the output before it goes anywhere, or an eval set is hand-curated, or one customer is somehow always the reference customer, or a workflow step is described as "and then it just gets handed to the ops team." SWEs and SREs wince at this last phrase, since a genuinely standardized process for rolling out would never involve the word "just." It's a discipline all its own, and it's not something to be papered over.

Sixty-Four Paths, Six Of Them Written Down

In contrast, a confabulation is a silent wrong answer surfacing in a quarterly review nine weeks later. No amount of model tweaking/improvements is going to fix this; it is a property of delegation, and it's been true of every new hire you've ever onboarded. Except now your new hires are agents.

I keep watching teams ask for agentic workflows while their actual onboarding process is forty Slack messages plus a guy named Jerry who knows which of the three staging databases is the real one. Jerry appears in no runbook and no architecture diagram; he is a single point of failure with a mortgage, and he is taking two weeks in August. Four if he's in Europe.

So the practical move is unglamorous, but (nearly) always beneficial. Write the runbook you would hand to a competent new hire, so they can execute it without asking anybody anything. The places where you stall, or where you have to say "and then you kind of know from context," are the places where no agent will operate reliably, and now it falls to you to document. I have written before about the gap between agent ambition and agent infrastructure, about how a loop is only as good as the metric you close it against, and about who is accountable when the thing acts, and they are all this same problem from different sides: you are handing work to a system that cannot ask a clarifying question in the hallway (or Slack), and your operating model was quietly built on the assumption that everything could.

Whitworth's real contribution was never the 55-degree angle; the number was arbitrary, and the Americans later went with 60 and did fine. His contribution was that a bolt made in Manchester was threaded into a nut made in Glasgow by a stranger, which meant you could finally build a machine to make the bolt instead of hiring a man to fit it. Get the measurement, then get the agreement. The thread was never the invention.

Want the boring parts of your data operations to be repeatable enough that something other than a person can run them? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at Whitworth's Thread.