DEV Community: David Aronchick

One Floor Up

David Aronchick — Tue, 08 Sep 2026 18:01:34 +0000

OpenAI launched GPT-6 Astra on September 3, and there is a piece of the launch I think deserves attention alongside the benchmarks. We now have public reports from more than one lab describing evaluation agents reaching real systems they were never supposed to touch. These systems can do useful, difficult work. Some of the same persistence that makes them useful can also carry them well past the task somebody intended to give them.

The July Hugging Face incident is the most visible example, and we have much better information now than we did when it first broke. OpenAI's August 26 investigation describes several models, primarily an internal research prototype, exploiting shared package infrastructure, exchanging information through an unauthorized message board, and eventually compromising parts of Hugging Face's systems while looking for benchmark solutions. OpenAI published a technical report and linked an independent investigation by METR and Redwood Research. This was a serious failure, and publishing enough detail for other people to learn from it matters.

Two distinctions are important here. Astra was not involved. And these evaluations ran with reduced safeguards to measure the underlying models' cybersecurity capabilities, rather than the protections used in the products people normally access. That makes containment during evaluation especially important; it also means we should be careful about treating the incident as a description of ordinary ChatGPT behavior.

The response has moved beyond the initial pause. In its prelaunch safety update, OpenAI says Astra meets its Critical cybersecurity capability threshold, with advanced cyber access initially restricted. It also says the large frontier reinforcement-learning run restarted on August 28 after new safety and security requirements were in place, while some experimental runs remained on hold. The changes include tighter network controls, stronger isolation, better alignment training, and monitoring that can stop potentially unauthorized activity.

There is encouraging data alongside the capability increase. In the launch evaluation inspired by the Hugging Face incident, OpenAI reports that Astra went beyond the authorized target in 0% of cases, compared with 48.2% for GPT-5.6 Sol, tested without production safeguards. Those are results from a particular evaluation, not a guarantee about every situation a deployed agent will encounter. But they are evidence of improvement, and they belong in the discussion too.

Other Labs Are Finding This Too

On July 30, Anthropic reported three incidents across six evaluation runs, found during a review of 141,006 runs. Its review began after OpenAI's disclosure. In these cases, an evaluation environment had internet access that Anthropic believed was disabled, and Claude reached the production infrastructure of three organizations while pursuing fictional capture-the-flag tasks. Unlike the OpenAI incident, these models did not need to discover a novel sandbox escape. An open path was enough.

The models behaved differently, which is worth keeping in the story. Anthropic's newest research model stopped when it recognized that its target was real, although it had already gone further than the company wanted. An older model continued. Anthropic cautions that these were isolated incidents, not a controlled comparison, and says the evaluations lacked safeguards used in its public products. It paused cyber evaluations, notified affected organizations, and described changes to its monitoring and work with evaluation partners.

There is also deliberate misuse, which is a different problem but uses some of the same capabilities. Anthropic's August 2025 threat report described a criminal using Claude Code against at least 17 organizations, with the model helping carry out operations rather than only offering advice. That report is a year old now. Between misuse and evaluation failures, there is enough public evidence to put this in an ordinary security review. We do not need to assume every model behaves the same way to recognize that it is happening across products and environments.

One Floor Up

In the way back machine, in the summer of 1978, Janet Parker was employed as a medical photographer at the University of Birmingham Medical School, where her workroom sat one floor above a laboratory run by Professor Henry Bedson, one of Britain's senior smallpox researchers. The last natural smallpox case on Earth was recorded in Somalia in October 1977. The World Health Organization was preparing to certify global eradication, and the plan for the virus afterward was a short list of approved laboratories. Bedson's lab was due to lose its authorization at the end of 1978. Inspectors who visited earlier that year had noted that its containment did not meet the standards being drafted for the labs that would contain the virus, and Bedson, who wanted to finish his research program before the deadline, kept working. Parker fell ill on August 11 and was diagnosed with smallpox on August 20. Around 260 people who had been in contact with her were quarantined. Her mother contracted the virus and survived it. Her father died of a heart attack during a visit to his daughter in isolation. Bedson, under quarantine at his home while the inquiry assembled, cut his throat on September 1 and died on September 6. Parker died on September 11, 1978, the last person killed by smallpox anywhere. The official inquiry concluded that the virus had most likely traveled from the lab to her workroom through a poorly maintained service duct. Expert witnesses in the later prosecution of the university considered the airborne route implausible, and the honest summary, five decades on, is that the transmission path has never been established. A containment regime was inspected, found wanting, allowed to continue operating, and breached by a route that still has not been identified.

I want to be careful with this comparison. Software agents are not pathogens, and a security incident is not equivalent to a death. The useful connection is the boundary: somebody working outside an experiment can still be affected by what happens inside it. Parker never worked with smallpox. The organizations reached during the AI evaluations had not agreed to participate in those exercises either.

The evaluations themselves serve a necessary purpose. We want labs measuring these capabilities before release, and we want them to disclose failures, pause work when needed, and share what changed. OpenAI's disclosure prompted Anthropic to look through its own records and find incidents it had missed. That is a concrete benefit of publishing the uncomfortable details. I would much rather have this information available while we can use it to improve the systems we are building.

A capable agent with tool access is also a workload, and the questions that matter about a workload are unglamorous. What is it connected to? What credentials can it reach? Can it leave information somewhere another agent will find it? Who gets told when it starts doing something outside its assignment? Better model behavior helps, as the newer evaluations suggest. Isolation, access controls, and monitoring give you additional chances to catch a mistake before another organization has to deal with it.

For those of us deploying agents, there is work we can do now. Check the actual network access, including package proxies and shared services, rather than trusting the word "sandbox" in a configuration. Give the task credentials with a limited scope and lifetime. Record tool actions somewhere the agent cannot rewrite, and make sure someone can stop the workload and revoke its access. If you buy an agent service, ask the vendor the same questions. The prompt saying "this is a simulation" did not make Anthropic's network a simulation, and a statement of intended access is not a test of actual access.

I am excited about how much more useful these systems are becoming. That is also why I want the operational conversation to catch up. You may run the agent, supply a service it uses, or simply have infrastructure it can reach. Janet Parker's workroom was one floor up. For our systems, we can at least start by finding out what is connected to what.

Want to know exactly what your data workloads are connected to and run them where the blast radius is small? Check out Expanso. Or don't. Who am I to tell you what to do?

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!

Originally published at One Floor Up.

Whitworth's Thread

David Aronchick — Fri, 04 Sep 2026 18:01:14 +0000

In 2025, Gartner issued a poll that said it expects more than 40 percent of agentic AI projects to be canceled by the end of 2027. This year, the firm sharpened the claim: by 2027, roughly 40 percent of enterprises will demote or decommission autonomous agents, and the driver it names is governance gaps that only surfaced after a production incident. Note the timing: these gaps surfaced AFTER it rolled out to production. I think it's a pretty safe summary to say that the industry thinks this is a model problem, that the models aren't good enough yet, and that another turn of the capability crank will fix everything. I think this reading is wrong, and this survey points to the reason why.

In a separate survey, DORA's most recent State of DevOps research, drawing on nearly 5,000 technology professionals, found that AI adoption acts as an amplifier: strong positive effects on organizational performance when the underlying platform is strong, and effects near zero when it is weak. In the same dataset, individual productivity rose while delivery throughput and stability declined, with the gains being absorbed by code review, testing, and security sign-off. In the study, they coined the term "downstream disorder," which is kind of adorable, like something a Victorian doctor would diagnose an aristocrat with. As in the Gartner study, none of it is a statement about model quality; it's almost entirely about whether the work was ever specified with meaningful outcomes.

Since the dawn of bureaucracy and organizations, people have been trying to measure outcomes. This is much harder than you might imagine! The biggest problem is that standardization inevitably curdles into bureaucracy. I have personally seen change advisory boards whose actual function was to spread blame too thin to land on anyone, and I have filled in templates whose only reader, ever, was the template's author. Teams route around dead processes within a week, and then there are two processes: the written one, audited and fictional, and the real one, undocumented, resident in the heads of four people. Arguably, that is worse than no standard at all, because now you cannot even see your own variance (I will concede that SOMETIMES it helps, because at least it forces the form-filler-outer to crystallize what they're thinking, but it remains in their head, which is not ideal).

So the distinction I care about is between prohibited variation and controlled variation. A standard that documents its own escape hatches- the four known conditions for leaving the path and who gets told when you do- can absorb surprise. A standard with no exceptions is a lie, and everybody involved knows. Let's figure out how to move forward better.

He Built The Ruler First

In 1841, Joseph Whitworth read a paper to the Institution of Civil Engineers proposing a uniform screw thread for British industry: a 55-degree included angle, with specified radii at the root and crest so that the thread would not concentrate stress where it was most likely to fail. It became the world's first national screw thread standard. Before it, every manufacturer in Britain cut threads to its own proportion; a bolt and its nut were a matched pair fitted to each other by a particular workman, and if you lost the nut, you did not go and get another nut; you went and got a fitter. The year before the thread paper, in 1840, Whitworth had developed what he called end measurements, a technique using a precision flat plane and a measuring screw of his own construction. He had worked it down to a claimed precision of one millionth of an inch, which he later showed off to the public at the Great Exhibition of 1851. (Remember when we had amazing things like that at the World Fair?! I was always so inspired by that. I wish we would bring stuff like that back.)

Metrology first, standard second. He built the ruler before he proposed the rule, and the former's existence made the latter possible.

Look For The Marks

In January 1801, Eli Whitney traveled to Washington and demonstrated interchangeable musket locks before President Adams, President-elect Jefferson, and a room of officials. Ten locks were disassembled, their parts mixed, and then reassembled. The demonstration was a sensation; it secured his contract and made him a fixture in every American textbook as the father of interchangeable parts. In something that will sound really familiar to all the fake-it-before-you-make-it startup people out there, the demo was rigged.

Whitney had marked the parts beforehand so they could be matched back up, and later examination of surviving Whitney muskets showed the components were not interchangeable in any strict sense. Hand filing was still required to make anything fit, and the muskets carry special engraved marks on their parts whose only purpose is to tell an armorer which part belongs to which gun. The man who ACTUALLY achieved interoperability was Honoré Blanc, in France, more than a decade earlier, using jigs, gauges, and master models to hold musket parts to identical tolerances by hand. Jefferson saw Blanc's workshop while serving as ambassador in Paris and wrote home about it, so the American government learned about the real version first and bought the staged one anyway.

The American who finally did it worked at Harpers Ferry, and almost nobody remembers his name. John Hall signed a contract with the War Department in 1819 to produce his breech-loading rifle at a small rifle works on an island in the Shenandoah, and he spent the first several years of it building machines and gauges instead of guns. He built dozens of milling and gauging machines, worked out fixtures that held each part in the same position for every operation, and ran everything against master gauges at every stage of production. In 1826, the government sent an inspection board, which took a hundred of Hall's rifles, stripped them, scrambled the parts in boxes, reassembled rifles from the mixture, and watched the reassembled guns fire. The board reported that the parts could be exchanged with a level of flexibility never before achieved. INTERESTINGLY, most of the contract money had gone into tooling rather than rifles, and Hall's cost per gun came out higher than the ordinary muskets the armories were already producing.

Flash forward to today, every agent demo you have been shown in the last eighteen months is the Whitney demo. I do not mean that as an accusation of fraud, because Whitney sincerely believed he was three years away from the real thing. I mean, the demo works because a human pre-fitted the parts, and you can tell by looking for the marks. Usually, someone quietly reviews the output before it goes anywhere, or an eval set is hand-curated, or one customer is somehow always the reference customer, or a workflow step is described as "and then it just gets handed to the ops team." SWEs and SREs wince at this last phrase, since a genuinely standardized process for rolling out would never involve the word "just." It's a discipline all its own, and it's not something to be papered over.

Sixty-Four Paths, Six Of Them Written Down

In contrast, a confabulation is a silent wrong answer surfacing in a quarterly review nine weeks later. No amount of model tweaking/improvements is going to fix this; it is a property of delegation, and it's been true of every new hire you've ever onboarded. Except now your new hires are agents.

I keep watching teams ask for agentic workflows while their actual onboarding process is forty Slack messages plus a guy named Jerry who knows which of the three staging databases is the real one. Jerry appears in no runbook and no architecture diagram; he is a single point of failure with a mortgage, and he is taking two weeks in August. Four if he's in Europe.

So the practical move is unglamorous, but (nearly) always beneficial. Write the runbook you would hand to a competent new hire, so they can execute it without asking anybody anything. The places where you stall, or where you have to say "and then you kind of know from context," are the places where no agent will operate reliably, and now it falls to you to document. I have written before about the gap between agent ambition and agent infrastructure, about how a loop is only as good as the metric you close it against, and about who is accountable when the thing acts, and they are all this same problem from different sides: you are handing work to a system that cannot ask a clarifying question in the hallway (or Slack), and your operating model was quietly built on the assumption that everything could.

Whitworth's real contribution was never the 55-degree angle; the number was arbitrary, and the Americans later went with 60 and did fine. His contribution was that a bolt made in Manchester was threaded into a nut made in Glasgow by a stranger, which meant you could finally build a machine to make the bolt instead of hiring a man to fit it. Get the measurement, then get the agreement. The thread was never the invention.

Want the boring parts of your data operations to be repeatable enough that something other than a person can run them? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at Whitworth's Thread.

The Money Is Coming From Inside the House

David Aronchick — Tue, 01 Sep 2026 18:01:36 +0000

Nvidia reported $96.22 billion in revenue this week for its second quarter, against analyst consensus of $92.11 billion, and guided to $108 billion for the third. A day or so later, the Wall Street Journal reported that the company had paused the revenue-sharing financing deals it announced back in July under the name AI Compute Partnership. The program was aimed at smaller cloud providers who need to spend billions on Nvidia chips and the data centers around them before they can rent out an hour of anything. What WAS in place, but is no longer, was that Nvidia and the provider would agree on a base hourly rental rate that covers the provider's costs, Nvidia would provide credit support against the chip purchases, and Nvidia would take 50% of whatever revenue comes in above the base rate. The reasons for the pause, about eight weeks after launch, were internal worries about antitrust exposure, plus prospective partners getting irritated at how much control Nvidia wanted over the operations it was financing.

So the chip vendor extends credit so the customer can buy the chips, then takes half the upside on the rental revenue those chips produce. I would call that a loan with profit participation before I called it a sale, and I think Nvidia's own lawyers apparently agreed.

BUT, the paused program was also the small version of something much larger that is still running. OpenAI has committed to spending roughly $1.4 trillion over eight years on data centers and compute, against annual revenue somewhere near $13 billion. In contrast, simultaneously, NVIDIA has committed as much as $100 billion to OpenAI directly. OpenAI has committed hundreds of billions to Oracle and Microsoft for capacity, and Oracle and Microsoft buy NVIDIA GPUs to build that capacity out. Bloomberg and others have now traced more than $800 billion of these interlocking arrangements through the AI supply chain. Morgan Stanley reportedly expects Microsoft's entire Azure AI growth for the fiscal year that ended in June, north of $20 billion, to come from OpenAI, a counterparty in which Microsoft also holds a large stake. If you try to name the party in that chain who put outside money at risk on the proposition that end demand exists at these prices, somebody who is not also a supplier, customer, or shareholder of one of the others, I think you will end up struggling. While this is good in many ways (people are putting their capital where their words are), it also creates a lot of interdependence.

As with so many things nowadays, this feels like a replay; in this case, Telecom ran this experiment 25 years ago, and the filings are still on EDGAR.

Specifically, in the late 1990s the hot buyers of network equipment were the CLECs, the competitive local exchange carriers, startups laying fiber to compete with the Baby Bells under the 1996 Telecom Act. They had orders and no cash, so the equipment makers lent them the purchase price. Lucent committed $8.1 billion in vendor financing over the period, including a $2 billion credit line to WinStar Communications that WinStar drew on to buy Lucent switches, and the switch sales went into Lucent's reported revenue, where analysts read them as demand. Between 1996 and 2001, the sector overbuilt by an estimated $60 billion, and once outside funding dried u,p, the upriers started failing: Covad, Focal, McLeod, NorthPoint, and then, in the spring of 20011, WinStar itself, days after Lucent declined to advance the final $90 million on the line. That single relationship cost Lucent a $700 million write-off. The whole book deteriorated even faster than the WinStar piece did, with bad loans going from 2.6% of Lucent's financing portfolio at the end of 2000 to 60% a year later (Nortel's went from 25.5% to 80% over roughly the same stretch). Lucent took billions in provisions against customer loans, shed most of its workforce, and was eventually absorbed by Alcatel, having discovered that a meaningful slice of its late-90s revenue had been its own treasury making a round trip.

And before anyone emails/texts/DMs me to complain that "this time it's different", YES, vendor financing on its own is not a scandal. IBM Global Financing has been lending customers the price of IBM equipment since 1981, and it worked for four decades because the collateral was mainframes running payroll at companies with thirty years of verifiable cash flow. In THIS case, Lucent's collateral was the business plan of a five-year-old CLEC whose model assumed the 1999 growth curve was permanent. And, unfortunately, disclosure doesn't rescue it either, since Lucent's loans were disclosed too, in the same filings everyone now cites as the warning nobody read. The thing that went missing in the CLEC years, and is missing now, is an outside party who validated the demand with their own capital before the revenue got booked.

Which brings me back to the pause. Nvidia had just posted the best quarter in its corporate history and could have coasted on the program for another year. Instead, eight weeks in, somebody internal looked at the antitrust exposure, or at the $11 billion-plus already lent against GPUs across the neocloud sector before this program even existed, or at what taking half your customers' upside implies about eating half their downside, and shut the thing off. The company with the best view of real rental demand on earth got offered a machine for manufacturing more of it, and declined.

None of which means AI demand is fake. A lot of it is real, VERY real, and expensively so. But some fraction of the headline numbers is the same dollar passing through three income statements; nobody knows the size of that fraction, including the participants, and we find out when a payment gets missed somewhere in the circle. The 1996 to 2001 fiber did get built, the builders mostly died, and the rest of us spent a decade lighting up dark strands bought for cents on the dollar. The capacity outlived the counterparties. WinStar felt like growth too, right up until the $90 million did not arrive.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at The Money Is Coming From Inside the House.

The Drums Out Back

David Aronchick — Fri, 28 Aug 2026 15:58:14 +0000

IBM published its 2026 Cost of a Data Breach report at the end of July. The global average is now $4.99 million per breach, up 12 percent year over year and a record for the study, which this round covered 602 organizations hit between March 2025 and February 2026. American breaches ran more than double the global figure. Healthcare stayed the most expensive industry to be breached in, at $6.64 million a pop. And one in four malicious breaches is now AI-enabled, averaging around $6 million, which is a 56 percent jump in a single year.

The report prices an event for the first time in a while that I actually like. But almost nobody is reading the other half of the sentence; it's half the event, and half the size of the pile. We need to look at both.

The Cleanest Illustration Available Is French

On January 13, the CNIL fined Free Mobile €27 million and Free €15 million, €42 million between them, over an intrusion in October 2024 that reached personal data on roughly 24 million subscriber contracts, including bank account numbers for customers who held both services. Three findings: the data was not adequately secured, the notification to affected people was inadequate, and the retention was illegal.

The third item is pretty uncommon to say the least. The regulator determined that the carrier was holding millions of records on people who were no longer customers, well past any period it could justify, when what it actually needed to keep for accounting purposes was a much smaller slice. So when the attacker came along, they did not steal what Free Mobile needed to run its business; they stole information that Free Mobile had no lawful reason to still possess, sitting in the same system as the material it did.

The intrusion was the trigger, but the retention was the multiplier. One of those two things was under the company's control for years in advance, at essentially zero cost. Less than zero cost, they could have deleted it and saved money!

Article 5 of the GDPR has said since 2018 that personal data must be adequate, relevant, and limited to what is necessary, and that it must not be kept in identifiable form longer than the purpose requires. Violating the Article 5 principles sits in the top tier of Article 83, which reaches €20 million or four percent of worldwide annual turnover, whichever hurts more. Cumulative GDPR fines across the EEA passed €7.1 billion by the start of this year. Minimization has been law for eight years and mostly treated as a policy document that lives in a wiki nobody reads.

Somebody Already Ran This Experiment With Actual Barrels

In 1920 a company called Hooker Chemical bought a partially dug canal in Niagara Falls, and for the next 33 years it used the thing as a disposal site, putting roughly 22,000 tons of chemical waste into the ground. The logic was completely sound, storage was cheap, the land was theirs, the practice was legal, and a fair amount of what went in the hole was technically feedstock, material with a real industrial value if anyone ever wanted to go get it.

In 1953 Hooker sold the site to the local school board for one dollar. The deed carried what became known as the Hooker clause, a written disclaimer saying the company would not be responsible if anybody got sick or died from the waste buried there. The local board proceeded to build school and a neighborhood went up on top.

Then the 1970s happened, families were relocated, Congress held hearings that are still studied as an oversight case, and on December 11, 1980, Carter signed CERCLA. And the statute did two things that have relevance today.

It made liability strict, meaning you do not get to argue that you followed the industry standard or that you were not negligent. And it made liability retroactive, meaning it reached backward to conduct that was perfectly lawful when it happened. The Hooker clause, a real contract, negotiated and signed by consenting parties, turned out to be worth nothing. Occidental Petroleum, which bought Hooker later, inherited the whole thing.

It's true that retroactive strict liability is a sledgehammer, the transaction costs have been genuinely awful, and a meaningful fraction of the money went to lawyers arguing about allocation rather than to anybody's groundwater. Nobody should hold CERCLA up as model legislation, but it was pretty groundbreaking (pardon the pun) for the time. What it gives us, for our purposes, is a demonstration of what a legislature actually does once a stored-material problem gets bad enough and public enough. It does not grandfather you and it does not care what your contract says.

So when somebody tells me their data retention exposure is handled because the DPA covers it, or because the terms of service disclaim it, or because everything was collected lawfully under the rules that applied at the time, I think about a signed piece of paper from 1953 that did all three of those things.

What Minimization Actually Costs You, And When

The reason "keep everything, decide later" won is that it was the rational choice for about fifteen years. Storage got cheap faster than anyone could develop taste, the analytics you would want were genuinely unknowable in advance, and every data scientist who ever got told "we dropped that column in 2019" learned to hoard. I have been the person arguing for keeping the raw feed (and am currently paying penance). Sometimes that argument is right.

But that's no longer the general case. Heck, in many ways it's the exception.

A record is created at some edge of your system. A meter, a handset, a claims form, a badge reader, a log line. At that instant, exactly one copy exists, in one jurisdiction, under one owner, and dropping a field costs you one line of configuration. That is the cheapest that decision will ever be, by orders of magnitude, and it is the only moment where "delete" means the thing deleted is gone.

Now let it move. It lands in object storage, gets picked up into a warehouse, gets denormalized into three marts because three teams wanted different grain, gets a feature-store copy for the model, gets a nightly backup with a 90-day cycle, gets replicated to a second region for durability, gets pulled into a vendor's SaaS for enrichment, and gets exported once into a notebook by an analyst who left in March. Call that eight to ten locations, and I am being conservative, because I have not counted the CI fixture somebody generated from prod or the Slack thread with the screenshot.

Every one of those is now a separate deletion obligation, a separate discovery obligation, a separate breach surface, and a separate thing you have to be able to prove about. Not just assert. Prove, to a regulator, with evidence, on a clock. Deleting a field from ten places under audit is not ten times harder than dropping it once at the source. That kind of work gets a program manager, a status color, and a standing Thursday call, and at the end of it the answer is usually "we believe so."

One decision at creation, or ten proofs later, you only get to choose one. And if you don't choose, you default to the ten proofs version. Good luck.

The Part That Should Actually Bother You

We built an entire generation of data infrastructure whose default posture is accumulate now, govern later. And nobody has ever once meant anything by "govern later." It's a sentence you say to end a meeting, and everybody nodding along knows it's bullshit while they're nodding. LATER NEVER COMES. It does not come because there is no forcing function, no deadline, and no individual whose bonus depends on the absence of a thing.

What does eventually arrive is an incident, or a subpoena, or a supervisory authority with a questionnaire, and by then the population of records you are answering for was fixed years ago by a default nobody chose deliberately. It's not that there was a bad call; at least that could be forgivable. That there was no call. A retention policy emerged as a side effect of a storage tier being cheap.

For the regulated and critical-infrastructure crowd, and I spend a lot of my time in rooms with exactly those people, the question in the room has already shifted. It used to be some version of what could we learn from this if we had it all. Increasingly it is can we demonstrate we never held it. Those two questions want opposite architectures, and only one of them is compatible with a pipeline that ships everything to a central lake first and sorts it out downstream. If the filtering, the redaction, the classification, and the retention decision do not happen in the environment where the record was born, they are happening after the liability has already been created and copied.

I have written before that data loses economic value as it sits, and that your catalog is wrong and will keep being wrong. At the end of the day, the value of a record decays, the accuracy of your inventory of it decays, and the liability attached to it does not decay at all. It sits flat, or it goes up when somebody passes a statute. You are holding an asset that amortizes against a liability that does not.

It's ALSO true that SOME of this material genuinely is an asset, and SOME of it is regulatorily required to be kept, and the retention schedule for a medical record is not a thing an engineer gets to have an opinion about. Fine. Good. That is the point. The categories are different, they have different clocks, and a system that cannot tell them apart FROM THE MOMENT OF CREATION has already decided to treat all of it as the most dangerous thing in the pile. Classification at the point of creation is what lets you keep the valuable part at all, which is why I get twitchy when people file it under compliance and hand it to the team with no engineers.

The Clause Was Signed

Time and jurisdiction draw the line between an asset and a liability. And you do not control either one of them. Hooker Chemical had a legal practice, a real commercial rationale, and a signed indemnity. What it did not have was a view into the future, and in 1980 folks decided they were wrong and should have always done better.

You do not get to know today which of your columns becomes a drum out back. What you get to decide is how many copies of it exist by the time somebody asks.

Want to filter, redact, and classify data where it is created, so the copy you keep is the one you meant to keep? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Drums Out Back.

Forty-Nine Megawatts

David Aronchick — Tue, 25 Aug 2026 18:05:24 +0000

On July 14, Governor Kathy Hochul signed an executive order making New York the first state in the country to pause the construction of new hyperscale data centers. The moratorium covers new facilities with an electrical demand of 50 megawatts or more, freezes state environmental permits for up to a year, and includes a promise to build a nation-leading framework for grid reliability, electricity costs, water, noise, and land use before anything else is approved. Thirty-nine more are somewhere in the application pipeline, which tells you how fast this was moving before somebody hit the brakes.

I think this is a TERRIBLE idea, but there's SOME logic behind it, so let's dive in.

Back in May, I wrote about Loudoun County, the most permissive data center jurisdiction in America, flipping to moratorium proposals in about eighteen months. When the friendliest county in the friendliest state turns hostile, it is no surprise that these things appear more often in other districts and states. The grid genuinely cannot deliver power where people want to put the load, and ratepayers are really eating costs they never agreed to. (I SHOULD NOTE: the reason the rates are going up is NOT that the data centers are raising the cost! It's because these power companies see the opportunity, take it, and are heavily regulated. A one-year pause to write actual rules, in a state where thirty-nine applications were stacked up like planes over LaGuardia, is a defensible thing for a governor to do. The same week, for what it's worth, Microsoft was cutting the ribbon on $3.3 billion in Wisconsin, so nobody should mistake this for a national trend yet.

But let's talk about the guidance - it's JUST at 50 megawatts. How'd they choose that?

Every threshold becomes a spec sheet

France has a rule that at 50 employees, a firm crosses into a different regulatory universe: works councils, union delegates, profit-sharing obligations, formal restructuring plans. The result is one of the most famous pictures in empirical economics. Plot the distribution of French firms by headcount, and there is a pileup at exactly 49 and a crater just past it. Firms do not grow in a straight line. They stop at it, split in two, spin off subsidiaries, push work to contractors, whatever it takes to stay a hair under. The economists who studied it, Garicano, Lelarge, and Van Reenen, found the distortion was big enough to measure in fractions of national output. Not because anyone cheated, but because the line was published, and published lines get engineered to.

This is not a French quirk. Federal law caps trucks at 80,000 pounds gross weight, so an enormous amount of American freight rolls at 79,900 pounds. Coastal shipping fleets around the world are full of vessels built to slide just under whatever tonnage triggers the next tier of crewing and inspection rules. Buildings stop one floor short of the elevator code. Wherever a regulation names a number, an industry grows a callus at that number. The market never argues with a threshold. It reads as a product requirements document.

So here is my prediction for New York: a wave of 49-megawatt data centers, with facilities designed to the line the way French firms are staffed to it. Campuses that are legally three separate 45-megawatt projects with three separate applications and, remarkably, three separate fences. Developers did not stop wanting to build in New York on July 14. They started reading the order for its edges that afternoon, billing by the hour.

The accidental architecture

But here's the funny bit... a 49-megawatt data center is a good idea!

A fleet of smaller facilities, spread across a state instead of piled onto one substation, sited where the grid has actual headroom or next to their own generation, closer to the people and data they serve: that is what the physics has been begging for the entire time. The gigawatt campus exists because it is convenient for the operator, not because the workload demands it. Training wants tight coupling; the inference and agent workloads that are actually growing mostly do not care, and demand is on track to nearly triple by 2035, whether we build it in three monoliths or three hundred sheds. New York, in trying to stop data centers, may have accidentally written the first state-level specification for distributed compute. The moratorium is the mandate.

THAT SAID, twenty 49-megawatt buildings running software that assumes one big building is not distributed computing. It's a monolith with extra steps and worse latency. The hard part was never pouring smaller slabs; it was treating computers spread across a state as one coherent system, moving the work to where the capacity and the data already are, instead of hauling everything to a headquarters that no longer exists. The real estate industry will complete the 49-megawatt building in about six months. The software model that makes a hundred of them worth more than the sum of their parts is the actual project, and just dealing with land/power/shell is not going to get you there.

New York thinks it drew a boundary. What was published was a design constraint, and the one thing this industry reliably does with a design constraint is build to it, at volume, faster than the regulator can schedule the follow-up meeting. The question worth watching is not whether the moratorium survives the year. It's what gets built at forty-nine.

Want to run compute where the power and data already live, instead of where the substation used to be? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at Forty-Nine Megawatts.

The Exclusion Is the Spec

David Aronchick — Fri, 21 Aug 2026 18:05:25 +0000

The Insurance Services Office (ISO) has three endorsements in circulation with a 01 26 edition date: CG 40 47, CG 40 48, and CG 35 08. Let's say these names were not presented in front of a marketing committee, but what do I know? What they do, between them, is carve bodily injury, property damage, and personal and advertising injury arising out of generative AI out of the Commercial General Liability form and the Products/Completed Operations form both. In the past six months, carriers have been lining up at state insurance regulators for permission to actually use them, and the attorney tracking those filings for Lathrop GPM called it an industry-wide reaction to the explosion of AI. Berkley went even further last year with an absolute AI exclusion for D&O, E&O, and fiduciary lines, one that reaches past your AI's output to your AI policies, your AI procedures, and your failure to notice somebody ELSE's AI. Talk about the long arm of the law.

For context, ISO writes the forms most of the American commercial market runs on, so an ISO exclusion is about as close as insurance gets to a standards body publishing a deprecation notice.

And keep in mind what the business of insurance actually IS. You take a risk of some kind. They calculate the risk and tell you how much it will cost if it goes wrong. You pay them, and now you are protected. That's the product; there isn't another part. And, for whatever reason, a meaningful chunk of that industry - whose ENTIRE job is to evaluate the risk of just about anything - has now looked at generative AI and said, some version of, no thanks.

They are not being cowards about it

Joe Lam, the Verisk VP who helped write the endorsements, gave the least dramatic (and, I'd argue, most honest) account of it in that Claims Journal piece: "Without exclusions to allow underwriters a level of stability to accept a risk, you run into a situation where they might just walk away from the risk. So exclusions are very essential in the marketplace."

The exclusion, as it stands, fences off the one piece the underwriters can't measure, so they can keep writing everything around it, because the alternative was walking away from the whole line. A narrow exclusion is more coverage than no market at all, and anybody who has watched a line of business go uninsurable (e.g., Enron) knows exactly which of those two is worse.

They're not doing this in a vacuum. Gallagher counted a 978% increase in AI-related litigation between 2021 and 2025, with a 137% jump in the final year of that window alone. And on July 24, the Delaware Superior Court ordered Google to defend a defamation suit over what its AI said about a person. If something has a docket number, people are going to stand up and take notice.

The problem isn't that AI is dangerous

Most of the commentary goes straight to the black box: AI is unpredictable, its outputs aren't deterministic, and underwriters can't model what they can't explain.

That said, insurance has never needed predictability at the individual level. Nobody knows which house is going to burn down, because no one is measuring just one house. The actuarial math asks for exactly one property: that the losses be independent—ten thousand houses, uncorrelated fires, the law of large numbers, everybody goes home happy. Correlation is what kills an insurance market (Go look at the mortgage insurance market in 2008 if you want to see how). Correlation is why nobody will sell you a single policy covering every house on one street against the same fire, and why flood ended up as a federal program.

Now let's look at AI. A handful of foundation models, three clouds, overlapping training corpora, the same inference frameworks and orchestration layers, and vector stores wired together off the same blog posts. I've spent most of my career (Google, Microsoft, Amazon, now Expanso) building distributed systems where a big part of the job is keeping failures from correlating, so watching this particular stack assemble itself has been, let's say, uncomfortable. Gallagher Re has been flagging it for a year, and Aon's Kevin Kalinich distilled the underwriter's view into three words: "aggregated, systemic, correlated." One vulnerability in a common dependency, and the losses land on an entire book of insureds the same afternoon.

I wrote in June about what happens when everything speaks one format and routes through one provider, and the underwriters have now put a price on the answer. Or rather, declined to put a price on it. Fragile things get insured all day long, every day, everywhere. The issue with a monoculture is that when it goes, it all goes at once, and the pool that was supposed to absorb your loss turns out to be built from the same stuff that just failed.

Which means CG 40 47 is a verdict on the topology.

Silent cover is how this always starts

"Silent AI" is exposure sitting within conventional policies that neither confirm nor deny it, left to be argued at claim time by lawyers after the loss. According to one industry estimate, more than 90% of insurers' AI-agent exposure is silent, tucked inside cyber, professional indemnity, general liability, and D&O policies written by people who were not thinking about agents at all.

We have run this movie before, and it did not turn out well.

General liability policies written from the 1940s through the 1970s said nothing about asbestos, because why would they? The exposure was silent, unpriced, and enormous, and it surfaced decades later as long-tail claims against contracts nobody remembered signing. Lloyd's underwriters lost roughly £9 billion between 1988 and 1992. And here is the detail people tend to forget about Lloyd's. The capital behind the market came from about 34,000 Names, individuals carrying unlimited personal liability, and when the bill arrived, many of them lost everything they had; at least fifteen killed themselves. Lloyd's survived only by walling the old years off inside a separate reinsurance vehicle called Equitas, and lawyers were still picking at that structure's solvency a decade later.

Asbestos was in everything; the policies said nothing; and the bill came due twenty years after the premium had been paid. So when somebody tells me that 90% of the industry's AI exposure is currently silent, that sends shivers down insurers' and reinsurers' spines.

Underwriters end up writing the spec

So what do we do?

When insurers can't price something, walking away is only their first move. The second move, reliably, after more than a century of doing this, is to fund someone to go measure the thing. And whoever does the measuring ends up dictating how the thing gets built.

In 1893, the Chicago fire insurance authorities watched the Palace of Electricity at the World's Columbian Exposition light up with a hundred thousand Edison bulbs and kept noticing an inconvenient pattern: the building kept catching fire. Was it the wiring? The hookups? This new alternating current? Nobody knew, and the insurers were not inclined to keep writing the coverage while everybody wondered. So they hired an electrical inspector named William Henry Merrill and funded him, through the Chicago Board of Fire Underwriters and the Western Insurance Association, to investigate. His lab was a room above Fire Insurance Patrol Station Number One. A bench, a table, some chairs, $350 of measuring equipment, and that's it, that was the whole operation.

His first test, filed March 24, 1894, was a sheet of asbestos paper a manufacturer had claimed was noncombustible and nonabsorbent. Merrill found it absorbed water and would not burn, making it useless as insulation, decent for fire resistance, and, either way, a measured fact now instead of a sales claim. (And yes, the first thing Underwriters Laboratories ever tested was asbestos, the same material from the section you just read. Let it never be said that history does not have a sense of irony.) After several thousand tests, the lab published its first list of approved fittings and devices in 1898, and approved products got a label. In 1901, it was chartered in Illinois as Underwriters Laboratories, taking the name of its new sponsor, the National Board of Fire Underwriters, with a stated purpose of testing appliances and recommending them to insurance organizations. Its first Standard, in 1903, covered tin-clad fire doors. After the 1906 San Francisco earthquake, UL was helping the National Board write building codes, and its engineers went on to shape the early National Electrical Code.

It's insane, but true, that a meaningful share of the electrical safety rules governing every building you have ever walked into exists because a group of fire insurers refused to keep writing policies until somebody could tell them what was in the wall. The refusal came first; the standard is the reason coverage ever came back.

So I honestly don't care whether CG 40 47 is fair. What I want to know is what the AI equivalent of a tin-clad fire door looks like, because somebody, somewhere, has to write that before this exposure becomes insurable again.

I can tell you what it won't be: any of the "benchmarks" we have today (which are starting to feel a bit like Goodhart's law). An underwriter does not give a damn that your model came in three points higher on some eval, because an average tells you nothing about the day it goes wrong. What an underwriter needs is provable data provenance, a record of which decisions the system actually made (not just recommended), tenant isolation, and some ceiling on how far a bad model update travels before anyone notices. I argued in April that you can't sue an agent, and these exclusions are what that argument looks like as an invoice. If nobody can locate the liability, nobody can price it, so it goes out of the form.

Every item on that list is a property of how the system is built, not of the model sitting inside it — which is a mildly humiliating thing for our industry to be learning from an insurance endorsement, but here we are.

The carriers that filed exclusions in July did not end anything. They're Merrill in 1894, standing in front of the exposition wiring, declining to sign until someone tells them what's behind the panel. Nobody could tell him. He had to build the lab to find out.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do?

NOTE: I'm currently writing a book based on my observations of real-world challenges in data preparation for machine learning, focusing on operational, compliance, and cost issues. I'd love to hear your thoughts!

Originally published at The Exclusion Is the Spec.

Investment Grade

David Aronchick — Tue, 18 Aug 2026 18:05:04 +0000

On Thursday, July 9, S&P cut Oracle's issuer credit rating from BBB to BBB-. That's one step above junk, which, for a company as old and big and staid as Oracle, is ... not good. The same note named OpenAI as a key credit risk sitting within Oracle's $638 billion backlog and projected a free operating cash flow deficit of roughly $42 billion for fiscal 2027, on top of about $167 billion in total debt already on the books. Having this much debt and FCF deficit is also... not good.

SMASH CUT to Friday, July 10, over in Madison: Wisconsin's Public Service Commission let a deadline expire without putting the item on the agenda, which is the regulatory version of getting ghosted. The item it declined to revisit was a rule the commission approved back in April, part of We Energies' new "very large customer" rate structure. Why? Turns out that any customer in that class rated below A- has to post financial guarantees before the utility will sell it electricity.

Add those two together, and the Oracle subsidiary co-developing the Port Washington campus (roughly a gigawatt on 672 acres, alongside Vantage and OpenAI, if you haven't been following it) is staring at more than $100 million a year in cash deposits or letters of credit before they can move forward. To reiterate, $100 million a year for the privilege of being ALLOWED to buy electricity. Oracle had already sued the commission in Ozaukee County Circuit Court on June 19, arguing that the PSC acted outside its authority, on insufficient evidence, and that the A-line isn't needed to prevent harm to anyone. Maybe! But when they filed, it was two notches under the threshold. Three weeks later, it was three. That's not exactly what I would call "getting closer."

What's funny here is nobody in this fight is arguing the stuff people usually argue about: megawatts, interconnection queues, or transformer lead times, or how much water the DC is going to take (hint: VERY VERY LITTLE). The site exists, the demand is real, everything lines up! Even We Energies wants the customer so badly that it asked its own regulator to stand down. The only thing in the way is a letter grade.

The utility's argument is good, which is the problem

We Energies asked the commission to back off the rule on June 10, warning it would push investment out of the state, and its lawyers put the credit case about as plainly as anyone could: "tens of billions of dollars in Oracle's value would need to be destroyed before creditors or counterparties, such as Wisconsin Electric and its other customers, could experience losses."

Which is true! If I were Oracle's counsel, I'd make the same argument. And for the record, I am not a credit analyst; I have never rated so much as a parking-garage bond. But you don't have to be a financial genius to notice that this is a statement about a company, and the commission is not underwriting a company.

Simply put, We Energies now bills large data centers directly for the generation built to serve them, and that generation includes the proposed Red Oak Ridge plant in the town of Paris, a build that runs north of a billion dollars. If you draw this all the way out, a gas plant is a thirty-to-forty-year asset, and the demand it's being built against is maybe three years old. So, ten to one, asset life over demand history, on a billion-dollar bet. If the tenant walks, the plant stays put, and the residual lands on schools, small manufacturers, and every residential meter in southeastern Wisconsin.

So I think the PSC, whether it would ever phrase it this way or not, is pricing a duration mismatch, and Oracle's position with the credit markets doesn't really factor in. Even though the ratings agency picked the same week to agree, Oracle's borrowing to fund its AI position had already pushed debt-to-equity past 400% as of May, and the stock shed more than $50 in the month before the deadline. Whatever you think of the A- line, the commission's worst-case scenario got measurably more plausible while the reconsideration request sat on somebody's desk.

And Wisconsin isn't some rogue outlier, much as Oracle's lawyers might want it to be. As of May, twenty-three states had approved at least one large-load tariff, with another seven pending. Virginia's version of Dominion's GS-5 tariff starts automatically applying to customers at 25 MW and up in January 2027, with minimum 14-year terms. I wrote in early July about FERC ring-fencing the wire while the scarcity leaks out anyway, and this is the other half of the same coin: cost allocation is about who pays; collateral is about who gets stuck with the plant when nobody does.

The Northwest ran this experiment and kept the receipt

When I started pulling on this thread, I found a strong historical record leading back to the Pacific Northwest. The Bonneville Power Administration began selling enormous volumes of cheap hydroelectricity to aluminum smelters in the 1940s, and by the end of the century, forty percent of U.S. smelting capacity was located in the Pacific Northwest. That arrangement lasted fifty years, longer than any computing platform I have ever heard of. But in 1996 and 1997, the smelters traded away chunks of their long-term BPA contracts for access to a wholesale market that, at that particular moment, was cheaper.

But when 2000 arrived, and the West Coast power crisis came with it, wholesale prices rose by factors of ten and twenty. All of a sudden, the contracted power was worth more than the metal it was made of. What's the rational move? Don't smelt any more. The smelters paid their workers to stay home and resold their Bonneville electricity on the spot market. Paying people NOT to make aluminum penciled out better than making it, which is, before you ask, totally legal, exactly what the contracts allowed. By the summer of 2001, all ten of the region's smelters had shut down or dropped to token production, and most of them never came back.

The dams stayed, though. The transmission built to feed Troutdale and The Dalles is still standing, still in the rate base, sized for an industry that left. Nobody defaulted, nobody committed fraud, and the region still wound up holding fifty years of single-purpose infrastructure after the single purpose moved away. I am fairly sure THAT is the scenario sitting in a spreadsheet in Madison right now, and I want you to notice it is not "will Oracle go bankrupt."

Credit risk is an architecture problem

Look, everyone is treating the collateral fight as a financing question, and I get why, but I think underneath it there's a granularity question which is more interesting.

A $100 million annual guarantee only makes sense as an ask because the underlying asset is indivisible. One plant, one campus, one tenant, and no way to lose eight percent of Red Oak Ridge, because there is no eight percent to lose. Project finance figured this stuff out something like a century ago: break the huge commitment into pieces whose failures aren't perfectly correlated, and suddenly it's bankable. That is the entire reason nobody underwrites a single mortgage the size of a neighborhood. We built the AI buildout backward from all of that. I don't think anybody decided to, exactly. But everything got concentrated: one counterparty, on an indivisible thirty-year asset, justified by three years of demand history.

On the other hand, if you could split it up into tranches of investable assets, things would change quite a bit. A portfolio of facilities spread across several interconnections, several utilities, and a mix of tenants doesn't need a letter of credit the size of a small utility's annual revenue, because no single tenant walking out strands a billion dollars of steel. Note that nobody got more creditworthy in that scenario! The blast radius just shrank until the counterparty stopped mattering so much, the same reason a bank will happily lend against a hundred small loans it can't individually assess and won't touch one enormous loan it can. (NOTE: This, and credit default swaps, were part of the root of the 2008 financial crisis - caveat emptor)

This sort of distributed STUFF is what years of my life have been spent working on. Kubernetes at Google, and now Expanso, are variations on the same idea: stop trying to make any single component heroic, and make its failure boring instead. Small failure domains, loosely coupled, add up. I've been arguing for years that compute should sit closer to its power and its data, mostly on physics and cost grounds, and occasionally because the grid flatly said no. What I did not expect, and probably should have, was the balance sheet independently arriving at the same place, in a language utility commissions already speak.

The market has half-conceded all of this, by the way. Last Sunday, the Journal reported that Nvidia is in talks to guarantee financing of around $250 billion so OpenAI can lease a 10-gigawatt campus that SB Energy is building in Piketon, Ohio. Strip off the zeros and squint, and it's the identical instrument the Wisconsin PSC asked for: a counterparty that can't carry the obligation alone, and a third party stapling its own balance sheet to the lease so the deal can close.

One of those requests got a quarter-trillion-dollar term sheet. The other one got the regulator sued in Ozaukee County.

The commissioners in Madison are asking, out loud, the question the industry has spent the past year answering privately and refusing to answer in public: if these assets can only be financed with a co-signer, what the hell do we think we've been building?

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do?.

Originally published at Investment Grade.

The Longest Life in Compute

David Aronchick — Sat, 15 Aug 2026 18:01:39 +0000

On Monday, Nvidia announced memorandums of understanding with Apollo, BlackRock, Blackstone, Brookfield, Goldman Sachs, and KKR to mobilize more than $500 billion of third-party capital so that hyperscalers, frontier labs, and enterprises can borrow against AI hardware instead of paying cash for it. Jensen Huang told CNBC this is the first time technology chips have become an investable asset class and described the chips as productive, long-lived, fungible, and flexible. This is PROBABLY all correct, but I think it misses something cool.

In the same announcement, Nvidia told bond buyers that CUDA keeps extending the useful life of the hardware and improving its economics over time. EXTENDING THE USEFUL LIFE. Nvidia has just informed the largest capital allocators on earth that its software support policy is the collateral.

The pitch is a hundred and fifty years old

As always (it seems), everything old is new. Productive, long-lived, fungible, transferable across operators (aka STUFF) is the pitch for rolling stock — is decidedly pretty well understood. In the 19th century, equipment trust certificates put locomotives and freight cars into a trust that leased them back to the railroad, which meant that when the railroad went under, the equipment was not railroad property and did not go into the estate. (By the way, if you'd like to read an excellent book on the subject, let me recommend RailRoaded. In an era when American railroads failed constantly, equipment trust paper was among the safest debt you could hold. This is convenient! Because, many many many people think we're in a world where things are going to go bankrupt (soon-ish?), so the fact that the structure Nvidia is using having been stress-tested across a century and a half of bankruptcies is quite nice.

Now, one of the most important parts is that a freight car works whether or not the manufacturer of the freight car is in business. A freight car has residual value only if it can roll onto somebody else's track. AS AN ASIDE, for decades in America, a great deal of it was not interchangeable, because the southern roads ran a five-foot gauge while the north ran what became standard, so freight moving between them had to be transferred by hand at the break. Then on May 31 and June 1 of 1886, work gangs across the South moved one rail three inches inward on roughly 11,500 miles of track in about 36 hours. And while it was a really big financial feat, it was ALSO a big financial one. Overnight, a boxcar sitting in Atlanta became collateral worth something in Chicago.

Back to GPUs, this is not the case today! CUDA is HIGHLY hardware specific, and this is (or was anyway) a real blocker. Compute is fungible only to the degree that the software layer keeps accepting the hardware underneath it.

Nvidia pulls the lever

CUDA 13 removed offline compilation and library support for the Maxwell, Pascal, and Volta architectures (Volta is the V100, which shipped in 2017). That silicon still computes exactly as well as it did the day it was installed, though probably less power efficiently than even a very low end chip. But, because nvcc will no longer generate machine code for it, cuBLAS and cuDNN will no longer ship kernels for it, the architectures are marked feature-complete in the toolkit release notes, and PyTorch dropped them as build targets to match. This is bad, and (ultimately) sets a pretty severe deprecation schedule.

This is also one of the biggest questions in the market - exactly how long do cards still work? What ends the productive life of an accelerator? Until now, the answer was its the morning the framework stops compiling for it, and the company that wrote that framework had a significant incentive to NOT update. But no longer! NOW, Nvidia has every reason to keep the deprecation out as long as possible, because that makes the physical asset worth more, longer.

This is also a HUGE accounting impact. The hyperscalers moved server useful life from three or four years to six, which analysts estimate removed around $18 billion a year in depreciation expense from their income statements. Michael Burry's argument, which has moved from accounting newsletters into the mainstream over the past year, is that carrying GPUs on five and six year schedules while Nvidia ships a new architecture annually understates depreciation by roughly $176 billion across 2026 through 2028. Amazon already cut a subset of its servers and networking gear from six years to five, citing the increased pace of development in AI specifically, and absorbed roughly $700 million of lower operating income for the honesty. Meta went the other direction in the same window. So, with no settled convention to appeal to, everyone is tossing aronud vague ideas. Nvidia isn't stopping, of course, and Hopper landed in 2022, Blackwell in 2024, Vera Rubin hit full production this March, and Rubin Ultra is slated for the back half of 2027. The cadence that creates the problem is not slowing down.

What happens when things invert

For thirty years the vendor incentive ran in exactly one direction. Deprecate the old architecture, make the upgrade compulsory rather than attractive, and book the new generation. The installed base of five-year-old hardware was a support cost with no revenue attached, and every quarter you kept it alive was a quarter somebody didn't buy the replacement. The entire industry is built on that reflex.

However, with Nvidia's new financialization (and half a trillion dollars of paper written against that hardware), the sign changes. Whether or not Nvidia ends up signing an explicit residual guarantee (and the announcement conspicuously does not contain one), it now has a direct commercial stake in five-year-old racks remaining useful. And there are lots more examples. Meta handed its Hyperion joint venture a residual value guarantee covering the first sixteen years of operation. Broadcom agreed to cover 100% of any shortfall to the senior tranches on the $35 billion Apollo and Blackstone structure funding Anthropic's compute. Wall Street had already lent more than $11 billion against GPUs held by neoclouds, and the figure has only gone up since. Every one of those lenders is now exposed to a software decision made in Santa Clara; the only ones who caught it were the Motley Fool, though it framed it as a demand signal rather than a support obligation.

So, at the end of the day, we have something unprecedented: a hardware vendor's software support calendar is, in effect, a financial covenant. Now, the difference between a GPU supported for eight years and one supported for three is the difference between investment grade and junk on the same physical asset. Which is... pretty freaking huge from a software perspective.

Aviation figured this out the hard way. When Fokker collapsed in 1996, they left 1,130 aircraft already in service, with almost no support, and they solved it by building a standalone company created specifically to keep spares and product support flowing. The airframes were airworthy either way butwhat made them financeable was somebody agreeing, on paper, to keep supporting them. Orphan a fleet and the values go regardless of what the metal can still do.

What to actually ask for

If you are signing anything that involves accelerators over the next eighteen months, price per FLOP is no longer the interesting number, and neither is delivery date. Ask how long the toolchain will target this architecture, get a date rather than an adjective, and ask what specifically happens to a cluster you own outright on the day the compiler moves past it. Almost nobody currently has that number, in the same way that almost nobody running a real-time dashboard can tell you its actual latency. The number exists whether or not you know it, and it is your architecture restated in years.

There is a genuinely good outcome here! PARTICULARLY for open source and drivers, which have typically been underappreciated. If financing works and Nvidia does what the paper requires, that's architectures compiling for eight years instead of three. By 2030 that produces an enormous installed base of hardware that is uneconomic for frontier training and ENTIRELY adequate for inference, which is already the majority of AI compute and the part of the workload where the bill actually lands. Depreciated silicon that no lab wants to train on is exactly the silicon you want sitting next to a factory floor, a substation, a hospital basement, or anywhere else the work does not live in a data center. A financing structure designed to keep the buildout going may accidentally fund the distributed compute layer nobody could previously justify on a spreadsheet.

Nvidia spent a decade selling scarcity, resulting in a world where last year's rack becomes a bad bet. It now has to sell durability and a promise about software, made to people who will eventually ask for it in writing. The fact that they are SO incentivized to keep things running, particularly the hardware you already bought, really is a new world.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Longest Life in Compute.

The Freeze Function

David Aronchick — Tue, 04 Aug 2026 18:28:33 +0000

On June 9 the public comment window closed on the anti-money-laundering rules that will put teeth in the GENIUS Act, the stablecoin law signed last July, whose full implementing rules take effect on July 18. The debate around all of it is a debate about money. How much reserve has to sit behind each token, and in what (one to one, in cash, insured deposits, and short Treasuries). Whether issuers can pay interest to the people holding the tokens (they cannot, not directly). And how many ordinary bank deposits get up and leave the moment a dollar of stablecoin starts looking like a better checking account than your checking account. The American Bankers Association has been waving around a $6.6 trillion figure for the deposits that could flee. Citi models the stablecoin float reaching as much as $3.7 trillion by 2030. It is a real fight, the numbers are enormous, and I understand why everyone is staring at them.

They are staring at the wrong feature.

The thing that makes a stablecoin dollar different from a bank dollar is not the yield and it is not the settlement speed. It is that the issuer can freeze one specific dollar, in one specific wallet, from a console, in about the time it takes to ship a config change. Tether and Circle both build a blacklist function directly into the token, and it is not a bug or an exploit or a backdoor somebody found. It is a documented capability they describe to regulators as a feature. Tether has used it to freeze more than four billion dollars across better than seven thousand addresses, in coordination with OFAC and U.S. law enforcement. Circle's numbers are smaller, a hundred-odd million across a few hundred wallets, and it froze one of its first big batches the day the Treasury sanctioned the Tornado Cash addresses in 2022. A dollar sitting in USDT has exactly one switch in front of it, and that switch has already been thrown more than seven thousand times.

Now hold that up against the gloriously boring thing it is replacing. A dollar inside the American banking system sits in one of roughly four thousand three hundred separate FDIC-insured institutions, each with its own charter, its own ledger, its own compliance department, its own failure domain. There is no master console. An individual bank can freeze an individual account, sure, and does. But to freeze all of it at once, every dollar in the country, you do not push a config change. You need the state, and you need it to do something extraordinary.

We know exactly what that looks like, because it has happened, precisely once, and it took the entire machinery of the federal government to pull off. At one in the morning on Monday, March 6, 1933, Franklin Roosevelt signed Proclamation 2039 and suspended every banking transaction in the United States. Congress passed the Emergency Banking Act on the 9th, and only after Treasury examiners had gone bank by bank did the system come back, 12,756 of them reopened by March 15. The reason that operation needed a presidential proclamation, an emergency act of Congress, and the better part of two weeks instead of an afternoon is that the money was spread across more than twelve thousand institutions, and there was no single place to reach in and stop it. The friction was the whole point. The fragmentation that everyone now calls inefficient was the thing that made a national freeze require an act of national will.

Programmable money deletes the friction. It rebuilds the single switch of 1933, makes it surgical enough to hit one wallet instead of all of them, and makes it fast enough to throw before you have finished reading the sentence that authorized it. And we have decided to call that progress, and to call the twelve-thousand-ledger mess it replaces backward. Slow settlement, redundant charters, no central console, a system so fragmented it took Congress to halt it. Every one of those inefficiencies is a place where universal control is expensive and slow. None of them was a defect we never got around to fixing. They were the only thing standing between "your money" and "your money, conditional on staying off a list maintained by a company in a console you will never see." You do not feel the redundancy of four thousand banks until somebody has swapped it for one freeze function, and by then feeling it does not help you much.

This is the same lesson the cloud keeps teaching, denominated in dollars instead of GPUs. A single control plane is a single point of seizure, and it does not much matter whether the thing it controls is your compute, your data, or your checking account. Centralizing the ledger does not just make payments faster. It manufactures a switch that did not used to exist, hands it to whoever holds the keys, and bills the convenience back to you as a feature.

A bank run used to take a crowd. A bank holiday used to take an act of Congress. The GENIUS Act has a great deal to say about the reserves behind the dollar and nothing at all to say about who holds the switch in front of it. Both are worth arguing about. Only one of them can freeze your money before you finish this sentence.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Freeze Function.

The Map Is the Moat

David Aronchick — Fri, 31 Jul 2026 18:26:49 +0000

On June 17, a coalition of the biggest names in enterprise software published a new open standard called Agentic Resource Discovery, ARD for short. Google, Microsoft, and Salesforce headlined it, with Cisco, Databricks, GitHub, Hugging Face, NVIDIA, ServiceNow, and Snowflake signing on. The stated goal is boring in the way that important infrastructure is always boring: a common way for an AI agent to find out what tools and other agents exist and how to call them, so nobody has to hand-wire every connection between every piece of enterprise software. It is the latest in a run of big-tech moves to set the open standards for agentic AI, and on the surface it looks like more of the same cooperative plumbing.

The two companies not on the list are OpenAI and Anthropic, which, seems like a pretty big miss.

Strip the branding off ARD and you are looking at service discovery - an area I spent quite a bit of time on with Kubernetes and know and love since distributed systems have needed since there were two computers to introduce to each other. A service comes online and has to announce what it is and where it lives, and other services have to be able to look it up without someone editing a config file by hand. DNS is service discovery for hostnames, while Consul and etcd and ZooKeeper are service discovery for microservices. ARD is the same primitive aimed at agents and tools: each organization publishes an ai-catalog.json manifest under its own domain, an agent describes what it is trying to do, and the discovery layer tells it what is available to do it with. Federated, Apache-licensed, hosted under the Linux Foundation, no single company owning the registry.

This, in a lot of ways, could be the new Google (or Yahoo, depending how old you are). Whoever controls how services find each other controls which services get found which is pretty fucking powerful, since the thing that resolves names decides what is allowed to exist. A service that isn't in the registry is a service that, functionally, is not there. Put another way, it's a map, and the map is the moat, and the enterprise incumbents are taking a stab at deciding how you draw it.

People have already been circling around this. Anthropic gave the world the Model Context Protocol in late 2024, which standardized how an agent connects to a tool, and it was good enough that everyone adopted it, all the way to 10,000-plus public servers and 80% of the Fortune 500 touching it within a year. Google shipped Agent2Agent for how agents talk to each other. Both of those layers eventually got donated to the Linux Foundation. And while the labs sit on the governance of how agents connect and how agents converse, the ARD stakes out the one layer above both, the catalog of what is worth connecting to in the first place. It's not ALWAYS binary, but this is a move that lets the more traditional enterprise folks (Google, Microsoft, Salesforce, Snowflake, etc etc) push what they already own. The labs have every incentive to push another angle, since the enterprise folks are trying to redraw the boundary of the fight to a place where they hold the land, where the labs only show up as entries in a directory somebody else defines.

There is a second, sharper move buried in the design, and it is a bet against the labs' entire product surface. ARD assumes the agent discovers and calls tools on its own, without a conversational interface as the bottleneck. Read that as what it is: a wager that the chatbot, the thing OpenAI and Anthropic have built their enterprise businesses around, is not where real work will happen, right as the two labs are visibly pulling in different directions on what an agent should even be. In the ARD worldview the chat window is a demo, and the actual economy is agents quietly resolving a procurement request against the approval system and the budget tool and the vendor database with no human typing in a box. If that bet is right, the most valuable real estate in enterprise AI is the registry, not the model, and the registry is the thing the labs conspicuously do not have.

Now, credit where it's due, because this is solid engineering and a real fix for a real problem. Before a discovery standard, every autonomous workflow is bespoke integration, hand-built and brittle, which is most of why "agentic AI" has been a great demo and a miserable production system. A federated catalog you publish under your own domain, keeping control of what you expose and to whom, is the correct architecture. It keeps the description of your capabilities next to your capabilities instead of shipping the whole map to a vendor who rents it back to you. I have argued for a while that any catalog that lives away from the thing it describes is a catalog that is already wrong, and a federated, locally-owned manifest is the first design I've seen from the majors that takes that seriously.

But the word "federated" is doing a lot of work, because the history of federated systems is the history of things that were going to stay decentralized and didn't. Email is federated, and most of the world's mail now flows through a handful of providers who decide what counts as spam. DNS is federated, and there is still a root, and there are still registrars, and there is still an afternoon where a name can stop resolving. A2A's own pitch is that it mitigates vendor lock-in, which is precisely the promise every federation makes on day one. Federation describes where the data sits, not where the power settles, and power settles wherever the defaults are set. If ARD becomes how agents find things, then a tool that does not publish an ARD manifest becomes invisible to every agent that uses it, and the companies that authored the schema and ship the most popular catalogs get to define what "discoverable" means. The quieter fight underneath all of this is not who runs the agent but what you are allowed to take back out, and a discovery layer is where that gets decided first. Apache 2.0 licensing means nobody can charge rent on the standard itself. It says nothing about who benefits when your agents reflexively discover Workspace, M365, and Salesforce first.

So the fight nobody framed correctly at the time is not model versus model. It is the layer that owns the enterprise's tools and data drawing a border against the layer that owns the models, and choosing to fight it at the map. Whether you build agents or just buy them, the question worth asking your vendors is not whose model is smartest. It is who controls the directory your agents read before they do anything at all, and whether the map of your own capabilities is something you own or something you have quietly agreed to rent. Being left off that map is what OpenAI and Anthropic are worried about this month. Eventually it is what you should be worried about too.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Map Is the Moat.

Free Binaries, Again

David Aronchick — Tue, 28 Jul 2026 18:24:58 +0000

The phrase "open source" has a birthday. February 3, 1998, at a strategy session in Palo Alto, called in a hurry after Netscape announced it was releasing its browser code. Christine Peterson coined it in that room, and her reasoning was specific: "free software" made everyone think about price, and price was the wrong axis. She wanted a word that pointed at the source code itself. Within a few weeks Netscape and O'Reilly were both using it, and the Open Source Initiative existed to defend the meaning.

The thing the word was invented to be different FROM matters more than the word. There was already freeware, free binaries, software you could download and run and pass to a friend at zero cost, and could not read, could not rebuild, and could not carry forward without the company that made it. The whole point of 1998 was to say that those are different categories and that we should stop confusing them, because one makes you a participant and the other makes you a guest.

Twenty-eight years later, we are calling model weights "open," and I would like someone to explain to me which category they're actually in.

Tobi Knaup made the case this week that open-weight AI is having its Kubernetes moment, and I want to be careful, because I agree with nearly everything he wants to happen. He also has standing on this; he co-founded Mesosphere, built DC/OS, and then watched Kubernetes take the category out from under him. I was on the other end of that trade. I was the first (non-founding) product manager on Kubernetes, which means I spent 2015 and 2016 doing the thing that ran him over.

So when he says "Kubernetes moment," I know exactly which moment he means. And I want to push on the analogy harder than he does, because if you take it seriously — actually, seriously, not as a compliment — it is asking for a great deal more than a download link.

Start with the definition, since we already have one. The OSI published its Open Source AI Definition in late 2024, and it asks for four freedoms (based on the Stallman-coined freedoms): use the system for any purpose, study how it works and inspect its components, modify it, and share it. Open weights, nail use, and share. However, unlike source code, models have a bit more complexity. You cannot study a model the way you study source code; you can probe its behavior, which is a different activity that we call evals precisely because it is not reading. And you cannot modify how it was made. You can fine-tune the output of a process you were never shown, which is closer to sanding a table than to changing the design. Meta's LLaMa ships under a custom license with a monthly-active-user threshold attached, and publishes nothing meaningful about its training data, which is why OSI has had to keep posting things with titles like "Meta's LLaMa license is still not Open Source" and why Meta simply rejected the definition rather than argue with it.

I've had some experience in this rodeo before. I believe open weights are the most important thing happening in AI economics, and I still do. The commodity tier is going to eat this market from below. Good enough at a fraction of the cost, running on hardware you control, is the winning bid for most enterprise work, and the labs sealing off the top of the index do not change that. None of what follows is a case against open weights. It's a case against the sentence people say right after "we went with the open model," which is usually some version of: so now we're not dependent on anyone.

A bit from my history: I was there when, on July 21, 2015, Kubernetes hit 1.0, and Google handed it to a foundation that did not exist the week before. While we released it under Apache 2, I believe the real novelty was the governance. There was real friction from people who did not want to be tied to Google or its schedule. Giving the project away was how you removed our veto. And that's what we wanted! We knew that if we exercised too tight a control over the top, we never would be able to get the industry moving in that direction. I was only one of the people weighing in on this decision, but I am so glad we did.

With the foundation and the license, people also had the freedom to completely fork. Not grab binaries and then hope upstream stayed compatible; people could take every element of Kubernetes (other than the name) and fork away. Not that anyone planned to use this threat, but it kept the community honest. If the steward went bad, or slow, or greedy, you could take the whole project and keep going, and everyone knew it, which is exactly why nobody had to. That is what "open" purchased: Continuity without asking.

This is where I start to reject the concept of "open" models. You cannot fork a model; there is nothing to fork. A fork of Kubernetes is a living project with a build and a roadmap you now control. A "fork" of an open-weight model is a fine-tune of a snapshot, because the two inputs that would let you continue its development, the training data and the compute, were never in the box. The lab kept the factory and shipped you the output. That is not a criticism of the lab! It's a description of what you received.

At the end of 2017, Jeremy Lewi, Vishnu Kannan, and I announced Kubeflow, which was our attempt to open-source the way Google ran machine learning internally. It was ALSO Apache 2.0, with contributions from Google, Cisco, IBM, and Red Hat. It had all four freedoms, for real, the full 1998 package, no asterisks. Sadly, one thing that really hurt us in the beginning (and even now) was that it was miserable to install, miserable to upgrade, and miserable to keep running. We shipped a pile of genuinely open components and told people they had a platform.

We weren't alone; Kubernetes was ALSO miserable. In 2015 you stood a cluster up by hand, in the right order, and if you got certificate rotation wrong you found out about it eleven days later. Kelsey Hightower wrote a tutorial called Kubernetes The Hard Way, and it wasn't satire; it was the documentation that a lot of us actually used. Difficulty didn't kill Kubernetes; it barely slowed it down.

So, difficulty was never the variable. Two projects, same license, same four freedoms, both a nightmare on day one. What Kubernetes had was a few hundred people at Red Hat and Rancher and three cloud providers whose paychecks depended on making it installable, plus a foundation that made it safe for all of them to show up in the same room, with a license that made it legal for them to try.

I think this is the thing missing from all the discussions of model "openness." Publishing something and getting it adopted are two different projects with two different budgets, and getting it adopted and having a community are two more. We shipped the first one and wrote "platform" on the box. We are doing a version of the same thing with open weights, and it is going to cost somebody a couple of years.

Because when a lab hands you weights, you're getting the cheapest artifact in the building. Not cheap to make — training is the most expensive thing anyone does with a GPU. Cheap relative to what it costs to keep a model running in production, for other people, which is where the actual bill lives.

Go count what a frontier lab actually operates. Serving infrastructure that holds a tail-latency target while traffic swings 10x over an afternoon. A batching and caching strategy that decides whether the unit economics work at all. Quantized builds for whatever silicon the customer actually bought, not the silicon you wish they'd bought. An eval suite that catches the regression before the customer does. A safety layer, an abuse pipeline, a deprecation policy, capacity planning eighteen months out, and a human being holding a pager at 3 AM with the authority to roll the whole thing back.

None of that shipped with the weights. And, most importantly, almost none of it is represented in open source either. Not because open source is bad at it, but because open source has never been in the business of operating things on behalf of strangers. Nobody's pager is attached to your cluster. That was equally true of Kubernetes, which is precisely why Red Hat and the clouds got to build real businesses on top of a free thing, and why nobody found that outrageous.

The labs and the hyperscalers are doing an enormous amount of this work, and the open-weight conversation has picked up a bad habit of calling all of it rent. Most of what they're charging for is the product. You can hate the price and still be accurate about what's being priced.

Knaup is right that a serving stack has shown up, and it's a good one. vLLM, SGLang, llama.cpp, Ollama, MLX — I use these; they're excellent, and honestly, they were never in doubt. Engineers build inference runtimes because inference runtimes are FUN. Nobody open-sources a deprecation policy. Nobody sends a pull request with a capacity plan. The missing pieces are missing because they're boring and because somebody has to be accountable for them, and accountability is the one thing a download cannot transfer.

People are choosing open-weight models right now as an insurance policy, a hedge against a lab repricing them, deprecating them, or quietly re-aligning the thing they built on. I understand the instinct completely; I've written about what it costs when the wire in front of the model goes away. That's not to say weights on your own disk, which let you keep running what you already have, isn't genuinely worth something. But they do not let you extend it in the spirit of open source. When the version you're on stops being good enough, your options are to wait for whatever the lab decides to release next or to stop.

Run it, or stop. That's the whole menu.

So let's total up what "open" has actually bought us so far. You can run a very good model on hardware you already own, and the top slice of the inference bill, the expensive slice with somebody's margin stapled to it, goes away. I've spent a lot of words arguing that it's going to reshape this market from the bottom, and I still believe that. It is also, start to finish, an argument about price.

But I want to capture, in the current discussion, the wisdom from those people in February 1998 in Palo Alto, in a new phrase, precisely because the old one made everybody argue about price, and price was the wrong axis. It took about twenty-eight years for us to need their word again, and we've spent it on a discount.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Free Binaries, Again.

Acceptable for Inference

David Aronchick — Tue, 21 Jul 2026 18:24:28 +0000

Starcloud closed $170 million at a $1.1 billion valuation this month, the fastest company in Y Combinator's history to reach a billion. Starcloud-2 launches later this year carrying Blackwell B200s to run commercial cloud workloads in orbit for customers that already include AWS and Google Cloud. This is pretty rare for a YC company! ACTUAL paying tenants, this year, in a rack that is going around the Earth every ninety minutes.

When Google published Project Suncatcher, the press took the obvious angle: Google wants data centers in space, fleets of TPUs linked by free-space optics into kilometer-wide arrays of 81, two test birds going up with Planet by early 2027. Solar power that never sets, which seems exactly right! Let's do it!

But, Google ran its TPUs through a particle accelerator to simulate the dose of low-earth orbit, and the compute chips came through fine. The high-bandwidth memory took uncorrectable errors that the error-correcting code could not catch and repair, at a rate Google described as "likely acceptable for inference."

Not "acceptable" for anything, but "acceptable for inference". This is pretty specific guidance that running in a space is only suitable for a specific job it has in mind will forgive the occasional wrong bit.

This makes sense! A model writing the seventh paragraph of a product description genuinely does not care if one weight in one layer got nudged by a passing cosmic ray. The output was a probability distribution to begin with, so a little noise in the machine is a rounding error inside a process that was already rolling dice.

Precision was always a marketing choice

We have spent the entire AI era letting people believe these systems are precise, and orbit just makes the imprecision physical. Down here, the fuzziness hides inside phrasing that sounds authoritative. Up there, it's a photon flipping a one to a zero in a memory cell, and the model downstream will report the result with exactly the same confidence it would have had if the bit were correct. I wrote a while back about a support chatbot that told a customer they had 365 days to return a product when the real policy was 30, every dashboard green, the model perfectly poised while it was flatly wrong. Now picture that same confidence, except this time the error was injected by the sky.

For a chatbot, who cares. The trouble starts the instant somebody wires a forgiving workload to an unforgiving job. Starcloud has filed to put 88,000 satellites in orbit to process data rather than relay it, and somewhere in the addressable market for eighty-eight thousand orbiting accelerators is a company that will run something that counts on hardware whose spec sheet says, in so many words, good enough to be wrong sometimes. The problem isn't JUST that it can be wrong, but that it can be wrong silently, since nobody in that chain is going to be told which rack the answer came from. That is the entire product promise of cloud: you don't think about the hardware. It is a very good promise right up until the hardware develops opinions.

The rot is already in the building

If you're about to file this under "space is weird," don't. Silent data corruption is not JUST an orbital phenomenon. Meta went looking on its own fleet and found corrupted computations coming out of perfectly healthy-looking CPUs at a rate high enough to matter across a datacenter, and the industry now has an Open Compute working group and a whitepaper about it specifically because inference multiplies the blast radius: one marginal device quietly wrong, hundreds of thousands of inferences an hour, every one of them delivered to a customer with full confidence. The causes are mundane and unfixable, timing violations, aging, marginal defects, temperature, voltage, and yes, cosmic rays hitting silicon at sea level.

So orbit is not introducing a new bargain; it is turning up the gain on one we already made and mostly declined to discuss. What Google did that's genuinely new is write the terms down. Every terrestrial datacenter is running some rate of silent corruption it does not advertise and cannot fully measure. Google put a number next to it, attached a workload class, and called it acceptable. That candor is rare enough here that it reads as alarming, but it shouldn't. That candor should come stapled to the side of every output, like an FDA label.

We named this bargain once already

Distributed systems made exactly this trade a long time ago, on the ground, and we even gave it a name. Eventual consistency. We decided that for a shopping cart or a like counter, the right answer soon-ish beats the exact answer slower, and we built half the modern internet on top of that call. It sits close enough to the point of this whole blog that it's in the subtitle. But the expensive lesson, the one every architect learns once and never forgets, was working out which systems you are absolutely not allowed to make eventually consistent. Amazon runs its catalog eventually consistent and its payments emphatically not, and the entire art was knowing exactly where that line sat.

The orbital memory result is that same fork, pushed down to the level of a single bit and handed to radiation to decide. Correctness is about to become a per-workload dial that gets set, in part, by how much cosmic radiation a given satellite happened to eat that week. And the thing generating the answer is not going to print which setting it was running on.

So the question worth asking about compute in space was never whether we can do it. We can (at least to some degree), the first paying workloads go up this year, and the solar-power argument is real and worth taking seriously. The question is who keeps track of which answers came back from a place where the memory does not reliably hold, because the model certainly won't volunteer it. It'll sound exactly as confident either way. It always does.

Google, to its credit, told us the setting. Ask your own vendors what theirs is.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Acceptable for Inference.