DEV Community: David Aronchick

It Worked Before We Understood Why

David Aronchick — Sat, 14 Nov 2026 00:01:48 +0000

There is a critical hunk of metal inside a jet engine that I think is a wonderful analog to how we should think about AI.

In the hottest part of the turbine, gas can be flowing past the blades at temperatures above the melting point of the alloy they are made from. Those blades are also spinning, carrying enormous loads, and expected to keep doing this while the people in the cabin argue about whether the person in front of them should be allowed to recline.

This seems like a fairly aggressive thing to ask of a piece of metal.

Veritasium's deep dive into jet engines is wonderful on the engineering involved. The survival trick is a combination: nickel superalloys, protective coatings, and cooling air routed through and around the blades. The gas can be hotter than the metal's melting point because the metal itself is kept cooler. Rolls-Royce describes gas temperatures around 1,700°C, several hundred degrees above the blades' melting point. Even with cooling, the material has an big, big job.

Part of what makes these alloys work is a microscopic structure called gamma prime. Tiny particles, often shaped like cubes, sit inside the nickel matrix and interfere with the movement of dislocations, the defects that allow a crystal to deform. Under certain conditions, heating actually makes those dislocations lock up more readily. Over a limited temperature range, the material can become harder to deform as it gets hotter. This is not unlimited, and it is certainly not permission to turn off the cooling, but it is a lovely reversal of what most of us expect heat to do to metal. The microscopy alone is worth rabbit holing on.

The fascinating part of it was the order in which things happened. Useful alloys came before a complete explanation of their behavior. Engineers developed materials, measured their properties, changed their composition and processing, and learned which ones survived. The microscopic explanation was developed alongside and after that work; early nickel alloys were being developed in the 1930s and 1940s; the detailed account of what was happening inside them took longer.

They knew something was consequential before they understood it.

I think we are having a surprisingly hard time allowing that distinction in AI.

Around the same time, I watched Grant Sanderson and Steven Strogatz discuss what AI means for mathematics. There is a section, starting around 33:55, on whether understanding is overrated, which is an entertaining question to hear in a conversation involving the creator of 3Blue1Brown. Making mathematics understandable is pretty much the man's entire thing.

The question I was thinking about was how much do we value understanding for itself and how much because it has been such a powerful way to get something done?

For a mathematician, understanding may very reasonably be the point. For the passenger in 23B, there is a more immediate requirement, which is not falling out of the air.

Of course, humans have always made useful things before fully explaining them. Fermentation predates microbiology by a fairly substantial margin. We did not wait for a molecular account of bread before opening bakeries. AI did not invent the separation between a useful result and an explanation we find satisfying. It is making that separation show up in work where a lot of us had assumed our understanding was indispensable.

That is uncomfortable. I like understanding things. Quite a lot. A good explanation lets you transfer an idea, spot a mistake, and predict what will happen somewhere you have not tested yet. It is enormously valuable. But the feeling of understanding a system and the evidence that it will do its job are different things, and I think we sometimes let the first stand in for the second.

An explanation can be convincing and wrong. A result can be correct even when the process that produced it is opaque. Neither observation tells you, by itself, whether it has value.

The turbine blade is useful here because nobody has to settle a philosophical argument before deciding what to measure. You need to know how it behaves under heat and load, how that behavior changes with time, which defects matter, and when the part has to come out of service. Incomplete microscopic understanding does not eliminate that work. It makes that work more consequential.

And this is where the analogy gets demanding. A turbine blade operates inside a comparatively constrained physical environment. A general-purpose model can encounter a new kind of request, retrieve hostile instructions, or be given access to a tool its designers never anticipated. There is no universal test suite that turns all of that into a certified component. If we want to borrow aviation's confidence, we have to borrow its discipline about the conditions under which that confidence applies.

What interests me, though, is why this feels so personal.

In the video, they talk about the particular pleasure in finally understanding something that has been bothering you when a collection of disconnected facts suddenly becomes a thing you can turn around in your head. You can explain it to somebody else. You can imagine what would happen if one piece changed. The world has become a little less mysterious, and you have become a little more capable of living in it. I don't think we should be embarrassed about wanting that or pretend that getting an answer from a machine provides the same experience.

But I do wonder how much of our discomfort comes from the possibility that the world can become more manageable without becoming more understandable to us. We have spent a long time occupying both roles: the sentient beings who want things to happen and the sentient beings whose understanding makes them happen. It is deeply unsettling to discover that we might get more of the first while being less necessary to the second.

If a machine finds something useful that I cannot explain, my inability to explain it does not make it less useful. It does, however, leave me with a choice about what evidence I would need before depending on it. And that is a harder question than whether its explanation sounds like something a smart person would say.

We already live with an enormous amount of borrowed understanding. I cannot personally explain everything (anything?) about keeping an airplane in the sky, and neither can any one of the people who built it. My confidence rests on other people knowing different things, checking one another's work, recording what went wrong, and having reasons to take those failures seriously. Somewhere in that arrangement, knowledge that no individual possesses becomes something all of us can use.

AI puts pressure on that arrangement because some of the missing understanding may not belong to anyone. There may be no expert down the hall who can sit me down and explain why this particular answer emerged. That is a real difference. It means we have to be much more explicit about what we have observed, what we can establish independently, and what we are merely hoping will continue to work.

It also forces a question we tend to skip when we are impressed by a result: useful to whom?

The passenger wants to arrive safely. The airline wants to operate economically. The engineer wants a component that survives its specified conditions. Even in that relatively concrete example, “it works” contains a collection of human purposes. Move into education, relationships, or the way we organize a society, and agreeing on what counts as success becomes a substantial part of the work. A system can be exceptionally good at producing the thing we asked for while leaving us to discover that we asked for the wrong thing.

Understanding matters here in a different way. We need enough understanding of ourselves to decide which outcomes are worth pursuing, what we are willing to risk, and whose experience counts when we decide whether something succeeded. Greater capability makes those questions more consequential. It does not answer them.

And understanding can still be worth pursuing even when it is no longer required to obtain a result. I would still want to know why those tiny cubes make a turbine blade stronger if every flight I ever took were guaranteed to land safely. There is something wonderful down there, and I want to see it. The practical usefulness of curiosity has helped justify it for a long time, but I would hate to discover that we only valued curiosity because it paid.

Maybe that is part of what we are being asked to get comfortable with: understanding as something we love, sometimes something we need, without insisting that it must precede every useful thing we allow into the world.

That requires humility in both directions. We have to be willing to accept evidence for something we cannot yet explain and equally willing to admit when the evidence is too weak, when the consequences are too large, or when we have confused getting an answer with knowing what to do. We remain responsible for the decision to trust it, including the consequences for people who never got a vote.

The engineers working on those alloys had somewhere they wanted to go and an incomplete account of what would get them there. They measured, tested, learned, and kept investigating. I find that a much more human story than the idea that progress happens only after somebody has understood everything.

We have always lived between what we can explain and what we are prepared to try. AI is making that space larger, and we will have to learn how to inhabit it carefully.

There are still people on the plane.

Further reading: NASA's 1972 historical review of high-temperature alloys, Superalloys: A Primer and History, and Cambridge's introduction to turbine-blade coatings.

Want to run the filtering and the fast decisions where the data is born, instead of two hours downstream of it? Check out Expanso. Or don't. Who am I to tell you what to do.

NOTE: I'm currently writing a book based on what I have seen about the real-world challenges of data preparation for machine learning, focusing on operational, compliance, and cost. I'd love to hear your thoughts!

Originally published at It Worked Before We Understood Why.

Two Hours Behind

David Aronchick — Wed, 11 Nov 2026 18:01:09 +0000

On December 24, 1814, British and American negotiators signed the Treaty of Ghent. The document then got put on a boat. On January 8, 1815, fifteen days after the signing, British troops attacked Andrew Jackson's defenses outside New Orleans. The American Battlefield Trust counts 2,034 British casualties, including 285 killed, against 71 American casualties.

The familiar version of this story says they fought a battle after the war was over. They didn't. The treaty still needed American ratification. The Senate approved it on February 16, and the exchange of ratifications on February 17 ended the war. On January 8, the armies still had orders to fight. An agreement had been reached on the other side of the Atlantic, but the process needed to make it effective had not finished.

Nobody blames those commanders for not having a telegraph. I am less sympathetic when we build a fast network and then deliberately leave the data waiting at every stop.

Ask whoever owns your real-time dashboard how long an event takes to become a pixel. The query latency is only part of the answer. In front of it sit a scrape interval, a transport hop, a queue, an ingest batch, a transform job, a materialization schedule, and a cache TTL nobody has looked at since it shipped. A dashboard can load quickly and show you something that happened two hours ago.

Count the waiting

Take a deliberately simple example. Collection runs every 60 seconds. Transport takes two seconds. Ingest flushes every 30 seconds. Enrichment runs every two minutes. After the enriched data reaches the alert evaluator, the condition must remain true for ten minutes before it fires. That last delay might be entirely reasonable after last year's on-call burnout.

For an event that just misses each scheduled step, the budget is 60 + 2 + 30 + 120 + 600 = 812 seconds. Thirteen minutes and thirty-two seconds, before notification delivery or human response. That's an illustrative sum of maximum scheduled waits, not a measurement or a prediction that every event takes that long. Some stages may line up more favorably. Backlogs and retries can make the trip longer.

The alert rule needs particular care. A five-minute lookback window does not automatically mean a five-minute wait. A pending duration does: in Prometheus, a for: 10m clause requires the condition to stay active through evaluations for ten minutes before the alert fires. Evaluation cadence adds its own delay. Read the rule you deployed before adding numbers to the budget.

If the promise is to act within five minutes of the event, this design cannot meet it. If the promise starts at detection, write down what counts as detection. Starting the clock after ten minutes of waiting makes the metric easier to satisfy without getting anyone help sooner.

Each team may have made a sensible choice. Bigger batches reduced storage overhead. Less frequent enrichment saved compute. A longer pending period cut noisy pages. The person on call gets the sum of those decisions. You have a fucking architecture problem wearing a dashboard as a disguise, and the postmortem will blame the dashboard.

Buying more query compute will not shorten a configured ten-minute hold. Someone has to decide whether that wait belongs in this path.

Give each signal a budget

Industrial systems make this distinction explicit because a late response can damage equipment. Sielco's comparison of SCADA and industrial IoT separates production-critical supervision from supplementary sensing and analytics. A plant-wide energy dashboard and a machine's protective control loop have different jobs, even when they use readings from the same equipment.

That is a useful discipline to bring into a dashboard review. Decide how late each answer can be before choosing its route. A maintenance trend may tolerate minutes. A protective response may need a dedicated local control system, with requirements a general-purpose telemetry pipeline cannot satisfy. Moving an analytics rule to an edge server does not turn it into a safety system.

You also need to decide what information can survive aggregation. An average can be useful for a trend and useless for detecting a brief spike. Keep the resolution the decision needs, whether you evaluate the signal locally or centrally.

Keep the center where it helps

I have architected plenty of centralized telemetry and would again. The mid-incident questions are often cross-system questions. Correlation, retention, and the auditor's copy are good reasons to keep central processing. A centralized streaming system can also handle fast reactions when its transport and processing fit the budget.

When they don't, move the fast rules earlier in the pipeline, sometimes to the source. Filter, enrich, classify, and evaluate what can be decided there, then send the center what it needs. Some decisions need the wider context. Others should not wait for a warehouse refresh to discover that one machine crossed a threshold.

In March I wrote about the time value of data. The practical follow-through is to measure where that time goes. Keep the event timestamp through the pipeline. Record when the data arrives, when the rule evaluates, when the notification goes out, and when someone acts. Account for clock skew before treating differences between machines' timestamps as precise measurements.

Then test the whole path under load, including the batch boundaries. Show the age of the data on the dashboard alongside the result. Give someone ownership of the event-to-action budget, because owning the query is only a small part of that job.

Before the next incident, put your promised response time beside a measured trip through the system. If the waiting alone exceeds the promise, you have a specific configuration or architecture decision to change.

Want to run the filtering and the fast decisions where the data is born, instead of two hours downstream of it? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Two Hours Behind.

Black October: Your SBOM Is Not Enough

David Aronchick — Tue, 03 Nov 2026 18:02:32 +0000

October has been a miserable month to be responsible for keeping systems secure. During the week of October 19, teams were dealing with critical PHP flaws, a poisoned LiteLLM release, a critical GitLab patch, a Brotli vulnerability in Node.js, compromised certificate keys, and cloud platforms telling customers to rotate their secrets. Several disclosures described attacks that had started earlier. The remediation arrived together.

The industry has taken to calling it “Black October.” The “cybersecurity apocalypse” warnings predate this week, but the exhaustion is easy to understand. You could be halfway through rebuilding your application images when the next advisory told you to replace the credentials those applications use.

There is a data problem running through these incidents. Sensitive information sits in process memory, environment variables, agent configurations, CI systems, and backups. Some of that information is a credential granting access to another system. An attacker who obtains it can keep going after the original entry point has been patched.

We need software bills of materials. We also need to stop treating a software inventory as the end of the security review.

A week of different ways in

PHP2Shell is the most visible example. Wiz demonstrated unauthenticated code execution and, separately, the ability to read sensitive PHP process memory, including passwords, tokens, and API keys. The researchers say AI helped turn longstanding flaws into practical exploits across different deployments. The NSA called for immediate patching and advised considering taking affected systems offline when immediate updates were not feasible.

The same week, Node.js disclosed a Brotli decoder flaw in which malformed compressed input could corrupt memory, potentially causing a crash, memory disclosure, or remote code execution. The input your application decompresses is part of its attack surface. That includes data received while doing ordinary application work, not only requests to a login page.

GitLab's October 21 critical release fixed a maximum-severity flaw that could allow an unauthenticated attacker to execute code through a specially crafted regular expression. A vulnerability advisory establishes a dangerous capability, not proof that every exposed installation was breached. It still puts the system holding source code and release automation at the front of the patch queue.

Then there was eMudhra. Mozilla reported that private keys for eight intermediate certificates had been compromised. Someone able to intercept a connection could use fraudulent certificates to impersonate websites and potentially read or modify traffic. Mozilla blocked the affected certificates through OneCRL, then updated NSS to help protect applications using its trust store without OneCRL updates. Updating a browser and updating every application's certificate trust are different jobs.

The platform incidents made the credential problem explicit. Railway's preliminary report says attackers accessed encrypted environment variables and decrypted a subset using compromised credentials. Zeabur reported that attackers read customer environment variables from its platform database after entering with leaked internal credentials. Its report describes activity on October 14–15, before the PHP disclosure.

These were separate failures with different mechanisms. The reports do not establish one coordinated campaign or an AI cause for every incident. The workload for defenders was simultaneous anyway.

LiteLLM shows how a data leak becomes a software attack

The canonical LiteLLM investigation is worth reading all the way through. It corrects parts of the initial incident report, including the early theory that the first GitHub token came from logs.

In May, a configuration file containing an internal Agent Platform master key was committed to a public repository. The commit was overwritten. The key was identified as needing revocation. It was not revoked.

On October 10, the attacker used that key to query the Agent Platform. The response exposed a GitHub CLI OAuth token belonging to an organization owner. A system intended to let agents file pull requests had been given an owner's token without narrowing its scope.

The attacker used that access to extract more credentials through CI workflows. On October 21, stolen identities allowed the attacker to change the release workflow, overcome repository protections, and trigger and approve a malicious release. The pipeline signed and published the poisoned artifacts. Version 1.107.0 contained credential-stealing malware.

An SBOM helps downstream users identify that version. It does not revoke the master key exposed months earlier, restrict the owner's token, or prevent a compromised release system from signing malicious output.

The sequence crossed several systems: configuration data exposed a credential; that credential unlocked agent configuration containing another credential; repository access enabled further secret extraction; those identities controlled a release consumed by other organizations. The software supply chain and the movement of sensitive data were part of the same attack.

The backup copied the replacement key

Zeabur's report contains a detail that should change how teams think about incident response.

During containment, a new credential was put in place at 15:40. A daily cluster backup captured it at 16:00. The attacker used it at 20:30. Zeabur confirms the backup captured the new key and that the key was later used; downloading that backup is its assessed explanation, rather than a directly confirmed observation.

The team rotated a credential while another route to replacement credentials remained exposed. A backup operation meant to preserve recoverability also preserved sensitive access information.

The same report offers a bounded example of data minimization working. Projects that had moved their environment variables off the platform database after an earlier incident were outside the scope of that database read. Zeabur explicitly warns that this did not rule out exposure from a compromised host. Removing a copy reduced one exposure without making the entire system safe.

Railway's account reinforces the point about encryption. The variables were encrypted, but the attacker acquired credentials that enabled decryption of a subset. Encryption at rest matters. So does the separation between the encrypted data and the identities capable of retrieving its plaintext.

The inventory needs the actual data paths

CISA defines an SBOM as a nested inventory of software components. Keep it current and connect it to what is actually deployed. During a week like this, an accurate inventory saves responders from hunting through every team and every image to find an affected runtime.

Alongside it, we need evidence of what each workload can read, which identities it uses, where its output goes, and how long the resulting copies survive. A second spreadsheet is not sufficient either. Permissions and destination restrictions have to hold when tested.

For a data pipeline, that means following the raw input through transformations, logs, retry queues, caches, and backups. A job that needs one file should not inherit a credential that reads the entire archive. A text extraction worker should not carry release credentials. Sending a redacted result downstream is safer than sending the raw record and hoping every recipient applies the same policy.

Processing near the source can avoid unnecessary exports of raw data. It still requires constrained workloads and controlled outputs. A worker running beside a database with unrestricted access can do plenty of damage, and an aggregate can still disclose sensitive information. Location does not substitute for authorization.

AI applications add another reason to examine incoming data. Indirect prompt injection can turn a retrieved document into an attempt to redirect an assistant's behavior. This is an additional mechanism, not an explanation for the incidents above. A valid file format or known source does not make the contents safe to obey. The application must enforce access and action permissions outside the model.

A patch is one part of recovery

The surrounding incidents show why the work continues after an update. OpenCode reported a breach affecting a subset of Console users with external providers linked. ConnectWise's October 26 advisory described unauthorized access to Cloud Backup application credentials and continued to direct customers to investigate their Microsoft 365 environments. ConnectWise also said it had no indication that existing backup archives were compromised. Stolen application credentials and stolen backup contents are different claims, and responders need to know which one the evidence supports.

I build data infrastructure, so the operational details in these reports are where I keep stopping. A configuration API returns an owner's token. A backup copies a fresh secret. A compromised identity decrypts stored variables. These are specific data flows we can restrict, observe, and test.

Pick one production service at the next security review. Assume its process is compromised. Verify which records and credentials it can read, which records it can change, and where it can send them. Include backups and intermediate copies. Then test whether revocation holds after the next automated job runs.

Keep the SBOM open while you do it. The response is unfinished until you have checked what the vulnerable software could reach and closed the routes that remain useful after patching.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at Black October: Your SBOM Is Not Enough.

The Donation Bin

David Aronchick — Wed, 28 Oct 2026 00:01:47 +0000

A recycler in Bellevue, Washington, opened one of his donation bins in July expecting the usual: dead laptops, tangled cables, and a printer somebody was too embarrassed to sell. Instead, he found a radiation-hardened Texas Instruments SBP9900X microprocessor from 1977, stamped "Experimental," the family of chips that flew in missile guidance and deep-space hardware. Next to it, an 8x8 optical star coupler from Canstar, a component from the era when "fly-by-light" aircraft controls were a laboratory bet. And around those, a scattering of hand-built hybrid microcircuits, gold wire bonds laid down by a human being at a microscope, the signature of thirty years of military and aerospace engineering somewhere in the Seattle area.

No paperwork. No notebooks. No names. Nothing. Tyler Rivers, who runs the recycling outfit, is an economics grad with no aerospace background, and he set out to figure out who built these things and why. GeekWire's July report left those questions open.

Bunnie Huang, the hardware hacker, looked at the photos and guessed it was a collage of souvenirs a technician kept from various projects. Perhaps a technician retired or died, and somebody cleared a garage. Which makes sense, but it's kind of crazy we don't know. The objects span decades of engineering work, and because their provenance did not arrive with them, their survival depended on one curious recycler recognizing what was in the bin. In this case, preservation depended on someone noticing what had been thrown away.

The bin decides.

We are the most storage-rich civilization that has ever existed. Standard object storage is cheap enough to make retention an easy default, and so we keep everything: exabytes of debug logs nobody will ever read, seven years of clickstream from a product that shipped its last release in 2019, and every intermediate table of every pipeline, retained forever because deletion requires a meeting and storage does not. Meanwhile, the irreplaceable stuff—the annotated schematic, the prototype with the hand-laid gold bonds, and the engineer who remembers why the third revision moved the coupler—has no budget line, no bucket, and no lifecycle policy except the estate sale.

We often keep the logs and lose the engineering without making either choice explicit. Keeping is a decision, and we have delegated it to defaults. The default for bits is retain, because it's cheap and nobody gets fired for a full bucket. The default for atoms, and for the context in people's heads, is the bin. So the archive fills up with the stuff that was easiest to keep, whether or not it was worth keeping.

And notice what actually makes the Bellevue trove a mystery. The silicon survived. Those chips were built to survive radiation that would scramble anything in your phone. What's gone is the metadata: whose bench, which program, and what problem was the experimental part experimenting on? The bits survived, and the lineage died, and without the lineage, the bits are just very well-made shrapnel. Every organization I talk to is sitting on petabytes that are one retirement, one reorg, and one "who owns this table?" away from being gold wire bonds in a donation bin: perfectly preserved, meaning discarded. I've written before about why catalogs drift out of truth; this is the endgame of that drift, just with a fifty-year fuse.

Three ways to lose what you kept

NASA spent three years searching for the original slow-scan tapes of the Apollo 11 moonwalk, the highest-quality recording of the most-watched event in human history, and concluded in 2009 that they had probably been erased and reused for Landsat satellite data sometime in the early eighties. Not stolen. Not burned in a fire. Taped over, because magnetic tape was expensive, the erase-and-reuse policy was the default, and the moon landing was, by then, old news.

Or take the F-1, the engine that put the whole thing up there. When NASA engineers wanted to revisit it in the 2010s, the blueprints still existed. Drawings alone could not convey every manufacturing decision. Engineers examined surviving hardware, scanned it, and test-fired its gas generator, with help from heritage Rocketdyne engineers who remembered the work. Recovering the knowledge took both the artifact and the people who could explain it.

My favorite version, though, is the BBC Domesday Project. In 1986, the BBC celebrated the 900th anniversary of the Domesday Book by building a new one: a national survey of Britain, contributed by a million people, published on state-of-the-art laserdiscs. By 2002, the specialized computers and drives needed to read the discs had become rare, and preservation projects turned to emulation to keep the material accessible. The original Domesday Book, written on parchment in 1086, remains perfectly legible at the National Archives. The eleventh century out-archived the twentieth by roughly nine hundred years.

Back in Seattle, the stuff that ends up at the Museum of Flight got there because, at some specific moment, a specific person made an active decision to keep it and wrote down why. That decision is the whole difference between an archive and a landfill.

And decisions get unmade. The city had a museum built for exactly this kind of hardware, Paul Allen's Living Computers: Museum + Labs. It closed at the start of the pandemic, never reopened, and in 2024 Christie's auctioned the highlights while the rest went to a museum near Atlanta. Its former director told GeekWire that most museums would not exist without collectors, people who look at a box and say, "Well, that seems like it's worth keeping." Rivers is doing archaeology on nearly fifty-year-old hardware because its context did not arrive with the hardware. Somebody, someday, is going to do the same thing to your data lake.

Check your defaults. And if you recognize the gold bond work, GeekWire asked readers to get in touch.

Want your data to keep its lineage attached while it's processed, instead of reconstructing it later like a crime scene? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at The Donation Bin.

A Kill Switch of One's Own

David Aronchick — Sun, 18 Oct 2026 18:01:02 +0000

In June, the European Commission unveiled its Technology Sovereignty Package, a stack of proposals built around a Cloud and AI Development Act and a second Chips Act, and the people selling it were unusually honest about the fear underneath. The worry, in the framing they chose themselves, is a kill switch: the possibility that the cloud running Europe's hospitals and ministries and grid operators answers to a jurisdiction on the other side of the Atlantic, and that the access could be throttled or cut from there. Executive Vice-President Henna Virkkunen said the goal was to guarantee that critical workloads do not carry a remote off switch somebody else controls. Von der Leyen said Europe cannot afford to depend on others for the technologies that keep its hospitals running and its grid stable. The Act proposes to triple EU data center capacity within five to seven years, sort cloud services into four "assurance levels", and push sensitive government workloads away from AWS, Microsoft, and Google. The Commission puts the data center build alone at around two hundred billion euros by 2036, mostly private money. Its own target date for Parliament and Council merely agreeing on the text is the fourth quarter of 2027. The kill switch, presumably, will wait.

The jurisdiction problem is still real. The CLOUD Act concerns access to data under qualifying legal process; it is not a general authority to shut down a cloud service. Sanctions, commercial decisions, and operational failures create other dependencies. On May 7, a cooling failure in one Northern Virginia Availability Zone forced servers to shut down and left customers working through a prolonged recovery. That was a localized infrastructure failure, not the whole region going dark. It still showed why the location and control of a dependency deserve separate scrutiny.

The proposal needs to be judged by what operators can do when a dependency fails.

The Commission includes an open-source strategy, and its sovereignty framework considers operational autonomy, control, resilience, and exposure to third countries, so they are going to more of the primitives involved. My concern is what happens after certification. For example, can an operator move a workload, keep its data usable, and recover without permission from the failed provider? More European capacity can reduce some dependencies, but capacity and assurance levels alone do not demonstrate that those recovery paths work.

Europe has run this experiment before recently, and it kept the receipts. GAIA-X launched in 2020 as the continent's sovereign-cloud answer to exactly this anxiety. Within a year the American hyperscalers were members; one of the French founders, Scaleway, walked out, saying it had no more time to waste on a project rotting from the inside with American Big Tech, and the European share of Europe's own cloud market kept sliding while the project that was meant to reverse the slide held conferences. It is hard to build a moat when you invite the besiegers onto the committee that designs the moat.

There is an older warning in France's Plan Calcul, launched in 1966 to reduce dependence on foreign computer suppliers. Its national champion, CII, was eventually absorbed into Honeywell-Bull in 1976. That history does not prove the current European proposal will fail. It does show that domestic ownership at launch is a poor substitute for control that survives later mergers, suppliers, and operational changes.

Sovereignty has both legal and operational parts. Workloads that can run across independent providers and your own hardware can reduce the reach of any one service-denial decision. That takes more than placing a second logo on the architecture slide. Identity, DNS, networking, data formats, and control planes can preserve the same dependency across nominally different providers. Failover has to be tested, the dependencies have to be independent, and the people operating it have to know what they can keep running when one supplier disappears.

Europe is right to examine who can interrupt its critical systems. The useful test for the package is whether it buys operators a working alternative: a workload moved, a dependency isolated, and a recovery rehearsed. Ask for that evidence alongside the capacity targets and assurance certificates.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at A Kill Switch of One's Own.

The Chaperone

David Aronchick — Wed, 14 Oct 2026 18:01:17 +0000

IBM's biggest quantum chemistry result depended on a conversation between very different machines. In May, Cleveland Clinic, RIKEN, and IBM simulated a protein–ligand complex in water totaling 12,635 atoms, which IBM describes as the largest heterogeneous quantum-classical electronic-structure calculation to date. Two 156-qubit IBM Heron processors, one at Cleveland Clinic and one at RIKEN in Kobe, did the quantum part. Classical computers did everything else: cutting the protein into fragments, sampling the nastiest fragments, and then... handing it off. Then two classical supercomputers took those samples: Fugaku, which topped the TOP500 rankings before Frontier, and Miyabi-G. The algorithm (called EWF-TrimSQD, because quantum computing has never met an acronym it didn't want to marry) made progress by stitching the molecule back together.

It's kind of crazy to see the pace of things. In January, at Supercomputing Asia, IBM's Jay Gambetta presented what was then the largest and most accurate chemistry experiment ever performed on a quantum computer: a pair of iron-sulfur molecules, solved by one Heron chip in a closed loop with all of Fugaku, the two machines passing data back and forth in an unbroken cycle. The protein work started smaller: Trp-cage, 303 atoms. Five months later, in May, it had a T4-Lysozyme–ligand complex in water at 11,608 atoms and a trypsin–ligand complex in water at 12,635, a forty-fold jump. That's a big jump! And I don't want to take that away from anyone at Cleveland Clinic, IBM, or RIKEN who spent the preceding year getting a quantum processor to provide a real-world use case. BUT.

Quantum computing has been sold for something like two decades as the technology that gets you out from under classical computing's limits, the thing that solves what a normal machine can't solve no matter how many racks you buy. Chemistry simulation is the textbook example: the compute required to track every electron's interaction with every other electron grows exponentially with the size of the molecule, which is why a high-accuracy classical treatment of a whole protein is still impractical and why quantum computing has always used chemistry as its opening pitch. So, we have the biggest (quantum) chemistry result of the years still needing classical machines at every single step, and... it still does not yet outperform the best classical approaches. There's a good reason (fault tolerance) which will be addressed by 2029, which will still not remove classical control, error decoding, or the classical steps of a hybrid chemistry algorithm, but it'll be ... more quantum? IBM describes the future as quantum-centric supercomputing, with the machines working together.

On narrow tasks, researchers have reported results beyond the classical methods they compared. On July 30, IBM and Qedma claimed a quantum advantage on the dynamics of a driven quantum magnet, up to 74 qubits, using error mitigation and comparisons with classical methods where those methods worked. A September report on an unreviewed Nighthawk preprint described a million random-circuit samples in 19 seconds. BlueQubit's 110-year Frontier estimate was for a particular tensor-network method at comparable fidelity, under an unlimited-memory assumption. But benchmarkmaxxing random-circuit sampling is great for press, not a chemistry result. Useful chemistry still needs a comparison against the best classical workflow for the same problem.

There's an almost too-on-the-nose precedent for this from transport. Britain's Locomotives Act of 1865, called the Red Flag Act, required every road locomotive to be preceded by a person on foot carrying a red flag, walking sixty yards ahead to warn horses and riders that the machine was coming. These were steam engines of up to 14 tons, rolling down roads built for horses. They could genuinely propel themselves, but they couldn't (yet) do was be unsupervised, so the law bolted a human chaperone onto the front, capped the speed at four miles an hour in open country and two in town, and made the astonishing new machine walk at the pace of the person escorting it. Parliament dropped the flag itself in 1878 and kept the man, who now walked twenty yards ahead with nothing in his hands. He lasted until 1896, when light vehicles under three tons, the new motorcars among them, were finally let off the leash. On November 14 that year, the Earl of Winchilsea marked the occasion by tearing up a red flag that nobody had been legally required to carry for eighteen years.

The quantum computers in Kobe and Cleveland contribute a specialized step inside a larger classical workflow. The result is real progress. The test is whether the whole workflow produces a useful answer better than the best classical alternative, including its classical processing and coordination costs. The red-flag story is a reminder to inspect what surrounds a new machine. It does not establish a date when that support disappears. IBM's 2029 target concerns fault tolerance, and its chemistry researchers expect the hybrid workflow to continue.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at The Chaperone.

The Clock Is a Server

David Aronchick — Fri, 09 Oct 2026 18:01:29 +0000

At ten to three in the morning on July 8, 2026, Telstra's network forgot what time it was. One of the carrier's time servers, the boxes that tell the rest of the network what time it is, had reset its calendar to 2006, and the bad time was propagating outward through the synchronization layer. By breakfast, 8.8 million Telstra customers were on services that were degraded or dead. Victoria's V/Line regional railway stopped running trains. National freight operators suspended movements as a safety precaution. Tyro, the payments company that handles card terminals for about 80,000 Australian businesses, started dropping transactions at shop counters. 612 triple zero emergency calls failed on Telstra's network, more than 430 of them without reaching anyone at all, and every one ended with a welfare check on the person who had tried to call for help. Telstra had the time fault fixed by about 4 p.m., some thirteen hours in, and knock-on problems ran into the next day.

Nobody attacked anything at any stage.

The independent review Telstra released on September 1, 2026, found that a power supply swap on a Melbourne time server kicked it back 1,024 weeks, to 2006, the classic GPS week-number rollover. (This is an excellent time to remind everyone to read Falsehoods Programmers Believe about Time) The vendor had issued advisories about that bug as early as November 2000 and Telstra's own technicians logged it in September 2022 and decided it didn't apply to them, because the patch fixed a feature they weren't using. Then an undocumented design change in October 2025 started using it, which, of course, nobody wrote down, so in January 2026, with the bug now very much live, they looked at the patch again and passed again. The alarms that would have flagged the timing drift were only watched during business hours, by a limited number of people, and two of Telstra's key engineers were on mandatory leave that night. The review's overarching finding was that Telstra never treated its timing system as a critical capability. CEO Vicki Brady's summary: "That is a miss on our side." This was a known defect in a component nobody treated as critical, because time does not feel like a component until it stops.

As I write this in early October, ACMA is still investigating, with fines of up to A$30 million on the table. On September 21 a Senate committee recommended taking Triple Zero away from Telstra and handing it to a new public body, and on October 1 the three big carriers switched on roaming onto each other's networks for when a disaster takes one of them down. Bushfires and floods, mind you. Nobody has a roaming plan for a clock that thinks it's 2006.

To see why one bad calendar could do that, start with the radio network. In the time-division duplex modes used by modern LTE and 5G networks, the tower and the handset share one slice of spectrum by transmitting in alternating slots, and the specifications require every cell to hold its slot boundaries within roughly 1.5 microseconds of a common reference, so no two neighbors drift more than about 3 microseconds apart. Beyond that tolerance, adjacent towers begin transmitting over each other's frames. Carriers therefore operate a dedicated distribution system for time itself, in which receivers discipline themselves against atomic clocks in orbit, the Precision Time Protocol carries the reference across the backhaul, and holdover oscillators at each site keep an approximation running whenever the upstream reference goes quiet. Time on a network like Telstra's is produced in a small number of places and consumed by tens of thousands of sites, and in July the production run was defective.

Distributed systems people have known the underlying truth for decades, and everyone else keeps rediscovering it at three in the morning. Computers cannot vote their way to the correct time, because every voting protocol needs timeouts and every timeout needs a clock, so Leslie Lamport's famous 1978 move was to give up on clocks and derive ordering from messages instead. Google went the opposite direction with Spanner's TrueTime, installing GPS receivers and atomic clocks in its data centers and having the system report an uncertainty interval and wait it out rather than pretend the time is exact. Opposite philosophies, same admission: somewhere in every distributed system sits a reference authority everything else syncs to. You do not get to not have one. The design choices are how many you have, who operates them, and how far a lie travels when one of them lies. Most of the internet answers those questions with "a handful," "volunteers and hyperscalers," and "we'd rather not think about it," courtesy of NTP.

On the night of June 30, 2012, a leap second was inserted into civil time to keep atomic clocks aligned with the earth's rotation, and a bug in the Linux kernel's handling of the extra second sent processes on servers around the world into high-CPU spin loops. Reddit went down for a while. Mozilla, LinkedIn, Yelp, and Gawker all reported problems that night. The Amadeus airline reservation system failed in Australia, and Qantas staff checked passengers in by hand for hours until it recovered. Four years later, in January 2016, the US Air Force decommissioned a GPS satellite designated SVN-23, and an erroneous ground-software upload put a 13-microsecond error into the time broadcast by 15 satellites, about half the constellation. Over the following hours, the BBC's digital DAB radio listeners lost reception because the transmitters took GPS as their timing reference, and one timing-equipment user logged nearly 2,500 alarms before the problem cleared.

We did solve this once, at continental scale, and the people who solved it ran trains. On November 18, 1883, the North American railroads imposed five time zones on the continent by private agreement, thirty-five years before Congress ratified the arrangement in the Standard Time Act. And when synchronization still failed, as it did near Kipton, Ohio in 1891, where a crew member's watch ran slow and two trains met on the same track, the response was Webb C. Ball's watch inspection regime: approved movements, thirty seconds a week of allowed drift, inspections on a schedule, paperwork per timepiece. The reference was standardized, and then every consumer of it was audited as an independent clock, so a watch could still fail but a defect could not propagate. Telstra's design inverts that: centralize the reference so everything downstream can be a dumb consumer, and one bad clock becomes everyone's outage at the same instant.

Time is upstream of everything, so a time failure surfaces downstream everywhere, wearing everyone else's costume. On July 8 it looked like a train problem, a payments problem, and an emergency-line problem, and the fix took half a day at one of the most sophisticated operators in the hemisphere, because you cannot debug a distributed system when the broken thing is the thing you use to order events. Every log line on every node carried a timestamp from the same poisoned well.

So, a suggestion, and it is the same one the railroads landed on after Kipton. Go find out where your time comes from. Not conceptually; specifically. How many independent sources, owned by whom, checked by what, and what your systems do in the hours after the holdover oscillators start to drift. If the answer is "one vendor's servers in two cities," you do not have a clock. You have a subscription to someone else's, and on July 8 all of Australia got to read the terms.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Clock Is a Server.

Estate Sale

David Aronchick — Tue, 06 Oct 2026 18:01:26 +0000

A bankruptcy court has finally put a price on losing your genome. About $2.60, and that's for the whole thing, in case you were hoping it was per gene.

And $2.60 is only the states' share. Throw in the consumer class action and the price climbs all the way to about $9.40.

Every step of this got covered as its own story, and it's one story. In 2023, a credential-stuffing attack exposed profile data on 6.9 million people. In March 2025, 23andMe filed Chapter 11, and its most valuable asset, by a wide margin, was the DNA database itself. Twenty-seven states and the District of Columbia sued to block the sale of genetic information without customer consent. The court approved the sale anyway, with privacy commitments attached, and TTAM closed the deal on July 14, 2025. The states' settlement over the breach landed July 14, 2026, a year to the day. First the estate sold the genomes, and then it settled up for losing them with the change.

That day, a coalition of more than forty attorneys general announced a settlement with 23andMe's bankruptcy trustee over that breach. The states were granted $150 million in allowed claims. What they will actually collect is $18 million, because that's what remains in the estate after everyone ahead of them in line got paid. Pennsylvania and New Jersey, two of the bigger states in the coalition, walk away with about $492,000 and just under $410,000. Spread $18 million across the 6.9 million people whose data walked out the door, and you get about $2.60 apiece. Add the $46.75 million the same court approved for the consumer class action a week earlier, and you get to about $9.40. I ran that math twice, hoping it would improve. It didn't.

Don't mistake the $18 million for a fine. A fine is a punishment, calibrated to deter. This was a claim in a bankruptcy, calibrated to whatever was left in the drawer. What your genome is worth depends entirely on which room you ask in. In the marketing room, it was worth a $99 kit and a lifetime of insight about your Neanderthal percentage. In the auction room in June 2025, Regeneron thought the database was worth $256 million for drug discovery, and Anne Wojcicki's nonprofit TTAM won it at $305 million. In the courtroom, a year later: about $9.40, with the same molecules and the same double helix. The price is a property of the venue, and the only room that sent any money back toward the people in the database was the cheapest one.

I think there's a fundamental unpleasantness that turns over all this. A privacy policy is a promise, and in bankruptcy a broken promise becomes an unsecured claim, the cheapest seat in the house, with the secured lenders eating first. Then the professionals administer the estate, billing it by the hour, and FINALLY, from whatever is left, the unsecured claims. Which is where "we will never sell your data" lives, right next to the unpaid landscaping invoice. Twelve cents on the dollar, which is what $18 million against $150 million works out to, is honestly a respectable recovery for that tier, and that is a hell of a sentence to write about your DNA. The system worked EXACTLY as designed, and the design is the problem. You cannot repossess a promise.

(And before anyone emails me: yes, the court attached privacy commitments to the sale, and yes, the buyer promised to keep the data safe. Maybe it keeps every one of those promises! But they are still promises, and we just went over what a promise fetches in that courtroom.)

With most data, this is bad. With DNA, it's a different category of bad, because a genome is the one credential you cannot rotate. Your password leaks; you change it. Your credit card leaks; the bank mails you a new one. Your genome leaks, and nobody mails you new chromosomes. It stays leaked for the rest of your life, and for a meaningful fraction of your siblings' and children's lives too, since they share the sequence and never clicked "I agree" on anything. The 6.9 million people in the breach consented, in the loose sense that word has acquired. Their relatives were simply along for the ride. Data outlives the company that collected it, and this particular data outlives the people it describes.

Genomes are the newest entry in a bankruptcy playbook whose receipts go back a quarter century. In 2000, a Disney-backed toy retailer called Toysmart went under with a privacy policy that promised customer data would never be shared with third parties. The bankruptcy listed the customer database, including data on children, as an asset for creditors, and the FTC sued to stop the sale. The settlement restricted the sale to a buyer in the same line of business, and in the end Disney paid to have the list destroyed. Fifteen years later, RadioShack (yes, that RadioShack) went into bankruptcy carrying more than 65 million customer names and 13 million email addresses on its asset schedule, auctioned them, and got 38 attorneys general objecting before a deal destroyed most of the data. Toy shoppers, battery buyers, genomes. The asset class keeps upgrading, and the playbook hasn't changed at all. I'll let you guess what ends up on the folding table next. (Hint: ask your chatbot what it remembers about you.)

I keep coming back to the same conclusion, and I promise it's not just because of where I work. (Okay, it's a little because of where I work). Paper guarantees have a documented lifespan, and it's shorter than a mortgage. The settlement's forward-looking fix, for the record, is a risk analysis, an advisory board on data security, and continued deletion rights at the buyer, which has since re-registered as 23andMe Research Institute. More paper, under the old name. The version of "we will never sell your data" that actually survives a bankruptcy is where the engineering and data teams take on the work long before bankruptcy is ever considered. Data that gets processed where it's generated, and never piles up in one central database, can't be inventoried by a trustee, can't be auctioned to the highest bidder, and can't be marked down to less than ten bucks in a courtroom you will never see. We usually sell data locality on latency and egress fees, and those are real. The 23andMe estate just made the other argument for us: blast radius. A trustee cannot sell what the debtor never possessed.

An estate sale can only sell what's in the house. Maybe stop filling the house.

Want to process data where it's generated, so there's never a pile for a trustee to auction? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Estate Sale.

The Shortest Stave

David Aronchick — Fri, 02 Oct 2026 18:01:34 +0000

A wooden barrel built from unequal staves (I love this term, by the way; I've never heard it!) holds water only as high as its shortest stave. Chemists call this the "limiting reagent," but I'm particularly fond of talking about these kinds of constraints in the living world, where things grow ... organically. Long before anyone had heard of a GPU, Carl Sprengel sketched the idea in 1828, and Justus von Liebig made it famous a couple of decades later, focusing specifically on how a plant's growth is set by whichever nutrient is scarcest. Pour on all the nitrogen you want, but if the soil is short on phosphorus, the plant stops growing at the phosphorus line, and the nitrogen sits there, unused, a resource with nothing left to do. So, back to our barrels: you can spend a fortune making the tall staves taller, but the barrel doesn't care. Every dollar spent elsewhere runs over that one short piece of wood and out onto the ground.

During 2026, the price of a stick of server RAM stopped being "Oh, I guess we'll just have to add a little bit to the price of this server, but it's diminimus compared to this Rolls Royce GPU we have in here." Conventional DRAM contract prices rose 93 to 98 percent in the first quarter alone, and TrendForce expected another 58 to 63 percent in the second, roughly tripling in six months. The reason? High-bandwidth memory will take about 22 percent of the world's DRAM wafer starts by the end of 2026, up from 18 percent a year earlier, and turn them into only about 9 percent of the bits, because a bit of HBM eats three to four times the wafer of an equivalent bit of DDR5. SK Hynix, Samsung, and Micron, the only three companies on Earth that make HBM at volume, reported their 2026 HBM capacity sold out under existing contracts, and Digitimes reported in August that most of 2027 is already allocated too. Try to buy a stick of RAM for a gaming PC right now, and you're competing with Nvidia's supply chain. Nvidia is winning.

Everybody saw this coming from the wrong direction. For two years the story was GPUs: how many Nvidia could ship, how long the allocation list revolved around, and whether the shortage had somehow become a permanent feature of the industry rather than a temporary one. Then the story became power. Gigawatts, interconnection queues, and data centers in Virginia are waiting up to seven years for a grid connection. Nobody was watching the beige stick of memory sitting next to the accelerator on the board, because memory has been the least glamorous component in a computer since the 1960s, and unglamorous things don't get congressional hearings. Now it's the thing capping how much of this you can actually build, and a fab that makes memory cannot be told to make more of it by Thursday. Wafer capacity is a multi-year build; wanting more sooner doesn't make it sooner.

The AI industry has been building staves. It built an enormous GPU stack, then an enormous power stack in roughly that order, because those constraints tailed first and made the loudest noise going down. Memory didn't make a loud noise, and so it quietly became the shortest piece of wood in the barrel. The irony is that it's not even HBM's fault! It's just that HBM and consumer memory come out of the same fabs, competing for the same wafer starts, so every additional stack of HBM that Nvidia's next accelerator needs is DRAM capacity that doesn't go into a laptop, a phone, or, this being the actual news story of the last few months, a stick of RAM for the machine on your desk. The industry solved for the two constraints everyone could see coming, and the barrel filled to a different line anyway. One nobody had thought to check.

Even worse, Samsung, SK Hynix, and Micron can't ship a patch. Building new wafer capacity takes years, and the contracts locking up existing capacity run multiple years past that. Which means the memory constraint, unlike the GPU shortage and unlike the power shortage, won't get quietly fixed by a new chip generation or another grid order. It's poured into the concrete of fabs that haven't broken ground yet. The people planning AI infrastructure right now are, whether they've noticed or not, planning against a stave whose length was mostly fixed a few years ago and cannot be argued with, subsidized, or ordered into growing faster by anyone in Washington.

The GPU story and the power story were both real. They ARE both real. Solving them was and is necessary work. Solving the constraint just moves the waterline to the next-shortest stave, and that one is rarely the one the headlines prepared you for. Liebig figured that out, staring at dirt in the 1840s. The AI industry is finding it out staring at a spec sheet in 2026 and paying triple for the privilege.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at The Shortest Stave.

The Elevator Receipt

David Aronchick — Tue, 29 Sep 2026 18:01:35 +0000

On July 1, Bloomberg reported that Meta is building a cloud business to sell access to its spare AI compute, an initiative reportedly called Meta Compute. This is the same company that, as of June 30, had $279 billion in data center leases signed but not yet started (and signed another $68 billion in July), with an Ohio campus coming online this year and a Louisiana campus behind it, which Zuckerberg pitched as covering a significant part of the footprint of Manhattan. Zuckerberg had said in May that a cloud business was "definitely on the table." Meta was not even first.

In early May, the entire capacity of Colossus 1, the data center xAI built before it was folded into SpaceX, went to Anthropic in a buyout deal, followed by similar leases with Google and Reflection AI. In August, SpaceX reported that second-quarter revenue had nearly doubled, from $4 billion to $7.8 billion year over year, with compute deals contributing close to $2 billion of the growth, plus another $6.7 billion in cloud revenue under contract for a six-month stretch that starts ramping in October. The exchanges moved in the same window. On May 12, CME Group and Silicon Data announced the first compute futures market, built on daily GPU rental-rate indices, and days later ICE and Ornn announced cash-settled GPU compute futures on an index tracking live spot prices for H100s, H200s, and B200s. SF Compute already runs a live order book where GPU-hours trade like any commodity, and the spot price of a Blackwell GPU-hour rose 48 percent between mid-February and mid-April, from $2.75 to $4.08.

And on that ticker, CME has already set the date its first two contracts, H100 and B200 rental index futures, would start trading, pending regulatory review: October 5, 2026.

A bit of a detour to the history of commodities. Before the 1850s, American grain traveled in sacks. Each sack belonged to a specific farmer, was moved under his name, and was sold on inspection, so a farmer with better wheat than his neighbor got paid the difference. Joseph Dart opened the first steam-powered grain elevator in Buffalo in 1843, a machine that ran grain up a bucket conveyor into storage bins and unloaded lake boats several times faster than the crews who had needed days to empty one by hand. The Chicago Board of Trade began grading wheat in 1856, sorting everything that came through into categories: No. 1 spring, No. 2 spring, and so on down the list. Once graded, a farmer's wheat went up the elevator leg and blended into a bin with every other load that met the same grade, and what the farmer carried away was a paper receipt for so many bushels of No. 2. Receipts could be sold without moving the grain. Then they could be sold before the grain existed, and in 1865 the Board formalized the practice into futures contracts.

Within a generation, Chicago was clearing trades on wheat that had not yet been harvested, and the farmers were organizing a national political movement, the Grangers, aimed in large part at the elevator operators and the railroads that fed them. In 1877, the Supreme Court upheld state regulation of elevator rates in Munn v. Illinois by dusting off a phrase from a seventeenth-century English judge, "business affected with a public interest," to address the fact that a handful of warehousemen had become the chokepoint for the national harvest. William Cronon's Nature's Metropolis tells the full story, and it remains the best systems book I know, even though it's technically about corn.

Two things happened when wheat became fungible. Liquidity exploded, which was good for nearly everyone. AND power moved away from the people who grew the wheat and toward the people who ran the elevators, defined the grades, and cleared the trades. The farmer gained a global market and lost every ounce of pricing power that came from his wheat being his.

I think this is happening again. The exchanges are building the grading system, and Meta just volunteered to pour its harvest into the bin. Bloomberg says Meta may sell "raw" capacity the way CoreWeave does, which is an admission, stated in infrastructure rather than words, that an H100-hour in Meta's Ohio campus is interchangeable with an H100-hour anywhere else. And remember the story that justified the capex in the first place: our compute trains our models; our models power our products; the flywheel compounds; none of it is for sale. Meta AI and Llama still do not appear as a revenue line anywhere, and the company has now conceded the first link in the flywheel by renting it out. (Zuckerberg's version, on the July earnings call, is that selling intelligence carries "a significantly higher margin" than selling compute, and that it "would be foolish to basically just sell all of the compute." He also said buyers are already offering "a meaningful premium" over what Meta paid for it.) The megawatt turned out to be the crop. The models were supposed to be milled flour, and there does not appear to be enough demand for it.

I don't think selling the surplus is a mistake, to be clear. Pretending your commodity is a moat costs money and adds extra steps. (And before anyone emails me: no, AWS was not built from Amazon's spare holiday capacity. That story is a myth Amazon's own engineers have spent fifteen years trying to kill. AWS was a deliberate business from day one, which is exactly the point. Companies that win infrastructure markets decide to be infrastructure companies; they don't back into it.)

But the history is blunt about who captures the profits in a commodity market, and it is not the growers. It is whoever defines the grade, runs the elevator, and clears the trade. In compute terms: the index publishers, the exchanges, the interconnects, and above all, whoever controls where a workload can physically run. A futures market only helps you if you can take delivery, and taking delivery of compute means your workload can actually move to where the cheap capacity is. If your pipeline is welded to one region of one provider, the spot market is a spectator sport, and you get to watch the price of the thing you are overpaying for fall in real time. Fungibility is a property of your architecture, and most architectures I see were built on the assumption that compute would never be worth arbitraging.

Nobody in Chicago planned to become the chokepoint for the American harvest either. They just printed the receipts.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do?

Originally published at The Elevator Receipt.

Coase's Coffee

David Aronchick — Sat, 26 Sep 2026 18:01:09 +0000

An inventory system goes wrong before the coffee rush. Who gets the call? I made this argument about agents (you can't sue an agent), and it applies with full force here: a SaaS contract is a counterparty with skin in the game. When the inventory system is wrong today, Starbucks calls Microsoft, and an army of people whose careers depend on that phone call gets paged. When the in-house replacement is wrong at 4:30 in the morning before the rush, Starbucks calls Starbucks. Maintenance is where AI-built software goes to get quietly expensive. And AI in the stockroom has already bitten this exact company once: in May, Starbucks retired Automated Counting, a vendor-built AI system for counting milk and syrups, after it kept confusing products that looked alike, and sent stores back to counting by hand. Or ask Ford, which spent June rehiring the gray-beard engineers it thought AI had replaced. Starbucks' software bill does not disappear. Some fraction of it converts into engineers, pager rotations, and the slow institutional discovery that "the model wrote it" is not an acceptable sentence in a Sev-1 postmortem.

On July 9, 2026, Bloomberg reported that Starbucks is using AI to build in-house replacements for software it currently rents, including a Microsoft system that tracks inventory and an IBM tool that manages equipment maintenance. Starbucks spends about $400 million a year on software, according to CTO Anand Varadarajan, and the enterprise technology team expected roughly $30 million in budget savings this fiscal year, about $10 million of it from software. The first replacements could roll out by the end of 2027, pending testing. A coffee company looked at Microsoft and IBM and said: we'll take it from here.

If you want to understand why this is happening, and why it is going to happen a few hundred more times over the next three years, don't read an AI newsletter. Read a paper from 1937. Ronald Coase was 26 years old when he published The Nature of the Firm, which asks a question so simple that economists were mildly embarrassed nobody had answered it: if markets are so efficient, why do companies exist at all? Why is anything done inside a firm instead of contracted out on the open market? His answer, which eventually earned a Nobel, was transaction costs. You pull an activity inside the firm when coordinating it internally is cheaper than buying it outside, and you push it out when the reverse becomes true. The boundary of the firm isn't strategy, and it isn't culture. It's a price. When relative prices change, the boundary moves.

SaaS is a forty-year bet on one side of that equation. Building software internally was brutally expensive (hire the engineers, keep the engineers, maintain the thing forever), so vendors amortized one build across ten thousand customers and rented it back to you. This was correct. It was so correct that we stopped noticing it was a bet on a particular cost structure rather than a law of nature. Then AI coding tools cut the internal cost of producing working software by some large and still-unmeasured factor, and the equation started running backward. Gartner now expects agentic AI to put $234 billion of SaaS spending in play. And Starbucks is exactly the company you'd expect at the front of the line: its problems are specific (predicting inventory across roughly 40,000 stores, keeping espresso machines alive), the data feeding those problems comes off Starbucks' own machines and registers, and the vendor products it's replacing are general-purpose tools priced like moats.

We have run this loop before, in the other direction. In 1900, factories generated their own electricity, because that was the only way to get reliable power at industrial scale. Then central utilities got good, and by 1930 buying off the grid had crushed self-generation. Nicholas Carr wrote The Big Switch in 2008 arguing that computing would follow electricity into the utility model, and for fifteen years he was dead right: the cloud became the utility and SaaS became the appliance plugged into it. The lesson from both cycles is not that make beats buy, or that buy beats make. The lesson is that the answer is cyclical, it follows the cost structure of the era, and anyone who tells you the current boundary is permanent is selling you the current equilibrium.

The honest version of the math has three terms, and AI only changed one of them. Insourcing wins when your requirements are genuinely specific rather than generic. It wins when the data feeding the system is yours (Starbucks' machine telemetry is exactly that, and it never needed to leave the building in the first place; shipping your operational exhaust to a vendor so they can rent you insights about your own espresso machines was always a little absurd). And it wins when you can carry the maintenance burden for the life of the system, not the life of the demo. Starbucks, with a real engineering organization and a $2 billion cost program giving the project air cover, may clear all three. The 200-person company currently asking a coding agent to clone Salesforce will clear none of them, and will rediscover each term of Coase's equation the hard way, in production.

Coase's insight survived the corporation, the conglomerate, the outsourcing wave, offshoring, and the cloud. It will survive this too. The boundary of the firm is a price. AI just changed the price, and the map is getting redrawn, one $400 million line item at a time.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.

Originally published at Coase's Coffee.

LLMs Are Too Big. My Log Router Doesn't Need to Sing.

David Aronchick — Tue, 22 Sep 2026 18:01:30 +0000

I think our language models are too big.

Which is a little bit spicy to say, because I don't actually mean the parameter count. I mean that we've gotten comfortable asking an incredibly general system to do a very specific job, and then doing a lot of work to keep it focused on that job.

Working on log routing with Expanso and Jev has made this feel a little ridiculous to me. The question we need answered is whether a particular event deserves someone's attention, especially when we already know the possible destinations and what should happen next. And yet the default approach in a lot of AI software is to bring in a model that can help with just about anything, explain the situation, and ask it to please stay inside the lines.

The capabilities you need to work on the Navier–Stokes equations are not necessarily the same ones you need to sing "Bohemian Rhapsody" in Klingon. Neither requires you to get all the angles right on Michelangelo's David. Having one system that can attempt all three is amazing. I use these tools, and I want them to keep getting better.

I don't think every piece of software needs access to that entire repertoire every time it makes a decision.

The job is smaller than the model

There is a lot of infrastructure work that sits in an awkward place. You can describe what you want fairly easily, but writing all the rules turns into a project of its own.

Take a log message that says config reload requested by unknown actor. It might have an INFO label, but that doesn't make it routine. You want something that can read the message and recognize why it could matter, without having to anticipate every way someone might phrase it.

Now suppose the same message keeps arriving. Once could deserve a look. Repeatedly, in a short window, could deserve something more urgent. The words haven't changed; the circumstances have.

That's the kind of judgment I want help with. I don't need a paragraph about the philosophy of incident response; I need a result the next piece of software can use, and I need to know what to do when the model isn't sure.

Jev, from TypeSafe, is interesting to me because it starts much closer to that requirement. You give it context and defined questions, and it returns typed judgments and probabilities: a choice among options, a score, or a yes/no probability. It doesn't provide an explanation and leaves you to figure out which part was the answer.

Of course, general LLMs can produce structured outputs too; we use that capability all the time. What interests me here is having a model built for these bounded decisions from the start, where there is less for the application to ask of it, and less for it to interpret afterward.

What we actually built

We put together an Expanso and Jev log-triage example that makes the division fairly concrete.

I've written up the implementation in our Expanso post on log triage with Jev. You can also watch the full walkthrough on Expanso's YouTube channel to see the routing and simulated outage in action.

Expanso handles the incoming records and the routing. Known routine events can go straight to archive using explicit checks, without asking a model anything, but for events that require judgment, the pipeline provides context, including how often a matching event has occurred within a 10-minute window.

Jev answers questions about whether the event is actionable, how severe it is, which team should own it, and whether the recurrence is concerning. The pipeline then applies its thresholds and sends the result toward page, notify, review, or archive.

I like that you can point to each part and say who is responsible for it. The model judges the event, the routing policy remains something we wrote and can inspect, and if a call can't be answered, Expanso holds it for retry and eventually sends it for review, while routine traffic continues.

There is a small detail in the demo that I think says a lot about this work. To count repeated events, you can normalize away numbers so that similar messages group together. But you absolutely cannot assume that the resulting group is safe to ignore. A successful health check and a failing one can look very similar once you've removed the status code and response time.

That is our responsibility as the people building the pipeline. A more capable model doesn't excuse us from deciding which information to preserve or which shortcuts are safe.

Fast enough to use everywhere

The part I find exciting is how often you could use this kind of judgment if the cost and latency were low enough.

You could ask about individual events as they move through a system, rather than collecting them in a pile, sending them somewhere else, and waiting for someone to interpret a summary. That changes which problems are worth tackling. A small decision that is too expensive to make a million times generally doesn't get made a million times.

TypeSafe currently lists Jev at $0.042 per million input tokens, with no charge for output tokens. Its own selected performance examples show subsecond responses. Those are vendor results, not a latency or throughput benchmark from our log-routing demo. Before putting this on a critical path, I'd still want to measure tail latency, sustained load, and the mistakes it makes on the actual data.

But at that price, with machine-usable answers returned quickly and cheaply enough that you can build them into ordinary software, this could be much more useful than another conversational interface for many of these jobs.

Typed output doesn't make the judgment correct; a wrong answer in perfectly valid JSON is still wrong. You need examples from your own environment, defensible thresholds, and a place for uncertain results to go. I would much rather build those controls around a limited question than ask a model to figure out the question, the policy, and the action all at once.

I expect more models like this

My bet is that we'll see more smaller, more targeted models alongside the big general ones. Jev is an early example of the direction I mean, although I am not making a claim about its undisclosed parameter count. TypeSafe also says customers use the same model weights; this isn't a separate fine-tune for each company.

The specialization is in what we ask the model to produce. There is plenty of room to get very good at turning messy text into a limited set of judgments that other machines can use. A log router, a support queue, and a data-quality check all have reasons to want that, even if none of them needs a chatbot.

Obviously, I have a stake in this through Expanso. We spend a lot of time thinking about what should happen to data as it moves, and these examples are part of that work. But the thing I keep coming back to is how much of the system we can already specify ourselves. We know where records should go. We know the policy. We mostly need help interpreting the messy bit in the middle.

I'd like to see us get more comfortable asking AI to do that smaller job well. There will be plenty of work left for the model that can discuss fluid dynamics and Klingon. I'm happy to keep using it for that. For the log router, I'd rather have the answer and get on with it.

Want to learn how intelligent data pipelines can reduce your AI costs? Check out Expanso. Or don't. Who am I to tell you what to do.*

Originally published at LLMs Are Too Big. My Log Router Doesn't Need to Sing..