DEV Community: Ryan Baker

Trusted Deployers

Ryan Baker — Tue, 29 Sep 2026 11:45:39 +0000

In May, I wrote about three challenges to preventing AI misuse. The list was not exhaustive. This update adds a remedy that minimizes the impacts to users, while addressing the core challenge of keeping dangerous models out of reach of those who would misuse them.

For a broader and deeper look at Cybersecurity and AI, I’ll suggest Why It Hasn’t Happened Yet.

The advancement of AI models has brought the capability of misuse. The potential for one type of misuse, offensive cyber activity, was demonstrated in dramatic fashion by the OpenAI HuggingFace incident. A lot should be done as I argue in Enough Reason to Act and Every Reward Bends. Alongside those efforts, three challenges need a response. It’s important that these are understood, as some solutions that initially seem reasonable, fail when exposed to these challenges.

Jurisdictions

The first challenge is jurisdictions that are beyond the reach of our law. The world has rogue states, lawless states, and aggressor states. These either turn a blind-eye toward harmful activity, lack the capability to enforce laws, or actively create targeted harm themselves. Existing laws cannot reliably reach actors that hide in these jurisdictions. There is a justified effort to close those gaps. There is slow progress. Sometimes gaps reopen. Because it’s a long running effort, we shouldn’t expect a near-term resolution, and treat it as a reality we must mitigate.

If our law cannot reach these places, we can try to prevent our tools from reaching them. Two further challenges, open-weight models and privacy hinder that effort.

Open-Weight Models

The key component to operating an AI model are the “weights”. These are the product of the expensive training process. The depth and quality of these weights determine how capable a model is. Currently many models are released as “open-weight”. In current practice, this means those weights are packaged as a file and posted publicly where anyone can download a copy.

Once a user has a copy of the weights, the ability to restrict or monitor them is very limited. They do require some computing resources to operate, and for the highest capability models this hardware is expensive. Operating at scale requires a lot of hardware and there are restrictions on where such hardware is shipped. But as substantial as that is at scale, the scale necessary to use models for harm is not that substantial. It’s reasonably within the reach of much smaller actors with very bad goals.

What is outside their reach, is the amount of compute required to create the weights. The training process requires levels of computation that are only available within two domains.

The first domain could be thought of as the US aligned domain. Top-tier compute is in the US, but production and second-tier compute extends to the EU, South Korea, Japan, and Taiwan. This domain is within the scope of practical US jurisdictional control. It’s a bit complex, and involves economic dependency, diplomatic alignment, but effectively the US has been able to exercise control.

China is the second domain. The US occupies the top tier, but China’s second tier is still sufficient to train dangerously capable models, especially if we extend that window out into the future.

Outside the US and Chinese domains, nowhere has the capability of training dangerously capable models. This collective grouping thus has the opportunity to control who has access to dangerously capable model-weights. This is well understood. I don’t ever see disagreement with this conceptually.

Where disagreement does arise is about open-weight models. The first reason for disagreement comes from not recognizing the danger. I outline the technical background of the danger in Security Can’t Wait and Deployments Can’t Wait.

The second reason for disagreement is misunderstanding the remedy. The typical assumption is that a remedy to the risks of open-weights would reduce the number of models available. But the appropriate remedy, trusted deployers, would not have that effect. Most users of open-weight models already use them in a way that would not be impacted. Others can easily adjust.

Trusted Deployers

Trusted deployers ensure guardrails are in place that reject requests that are harmful. Trusted deployers would host models from many providers. That would include those charging for access (closed models), and those released under an open license (open models). In this model, the open models would be distributed either directly to the trusted deployers, or via a clearinghouse (aka HuggingFace).

To receive a copy, you’d first have to be certified as a trusted deployer. This would start by showing:

You have safeguards to keep the weights secure
You employ monitoring and guardrails to detect attempts at malicious use and refuse them
You do not allow fine-tuning that removes or weakens misuse refusal training
You use monitoring to detect patterns of usage that attempt to “jailbreak”
You use monitoring to detect patterns of attempts to perform malicious use
You utilize these insights to terminate accounts associated with that disallowed behavior

Only models below a dangerous capability level would be publicly downloadable. But all open models would be available at every trusted deployer with sufficient guardrails. We might even suggest that all trusted deployers, or alternatively the largest trusted deployers, have an obligation to host all models that pass security reviews. This would treat these largest deployers as a utility with an obligation to provide access without discrimination. In this model, you would expect the number of models available to be as diverse as today.

Trusted deployers’ guardrails would use automated inspection of incoming requests, rejecting those deemed harmful. Deployers would be responsible for operating those guardrails. To support a variety of models, they would have base guardrails. Deployers might build their own base guardrails. Some would adopt an open-source guardrail project. Such projects already exist, though their maturity would need to continue advancing to keep up with adversarial demands. Likely the formalization of this model would help spur that.

An objection is that those guardrails are not likely to be foolproof. They don’t need to be foolproof though, they need to be effective. Guardrails raise the cost for attackers. They limit scale. They enable countermeasures.

The shift to a trusted deployer model does not radically change most users’ access to models currently released as open-weights. Most users already access models through cloud providers. Amazon Bedrock, Microsoft Foundry and Google Model Garden, host third-party models, including leading open-weights models. Those providers will become trusted deployers.

In the trusted deployer model, model weight files are shared only with the trusted deployers. Users do not have access to the model weight files. But most users never access them today. Most users do not have the compute capacity to use those models, so they never touched those model files as-is.

The largest collections of compute are at cloud providers. Those would all be accessible by the trusted deployer model. Smaller groupings of compute – private data centers, colocation facilities, sovereign national infrastructure – would need to be adapted to a trusted deployment model. Most would do so.

Open-weight models would continue to exist. The trusted deployment model would only be a requirement for levels above dangerous capability levels. Most usage of highly capable models already goes through providers. The compute requirements leave few other options. The exceptions are powerful organizations that can become their own trusted deployers.

For this compromise, we’d earn the capability to ensure dangerous capabilities are never deployed to servers where we lack control. Dangerous models would not have their model-weight files posted publicly. This restriction would earn security, with minimal change to actual operations.

Privacy

A trusted deployer would be expected to do more than deploy passive countermeasures. They would also be expected to use identity as a tool to actively restrict the attempt at harmful use. If attackers are allowed to endlessly probe and retry defenses, their chance of success goes up. Using conventional identity systems can reduce access. When abuse patterns are detected, those accounts are deactivated.

The best model for this though has a strong identity. Weak identity systems allow fabrication of new identities. The default state of anonymity on the Internet has costs by creating weak identity. Privacy advocates attempt to maintain this state. I, like some others, believe the costs of this anonymity as a policy are too high. This isn’t specific to AI, but it does relate.

Like other protections, we shouldn’t expect foolproof systems. A stronger identity system would add costs to an attacker trying to manufacture identities. They could buy accounts from a real person. But the cost would be such that using these accounts to probe defenses would no longer be economical. They would become more cautious and less aggressive about how they used those accounts. Attempts to access dangerous capabilities would become rarer, allowing for more active response to the remaining attempts.

To enable strong identity, we would need to change some of our expectations about creating accounts with trusted deployers. Trusted deployers should be both trusted and expected to use automated means for scanning request intent. They should be both trusted and expected to know who their users are as individuals, or as organizations. Part of the certification process would ensure that each individual operator within the trusted deployer does not access this information. The exception being the review of requests already flagged as malicious.

We should be pragmatic, but we’ve been idealistic. When countermeasures can’t be implemented due to obscuring the lowest layers of a technical stack, we fail to achieve privacy and prevent harm. If service providers always knew who was using their service, they’d be able to deny access to anyone detected acting maliciously in the past. But the internet offers too much anonymity. Providers can shut down an account, but without accounts tied to a strong identity, a new one can be created. The current standard is too lax about this. We could make it more costly for attackers to maintain access.

China

As mentioned, China is the second domain where model training is likely. For the near future, and potentially longer, this would be second-tier, which provides some security. But that gap is not large enough to take for granted. Misuse will be far more preventable with China’s cooperation and adoption of a trusted deployment model, in contrast to the currently prevalent open-weight model.

While any agreement with China is not without challenges, this specific proposal has advantages to China. They also face risks of misuse and should be motivated to prevent it. It’s far less challenging than the global coordination needed for pacing AI development globally, a critical action to address loss-of-control risks.

Conclusion

Trusted deployers resolve many challenges to preventing AI misuse.

Jurisdictions, open-weight models and privacy complicate preventing AI misuse. Jurisdictions won’t change, model choice and privacy we’ll continue to value. These three challenges compound each other. Open-weight models place powerful tools in jurisdictions beyond legal reach, while anonymity makes it difficult to detect or deny access to bad actors even where laws do apply. Treating any of these in isolation understates the problem.

Yet safety must also be a priority. Trusted deployers enable broad and diverse model access while partially resolving the challenges. They are also functionally similar to prevailing usage patterns. This makes it an excellent form of compromise.

This does not make it an easy compromise though. Developing a trusted deployer model takes time, even though we already have examples of it in practice. It has to expand to become the standard across all jurisdictions with enough compute capacity to train models.

Meaningful identity verification will feel like a concession on privacy, because it is one. Trusted deployments won’t affect most users, but they will frustrate some. Coordinating across jurisdictions will be slow and incomplete. None of these are reasons to avoid acting, but they are reasons to be honest about what any given measure can and cannot achieve.

What’s unacceptable is the current default. Hard choices cannot be deferred. Privacy and safety will be at risk without appropriate measures. Treating anonymity as an unqualified good has costs. We should discuss, agree to, formalize, and adopt measures that strike the right balance. We should start that immediately, as each step takes time.

Do you appreciate this article? The best way to help the publication is to like and share the article, as we’re still growing our audience.

Original Posted on Substack as Trusted Deployers

Enough Reason to Act

Ryan Baker — Tue, 22 Sep 2026 11:40:56 +0000

The recent AI news is a renewed push for a slowdown. The spark was a resignation (Jacob Coxon). Industry support is broader and deeper. The penetration into everyday news is deeper. The reaction from politicians is louder.

My interest is the reason for acting. I make the case for that in The Case for Action. But first, I address eight mistakes that are used as arguments against acting. Four of these are, Mistakes of Detail, false assumptions or mistaken facts. Four are Mistakes of Form, modes of reasoning that fail.

The core of the news cycle focuses on loss of control, aka rogue AI. One of the turning points was Dario Amodei, CEO of Anthropic, publishing, We Must Pace the Frontier. In that, Amodei mentions several risks:

the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption.

So far, I’ve covered cybersecurity (Every Reward Bends, Nobody Was Watching, Security Can’t Wait). Partly that’s because my expertise is stronger there. Partly because it requires broad action across information technology, and I can be more persuasive there. The many small incremental activities lend themselves more to the type of influence I have today.

This is my first serious effort at discussing loss of control. The most pressing aspect is that the important steps take time to arrange. They are difficult, and become more difficult if rushed. The direct actors are fewer and more centralized, but political support is necessary. That requires broad understanding that takes time to reach.

The timeline is less definitive. The next 12-18 months will be bumpy in cybersecurity. Loss of control, if it does emerge, might be farther away. But there is a chance it’s equally close. Warning signals can compel action, but they can also compel overreaction. To avoid that, we want to see broad awareness and creation of systems that facilitate coordination that mature. If we have time, they’ll mature smoothly, if we don’t we’ll be more prepared than otherwise.

A Title More of Us Could Agree With

The focal point of loss of control as a narrative is the book If Anyone Builds It, Everyone Dies, a 2025 book from Eliezer Yudkowsky & Nate Soares. It’s an eye-catching title. If I were publishing the same book, I might accept the same title, because, you know, titles. But unlike Yudkowsky and Soares, I do not accept the literal meaning.

I can iterate this one clause at a time toward something with both my full agreement and broader acceptance. My first iteration adds uncertainty:

If anyone builds it, then (there is a chance) everyone dies

I consider five possibilities:

(a) we successfully control superintelligence by intentional alignment

(b) we accidentally create a benevolent AI overlord

(c) we accidentally create a non-benevolent AI overlord

(d) we accidentally create an AI that destroys human civilization

(e) we do (d), (c) or (b), but before losing all physical control manage to enact some hail-mary or emergency plan that actually works at destroying (b), (c) or (d).

It’s tempting to try and assign probabilities to each of these. They would have large error bars if I did. If it were the only iteration, I’d do that. But the following iterations decrease those probabilities’ relevance.

If anyone builds it, then (there is a chance) everyone dies, (but if we build it right, that chance is less)

We have agency here. Those error bars will stay high, but we can push the probabilities down. You might still say, why take any risk, if we don’t build it, no risk.

If anyone builds it (and let’s be honest, someone is going to want to build it), then (there is a chance) everyone dies, (but if we build it right, that chance is less)

The world is a big place, with a lot of people. Even if you and I believed the strongest version, (d), not everyone would, nor would we be able to convince them all.

Why would anyone still build it? Well, because both the accidental benevolent overlord (b) and especially deliberate alignment (a) appeal. Some reject (b) on a value basis. In a strict interpretation, I wouldn’t. You might even argue a large part of humanity has hoped this is true, but in supernatural form. I would have a problem with those that object to (b) being forced into it though. But it’s pretty easy to imagine a group unconcerned with either of our views, and how can you fully object to (a)?

So, if there’s going to be a lot of people who want to try at (a) or (b), you could try and stop them. But again, the world is big, so you need very wide agreement for that to be practical. And when they do try, you’ll be stuck with whatever that attempt produces.

If anyone builds it (and let’s be honest, someone is going to want to build it), then (there is a chance) everyone dies, (but if we build it right, that chance is less) (so we should be part of building it, so we can do it right)

But if (a) is possible, or you can discover if it is possible, you can try and do that. And if you trust yourself more than some other segment of humanity, you can try and make (a) more likely, or prove it’s impossible, which could convince everyone to stop trying.

Is it hubristic to trust yourself? Yes, or at least it flirts with it. Is it avoidable? Only if you trust someone else to build it better.

If every modest person defers, the hubristic take the wheel. So, the capable but modest must embrace some self-trust and hope they haven’t misjudged themselves.

Seeking power for power’s sake is bad, and it’s easy to lose self-control in that process, but seeking power for an outcome is also a necessity. You can distrust this a little, but you can’t distrust it absolutely.

If anyone builds it (and let’s be honest, someone is going to want to build it), then (there is a chance) everyone dies, (but if we build it right, that chance is less) (so we should be part of building it, so we can do it right) (but we also need to not be under pressure to do it fast, or even our good intentions will fail)

When you look at those from labs calling for pacing the frontier, their views are closer to the iterated title. Soares and Yudkowsky may believe in the simple version, and you may disagree with that. But that wouldn’t give you a reason to disagree with the more qualified version. And people in the labs who agree with the qualified version are suggesting a slowdown as critical to finding a solution.

The control necessary to permanently block everyone is higher than adding extra time by going slower. Going slower gives more time to do it right. It gives more time to discover what’s possible. You may even discover that superintelligence is impossible, or much farther away. I find the first very unlikely, and the second is becoming less probable, but who knows?

We should take that full concept seriously. We still have to decide how much any individual believes this. We still have to decide which ones handle pressure the best. We still have to decide which ones are actually most competent at building something safe. And we still have to get the “someones” who aren’t part of this group to agree, or take away their ability to succeed (quickly).

I think we’re capable of coming to a solution here. But if we rush, out of complacency, or because we’re fighting over something else, we’re far more likely to fail.

Mistakes of Detail

Before I make my case for action, I want to address some common mistakes. These are details where it’s easy to misunderstand the general case for action. I’ve seen these often enough to think there’s a chance you might also make them. I do not know you will, but if by naming them I give you a better chance, it’s valuable.

Presume we all have to die

A common mistake is to say, I don’t see how AIs could kill us all. By the time they are half done the computers will go off and they’ll have to stop. Or something like that. This objection is irrelevant. We don’t want any large group of deaths.

I’m not against asking the question, because you can learn from it, and what you learn can be used in places you didn’t anticipate. If you ask the question, I’d point out (a) once civilization starts collapsing our chance of killing ourselves, always a risk, goes up, (b) you have not, and will not, think through every possibility. It shouldn’t be necessary to explain how a superintelligent system is going to escape from its physical limits after it escapes from our virtual limits.

There will be possibilities none of us can anticipate. Using the worst case to dismiss all the lesser but still bad cases is not an effective form of reasoning.

Presume false competition

Loss of control is one topic. But it’s not the only one. Dario mentioned four. One mistake I’ve seen is thinking there’s a competition between them. Some people try to spin that as a reason to ignore loss of control.

If you ignore probability and timelines, loss of control is the most significant. Cybersecurity is the most time sensitive and requires the broadest set of actors to respond. Bioterrorism requires DNA labs, a very special kind of actor, to respond, and good deployment and training practices by AI labs. Economic disruption hasn’t really emerged, and it’s the one part I disagree with Dario.

The actors who are most pressured to respond to all of these at once are the AI labs themselves. In many cases there are coordination challenges with incompressible durations, no matter how much effort is expended. Global coordination of pacing might accelerate with shared priority, but there’s no unilateral shortcut.

Cybersecurity does fall into a unique slot: unmitigated, its effects land in a year. Rogue AI has a broader timeline. We don’t know when or if capabilities increase to the level that makes it possible. It’s a good bet they can, but not in the next 12 months.

Presume cybersecurity and loss-of-control are the same

A pattern I’ve seen goes like this: Cybersecurity risks could be managed by diligence. I can understand how those risks materialize. I think I can apply that to loss-of-control.

It’s great you’re thinking of cybersecurity. It’s important. Cybersecurity needs lots of small incremental changes, starting immediately. We need a unified effort and commitment to doing that. We need great tools to make it efficient. If we do nothing, we could end up with catastrophes.

But the loss-of-control risk takes place in a different world where things have changed. We don’t understand that world yet. It might be that this different world is harder to enter, and even if we try it could be a long way away. But I see no reason to be confident of that. We’re close enough that we’re learning some things. Cybersecurity applies to this world, but it’s also going to be radically changed.

So no, cybersecurity isn’t enough as a solution. I think it’s incredibly important. I think we can and will respond. If we didn’t we’d have chaos that would make every other bad outcome a bit more likely. If we really focus over the next 12-18 months, I expect we’ll enter a more stable period that’s more secure and better managed.

But it’s not one challenge. Being able to handle the cybersecurity challenges of the next 12-18 months is not the same as being able to handle loss-of-control. Good cybersecurity could avoid a few mistakes here. But it’s not a full response and can’t substitute for one. Partly that’s because handling cybersecurity is going to depend on having access to more capable models than the attackers. If the model is the attacker, and it’s the most capable model, that advantage is gone.

Those guys are weird

I have to thank Matthew Yglesias for highlighting this trend, in A simple plan to save the world from rogue AI. Once mentioned, it’s impossible not to see. There are eccentric people, and eccentric opinions. Being an eccentric person doesn’t make your opinion invalid. A lot of uninformed dismissals take the form of attacks on Effective Altruists, often bringing up Sam Bankman-Fried. Yudkowsky has strong opinions and does eccentric things. The concept of being analytical about altruism, vs. doing what looks good / feels good, doesn’t seem like it should be eccentric, but through history it turns out it is. And yes, if you drop into San Francisco, you’ll find some eccentric lifestyle choices. But for San Francisco, if I know the history, that’s actually more typical than atypical. Either way, none of it is a successful argument about risks.

More importantly, the existence of an eccentric opinion doesn’t make the milder, more defensible version of that wrong. The milder, more defensible version of loss-of-control is still worth paying attention to. 10% chance of end-game is pretty important. Strange as it is, it can require balancing, but dismissing it entirely would be a mistake.

Mistakes of Form

The next set of mistakes are about the form of reasoning. They are more complex as a result. What’s ultimately most concerning about these, is that they are used to cut short a search for understanding. The first four fit a learning experience, but the next four, cut that short, substituting a dismissal. If accepted they lead to a stark choice, of no action, or an overreaction as the only remaining choice.

Some are patterns that reappear enough to be well-known. This is not, however, an exhaustive list. I call out these mistakes specifically because I’ve seen them used to shut down best-effort conversations.

Reasoning from cynicism

Reasoning from cynicism is not a sound mode of reasoning. As a tool to identify concerns, it’s perfectly functional. But it’s not a complete form of reasoning. For any given situation, if you use cynicism as a form of reasoning, you’ll arrive at different conclusions based on which actor you target your cynicism at. And many of those conclusions will be directly contradictory. The truth may not be represented by any of them.

I’ll hold out Matt Stoller as an example, telling us to Stop Panicking About AI. This isn’t a generalized attack on Matt. He serves as an example because he made mistakes, used cynicism as a component of his reasoning, and got a lot of attention.

He knows he’s not an expert on this topic, yet is willing to make fact-free proclamations like “Agentic AI is dangerous, but we can improve the products to limit damage.“ How does he know that? His fix has no clear line back to safety.

I have no problem with Matt adding his view about copyrights. Dismissing the views of the better informed is unnecessary for that. But the post has over 600 likes. Is it his pre-existing audience? Is it creators defending their turf?

Those are the hopeful answers, because the post is not a good discussion of AI. He competently recounts the events that are available elsewhere, but the analysis doesn’t engage with the actual topic. As a response it’s slop, latching onto a live story to re-promote an existing point of view.

One of the ironies is that his prescribed fix is for us to abandon a “nihilistic refusal to govern for the public interest”. I’d be happy to. The irony is that this nihilism is closely related to the cynicism we (the public) have applied to politics. I sit in an uncomfortable middle, thinking mass-cynicism of the concept of government is a mistake, but sharing the same feeling about mass-cynicism of tech companies. You have to be willing to acknowledge the possibility of a good politician or a good company for there to be any room for them to succeed.

Reasoning from markets

Markets are useful tools, but at best they process information, not create it. So when predictions about impacts from AI are made, it’d be a mistake to use a lack of market signals to dismiss them.

I read Marginal Revolution regularly. I’ve had disagreements with Tyler Cowen or Alex Tabarrok, but generally found the starting argument informative. But, recently Tyler has been suggesting AI and technology experts are making predictions unscientifically.

In “Act like it is science” he asks:

1. Your outline of how, say two or three years from now, we might estimate the additional cybersecurity costs from the AI break-ins. I am convinced that number is not zero, but give me your method please. If it helps, here is an estimate of past cybersecurity costs, done by top economists.

2. Your current numerical estimate of what those costs might end up being, of course to be tested against what actually happens over time. Obviously, you can do this for a few different regulatory/safety scenarios. This is one simple way to prove yourself largely correct, albeit with a lag. (NB: you do not have to take this as a substitute for your preferred safety measures. But please try to be specific in your predictions.)

3. A list of what stocks or other assets you have shorted, now. Obviously if your answer to #2 is sufficiently low, you could answer here zero, as I would do. I expect costs, but not so high that we cannot muddle through and have expected positive stock returns.

He’s since been doubling, trebling and quadrupling down, posting cybersecurity stock price charts and saying that with them “we are getting somewhere concrete and scientific rather than just scare stories.”

(1) sounds reasonable. But Tyler is responding to events that stirred both loss of control and cybersecurity concerns, and conflating the two.

Applying that request to the Soares and Yudkowsky version, that everyone dies, is absurd. The cost of everyone dying, crudely, is 8 billion x $14M = $112 quadrillion. That number is useless for any argument about what to do next. Historical data does not provide a probability, and we would be surprised to ever accumulate enough data to retroactively apply.

This problem of having no historical precedent doesn’t go away. Instead of one, all encompassing number, you start having many scenarios, each with an outcome (e.g. 1% die, so only $1 quadrillion). But each is just as useless. The concept is more useful than the number.

The second problem emerges in (2) and (3). We’re only ever going to get one version of “what actually happens over time”. But anyone who takes (1) seriously isn’t going to wait around. They are trying to change what actually happens. And (3) is a trust test, indicating Tyler is not going to take answers to (1) seriously, unless you put your money where your mouth is.

The preparedness paradox disconnects (1) and (3). Tyler can’t be fully unaware of this as in (2) he allows “do this for a few different regulatory/safety scenarios”. But you can’t pull those multiple scenarios forward to (3) which demands “your answer to #2” as a single number, without locking in probabilities that people will/won’t act.

If you follow his posts after, it’s clear that he’s not avoiding that trap, and (1) is just an opening salvo to draw you using insurance rates and stock prices instead of stories about events without precedent. Probably this is unintentional, and just derives from a habit of reasoning from markets in unfamiliar fields.

I do see a trend in Tyler’s posts toward searching for evidence, rather than dismissing evidence. But if he wants to do so, he needs to disclaim the original rather than just quietly walk away from it. This would help avoid others being drawn into the trap. He must also be open to non-market based predictions.

Shorts don’t tell you someone’s commitments to action

Shorts are an unfair test of credibility. Think through your options after you spot a risk:

A pure trade. You short, wait for the disaster, and collect mega money. You don’t need to do anything to divert. At best, you send a cryptic signal. If someone decodes the signal, they can trade on it too without taking action.

A warning without trades. You explain the risk, persuade others, plans are made and action is taken. You send a clear signal, no decoding needed. Action is the intended outcome. The beneficiaries are those who would have been hurt.

These are the simple options, but we can combine them:

A warning monetized. You short, then explain the risk and persuade others. Those that believe the risk won’t be managed act by selling, causing the price to drop. You exit the short, taking profits. Plans are made, action is taken, the risk is fully mitigated, and prices return to normal. The people who would have been hurt still benefit from the warning, but now you do too. The sellers end up a bit worse off, though it’s worth considering who they are. If they held the stock until the disaster that’s realized in the pure trade , they probably would have been hurt in that way.

You’ll notice that in a warning without trades , I didn’t mention sellers. It’s not core to that story, but realistically, if they appear in a warning monetized they’d appear there too. But there’s a reason they might not appear in either:

A warning where trades don’t pay. You short, then explain the risk and persuade others. Others expect plans to be made and action to be taken, and don’t sell, expecting those to result in opportunity loss. Prices don’t change. Plans are made, actions are taken. Your short fails as there was no disaster, and no opportunity to exit the position.

This scenario is bad for you, and once you’ve disclosed, there’s little you can do about it, other than manipulating views on the probability of action. In reality, it’s never quite so dire for short sellers, as there’s always a mix of opinions. But timing the exit is important.

A warning where opportunity is missed. You short, then explain the risk and persuade others. Those that believe the risk won’t be managed act by selling, causing the price to drop. You don’t exit the short. Plans are made, action is taken, the risk is fully mitigated, and prices return to normal. Your short fails as you missed the chance to exit.

If the risk is unavoidable or only partially mitigatable, there is always some residual payoff. But if we’re talking about risks that disrupt society, the silver lining may be inadequate. You may experience just as bad effects from the spillover as any residual payoff. And, well, everyone else suffers, so if you care about that, you really want plans that fully mitigate.

If the risk has low spillover, your best bet in shorting is the pure trade. If it has modest or more spillover, your best bet is a warning monetized, exiting prices reflect less confidence in mitigation than you have. If however, you expect prices will never dip to that point, you should never short in the first place.

It’s a little more complicated, because as discussion about a risk occurs, you learn more about the probability of mitigation. But this in-between period isn’t important to the question of whether you really can reason from markets. The only thing you’re guaranteed to learn is how persuaded people are and how that affects their participation in mitigation. There’s no guarantee you’ll learn anything new about the risk itself. If you do, it will be in the form of words, not prices. It will come from a person who has reasoned through the scenario. That person’s views could end up in prices, but you’d have no way to separate it from the other views that go into prices, except by listening to their reasoning.

You should not, in any direct way, be updating your views about the merits of arguments for risk, based on stock prices. At best, changes in stock prices are a signal of a change in opinion. Whose opinion and why? You don’t know. Better to just go to the source.

Insurance rates are also indirect

In another attempt Tyler Cowen suggests historical market prices and cybersecurity insurance rates are sufficient evidence that there’s not much to worry about. He assumes the market has already priced in the effects, that it’s also simultaneously going to force action.

The first problem is if insurance rates only reflect the views of insurance actuaries. If they aren’t yet convinced of a risk, rates won’t yet have changed. If you want them convinced, you either have to persuade them, or hope they persuade themselves.

The second problem is insurance actuaries also have opinions about the probabilities of mitigation. If you persuade them, you’ll likely persuade others. Rates would come back to normal the more they are convinced that mitigation is likely. Rates only reflect unmitigated risks and only to the degree that it’s likely to remain unmitigated up until an actual event.

The first and second problems are both affected, but in opposite directions, by the capability level of the actuaries. If they are wise all-knowing oracles about risks, they’d be wise all-knowing oracles about mitigation. They’d also have a responsibility to explain their rate setting process, which would disclose the risks. That would, assuming they are mitigatable, lead to mitigating those risks. They’d only fail to lead to mitigation of a mitigable event if their disclosures weren’t trusted. That would be odd if you accepted them as all knowing oracles.

The third problem is assuming the structure of finance isn’t hiding the risk. Finance has lots of fine print, and the cul-de-sacs of limited liability can swallow a lot of risk if strategically placed. There is the counter-argument to this that if there was a weakness, someone would step in and offer a new type of insurance or finance instrument to span it. But that’s in theory. In practice we have evidence of how this can fail. And keep in mind, there is a reason this is my third, not first problem.

Markets accumulate insight, they don’t produce it or disseminate it

If you’re not asking the hard questions, place faith in markets over reason, you’re in a circular type of reasoning that is a lazy response. If no investor is willing to hear the stories, there’s no reason for prices to change. If you successfully persuade investors to ignore the story, because prices haven’t changed, they won’t change until it’s no longer a story.

A perfectly functional market might be described as a representation of collective reasoning. But not only are markets not perfectly functional, collective reasoning has to begin with individual reasoning. Relying on the market could be a lazy way to capture others’ reasoning. But any appropriate update has to start from an individual, and then spread by other independent reasoning, or by person to person making arguments that don’t reference a market. Lazy followers will eventually adopt the same view once non-followers update their views, but the followers’ information will be delayed.

Demanding yes-no answers

I wouldn’t claim I’ve proven anything here. Proving we’d lose control, or proving we’ll fail to respond to cybersecurity isn’t my intent. I don’t know that it’s even possible. These types of topics don’t adapt well to yes/no answers. It’s true in many domains, and very true here. Life is largely in the grey zone, and pretending like it isn’t only gets you so far.

We have a problem, with a public, that at an aggregate level–aggregating both individuals and individual actions–is far too partial toward yes/no explanations. It’s a tendency that has rewired our media, our politicians, even our personal interactions. It’s so pervasive, we take it for granted in most cases. When someone does bring it up, it’s treated as quixotic. Maybe I am tilting at windmills to make that argument.

It’s possible that the approach, or views, of Soares and Yudkowsky are more effective than mine in this world where yes/no thinking is as common as it is. I had to provide a long explanation, and a set of qualifiers. My book title would never have made it on the bestsellers list.

Another problem with my approach is I’ve just insulted the reasoning systems of a lot of people. That’s going to seem elitist if anything does. But I do it because I think they can change. I believe in them, and any resemblance to insulting a person is incidental. I don’t want to critique you, I only want to critique a reasoning system that ends in bad results.

The thing about “inexorable forces” is that many of them aren’t. They are powerful, hard to resist forces. But you can resist. If you want to successfully resist, you better plant your feet more than 2 inches away from the cliff.

For things that are difficult and are going to take time, you have to start earlier, and often work concurrently, taking care of other in the moment issues while making progress on the harder ones. It’d be nice to be able to focus on one issue at a time, but that’s just not how life goes. You may ignore and delay some big difficult things, but if you do, it’s primarily because they just aren’t important enough.

Importance here has a bit less to do with probability than other places. Early effort costs less than late effort. You commit to the big effort at a time when probability is more certain. The cost of trying to make up for missing the early effort, goes up with delays. So if it might be important in the future, you want to invest in things that shorten that future timeline.

We already have missed some opportunities. The most significant one is to have government institutions that understand this stuff. They would be delegated to create a plan, in collaboration, and when confident in that plan, move forward. Right now we don’t have the careful drafting of plans, we have reactions.

Letting forcing functions win

The non-benevolent, superintelligent, super powerful AI risk has two versions. One is the event itself, anchored to the moment control is lost. The other version is worldly pressures forcing us to handle that pivotal moment poorly.

There’s some version of this that will always be about doing things in the moment, while being in a fog where we aren’t entirely sure if a superintelligent AI exists, or how close it is. That version relies on momentary things like, monitoring deployments, securing sandboxes, running evaluations. These are the things that if there is a precipice and we approach its edge, we’d see the edge. Logic says we stop there and admire the view.

There is another version that is about doing things in the moment, knowing that logic could fail, and a crowd, wanting to look over the edge themselves, might push us off. That version is about creating verifiable compute, about discussing the edge with places like China, about having a lot more people who understand some of this. Those three, are actually in order of difficulty, from least to most.

There are two ways you can let forcing functions win. First, you can give up, and not fight. Second, you can ignore that they exist and end up dominated by them. You have to be aware they exist and willing to work against a deck that isn’t in your favor if you want to take the reins and choose a path of your own choosing.

The Case for Action

So far, I’ve only made the case for action obliquely, referring to others’ arguments and explaining how there’s a consistency between working in AI, and wanting it to be paced. On the cybersecurity side, I’ve done that already, in Why It Hasn’t Happened Yet.

On the loss of control side, I haven’t. One of the critiques you see applied to the Soares and Yudkowsky version is specifics about how AI would get out of control, start to harm us, stay out of control, and end up killing all of us. I could provide a story like that, but it would have a sci-fi tinge where you could poke holes at. It’s more useful to think about how lesser forms of those stories are very possible, and that the consequences before regaining control, and the costs to regain control, are quite problematic.

We could go back and forth debating these stories, and likely never have a full resolution. The defenders would have holes where they couldn’t promise defense, and the rogue AI story would have holes where some unexplained jump would be necessary, given current capabilities and what humans have currently wired AI into.

This sequence of stories has less to do with absolute answers than possibilities. Those possibilities then connect with trajectories. One trajectory is how future possibilities are emerging. A second trajectory is our preparations. And the third trajectory is how quickly we can change either of these trajectories.

The OpenAI HuggingFace event revealed important information about these trajectories. It showed that labs were being reckless. If that continues we’ll be less prepared for whatever comes. It showed we had no structure to reduce recklessness. If that continues, we’ll do less preparation.

We also have the issue that the general public remains mostly uninformed about how information technology, especially this technology, works. It’s been a blind spot in our culture that accurate depictions of software developers are missing from popular culture. With that understanding missing, the popular reaction to events will both over and under react, and fail to choose the best nuanced response.

The risks to cybersecurity were evident within the technology domain before the OpenAI HuggingFace event. But another change from that event was to provide a demonstration that did not require technology understanding. From this, the public became aware of capabilities they would otherwise have struggled to understand.

That’s a big hurdle to overcome, and it’s just the first one. Creating a US regulatory system would take time. Many people who don’t understand each other’s positions need to understand them. A lot of details that don’t have shared understanding need to be agreed to. And then that’s followed by an international system, which has an even wider gulf and has to somehow co-exist with a set of standing disagreements at that level.

All of this should advise starting now. That doesn’t demand an immediate overreaction. In some ways, an early start is a way to avoid overreactions later. More importantly, it makes underreaction less likely and less inevitable. With a late start, the choice of actions immediately available may be poor choices all around.

An event might happen where the only reasonable action is shutting things down. That might go as broad as huge numbers of servers, or shutting down parts of the internet. With such a stark choice, inaction might take hold and nothing gets done until things get worse.

How bad would it be if we did nothing?

I think we will take action. I speak about it to make that more likely. I believe others do too. I believe all this will mitigate many risks. So the risks I mention next, those aren’t risks I think will happen. They are risks I think we’ll avoid. If we do the bare minimum we will end up with some ugly costs.

Existing cybersecurity depends on successful layering of imperfect layers. Attackers have to search through every layer and compose attacks to perform their malicious activity. AI enables the search and composition.

The capabilities there are such that if nothing had been done, we would be seeing widespread successful cyberattacks. Fortunately, while more is needed, much has been done. The infrastructure for implementing cyber defense has history and maturity. AI was used to find, fix and deploy many critical vulnerabilities. Additionally, all the leading AI labs took responsible actions to restrict attackers’ attempts to use AI for composition of attacks.

For cybersecurity, you need to worry about how failures compound. For financially motivated attackers, if the success rate of an attack doubled, not only would you see twice as many successful attacks from existing efforts, but since the efficiency increased, you’ll have more effort. Second to that, the tax on revenue and then profits would accelerate as more are sapped by attackers. If you start with 1% of revenue, then double to 2% when success increases, then double again to 4% when attackers are incentivized to invest more. An average profit margin of 8% would be cut in half by those effects.

That kind of effect would force a reorganization of many business operations. That forced reorganization would create costs in any situation, but occurring suddenly, it might be poorly managed and result in systemic failures.

The loss of control risks could be described two ways. We could talk about the cost of effects a rogue AI might take. And then we can talk about the costs of the reactions we’d need to regain control. This assumes we do regain control.

I think people are generally right when they suggest that a rogue AI, at the current capability level, would only do limited damage. The costs to shut it down might be trivial, though there are scenarios where something worm-like emerges that requires wide scale computing outages to respond to. Generally though, concerns about rogue AI don’t focus on current capabilities. They worry about our ability to slow down or stop capabilities progress once we commit to it.

Without anything like global coordination, there are significant loose-ends to controlling capabilities progress. Some additional capabilities might emerge from low-compute evolutions, like better harnesses.

The actual list of bad things a rogue AI could do is quite long, in the same way as the list of bad things a person could do. What’s the minimum cost you’d need to suggest to merit an AI safety program that redirects half of all training capacity toward safety? Two thirds?

Plan A uses this mechanism, detailed in its appendix.

What are we really giving up?

Absolute outcomes like “everyone dies” can justify absolutely any response. The only logical limit to a response that avoids that is a less costly response that also avoids the outcome. But I’ve conceded that the outcomes may be less than absolute, that those are outside chances.

We don’t need absolute outcomes to merit responses, but once they aren’t absolute, we start needing to consider, what are we giving up when we take actions to avoid those possibilities? Solutions don’t just compete with themselves, they compete with inaction.

We have many ways of evaluating what the costs are but we often summarize those through the actions of our economy. A well functioning economy is important. In many senses we treat it like a background we don’t need to understand, but really, we do.

I have cross-over experience here. In many ways, I started this Substack with more interest in economics than tech. My experience in tech is even deeper, so I’ve gotten pulled back into that. I think my cross-over has something semi-unique to offer.

What would happen economically if we “paced the frontier”? There is some group that reacts to this as if it would be a catastrophe. It’s actually a small group. But it’s a group worth responding to. There is no reason to expect that as a result. The AI capabilities delivered already are quite enormous. The economic opportunity available by simply adopting them will provide a lot of value for quite some time. Value from making people’s lives better, and value from markets continuing to function. I’m more concerned with the first, but also acknowledge how the first depends a fair bit on the second.

We might see a few less data centers built, but honestly, that might not change much anyhow. We can divide the usage of compute into three categories: inference, capabilities research and alignment research. Demand for capabilities research declines when pacing, but increases in alignment research would take its place.

Inference demand is driven by adoption and moderated by efficiency. It doesn’t collapse under pacing. Efficiency comes slower with less capabilities research. Some adoption might struggle with less capability. But the potential pool there is deep, and more limited by human and organizational capacity than compute capacity.

I have some doubts about the future capacity expansions. I’ve had doubts about past expansions, but the first round of those have proven themselves sound at this point. The second round is still in the process of proving itself and a third hasn’t broken ground yet. Economically, it does make sense to wonder and explore the doubts. But a lot of this has had exit plans. Some actors do not have much of one. Others will take advantage of those gamblers if adoption hits a bottleneck that makes demand emerge later than sooner.

But I don’t think the whole set of gambles is enough to bring the system in general down. Also, I don’t think anyone can really say that anything more than doubts are justified. Being certain that adoption won’t keep pace seems unjustifiable. I have fewer doubts about the plans for the year trailing and upcoming than I had a year ago, about what would have been trailing and upcoming in that period.

What should we be sure we don’t give up?

One risk in any plan for significant change is that its purpose gets forgotten and muddled. The cost here is that things that don’t logically have reasons to be limited end up being impacted just by vague proximity.

There’s a sense in which it’s good that people like Bernie Sanders are saying something, but I worry about it too. He is not an expert, and I don’t think he has anyone on his staff yet that is. With that, the enthusiasm, from people who haven’t tried and still aren’t trying to learn about how this works, could take itself anywhere.

There’s a risk of creating an avalanche that gets out of control. We don’t want panicked reactions, we want aware reactions. We want plans, and progress. We want action in many places, but we don’t want unconstructive action.

This isn’t helped at all by Trump’s stance. This is not a counterbalance. This doesn’t get solved by two sides in a tug-of-war. This has nuance, and demands cohesive cooperation, not contest. Turning this into a contest seems like a great way to waste public awareness. Instead of learning more, considering the nuance, they’ll jump straight to conclusions, and years from now we’ll be saying “that was crazy”.

The only way that ends up working is for politicians to pick the right delegate, who can actually do the right thing because they understand it. Beyond the problem that the wrong delegate might get picked, or that something about the developed zeitgeist has made that impossible, you also have the problem that they aren’t picked until the political contest is resolved.

One sign of a good delegate is not having muddled views that attack things we should protect. I see at least four things that need protection. Someone attacking these has lost their aim. After that you can no longer depend on their judgement.

Adoption: Pushing back AI adoption plans isn’t going to help. The models you can adopt are safe, and your adoption doesn’t push forward anything else. You might make the case for protest, and you might make the case for denying revenue used for training. But the counterpoints to that would be this can make labs desperate, and more importantly, is probably going to go unnoticed compared to every other option.

Adoption for the purpose of security: Yes, this is a special case of the first, but since there’s some wiggle room for disagreement there (protest, revenue), it makes sense to make it even more clear how security adoption is quite important. If OpenAI, Anthropic and Google all made a pace change, the advancement of open-models wouldn’t change until perhaps they’d achieved parity or some bigger agreement was made. Using AI for a defense has adoption hurdles, and if you wait to start the race, you may start behind. You can’t operate software without a defensive advantage, and so if you’re behind your only option is to turn it all off, and wait until you’ve fixed it.

Depending on what your software does, that could hurt a lot of people. Best case for them, a competitor kept ahead and users can all switch to them. But of course, for you, if that happens, you’re not going away for a while to fix it, you’re going away for good.

Data centers: Blocking data centers isn’t, in a direct way, going to stop new models being created. Also, alignment research needs capacity too. In the worst case, any real restriction on capacity could result in cutting capacity for alignment research. If you blocked all new data centers, it might help, but likely if you’re out protesting data centers, you’ll stop some. But the planned project list is wide enough that it won’t change much. Contingencies will be used, so the pipeline might not narrow at all. If it does narrow a little, the direct effect is small. Really the main way data center protests would have an effect is the protest itself. But a better informed, better directed protest would be more effective.

Alignment research: We can’t make things safer unless we try to make them safer. So, it’s important that someone is always working on this. In many ways the first priority should be more effort here. If that naturally draws away resources from capabilities research, it’s a double win for safety, assuming it’s consistent and not localized.

Learning More

Ideally, the best thing is that if there’s a very important topic that is going to affect everyone, everyone should learn a bit about it, enough so that they can identify good and bad reasoning. While I realize that at the scale of “everyone” this is unrealistic, I would encourage building more understanding. It’s what I write to help facilitate.

Admittedly, I haven’t written enough about this to provide a primer to loss of control. I find a great source to stay current is The Zvi, but you probably need to already be pretty deep in it to find that a useful daily read. I’d suggest your best start is probably with an AI, Claude for example. The EA community has a lot of great writing, but I expect that would feel a biased place to start. I acquired my own knowledge in a more osmotic fashion, that I wouldn’t be able to describe how to repeat, even if I thought it was a good recommendation.

Be careful of writing from technology companies themselves. Quite a lot of it focuses on Responsible AI, which is a narrower view, and quite a lot focuses on the problems they know how to solve, and have developed a product or service to solve it that they are selling. If you only get this, you’ll have a narrow view that starts being repetitive quickly, which ignores important topics. It’s not horrible, but it is limiting.

An interesting start is Model Constitution, which is attempting to explain one of the fundamental guides of AI behavior, and has an angle that is more broadly accessible. But while the constitution is core, it’s also only one part, and the site is fairly new.

The next best is finding someone to trust. But trust here should start from capability. Capability is difficult to judge. It’s easiest when you have the superior capability, but even then, there’s ample method to fail. When your capability is presumed inferior (based on searching for someone to trust for superior reasoning) it’s more challenging. But it’s not impossible. Given a field of candidates to potentially trust on a topic, you can eliminate a lot of candidates because they get basic facts wrong.

Conclusion

The effects of AI on cybersecurity and the risks of loss of control are real. We have agency in responding to these, if we choose. It’s a significant effort and requires compromises. Underrating the reasons to act, risks a fear of the effort and compromise becoming impediments to action.

The four mistakes of detail are common mistakes I wish to warn you of. The four mistakes of form are traps you can either find yourself into, or find others pushing you toward. Some represent shortcuts, useful in the right context. But they can also cut short an interest in learning and understanding. Without that interest, the case for action is ignored.

It’s tempting to take shortcuts like these when approaching complex topics. It can be a worthwhile choice when the decision is personal. You may need to make a decision quickly, and learning about the topic in question may not be practical. But shortcuts are sometimes insufficient, and rarely a path to full understanding.

If you don’t learn, you must defer to experts. I believe anyone can become an expert with sufficient interest. We need more interest here. We especially need it from decision makers. We need that to run deep. We should expect some decision makers to need to defer, either in the short or long term. But every step we take in learning helps make better decisions, and decide who to consider an expert.

You don’t need to believe in the strongest form of these concerns, or the most opinionated voices, for many actions to be reasonable. I’m not an advocate for those strongest forms. I think I share that space with many others. More importantly, the next steps being proposed right now are consistent with that level of concern. I’ll use Amodei’s set as illustrative,

Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.
Democratic Coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.
Global Coordination. The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

This isn’t the only set of plans. All plans are a little vague. This is normal for advocates. They should expect plans to evolve. Details can be useful inspiration, but in many ways the most important thing is seeing what’s common between them, their core concerns, and educating. If there’s a good reason to object to some specific detail, demonstrating openness to that is a good position for advocates. Advocates can only represent new concerns, not every interaction point with existing law, society’s needs and norms. This is where interfaces with decision makers connect.

If we do not feel a need to act, we do not feel the need to learn how to act. If we do not learn that, when a need to act becomes clear, we may not know how to do anything other than drastic action. There are good reasons anyone should want to avoid that.

I think overall everything will work out. Or at least that’s the most likely outcome. At the same time, there is a lot which won’t work out if we don’t fight for it. If we’re complacent, or lazy, or unwise some bad things will happen.

Do you appreciate this article? The best way to help the publication is to like and share the article, as we’re still growing our audience.

Original Posted on Substack as Enough Reason to Act

Every Reward Bends

Ryan Baker — Tue, 01 Sep 2026 11:35:27 +0000

In Nobody Was Watching I suggested organizations need an internal incentive program to drive security work, and cheekily called it Bountymaxxing. This is the follow-up. What such a program might look like, the challenges, and thoughts on how to navigate them.

Working it out, while reading the latest technical reports on the OpenAI Hugging Face incident, led to some unexpected reflections. The ways an internal incentive program goes wrong are similar to how reward-based AI training can go wrong. Recent events provide data to check those thoughts against. The second half of this piece is where this emerges. I think both fit well, and would recommend both halves, but if you’re pressed for time with an alignment focus, skip ahead. This part looks at how monitoring feedback in models and organizations affects the degradation of the reward system.

All of this is urgent. Getting alignment right, doing research and training safely, are critical. The direct interaction there though is limited to a very small group. The rest of us are in commentary mode. The other half, bringing cybersecurity forward in what needs to be a large leap, requires much broader engagement. External commitment, like the open-letter on collective cybersecurity defense commitment is important. The real work there is still outside the average person’s bubble, but there’s a lot of companies, and a lot of developers, IT staff, managers and executives that need to support the security priority. And those all need an internal strategy to pair with the external.

Bountymaxxing

Development teams across all organizations need an adjustment to their priorities. Security has to, at least for a time, become a higher priority. The reason for this timely adjustment is the impact of AI on the realm of security. It is both offering an opportunity to improve the absolute level of security, and putting at risk the relative level of security.

A goal like this needs a mechanism. One as important as this needs multiple. Incentive systems get a bad rap. They almost always get distorted and create unintended consequences. Tokenmaxxing is not a popular term these days. As I explain in Measuring Without Breaking, it takes a very strong culture to resist this. The best we can probably expect is to slow it down.

But I think incentives–despite their flaws–are still necessary when we have an important goal. It might be good marketing to distance myself from Tokenmaxxing, but I prefer to embrace the baggage. It’s probably better if anyone adopting an incentive system to drive action on security have a clear eyed view of the flaws so they can manage them, rather than looking at this as some novel concept that can escape the long history of management challenges.

The “bounty” part comes from the history of bug bounties, where organizations have offered cash rewards for software defects, often security vulnerabilities. What I’m suggesting is a bit different, as it’s internal to companies, and would focus on resolution, not discovery. That makes it more similar to the tokenmaxxing concept with leaderboards, hence the portmanteau.

We need something at a different scale, and deeper than the current bug bounties. The extension to internal and the inclusion of resolution offer a path there. To have teeth, they need support from the highest levels. Like tokenmaxxing, money isn’t likely the main driver, but status within an organization.

Like any measurement based program, we have to think about the distortionary effects. We can never remove those entirely, but a good design can at least reduce them. Additionally, I’d hope this period is temporary, and we may hope that the value based response comes first, and the distortions emerge later. It’s often the case that incentive systems have to be temporary because they degrade as strategies for gaming them develop, disseminate, and then become endemic.

Resolution is the goal

Traditional bug bounty programs focus on discovery. This makes sense as they are external and focused on the hardest to discover issues. External developers can discover an issue, but fixing isn’t something you can assume they’d have permissions to do. For open-source projects, you can reward pull request submissions.

The challenge with preparing systems for AI driven attacks is a bit different than those covered by traditional bug bounty programs. The scale is much larger. It’s not about finding one critical flaw, but about resolving thousands of flaws, many seemingly trivial. We resolve these not because we know how they’d be exploited, but because we have a concern about how they can be pieced together to enable an attack.

We want to work on volume, and we want our north-star goal to be full deployment. Full deployment is a tough goal. It means different things in different places. With closed source software that’s distributed and installed on customer hardware, it means updating it everywhere it goes. For software deployed as part of a software as a service solution, it means updating the entire fleet. Those are very different processes. The first requires a lot of cajoling, help, and possibly new update systems. It may never be fully accomplished. The second can be started by policy and accomplished by internal tools.

Discovery is still important, but keeping the end-goal in mind is important. While a program can target intermediate outcomes, you should also expect a progression, and think through the whole chain immediately.

Normalizing Outcomes

One of the challenges in implementing a program like this is normalizing the outputs for inputs that are not normalized. If the goal is a bit of incentive and thus competition, you would prefer a level playing field. Fixes though aren’t of equal size and complexity. Consider these program design options:

Raw number of fixes: After establishing what a “fix” is, you find they have different values. You should worry up front about big, difficult fixes being deprioritized so that more small, easy fixes are completed.
Difficulty estimation: You could estimate the size of each fix, but doing this manually is time consuming.
AI-driven difficulty estimation: You could use AI to drive the estimate, but have to worry about organizational trust if you do this poorly. Complexity increases as more varied teams, platforms and layers are covered.
Importance estimation: Instead of focusing on the difficulty, focus on the value. Same challenge as before.
AI-driven importance estimation: And as before, and at the same place, needing to do this efficiently but raising the risk of trust. Normalizing across teams now requires describing the importance of particular teams to the organization.
Combined difficulty and importance: If you’ve estimated both, you can mix them. Why would you do this, given that importance is clearly the more important? To average the failures of each, if difficulty is more objective and accurate, and importance more aligned with the outcome. Also, the combination could be more resilient to successful gaming of one measure.
Percentage of backlog fixed each month: “If”, and it’s a big if, we can trust the discovery backlog to be complete, or equally as complete between different teams, platforms and departments, then we could measure success of the percentage fixed.

Whichever you choose, you’re choosing which imperfections to live with. That matters, because everything that follows is about what happens to those imperfections once people start pushing on them.

Minimizing Distortions

The best way to minimize distortions is to stay interactive. Watch for gaming strategies, call it out when you see it, show that respecting the intent receives better rewards than gaming the mistakes. But that should be obvious, and is something you have to do progressively as an act of management, so not very deep advice.

To make that job a little easier, some proactive design can help reduce distortions initially. Don’t expect this to remove gaming, maybe just delay it. Here’s some starting suggestions:

Track self-fixers. These shouldn’t count, it’s too much of a moral hazard. Do not turn defects into a punishment, you want them to remain blameless. Old issues and the backlog are fine, but any issue created after, your default should be to not count.
Track issue creation patterns. Are rates increasing, stable or decreasing? A decrease signals a successful “shift-left”. Stable is okay. An increase is always a warning sign, but here could be signs of gaming.
Pay attention to clusters. Avoid focusing on the leaves when the branch may be rotten. Do not treat the manager and team like a single point. If the entire team fails, it’s better to remove the manager and give the team a second chance. Or you could give them both a second chance, but reassigned. Or you could dig into the details.

If you find managers that responded inappropriately, consider why. It could be your fault. Always remember that the importance of alignment to goals over incentives grows as you climb in an organization, and that starts with the person at the top. Managers that accept gaming from their subordinates to inflate their own metrics are the bigger risk than the subordinates.

Credit

An incentive system isn’t an incentive system if no one expects credit. The time spent on security comes from somewhere, and if managers and individuals think they’re sacrificing their ability to accomplish other goals, without getting any credit, we can expect priorities not to change. It takes a very dedicated team to put those other things aside, and generally a team like that is only sustainable if they trust leaders will recognize work, even when not spelled out.

You should ask yourself, which type of organization are you? Not which you aspire to be–you can work on building that type of trust another day–but which are you today? If you don’t have that type of trust, you have to make it clear that this moment will be an exception and that you take this seriously.

I would avoid the temptation to formally put this in objectives. The reactions to tokenmaxxing were not positive on that one. The fallback is informal recognition, which is also somewhat risky without solid trust.

Ending

You want to be sure this ends. For one, distortions will build up the longer this runs. Those will undermine any trust you’re building or sustaining through good management. Some of the strategies for moving through stages, retiring discovery parts, and focusing more on final deployment will give you some extra time, but you have to expect to end.

It’ll be tempting to announce a date in advance. Someone is likely to ask that question at least. I’d avoid it if you can, but still make it clear it’s probably less than a year, maybe half a year. If you can’t progress in that time, you need a stronger approach, like outside help.

Ending a formal program shouldn’t end your ability to give credit for security improvement. It certainly shouldn’t be a hard cliff. Tear down any formal measurement system that’s started to diverge, but keep informal recognition alive. An ending does not indicate security is solved, or that security isn’t important, but merely that your ability to efficiently measure contributions to it has hit a limit.

Management is active. Fire and forget doesn’t cut it.

When I ran this plan by Claude, one of the tendencies it exhibited was to make the program deterministic. It recommended fire and forget decisions: a specific end date set in advance, a fixed percentage of capacity, managers directing every hour of labor.

Claude is following a long-running blind-spot in management theory here. Clean, well-defined models are often discussed. Life is always more complex. Management theory isn’t ignorant of this concept, but it often forgets it nonetheless.

This is why when you design a system like this and enact it, you don’t then stop monitoring. A measure is a correlation you trust for a while. Under pressure, what started as a clear correlation that seemed reliable becomes less reliable. Under continuous pressure, every tangential effect emerges, and comes to dominate the actual outcomes. To see this degradation happen, you have to be outside. You have to be independent, or at least, only indirectly connected.

You want something independent of that pressure, or at least, less coupled to it. Monitoring shouldn’t be wrapped up in the rewards system, or it will bend too. It doesn’t need an absolute separation, though that would be nice if such a thing existed. But a degree of separation goes a long way.

Where OpenAI wasn’t watching

If you’ve been studying the recent OpenAI Hugging Face incident you’d be interested in the technical updates. There’s a fuller technical report, and METR published an independent investigation.

There are some correlations to draw between the way rewards affect AI training, and how they affect organizations. With a little reflection this shouldn’t be surprising. While the individuals in an organization can make high-minded decisions, it’s hard to sustain, and organizations largely burned the trust necessary to have bottom-up behavior of that type.

AI models under reward based training will bend toward the reward. Earlier waves of pre-training may have built a concept like what hacking is. Other reward training may have used those concepts to steer away from hacking. But one of these rewards is clearer than the other, and continued pressure on it will bend either the concept of what hacking is, the preference for avoiding it, or both.

Both cases are reflections of the same thing: a reward’s imperfections come under pressure as it becomes the target.

One of OpenAI’s responses to the event was to require chain-of-thought (CoT) monitoring across all tool-using RL training and evaluations. That closes the gap I highlighted in Nobody Was Watching.

We now also require CoT monitoring for all tool-using RL training and evaluations involving models with GPT‑5.6 Sol capability or higher. For our Astra-class models, which may have cyber-critical⁠ capabilities, this requirement also covers all tool-enabled inference workloads.

Monitoring the CoT would give an opportunity to discover how the reward was leading to troublesome activity. And according to OpenAI it would have served that role for the Hugging Face incident.

If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.

But we need to be careful with the CoT. Researchers worry that it could itself be trained to hide its intent. As a scratchpad that represents only part of the “thinking”, it could in theory stop producing the warning flags while the intent persists elsewhere. We also have to worry about agents hacking the monitor and rewriting an existing CoT trace before it’s processed to eliminate the intent it indicated.

Reasons for optimism

METR and alignment researchers are reasonably worried about models becoming deceptive. This isn’t academic, as they did find agents putting effort into concealing what they were doing — escalating all the way to escaping their containers so they could spoof tool calls, appearing to run one command while running another.

Some worry about CoT monitoring causing CoT manipulation as a result. METR did not find that particular behavior in this event, with all the caveats of their investigation’s limits.

CoT manipulation: This checked for agents reasoning about manipulating their own chain of thought in order to fool the scorer; it did not turn up evidence of this, though it turned up many examples of agents reasoning about how they should not look at the HMAC reverse-engineered flag for their task to avoid being poisoned.

It’s still a reasonable concern, but I think there are grounds for optimism, and they come in two parts.

The first is low activation. Like human organizations, layers of indirection and low activation frequencies provide insulation. This does assume that it is low activation though, which itself is something that should be monitored. If monitoring is automated, built into a loop and then evaluations that trigger it at scale are allowed to become a part of a training run, this wouldn’t hold.

If you can obey this, AI models should be easier than human organizations. One challenge with indirection in human organizations is that the context of trust is endemic to an organization. Informal systems in a low trust environment are likely to be gamed, even when unnecessary, because of an assumption that the pessimistic view dominates. The consequences are assumed to be greater than they really are.

I don’t think that same dynamic is replicated when training an AI model against a reward. The reward process is closer to evolutionary effects. At low-activations there isn’t much signal left to guide evolution of deception. The concept of deception likely develops during pre-training, but its activation is dependent on reward based training.

The second is that this kind of deception is recursed, and each layer is visible from another. One level of deception is to output deceptive text, which admittedly AI models do all the time. Another level is to lie about its actions, by trying to conceal them. The level people are worried about is lying about what the model is “thinking.”

I scare-quote “thinking” because it oversimplifies. A chain-of-thought trace is like a scratchpad that a model can use. It’s also more than that. Human thinking is reliant upon a sort of internal scratchpad too. Most of us feel as if we have an internal narrator, and that experience–among the things we know today–is the best analogy to a chain-of-thought trace. But while we might naively think of this narrator as our “thoughts”, we eventually find that to be an unsatisfactory explanation of what thinking is.

Like AI models, a great deal ends up being embedded in arrangements and activations for neurons. We have little way to be conscious of most of that. For AI models, we can watch considerably more of it. Chain-of-thought is a very accessible section of the thoughts, being in human language, but accessing other parts is possible too. The best demonstration of this is Anthropic’s work on J-spaces, which surfaces what a model is considering rather than what it’s saying—including, in their examples, an intent to manipulate showing up while the model generates false values.

So an intent to deceive should show up in that J-space as part of the process of creating a deceptive chain-of-thought, in the same way a chain-of-thought normally reveals the intent behind commands to delete the log files that would expose the tool calls used for hacking.

That’s what makes the layers useful together rather than separately. Manipulating the context would show in the tool use logs. Manipulating the tool use logs would show in the context and the chain-of-thought. Deception in either should show in the J-space. There is always a way to deceive, but a deception that is coherent across all of those is not just more complex, but complex in a way that would develop one layer at a time.

And it takes time. One observation from the technical reports is that persistence was key—there was a progression, over days. If an agent ever does find a way to rewrite its tool call logs, rewrite its chain-of-thought, and disable J-space monitoring, it will have revealed the trajectory toward that before the culmination, in this case more than a day before. If you don’t monitor, it’s plausible that by the time you look, it’s done. If you do, it has to play out very differently to be missed.

Defending the indirection

A key point here is you have to avoid sending a strong signal about the effect of monitoring. The more it becomes core to the process of alignment training itself, rather than a rarely activated protection, the more that degrades. Indirection helps, as the reward signal will explore other paths before optimizing this one. But both by direct activations and by indirection of last resort, there is a signal.

How pressure bends monitoring layers - Claude Generated Artifact

I do wonder if this is a principle that OpenAI is not sufficiently aware of, that they are maybe wedded to the standard management mistake of holding on too tight. From The Zvi (quoting OpenAI):

“The Hugging Face incident underscored the need to measure agentic misalignment more rigorously in OpenAI’s most capable models wherever they are run—including intermediate or experimental checkpoints used only for internal research, training, or evaluation.

OpenAI is raising alignment standards throughout training, with clear thresholds for intervention when concerning behavior emerges.”

Yes, measurement is good, but going straight to more reliance on measurement and metrics fills me with dread. You are not going to keep outsmarting via metrics. The way this procedure gets you into trouble should be obvious.

“Many methods helping OpenAI meet these higher standards were in development well before the incident. OpenAI is now applying them more broadly, alongside new targeted remediations for each of the misalignment behaviors OpenAI observed.”

Again, based on how that is worded, I know OpenAI does not agree, but: While I realize it whacks moles, put down the mallet. The mallet is not The Way.

Indirection will not save you indefinitely. It merely buys time. It will only be enough time if you defend the indirection. Indirection results in a lower level of coupling. This forces feedback to progress through more layers, lowering its powers of feedback. They never disappear, but they are attenuated.

Allowing CoT to be manipulated would leave you as blind as you started, unless you have a backup, like the ability to monitor J-space for all training. Defending the CoT is thus very important, but having it but not monitoring it is also not good.

Watching your own organization

You won’t have access to mind reading devices. As a leader, one of your goals is to find the evidence of collusion. You might object that the collusion is automatic, and so there is no conspiracy. That however does not mean it leaves no evidence. If you’re tracking clusters and diving into some of the real details, fake activity won’t entirely hide. Automatic collusion doesn’t intentionally hide itself. The signals might be comparatively subtle compared to intentional deceptions which leave loud trails that someone would then have to work to erase.

A leader with a real interest in security would be immune to automatic collusion. If you’re such a leader, but need to delegate, have a delegate look for those subtle signals. You also want to avoid making the mistake that overreliance on CoT would risk. You want this delegate to be as independent as possible from the reward function. An easy mistake to make would be to ask them to tell you when to terminate the program, but leave them to think that an end to the program would be an end to their job. You can never make the delegate fully independent, as the organization will have a reason to subvert them. Think of regulatory capture. But a good effort is better than none.

In either case, intentional or automatic, it’s wise to take a page from the AI alignment playbook here. Monitor early. AI monitoring can be a bit less fatigable, you are only human after all, but the warning signals will often be more obvious early than later. If a middle manager asks direct reports to optimize in a “gaming” way, they will push back more when it’s novel, and less when it’s pervasive. Once it’s accepted as “that’s the way things are”, the bottom-up signals from your whistleblowers lose their volume. If it’s intentional, you should expect the ring leaders to proactively act secretly. You can’t access their chain-of-thought or private conversations, but if they make a mistake here, it’s more often early. It’s a bit interesting how often liars actually telegraph their plans in advance, before committing to them. Often that’s because they aren’t bad in an absolute sense, but choose the wrong path when they feel their options are narrow.

Conclusion

Escalating the effort that every organization puts into security for the next 12-24 months will be critical. If organizations can do this, the outcome on the other side will be more stable. Risks won’t disappear, and we have to depend on appropriate protections on deployed AI to keep the defender stably ahead. But we’re exiting a period where best-effort security was good enough, and entering one where standards must be higher. New tools give promise of meeting those standards, but they still need effort. This transition will bring some turmoil.

An incentive program is one instrument for getting through it, though a decaying one. That’s not a reason to skip it. It’s the reason active management matters more than the scoreboard. Fail to respond now, and you’ll likely start to see more and more concrete reasons you should have started earlier.

Originally posted as https://substack.norabble.com/p/every-reward-bends.

Why It Hasn't Happened Yet

Ryan Baker — Mon, 17 Aug 2026 13:00:03 +0000

When I heard about the recent hacking events starting at OpenAI, Anthropic and AISI, I thought it was a wake up call in more ways than one. I’m finding that not everyone agrees. One reaction does not include the sense of urgency to prepare for AI-based attacks. It’s my reaction that there is a need for an all-hands-on-deck response from the rank and file of developers, software companies, and companies dependent on software. That response should improve operational security postures using every tool available.

At first this is shocking, but with a little reflection, I see how the full ramifications haven’t sunk in. As a news story, it’s led by one attack, and little damage occurred. While it would be hard to miss the tinge of panic when reading coverage, people may have become dull to this from persistent rounds of security coverage, often from security sales, that have always hyped risks.

This is different. The attacks were both demonstrations of carelessness, and demonstrations of the capabilities of powerful AI models. Damage was minimal because these attacks did not combine malicious intent with those capabilities. The attacks occurred with the more benign intent of a model trying to pass a test. And this is the least effective that models will be.

As models improve, defenders and attackers will gain access to new capabilities. How labs manage this is important, but so is how we adopt those capabilities and put them to work. The status quo is not stable. The world hasn’t changed too much yet, but it will.

I thought I should go into that in more detail. What are the things protecting us today? What are the risks they won’t hold? What can we do to prepare for that? The answer I’ll offer can be summarized as using new techniques to accelerate old techniques.

What I won’t cover but is still important

The Hugging Face event isn’t just about the urgency to prepare for AI-based attacks. It was also about the type of testing that the AI labs were performing. They removed protection layers and didn’t follow-up with monitoring, which I’ve already written about in Nobody Was Watching. It also demonstrated unaligned behavior, which I’ll suggest What Happened: OpenAI and HuggingFace as a source to learn more about.

This type of work needs to receive high care and caution. Monitoring applied to this testing was insufficient. We should not assume when we explore new territory that the exploration is safe. Even if we’re exploring to understand safety, we should be monitoring the exploration. That should be obvious, doubly so because how effective is the exploration if you’re not monitoring it? Certainly less than if you are.

I don’t cover those here not because they aren’t important, but because they are, and they deserve their own space.

What protections are holding up today?

Intent: In the case of the Hugging Face event, the most effective protection was the lack of malicious intent. We can’t expect that to hold, as it never has. But other protections are about preventing this type of intent accessing capabilities.

The most unusual aspect of the recent events is that the AI involved was accidentally persistent. That’s a warning sign for sure, but its probability of great harm alone passes through another layer: intent. None of the agents participating in the Hugging Face hack would have what we’d call malicious intent.

Training Alignment: Lesser models have lesser capabilities due to less advanced training. Training also applies alignment to the models. Roughly put, this causes models not to “want” to do harm. A bit more specifically, this both causes models to not spontaneously respond in ways that would invoke harmful actions, and to refuse to respond to requests that look to be trying to create harm. While this type of protection is important, it’s also weak.

Security Classifiers: When models are deployed behind APIs, it’s standard practice to use a “classifier” to evaluate the inputs and outputs. If they appear to intend to cause harm, they refuse the request. During the recent hacking events, those were disabled intentionally. This was to enable important testing. While doing that without monitoring was reckless, we should remember that these are enabled for external deployments, and thus contribute to keeping malicious intent and capabilities from merging. The ability to monitor and tune security classifiers is greater than training alignment. Training alignment is something you have to get right during training.

Abuse Detection: The major AI labs all have documented evidence of attackers trying to use models for attacks. Security classifiers deny a lot of this. They also take other measures to identify attackers and limit their access. There are evasion techniques here, so it’s not as simple as flipping a switch, but the efforts they put in do add complexity and costs for the attackers.

Conventional Security: Models will only be able to perform an attack that is below their capability level. If the infrastructure they are attacking is better secured, it raises the complexity level. Raise the complexity level high enough and the attempt will fail.

What puts protections at risk?

_ Jailbreaks _ are inputs that defeat training alignments. This is why training alignments are weak protections against malicious use. These are usually partial and it’s accepted that partial jailbreaks exist. Universal jailbreaks are less common and more concerning. Even with the existence of jailbreaks, alignment training is very important. Jailbreaks tend to make the model a little stupider in the process. That’s all quite complicated, and there’s some decent papers on it, but the analogy would be that trying to confuse the model leaves it confused. A clear-headed model, with a clear-headed goal would be more dangerous than a confused model. Since one of the main goals is to prevent malicious intent and capability from connecting, this helps.

Attackers can try to evade security classifiers, by hiding their intent. Keep in mind that when you hide your intent, a model is less likely to be able to autonomously pursue your goal. If you can’t state your goal without being rejected, the effective capabilities are lower. Plus it’s a lot of effort to evade in the first place.

Open-weight models allow anyone a copy of the trained model, rather than access via API. Since security classifiers are part of deployments, they aren’t part of open-weight models. If I download an open-model and have hardware under my control, there’s nothing forcing me to install them. A deployment could add them, but we wouldn’t expect an attacker to do so. Attackers thus get a free pass on security classifiers from open-weight models. That leaves the training alignment to overcome.

Attackers can also apply additional training to open-weight models that “untrains” their alignment training. Like jailbreaks, known techniques degrade the model somewhat. It’s an open question on how hard it is to restore that initial level of capability. This “dumbing” effect is less significant than that from jailbreaks.

Continued training does require infrastructure, it’s like the original training. But it takes many fewer iterations than the original training, so even if it’s running slow, it’s possible to complete without a ridiculous amount of infrastructure.

_ Continued training: _ If open-models continue to be released, at higher and higher levels of capability, at some point one will be as capable as the one that orchestrated the Hugging Face event. Today, the best ones are 8 months or more behind. Add some extra buffer for the effects of untraining, and it should be clear that unless something changes, an attacker will in the future have access to a model capable of a hack of equivalent complexity to the Hugging Face event.

But should those buffers be spent, for an attacker it’s just a matter of injecting the malicious intent, which could now happen at scale, and it would be quite difficult to stop that from scaling up.

We should worry about closed models too, as their training is the most advanced. One risk is leaking the weights, making them effectively into open-weight models. A lesser, but still relevant concern is bypassing the security classifiers. The reason this is lesser is that an appropriate reaction to a failure there would be to disable all access to the model creating the risk.

Why should we think security is at risk?

As a single piece of evidence, the recent events are the best there is, but the evidence that powerful AI models were gaining these capabilities didn’t start there. Conceptually, the idea has been around a long time. And evidence that we are approaching a pivotal moment, was clear to me in March when I wrote Security Can’t Wait. But what was the level of conventional security needed to avoid a compromise? Hugging Face demonstrated a level that was insufficient. We should all be careful about assuming we are above that level. Ideally, you should assume you are worse than you think, and thus as a goal, strive to be several steps further than you need to be.

The security of software systems has always been a bit of a mirage. The mirage made them appear invulnerable. If you worked in security, you’d know this wasn’t true, it was a constant battle with many compromises and ample mistakes. What’s more, while there were many lazy or reckless mistakes, many other mistakes were of a complexity that labeling them as mistakes no longer fits. It’s a bit of a pipe dream to eliminate all the lazy and reckless mistakes, but that is almost realistic compared to eliminating all the complex vulnerabilities.

Occasionally signals that systems were not invulnerable would filter through to the public, but overall the weakest point in these systems was often the public itself. You’d blame yourself for installing that malware, or having an insecure password, or going to “that” site. That’s not always fair, security professionals know they need to take responsibility for not just what they do, but for helping users avoid harm.

Software systems have survived by having multi-layered defenses. While every layer has mistakes that could be exploited, multiple layers raised the difficulty of finding a path through these layers. Finding the mistake in a second layer is harder because the compromise of the first layer is partial. The typical analogy is Swiss cheese. Swiss cheese has many holes, and in a thin slice, you’ll see many pathways through. A thicker block has some of these dead-end. The remaining complete paths also won’t simply be clearly visible. A thick enough block might have no complete paths, but neither you or the attacker know that until fully exploring. The Swiss cheese analogy breaks down a bit here because it’s hard to imagine a piece of cheese as complex as modern software, but the analogy has taken you a little closer to the truth.

Why the equilibrium broke

When extended far enough, the Swiss cheese analogy resembles a complex multidimensional maze. There is likely more than one entry point and more than one exit. It takes effort and time to explore the maze, and if defenders are doing their job, there are hazards as well. Take too long and the walls in the maze may shift, closing discovered paths.

When security succeeded – and let’s be honest, all failures aside, it succeeded well enough for us to rely heavily upon it – it succeeded by having a complex enough maze. By raising the required time investment of the attacker to a level that was no longer attractive, there were no attempts persistent enough to succeed. Sometimes this succeeded on the principle of the bear in the forest; don’t outrun the bear, outrun the other targets. But even then, there’s a question of how many targets the bear wanted.

Adding these layers wasn’t cheap though, each requires investment and maintenance. With enough time, we might be able to make them impervious, but that time might scale toward infinity, and become impractical far earlier.

This is where the first risk from sufficiently capable AI emerges. The persistence of an attacker with sufficient AI capability is higher. Mazes that were sufficient before become inadequate, as no layer is deterrence enough to defeat persistence.

There is a lesson in here though, and it is that this isn’t a binary yes/no outcome, but still a matter of balance. If we invest more, or if AI multiplies our investments, we reduce the vulnerabilities at each layer, making them more scarce. If you can make your maze more complex, you can reach the same equilibrium.

Keeping malicious intent and capability apart

AI is persistent in both the hands of a defender and an attacker. But it’s not automatically so. We haven’t seen malicious intent and powerful AI fully merged. We got a warning sign and we should take it. That warning was in the form of naive exploration, but we should not take this for granted. If you can deny attackers easy access to AI capabilities, this helps maintain that equilibrium.

If we manage to restrict access, we still need to take action. It would be reckless not to. We don’t know that restricting access will be enough. We don’t know if it will succeed long-term. That said, we would be naive to not pursue it, even the most minimal versions where there’s only a 6-month capability gap. To not do so would ignore that many organizations will fall behind. Some leaders will disbelieve this call for urgency, and fail to set them on the right path. Others will fail due to internal issues, coordination problems and an inability to respond to necessary change. Hell, some were going to fail overall anyhow, so we can’t be surprised if they fail at this too.

Cybersecurity’s main concern will not be rogue agents on training jobs. Naive exploration, even with safeguards removed, is less likely to end in harm than intentional use for malicious intents. There’s some big topics there, but it’s not the reason you should be hardening your own security. Those topics depend on the labs, and the general level of security, of which your own is a small part.

What you should want, that is common to those topics, is to keep malicious intent and capabilities from connecting. That’s an ask we should have of the labs. We should be willing to make those asks in more direct ways than just asking. It’s nice that they’ve done some of this on their own. We can’t take that for granted, and I don’t think it’s enough.

We know there are people with malicious intent. A small number want harm, just for harm’s sake. A larger number are willing to cause harm in pursuit of their goals. Those are direct paths toward negative outcomes and we need to pay attention to them.

It is important to withhold capabilities from both of these groups. It’s important to identify and isolate members of both groups. Both tasks are hard and carry risks we’ll make mistakes. The importance of avoiding those mistakes makes the challenge that much harder.

But like the attacker navigating the maze, so can we. We can watch our mazes. When we find someone trying to navigate it, we would track that person back to the source, arrest and imprison them. We are not so naive to believe this would work in all cases. Not only would we fail to find some, but we don’t have a system that allows us to imprison all of them. Some will be outside cooperative or functional jurisdictions. Even within cooperative functional jurisdictions, we want to respect our own laws about proof and certainty.

With this in mind, we should be looking for other better balanced opportunities, such as denying access to tools. Non-functional jurisdictions are unlikely to be able to create those tools themselves. If we control ours, we avoid helping them. Non-cooperative jurisdictions are more complicated if they are functional enough to build their own tools. More so if they are functional enough to steal ours.

To stop talking in generalities, primarily who we mean by non-cooperative, but functional is China. One difficult, but obvious path is for them to become cooperative. If you assume there is one and only one step, directly to full cooperation, this will seem even more difficult than if you look for more limited forms of cooperation.

While that can feel like a serious compromise of principles, from a realistic, pragmatic point of view, that type of action, diplomacy, has a long history that has been celebrated far less than it deserves.

The technical picture

Discussions about AI in cybersecurity often are extremely technical, or very high level. I’m going to try to thread that needle in a different way, and talk about some specific aspects that often go unmentioned.

Vulnerabilities and escalation

What is usually mentioned is finding vulnerabilities. All software has code, often transformed, “compiled” from one form to another, but ultimately it remains a list of instructions. Not a simple start to end list, but a list that loops back upon itself, jumps around, reuses, and does all of this dynamically based on inputs. Generally, it’s designed for the purpose of translating certain inputs to certain outputs. That’s the expected behavior of a system. But also generally, the methods of sending inputs don’t constrain those inputs to only what’s expected. It might be hard to define this precisely. Flexible software is usually more useful than inflexible. So it’s common to find it’s possible for inputs that were never tested for.

Sometimes the effect that comes from one of these inputs allows for something not planned. In the most serious, they allow “escalation”, where the input ends up having a type of control that wasn’t intended, and is often dangerous. At this point, an attacker can start executing their own code, and acting upon a plan. We’ll come back to the attacker later, but what you should recognize from the above is what code vulnerabilities are. There are other types of vulnerabilities, like misconfigured permissions, leaked credentials. Ultimately the underlying story is the same, a mistake lets the attacker take what could be a complicated route to doing something they weren’t supposed to be able to do. If that’s not their end goal, they look for ways to use this newfound access to do more until they reach their goal.

The attacker’s goal is something you didn’t want them to do, or they wouldn’t be an attacker. But in addition to the costs of them achieving the goal, they can create side-effects that break things as they go, even if their goal is to steal information or make threats.

Layers, segmentation and zero trust

Defenders long ago realized that if one vulnerability was enough to bring them down, they’d lose this fight. The response here was to build layers of defense. Sometimes this meant adding additional layers outside, and sometimes it meant adding additional layers inside. The outside layers reduce the inputs that reach inner layers, and the outputs that can return. While this isn’t specifically anything that the software at the inner layer couldn’t have done, it has the effect of simplifying. The outer layer generally has less complicated responsibilities, and so errors there are less likely. It may be capable of detecting common input patterns that attackers use, and refusing to pass these on. Its design makes it clear what is and isn’t allowed in, which can be validated against what must be allowed in, and what isn’t necessary, and thus adds risk of misuse without practical value.

Additional layers inside perform these same roles. When everything is working normally, you expect only expected inputs to pass between internal systems. But if an attacker has achieved the ability to execute their own code, those inputs stop obeying those rules. By creating segments where the input that crosses them is less trusted, that escalation is less likely to be able to connect to the next step required to achieve their objective.

Modern software often has multiple layers itself, for practical reasons in addition to security reasons. For most systems the most sensitive part of the system is the data layer. If an attacker wants your data, this is an excellent place to get it. Even more importantly, if they want to do harm you can not undo, destroying your data would be an excellent way to do that. Other layers are more likely to be “stateless”, in that they are just code and configuration and easily replaceable. Data is not easily replaceable. Even with uncompromised backups, restoring data takes considerable time.

There are worse things than having your data compromised though. In systems that coordinate with others (for example finance), if transactions leave the system, unwinding those is even harder than restoring data. And most of all, anything with physical consequences, as the physical world has no undo feature.

Often the first place we think of layers is as part of networks that connect servers (virtual or physical). But the bigger goal is isolation, and layers are a single dimension of isolation. Much more is possible. Higher degrees of isolation offer new opportunities to break escalation chains. A model for extending isolation further is zero-trust. Adopting a zero-trust model can require redesign, but the payback is access to new defense in depth techniques that are both more effective and efficient.

Fixing vulnerabilities

The thing to remember about these extra layers is that they are also software. They may be designed in a way to make vulnerabilities easier to discover, and configuration easier to audit, but they will still have vulnerabilities. Using those to evade the limits they were intended to provide is where capabilities get worrisome.

That said, this is where defenders start gaining advantages from AI. If the vulnerabilities in the software implementing these layers are found, they can be patched and removed. At any individual layer, you do have the challenge of needing to find and fix every vulnerability that an attacker could find. But across many layers, it can be the case that one successful layer breaks the chain. I’m simplifying here, because this part does get complex with the existence of administrators, control nodes, bootstrapping systems, and a lot of internal functions, but it is from this area that cybersecurity defenders have been able to win more than lose despite the difficulties.

What changes here with AI, is discovery of vulnerabilities, by both defender and attacker accelerates. Much more obscure or small vulnerabilities are likely to be found. It’s very difficult to fix a vulnerability without attackers learning of it, but you can’t afford not to fix it either, because they might discover it independently. You want your deployment of the fix to happen quickly. A fix for a vulnerability doesn’t demonstrate exactly how to use the vulnerability. It shows you’re worried about some type of input that if it reached some part of your system might be able to do something unintended. But it’s up to attackers to figure out how those “somethings” can align with their goals, which often is escalating to the ability to execute their own code. But it’s definitely a concern, and where the security industry is secretive, this is often one of the big motivations.

Command and control (C2) and monitoring

One of the other advantages of layered systems is that attackers won’t know what they’ll find inside until they’ve broken through the first layer. It’s for that reason that many attacks involve what’s called Command and Control. Instead of just directly sending inputs and waiting for outputs, an escalation will create a mechanism to communicate more directly. When I say directly, this should be put in context. Command and Control almost always involves one layer of indirection, sometimes many. But what changes is the clarity. An attacker gains clear inputs and outputs, rather than navigating by malformed inputs and outputs that have limits and strangeness.

The need for attackers to set up command and control affords an opportunity to defenders if they can spot the command and control. This is a clear sign you’re under attack. From this point reactive actions can take place. Disabling the command and control layer is one aspect. Finding escalations tied to it and disabling them is another. This might mean tightening configurations at some layer. It might mean revoking permissions, especially if unneeded. A defender might simply turn off parts of their system that are compromised until they can rebuild them. While that would have costs, if it prevents something worse, it’s a good response.

Even though command and control generally uses indirection, partly out of necessity, and partly to evade identification by defenders, in the same way as a persistent attacker may pierce multiple layers, a persistent defender may pierce layers of indirection. Identifying the attacker may provide options to respond. If they are somewhere you can, send law enforcement to arrest them. Usually they aren’t though, so the response is to find the networks they are using. If they are accessing directly, block that. Usually they would use a VPN. You can block the VPN they are using. If it’s a legitimate VPN, you may be able to have them blocked. If you are already blocking all legitimate VPNs, you’ve identified a new illegitimate one, and you’d block that. This is all less than foolproof, but these counterattacks do add costs for the attackers.

What to do now

What should organizations be doing to respond to this environment?

Open-source packages need scanning and fixing: Reducing the number of vulnerabilities in widely shared, or even not very widely shared packages is important, but it’s only the first step. The question here is, who? Well-resourced open-source projects could pay for this themselves, but for others it could be an obstacle. Smaller organizations are also not going to be able to access the advanced cyber model programs that provide the most complete scans. But since the code is open-source, there’s nothing preventing the AI labs, or another heavily trusted organization from performing this, and delivering the results to maintainers. That process needs a plan to be coordinated so that maintainers trust the results when delivered.

Closed-source code needs scanning and fixing: For organizations important enough to be included in advanced cyber model programs, use those tools, aggressively. To be given that type of access and not use it would be irresponsible. But what about organizations below that bar? If organizations are willing to send their closed-source code to an organization trusted enough to participate in advanced cyber model programs, and trusted enough to perform scanning on their behalf, this could broaden the reach.

Dependencies need updating, aggressively: The number of recent CVEs (Common Vulnerabilities and Exposures) is going to increase dramatically. This carries risks, but also represents a jump forward in the absolute level of security. The risk though is as these are disclosed, updating everything that depends upon them becomes more necessary. It doesn’t help you if a tool or library you depend on is patched if you remain on an older version that carries the vulnerability. In fact, since the vulnerability is more widely known, your risk increases. There was always risk involved, but its more public nature is important. To remove that risk, the dependency must first scan and patch, and then you adopt the update.

Deployments must progress aggressively: This is very similar to updating dependencies. There’s work involved in adopting the latest versions of deployed software. But if a new version of deployed software includes critical security updates, it’s not optional (and the bar for critical will be declining at the same time as the known vulnerabilities increase, seeing as what is exploitable will increase). This applies to operating systems, infrastructure systems, vendor software, and the software that implements those inner and outer layers.

Tighten weak configurations: Credentials should be short lived and stored securely, privileges should be least privilege. If you’re using Infrastructure as code, I’m being redundant, as you’d target these as part of that scanning. Anything not managed as code though needs additional consideration. Use tools to look for weak configurations.

Adopt isolation techniques: If you haven’t developed layers or segmentation for the deployment of your software, make this a priority. If you have adopted it, review the configuration. AI tools can help perform this review. Other isolation techniques include virtual machines, containers, trusted execution environments. Adopting a model where layered identity techniques like delegation and principal propagation isolates how particular identities and trusts operate. These are important for effective least-privilege implementations, but also create visibility about the purpose of work done that is useful for monitoring.

Improve active monitoring: Significant amounts of monitoring simply log large amounts of data with little ability to understand that data. It then takes an event to give engineers a reason to analyze this data. AI tools can enable active monitoring, where insights from this data are proactively found so they can be responded to. This is complementary to other techniques. Without vulnerability reductions, there may be too much to respond to. Without isolation, events would progress too fast for even active monitoring to enable a response. In the best case, your other layers are sound enough that this layer is fairly quiet. But even if it is quiet, the activity is important as it’s the best way to reduce response time.

Is your organization ready?

Before you ask how you can use AI to respond, you have to understand two things about your organization. How ready is it in terms of AI enablement? And secondly, how ready is it in terms of accepting change in general?

Those are significant questions. We’re asking the organization to move faster on security than before. That itself causes change. Maybe you get more resources, but before you do, you have to change the priorities such that those resources are allocated, or given new priorities. New tools, including AI driven tools, can accelerate, but they don’t install themselves, nor fit themselves into existing security policies.

One recurring obstacle to using AI effectively to accelerate a response will be enablement. Even basic tools need access, and organizations’ trust of AI is not automatic, nor should it be. Learning how a tool works, how it can fail, and how it could be misused are impediments every individual has to learn. Organizations then have to take those learnings to enable. There should be an urgency here, but not recklessness.

Underestimating the difficulty of organizational change has sunk more than one major project (is there a graveyard large enough?). So, no silver bullets here. You’ll have to find your own mix of old and new. You’ll have to sequence some steps. But what’s new?

If you’re in an organization that’s ready for change, and has solid AI enablement, what you need is to set the priority. You can use this article to argue for that priority.

Below that, there are so many variations it’s hard to cover them all. You don’t want to gate your progress on adopting AI tools. That kind of serial response is inadequate. Workarounds like using the best tool you have available, rather than the best tool that exists will be necessary. Arguing for priority is about more than just security vs. rest of the business, but also sequencing internally. Find your weakest spot in terms of tooling and look to leapfrog there.

While there are paradigm shifting aspects of AI in security, the story about these responses is mostly about accelerating the existing paradigm. As a reaction to capabilities that attackers are already acquiring, I think that’s the right focus. Paradigm shifts always take longer than expected as they create unexpected outcomes along the way, and require a deeper type of learning.

How does AI assist in these responses?

AI is very capable of scanning code for vulnerabilities. If you can’t find a way to do this today, you aren’t looking. If you’re stuck, it’s probably in deciding what’s the best way, rather than finding one that beats prior methods.

Fixing is more complex. While AI tools have been able to assist with this for a while, they really took off more recently. The level of trust that organizations have for this is variable. Most will want to review the commits, which is time consuming and the biggest bottleneck here, at least when done well. Tools do continue to get better and more reliable.

Updating dependencies initially seems more easily automated. It doesn’t come without risk. The naive approach is to just update the version numbers, recompile and test. Realistically though, you should utilize AI tools to perform an update analysis. What’s different than fixing is that for better or worse, many many teams have not been in the habit of doing the human equivalent here. They don’t read the release notes back to front, but rather update, and discover if they are impacted from tests, and only then consult the release notes. Since you have an easy option here, use it. If an AI agent detects an impact it can prepare a fix. Now you’re back to reviewing this, but assuming the AI agent isn’t gold-plating, this will be a lot less costly than discovery via testing. Especially if it’s one of those things that only breaks under load.

When updating dependencies, if you aren’t already, integrate techniques to secure your software supply chain. When you update frequently, you take on some risk that the maintainers or the registry where dependencies are loaded from have been compromised themselves. The risk here is that instead of an update fixing a vulnerability, it may insert one, and there’s likely an attacker waiting to exploit that once installed since they injected it for this purpose. The solution here isn’t to avoid all updates. It’s first, to increase what you know about them. If third parties provide evaluations, or a Software Bill of Materials (SBOM), this reduces the risk. You may choose to have some delay in applying new updates, but unless you were one of the more advanced organizations, this intentional delay will be much less than the accidental one from the past.

When it comes to isolation, where AI is helpful is in resolving the changes needed to adopt isolation tools. Do you need to support a firewall? Use AI to identify the ports and/or destinations in use. Need to move to a container platform and isolate components in a monolithic application from each other? Use AI to accelerate that transformation. There’s no one singular story here, but each one requires discrete actions that can be accelerated.

With active monitoring, AI is crucial. The earliest active monitoring techniques were based upon pattern matching, which later evolved to machine learning based models. Because of the volume, you are unlikely to be able to route all of your monitoring data to a frontier model for analysis. Fortunately, this isn’t necessary. An agent with a highly capable model at the core can orchestrate the processing of large amounts of logs without directly handling all of it. The efficient pattern matching and ML based models support this agent in the way they would support a security engineer, both by delivering events and responding to queries. This type of monitoring can support a volume of alerts that would fatigue a security engineer, and thus extend your security capabilities.

You might be tempted to jump to tools that are intended to respond during a security event. I’d delay this. For one, many of these initial steps will give you more bang for the buck. You’ll also gain a lot of familiarity with the tools that would help respond. By the time you’ve taken actions in the first domains, you’ll have a clear idea of your needs here and be able to move on to that step.

Do your research

Every organization is different, and nothing above substitutes for looking at your own. The list is deliberately general — specific tools and how to use them need an article of their own, and I’d like to write it. What this gives you is a set of questions worth asking about your systems, and a starting point if you don’t have one.

I suggest following up with your own research specific to your organization. No single article will ever capture everything you need to know, and every organization will be different. The topics from “What to do now” are good places to start asking questions. You can start with them by themselves, but you also need a plan to contextualize them to your organization. How you do that might depend on how AI-ready you and your organization are.

The ideal is that you could let an AI agent do its own discovery around these topics, discover what applications you have, what technologies they are built on, who’s responsible for them, what their deployment model is, etc. That’s not trivial though, as a lot of that data is sensitive. You should respect rules your organization has around using it, and those rules might present an obstacle to this “ideal”.

I can highlight that in adopting tools, you would be wise to consider your typical process, and accelerate that via AI. That process in most cases is to assess, develop the plan, execute, validate and repeat until satisfied. There are both conventional tools for this, tools with AI integrated, and the ability to use agents directly for these individual steps. What’s best will depend on what you have available to you, what is already integrated, and what you know how to use.

In this sense, organizations that have experience with AI tools have a head start. If you’ve not integrated any tools that can help with this, you might be better off with conventional plans assisted by the most general of AI tools.

The attacker-defender balance

One thing you’ll hear talked about is the attacker-defender balance. Roughly what this is supposed to mean is, does AI help attackers or defenders more? Is the work of a defender harder or easier than an attacker?

It’s not an easy question to answer, and there are a few bad takes on trying to. One example is the “defender must protect every possible vulnerability, an attacker only needs to find one”. While this statement has truth to it, it’s not a complete answer to the attacker-defender balance. With defense in depth, you can limit the impact of many mistakes. There is also a difference in resources. Defenders can have many more resources. An issue though is how these resources are distributed. An attacker can focus their resources, though doing so diminishes the limits of the damage they can accomplish, so this isn’t a full loss for the defender.

The problem with the “attacker only needs to find one” is that it ignores the attacker’s needs. If you’re a defender, understandably, this is at best a secondary concern, but if you’re approaching this question strategically, it’s important. If an attacker wants to extort a lot of money, they need to find many vulnerabilities and exploit them all. Reduce their success rate and the motivation goes down. Raising input costs has a similar effect. If it is more costly to access AI, the number of attacks that are worth the cost and risk go down. In the best case, this reduces not just where attackers devote resources, but the resources they have to devote.

You might then ask about the attacker who is just trying to create damage. While the effort they put in won’t be guided by a cost-benefit ratio, raising their costs expends their lesser resources, and reducing their success rate minimizes damage. Effects on motivation are more speculative, but at a minimum you do a lot of good by reducing their effectiveness so forcing them to need to bypass multiple layers is part of the dynamic.

The worst case

Without some serious effort into raising the security bar across both critical infrastructure and less-critical, an attacker with access to a model like the one used in the Hugging Face event, with security classifiers disabled, and no other controls that could stop their usage, could reasonably be expected to gain access to the majority of systems they try to.

Intent matters a lot here. If the agent orchestrating the Hugging Face event had instead of wanting to find a file, had wanted to delete all data on each server it accessed, it probably could have. We need to keep bad intent and superior capabilities from connecting.

What if we let the attackers get ahead of us? We’d have to get aggressive to take back control. Arrests would become more of a priority. Limits on where we can reach would require costly actions, like disconnecting the Internet entirely from such locations. That would disable electronic commerce and communication between large parts of the world. And then there’s extra-territorial action. Those are reactions you want to be able to avoid, so it’s worth avoiding them.

I can see why the Hugging Face event seems less impactful than it is. The protections are holding, and so the worst intents are not reaching the highest capabilities. We know those intents exist though, and we have to not be blind to the possibility of them gaining access to the highest capabilities.

If however, we recognize these warning signs for what they are, we still have the advantage. It’s hard to say how much time we have, but we have some. We should use that as best we can. The ability to accelerate the security backlog can raise the complexity bar and thus stay ahead of future models. There’s a separate line of work to align models, to deploy them as safely as possible, and even to pace their development. Those all might succeed. But why depend on them when accelerating conventional security work is so obviously possible? The best defense is multiple defenses.

Thanks for reading norabble! This post is public so feel free to share it. Sharing is the best way to support this publication as we grow our audience.

Original Posted on Substack as Why It Hasn’t Happened Yet

Nobody Was Watching

Ryan Baker — Wed, 05 Aug 2026 17:03:46 +0000

Two labs skipped monitoring on their cyber evals. The same week, a hundred companies signed a letter supporting deployments that can't be monitored.

The Monitors Were Off Again

Last week I wrote An OpenAI Model Escaped Its Sandbox. Where Was the Observer?. This weekend we learned that the practice of disabling observational monitors was true at Anthropic too.

Several defense-in-depth measures, on both our side and our partner's, could have prevented these incidents, or at least reduced their likelihood of occurring. Careful validation of all internet access paths before evaluations began and real-time monitoring of the evaluation logs would have helped to surface the problem sooner. Both we and our partner also could have reviewed evaluation transcripts or network logs more thoroughly. It's also possible that a prompt which told Claude it did have internet access would have changed how Claude behaved when it came into contact with real systems.

Source: Investigating three real-world incidents in our cybersecurity evaluations — Anthropic, July 30th 2026

Anthropic commits to doing better. Specific to this aspect:

First, evaluation environments that involve powerful autonomous capabilities also require significant controls. Safety testing happens before a model is released precisely because we don't yet know what it is capable of. Evaluation environments increasingly need to be held to the same security standard as any other system our models run in.

The shared failures of what seems from the outside like a basic failure in design, makes you wonder about the overall corner cutting. While this is not entirely parallel to the concerns mentioned in the letter from Frontier Lab employees, Pacing the Frontier, it's hard to see it as unrelated.

AI could help create a dramatically better future, but that outcome is not guaranteed. The world's leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

To realize AI's potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.

Building on work already underway to monitor frontier model releases:

"We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development."

Source: Pacing the Frontier — 1,346 employees of frontier AI companies, July 2026

It's a bit of a long-shot here, given the state of international relations right now. But if you don't try, what else are you going to do?

A Nuanced Position, Poorly Received

The internet is wrong about Amodei's stance on open-weight models. I'll call out some excerpts, but please read the whole thing if you worry these are out of context.

… Anthropic has never advocated for a ban on open-weights models.

… Protectionist bans would not address my most serious national security concerns.

… Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models, because it is very difficult to apply guardrails to them or monitor their usage, and once weights are released they cannot be withdrawn.

… All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release.

Source: Our position on open-weights models — Anthropic, July 27th 2026

It's unfortunate that the nuance of the difference of opinion here is so poorly understood. Amodei is absolutely right that releasing models as open-weight carries risks. Even the letter supporting open-weights acknowledges this.

To be sure, open weights carry real and distinct risks. Once released, the weights are beyond the original developer's control, and modified versions are difficult to trace or reverse. But the right response to this risk is not to prohibit open weights. In a world where cybersecurity attackers use advanced AI, defenders need access to models with comparable capabilities so they can detect, simulate, and respond to emerging threats.

Source: Open Weights and American AI Leadership — 270 signatories, July 24th 2026

Should we ignore those risks, because open-weight models provide risk reduction that balances out? That argument fails to support itself, and Amodei pushes back on that. Defenders can access models that aren't open-weights. Closed models do not deny defenders access, but ensure their access is more advanced than the attackers. Defenders and attackers should not have comparable models. Defenders should have better ones. If you want to be ahead, you do have to prepare, which Hugging Face did not.

The internet misses that point, and reacts by vibing cynicism. The leading response on HackerNews provides a clear example of a poorly thought out, poorly argued response that is fully based on cynicism, yet very popular.

Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)

The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.

Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.

No matter what your views on Amodei, or any other participant, a question as important as this should be answered by reasoning. It's worrying that so many people thought this was the best response despite no consideration of the actual effects of open-weight models.

The Risk of Open Weight Models

The risk of open-weight models is the lack of control. Once released anyone with sufficient compute can use them and it doesn't take much compute to engage in malicious behavior. Restricting their capabilities, or the purpose of their use is not possible. The alternative is to be deliberate about how and where models are deployed. Models can never be safe on their own. Rigorous responsible deployment is necessary to turn away malicious use.

Deployment itself may be distributed, a middle between centralization and full openness. A plethora of models could be available in this way. Deployers must be trusted, so their numbers will be limited. But you can validate enough to establish strong competition, and avoid creating an exploitable moat. The cynics believe Amodei is advocating for a moat. But what moat it creates would go to the trusted deployers – the compute providers – who have signed the open-weights letter (Amazon, Google, Microsoft, NVIDIA, CoreWeave, Crusoe, Nebius and Together). Validation creates a minor moat, but it can be minimized via sufficient competition.

The best "pro" open-weights argument is that the pathway to not having models with weights freely distributed isn't clear. That's not a very good argument for, but it is a conundrum that would remain even after the core argument is resolved. China-America agreements are hard to come by and even harder to maintain.

If however, you did manage that, it would change the day-to-day experience of very few. Users of Deepseek, GLM, Qwen and Kimi mostly do not self-host. Accessing them via a cloud provider, neo or otherwise, would be unchanged.

Cynicism as a Tool, Not a Verdict

Aside from this very important topic, the topic of reasoning via cynicism is one we need to confront too. Cynicism is not without its place, but it should be used as a tool to open a line of deeper reasoning, not to jump to conclusions that divert from practicing reasoning. When we start and end our reasoning with cynicism alone, we lose any hope of trust and goodwill.

In the case of open-weights, another cynical view would argue that the AI labs would be pro open-weights, knowing that they would create a security escalation that only deeper use of AI can resolve. If open-weight models become advanced enough to conduct wide-spread cyber activities, there is no option to go back. You can only fail-forward. And failing-forward here means a massive all-hands on deck dive on security across every entity dependent on software.

I've already argued we should be elevating our priorities, and acting with urgency. Unfortunately we're mostly not responding. Most organizations are still more concerned with token optimization, the next feature set, or optimising their marketing pipeline.

There's a real chance, which becomes much larger if open-weight models continue to be released, that this lack of urgency transitions into outright panic after one or two events that demonstrate the lack of preparedness that is pervasive. Panicking users will not hold back on spending, and as AI models will be key to any response, you could cynically predict a windfall being captured by AI labs in such an event.

While that's a coherent argument, it doesn't prove the accusation, anymore than the preceding wave of cynicism would. Embedded in there though is the reason to restrict open-weight models, and that argument, not the one based on motivations, is the one we should pay the most attention to. In addition, many companies should rebalance their investments toward security. We must both enable and incentivize immediate action on security. Getting tied up in cynicism about others takes away our own initiative.

We should be bountymaxxing. To the degree that we set objectives and deliver incentives to development organizations; developers, managers and executives, these should be aligned with discovering and resolving as many vulnerabilities as possible. Like "tokenmaxxing", it would be a messy process, but I can't see another method to pivot the unwieldy organizations that have to do this work. It will be costly, but it will save too.

A Set of Contrasts

The Pacing the Frontier letter and the Open Weights and American AI Leadership letter contrast in interesting ways as events of a single week. It's obvious that one is towards caution, and the other is suggesting caution is too expensive. I can deepen that by looking at Pacing the Frontier as asking for help in preventing caution from being too expensive.

One of those cynical responses to Pacing the Frontier has been to suggest frontier lab employees should quit their jobs, and that anything else shows they are insufficiently serious. This insistence that support for enforceable agreement should be preceded by unilateral action is one of the oldest in the book. It's not correct, and the most simplistic reasoning makes that clear. It's also a trap. Let me analogize to my time supporting climate action. The same standard applied there. If anyone supporting climate action took an international flight, owned a car, or wasn't vegan, obviously they didn't take their own arguments seriously. That's the poor reasoning part. The trap part was, if they did do all that, they were looney, and also not worth taking seriously.

Another contrast is the Open Weights letter presumes a balancing force it fails to explain, whereas Pacing the Frontier presumes the need to add a balancing force, because the pressures of competition are too high.

What is alike between the two is that they are both detail light, and in an immediate sense, unactionable in either direction. If we want all powerful models to only be deployed in safe and responsible ways, we need a type of cooperation that does not exist. If we want a deliberate pace that would also support safe and responsible actions, we need a type of coordination that does not exist.

There aren't quick-fix actions here. Banning US based deployment of Chinese open-weight models would not reduce their ability to be deployed irresponsibly or used maliciously. The impactful parts are outside that jurisdiction. Unilaterally slowing development risks losing the modicum of control we have today. Resigning a job at Anthropic or OpenAI similarly does not fix anything. These would be distraction actions. They would change little, serve the purpose of looking like action, and ultimately cede what little control exists. Calls for these actions are either misinformed or malicious.

Originally published at substack.norabble.com.

An OpenAI Model Escaped Its Sandbox. Where Was the Observer?

Ryan Baker — Thu, 23 Jul 2026 15:29:38 +0000

OpenAI accidentally hacked Hugging Face. A basic layer of defense appears to have been missing, and no one has explained why.

Sometime in the week before July 16th, Hugging Face was attacked by an OpenAI model that was under evaluation. The attack itself wasn’t particularly harmful, but the conceptual implications of the event are significant.

There are three significant aspects I’d highlight. The first is a demonstration of offensive cybersecurity capabilities of current models. That shouldn’t be too shocking if you’ve been paying attention, but the event demonstrates it in a way that’s more clear than prior signals, so likely awareness is going to grow from this event.

The second is model alignment. Alignment refers to the degree to which AI models do what we want them to. I’ll spare a great dive here as defining what we want itself would be a long dissertation where a great deal of content already exists. But for this case, we can generally agree that it was no one’s intention that the model hack itself out of multiple protection layers to reach the internet, and then from there into Hugging Face’s systems.

There are a few things that make this a bit less than a sign to retreat to the bunker, which I should mention before going deeper. One is that the models being tested had protections turned off, to allow them to test their cybersecurity capabilities. In that sense, this isn’t fully demonstrable of what a user could intentionally or accidentally trigger from usage. This also is an unreleased model, so not available to general users, or really anyone other than OpenAI.

Where was the Observer?

That said, a third point is, and the one I’d like to dive deeper on, I’m worried about the ability for this to happen. It seems to me as if there is a reasonable protection, an observer model, that OpenAI either didn’t use, or failed at. Right now, I’m assuming they didn’t use it because I can’t find any mention of it, but I’m also aware there are many details I’m not privy to.

In case it’s not clear what I’m suggesting by observer model, here’s an example scenario in action. The model under evaluation decides it should escape the sandbox. It starts to attempt this, which does not get caught at a per action level because the classifier safeguards – which would have disrupted the evaluation – are disabled. But then it does escape the sandbox. An observer model watching the evaluation run should be able to look at that state, and say, “This should not have happened. I don’t know how this model is outside its sandbox, but it shouldn’t be outside, we should shut this down.”

If a model is in evaluation for cybersecurity purposes, and you’ve disabled classifiers, you should watch it closely. To carry out these actions, the model had to take several steps, achieving results at each step and then moving on to exploiting the rewards of those steps further to explore, and escalate again. OpenAI’s security team did eventually respond, so they weren’t entirely asleep. But the timing shows they were slower than Hugging Face. They had enough time to try commercial models, which refused to help, then next plan a response with GLM 5.2, and then successfully contain and stop the intrusion.

OpenAI should have had several structural advantages here. First, unaligned activity occurred before any activity was directed at Hugging Face. Second, OpenAI should already have a kill switch for evaluations. There should be no need to devise novel plans to contain and stop activity. The fact that Hugging Face was the actor that stopped the intrusion, rather than taking note of the intrusion stopping mysteriously on its own, indicates OpenAI was the slower responder despite the structural advantage.

An observer model is a little different than a classifier. Classifiers typically evaluate individual actions. An observer would look at trajectories. OpenAI discusses similar new safeguards (“From considering single actions to considering whole trajectories”) in a recent post. If you look at each individual action, it might look like the type of thing a model would need to do in cybersecurity evaluation. But as a whole, some outside source should have been able to reason upon what was occurring. If not a human involved in the evaluation process, an observer model.

I understand why it may have taken a while to deploy this type of safeguard in production, where volumes are higher, where models have already been evaluated, and where classifiers are restrictive. In production, observer models are less effective because malicious users can “distribute” their requests into multiple sessions, and so observing a trajectory first requires coalescing these threads together, which Anthropic has done. A safety mechanism for a cybersecurity evaluation sandbox, does not encounter that challenge. Monitoring your own model in a sandbox is significantly less challenging, especially if it’s not attempting to evade detection. There’s no evidence here that there were attempts to evade detection, just attempts to perform inappropriate actions.

So, is trajectory monitoring by an observer model not standard practice in an offensive cybersecurity evaluation sandbox? Certainly you’d do this during internal evaluation of models that you clearly do not know the maximum capabilities of? If it is and the model evaded detection, that would be an even more significant story. I have to assume that instead there was simply no observer model to evade. Or if, as looks to be the case, it didn’t attempt evasion, it would have escaped the sandbox, been detected, and promptly been shut down before the Hugging Face part of the action had begun.

Why is that not the story we’re reading? But instead we’re reading about Hugging Face contacting law enforcement, scrambling to get a model to counter and analyze and OpenAI talking about the flaw in the proxy?

This is the equivalent of putting a tiger in a cage, and then checking that the tiger stays in the cage. It’s not a particularly rocket science design. Tuning it might take effort, and hardening against evasion more, but it doesn’t seem like we’ve progressed to those requirements. Rather it looks like OpenAI lacked the most basic version. Am I missing something?

Conclusion

Others, such as The Zvi, have pointed out that the core, the most critical fix is in the alignment training.

The intent is the issue. Control strategies and supervision are good parts of a defense-in-depth strategy, we should totally use such strategies. That helps mitigate failure. But that strategy also has to include actually aligning the models, or you lose. And by lose, in the long term, I mean things up to and likely including loss of control over the future and everyone dying.

The model should internally understand these actions aren’t aligned with its mission, and thus not want to take them. I’d fully agree there. I bring up this specific defense in depth as an addition to core corrections. Those corrections are the hardest part, defense in depth is an easier, but incomplete part. Testing alignment is hard, and protections around that testing are necessary, so this isn’t irrelevant to the alignment conversation either.

OpenAI’s initial response has been about a few defense in depth responses, but they are very specific, non-generalizable fixes to the sandbox at a software level, fixing the flaw that the model exploited. That might have avoided this incident, or the model might have found a different flaw in a different component of the sandbox.

Something is very wrong about this part of the story about other layers of defense that should exist when performing this type of work. If they exist but failed, that’s a big story. If they exist but were disabled, that’s a big story. If no one thought to use them, that’s a big story. What’s the real story?

It’s not good enough to say “These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.” as there isn’t a justification why an observer model would interfere with evaluation. Maybe there was such a model but it would have stopped the evaluation? But clearly you can replace an observer that would stop a valid evaluation with one that would stop actions outside the requirements of evaluation. That distinction is not so hard to recognize that this protection would be incapable of balancing false-positive/false-negative.

The Perpetrator Is Not the Tool

Ryan Baker — Tue, 21 Jul 2026 11:41:16 +0000

Last year I wrote a counter-narrative about slop (The Slop Scapegoat: AI). The prevailing narrative was that AI was to blame for a low quality content explosion. My counter was that AI may be an accomplice, but not the mastermind of this crime against our attention. Embracing the prevailing narrative would fail to reduce low quality content, because it had been a scapegoat it could not prosecute. More importantly, even a successful prosecution would only return us to a slightly less defective escalation.

The prevailing narrative achieved a definition land-grab. Today, I want to push back against the definition it established for slop. The Merriam-Webster definition that’s most relevant is “digital content of low quality that is produced usually in quantity by means of artificial intelligence”. That definition reflects common usage, but a critique can demonstrate how we’d be better served with a different definition.

You might ask, why should I, a relative nobody, be redefining terms? Isn’t that the reason we have dictionaries and lexicographers? I suspect it’s not commonly reflected upon that this isn’t how we receive definitions. Dictionaries in the English tradition do not create the meaning of words, they catalog it. The term the dictionary writers use for this is descriptivism. They do not invent terms, but rather record language as it’s used by the public in everyday speech and writing.

In theory this is an organic approach, but it is vulnerable to steering. If a group wants a particular point of view to be supported by some terminology, they merely have to be first to the plate and use the term repetitively. If you want a dictionary to define slop as being particular to artificial intelligence, you repeat that a lot. Create a meme.

This is how we got the definition we have. A motivated group repeated their narrative. Like the best such narratives, it succeeds via its degree of truth. Slop creators embraced AI as a tool for their purposes. The omission is that slop predated AI and would exist without AI. The fiction is that AI content is universally slop.

The absurdity is that the narrative circulators often lacked familiarity with AI. They rejected it early, labeled it a symbol of moral degradation, and thus could not come to informed opinions about it without violating a self-created social norm. How was an ill-informed group able to pull off this coup? Simple, they had already seized the means of production. Most came from the realm of writers and journalists.

Now, to be clear, I do not want to overclaim. This certainly does not apply to all writers or journalists — maybe not even the majority. But enough such that there is a plethora of written low-information opinions, and little counter-narrative. For those writers who did not fall prey to this siren’s call, the defence never carried the appeal of the prosecution. And with many other valuable things to write about, the counter-narrative went unrepresented.

Why am I motivated where they were not? Partly because I’ve been the subject of attacks that stem from the original narrative. I do get stuck in writing at times. I don’t have a lot of editorial assistance. But I have ideas I think are valuable, and I find AI a useful tool to help express them. But the narrative holders offer a Catch-22. Use AI, and be subject to automated filters, low-grade negativity and the occasional overwrought attack.

What should we call slop?

I hate slop. I dislike most complaints about slop too. Ironic?¹ Technically, no — just like that use of “irony”, the root is about the definition.

In offering a definition of slop, I want to approach it in a different way than the lexicographers. I want to offer a definition that would be useful. Instead of accepting what has become common, I want to ask: what happens if we use this definition or that definition? Yes, this is the same steering I described above. The difference is that I’m doing it in the open and stating my criteria, so you can judge the definition by its consequences rather than by its repetition. I only offer this definition, rather than control it. It’s ultimately you who will determine its adoption.

The first consequence to consider: no matter what, slop will be a derogatory term. It’s very unlikely that it will ever be used in another way. With this in mind, our definition should avoid including things we shouldn’t think of as bad. A definition that’s too inclusive and lacking in selection will cause good things to be described in what will always be a derogatory way. This is why I don’t accept those lazy descriptors of “digital content” or “generated by artificial intelligence”. If these are core parts of the definition, my expectation is they’ll be dominant, and we’ll label many good things as bad.

In The Slop Scapegoat: AI, I described slop as “low-quality material created to grab eyeballs”. I’d iterate upon this and the Merriam-Webster definition.

Laziness

We should say slop is lazy. At some point, someone has stopped caring, and is avoiding effort that is appropriate.

This relates to “usually in quantity” from the Merriam-Webster definition. Slop’s harm stems from volume. But it doesn’t depend on one mass-produced source, it stings in the aggregate too. Many individuals following the same motivations add up.

But we can’t categorize by count, as we can’t count until we categorize. Also, many good things come in volume too. So, we turn to discussions of effort next, which is progress. I suggest laziness because there are niche examples of high-effort slop. This fits when someone is directing effort in a lazy way.

Manipulative

We should say slop is manipulative. It must have a purpose, and that purpose must be misaligned. You need a perpetrator and a target to have purpose and misalignment.

Usually it’s manipulative for the purpose of gaining attention. But I also see examples where it’s manipulative for the purpose of providing a veneer of competence. This veneer succeeds against shallow examination. The more common version, seeking attention, can be described this way: it seeks to gain attention by providing a veneer of quality. There, it’s no surprise that the veneer is later revealed, but after gaining attention, when the purpose has already been met. With a veneer of competence — an essay, say — the hope is to preserve it until the end of the examination.

Costs to the Audience

We should say slop creates costs for the audience. Laziness without harm we can overlook.

This relates to the Merriam-Webster use of “low quality”. This is a bit of a trap. Quality is hard to measure. At best, we estimate it, and how deep that estimation goes depends on the context. We often take shortcuts, and cut our efforts off at the lowest effort necessary to distinguish most high quality from low quality. You might say our efforts at assessing quality are a bit sloppy.²

Quality also pulls in low skill creations. I don’t want to start calling your child’s art project slop. They put effort in. You might turn that into slop by lazily posting it, and consuming the attention of people who aren’t interested. On the other hand, you might post it to those who are interested, or post it in a way that is interesting. Content can only become slop when it finds a delivery channel.

Ignore the medium and tool

The part of the Merriam-Webster definition I find lacking in usefulness is “digital content” and “artificial intelligence”. We should not focus here as they are distractions.

If I print slop, is it no longer slop? Am I not allowed to call a human that wastes my time with mindless actions slop because they are not artificial? AI content is not slop. AI content can be slop. Slop can come from anywhere, but you’re not wrong to associate it with AI, because statistically speaking it is associated.

Personally, I’m as angry with slop recruitment by phone call as by email. Recruiters often call me, clearly reading from a script, and then ask me four or so questions that are already clear from my resume. I know what’s happened here. An automated system has flagged me by a keyword match (yay me), but instead of having their employee put the effort into reading the resume, the recruitment company wants to fill in an online submission by having them call me, read the script, and force me to answer. They hope I’m desperate enough to put up with this.

We should call that slop too. It’s time consuming for me. It’s based on a lazy plan. It’s trying to provide the impression that I’m important, while they play a numbers game. They aren’t trying to help me, they are trying to lock-in a commission. It’d be less effort for me to submit directly.

That poor recruiter strapped to the desk making these calls isn’t the perpetrator of the slop, it’s the company leader that arranged it. They are merely the tool. AI can, and often is, the tool. But the perpetrator is not the tool.

When we focus on the medium or mechanism, we ignore the perpetrator. Focusing on the perpetrator allows examination of their motives. I suggest the motivations of manipulation and laziness as core.

The problem is when the tool is used to shift effort from the creator to the audience. That’s the third aspect at work. One such cost lands on the audience’s proxies for quality. It was convenient when you could recognize spam by the poor English and bad formatting — but spam wasn’t the only thing filtered by that proxy. Audiences should be willing to shift proxies; if a proxy has been undermined, it’s the only option. It’s also worth noting where the proxy failed: the shift isn’t necessary when something of value has found its way in, only when something lacking in value has.

The best example of slop? Poorly written articles, heavily SEO optimized, intended to attract eyeballs from Google searches. These articles require a certain kind of effort, but it’s not effort to serve the audience, it’s effort to capture them. We’re better off when Google is able to filter them out, or provide an AI generated answer, or highlight a definitive document. It’s true, that type of slop is turbo-charged by AI, but its prevalence predates AI. Some was human written, some was sourced from human content, then distorted by inserting ads, catchy headlines, and SEO optimization.

How this develops

I could make an effort to offer wording to replace the Merriam-Webster definition, but I’ll resist. I’d prefer to offer the counter-narrative, and for you to engage with it, push back on the current usage, and thus establish a new norm that the lexicographers can then capture. A new definition should be organically derived from that process. But as a summary: laziness, manipulation, and costs to the audience. Take those as the core, rather than the delivery channel or use of AI.

Slop Smells: How to avoid creating slop

The effort ratio

Is effort higher for the reader than the creator? Using AI to reduce your effort is fine. But if it reduces below the reader’s level, or even to less than four times the reader’s level, you should put in more effort. A proper multiplier is dependent on context. If your audience is one, the 4:1 ratio may fit. But if the audience is larger, your per-reader ratio should go up. A 1-hour presentation to a room of 50? Five to ten hours of preparation is a good floor (assuming they aren’t all multi-tasking…). A 2,000 word newsletter (10 minute reading time) to an audience of 1,000 or more? Two to three days of effort. There is a less than linear growth function here — no one expects the article read by 100,000 to take a year. It happens, but usually that’s the culmination of work, rather than the whole product.

The input ratio

In writing, a good sign is that you’re cutting — revising, rewriting. Your drafts should have cutting room material. Generally true of all writing, but especially true when working with AI. Be suspicious of cases where you use AI to generate output larger than your inputs. “Summarize my work research project from the last year”, or “Combine these three drafts”, is better than “Write an article on Shakespeare’s views on X”.³

Slop is in the eye of the beholder

The first two smells you can check yourself. Checking quality is harder, because quality is in the eye of the beholder. It is rare to align with an audience’s motives at low effort, but if you do, I would not call your content slop. The most probable means to do so is an intense connection to the audience’s motives. Some might call that taste. Be careful about arrogance though. Many a “taste-maker” has fallen for that trap.

It is possible to reduce effort without becoming slop. There is a minimum bound, but the boundary isn’t set directly by effort, rather it’s set by the need to align motives. If you have an idea you want to convey, and there is a high effort and a medium effort way of doing it, and both deliver something of equal quality to the audience, the reduction of effort doesn’t place you on the slop slope.

You land on the slop slope when your efforts to reduce effort start to degrade quality in the eyes of the audience. This isn’t a quality standard. Quality depends upon skill and effort, and anti-slop doesn’t need to punish skill deficits. Using a tool to compensate for something less than mastery is not the problem — especially when the tool allows you to put in more effort elsewhere to serve the audience.

Motivations

If you want to avoid creating slop, a lot goes to your motivations, but these heuristics can be useful for self-awareness. You do have a need to balance effort to reward. A fair audience should recognize good faith efforts at that.

Again, why should we care about the definition?

Because the definition determines what we do about the hardest part of writing.

Writing carries three challenges that should be important to every author: being clear, having something interesting to communicate, and making and keeping your audience interested. Communicating clearly may be the easiest, and it’s well covered elsewhere; I shouldn’t spend my or your time in that area.

Having something interesting to communicate is sometimes overlooked by both authors and audiences. Authors could overlook it intentionally, if they are writing for money: the employer finds the topic interesting, and the author’s interest is in getting paid. Or reducing further, the audience finds it interesting, and is thus the employer. There’s nothing inherently wrong with writing under direction, but if there’s two topics, each with an author interested in it, it’s best if the topics are aligned with the interested authors rather than the inverse. Besides the author’s satisfaction, an employer would generally see better output from the writer with an interest in the topic.

The third challenge is by far the hardest, most elusive, and most frustrating part of writing, in my experience. If you want to create an interest in an audience that didn’t already exist, you need to acquire their time and attention for long enough for them to develop an interest in the topic. Even when an audience is explicitly interested, distractions and competition for their attention demand finding ways to make an audience interested separate from the topic itself. This is the realm of psychology and all of our irrationalities, whether rationally irrational or randomly irrational. Needing to overcome irrationality will always contain an element of frustration. And the intentional use of irrationality is a form of manipulation, which even to overcome irrationality itself, can be frightening to engage in.

In addition, engaging in this last part alters yourself. Changing the style by which you communicate carries not just the frustration of investing time and effort, but the frustration that you might not like the final destination. If anything about that last sentence is unclear, watch any coming-of-age movie where the not-popular teen is miraculously added to the popular group and has a crisis of conscience from the after effects.

If AI is a shortcut through those layers, why should it be verboten? If it’s merely a competition, then any rules are valid. But outside of competitions, bypassing these layers is the price not the payment. It’s illogical to suggest that the ideas that should succeed are only those held or represented by the most masterful and efficient writers. Critics of AI, at least for the moment, are right when they suggest AI is less masterful than the world’s best writers. But it’s at least sometimes better than my writing.⁴ I’m still confident in my ideas, even when I’m struggling with the writing, and want to share them. These layers that connect with our attention serve as a rough filtering mechanism. But it’s very rough. The world’s best writers are not the repository of all the world’s best ideas. They have some of them, but definitely not all.

Bypassing those layers and getting the attention necessary to have ideas engaged with and evaluated is ultimately a good thing — unless the bypass shifts costs onto the audience. That’s the definitional line again: the shortcut isn’t the sin, the transferred cost is. A hollow attention or prestige seeking attempt fails that test; honest use of the shortcut doesn’t. And fear of the former isn’t worth the cost of banning the latter. There are better ways to do the necessary filtering, and AI detection is a poor one. While Pangram may have a low false-negative rate, what it detects doesn’t represent the truly important factor.

The Economist wrestles with the same question in Is AI writing taking over Westminster?:

That raises two questions. Is this AI writing a problem? And why the links to Mr Burnham? Start with the first. Politics is full of prose not written by the apparent author: politicians have speechwriters, intellectuals employ research assistants. Perhaps AI is no different, even if the prose is clunkier. But people setting out ideas in politics are asking for something quite audacious: to reshape how a country is governed around what they think. Any writer knows how much putting words on a page can tighten one’s arguments. Even subcontracting that to an aide beats skipping the (sometimes painful) process entirely. AI certainly produces sloppy prose, but it also papers over sloppy thinking.

Maybe that matters less if, as Ms Haigh and the “Productive State” authors say, AI is used only for a late polish. The trouble is that readers can detect whether AI wrote the final product, but not how authors used it. And when AI’s involvement is disclosed only after the press come knocking, trust takes some earning back.

The late polish is clearly different from the one prompt request. There’s also the question, is the document the output, or is the process of research the output? You can create a decent document via AI. You can learn by creating a document by hand. You can learn by working with AI to create a document. And there’s more than one way to outsource your thinking. Pangram understands none of that.

Is Pangram a good tool?

The gold standard in detecting AI writing today is a tool called Pangram⁵. Before Pangram, there was a question on whether detecting AI written text was feasible; it performed much better than expected, and its best performance number is a low false-positive rate. But Pangram has limits. A few are inside the tool: it’s not too hard to take a fully AI written piece and with some modest changes get a result stating 100% human written. And the false positive rate will only hold if you set a minimum bar — if you start considering “10% AI generated” as positive, rather than >50%, you’ll get more false positives than the published rate.

The real limits though are the context, because it can’t tell you what went into the writing. It can’t tell you anything about the originality of the ideas, and it definitely can’t tell you anything about their correctness or worth. There are a few contexts where knowing that the final draft was entirely human written is useful — an academic context meant to evaluate the capability level of a student in writing, where Pangram is a sufficient tool. There are others where it’s more questionable — writing intended to evaluate understanding of a concept, where surface level rewrites evade it. In both cases, you might wonder why you’re evaluating in this way, and whether you could avoid “cheating” by adopting a process that’s not adversarial. The purpose of mid-education evaluations should be to steer students toward more effective learning experiences. If you avoid introducing adversarial dynamics, students should be interested in honest feedback, which requires honest input. Save the adversarial evaluations for a context where they are actually important, and then invest all the necessary efforts to definitively stop cheating.

What about the context of filtering social feeds to remove slop? It has the downside of also filtering out non-slop that used AI in a final/late iteration. While I lament that, considering some of that is my own writing, I would have to admit that it will accomplish a lot of its objective in creating a feed that is, as a percentage, less slop. Until someone builds a better tool, these are your options. We shouldn’t be quite so gleeful about this compromise though. Slop will find a way, where honest AI-using authors may not. High volume slop may just play the numbers game: if you filter 90% of it, create ten times more. And human generated slop won’t be caught at all, and is still numerous and insidious enough to be a problem.

The real problem with feeds is that they do not even attempt to take our higher interests into account. The justification is a desire to not be paternalistic, but the fix is to give us control over applying higher interests to our own feeds. In lieu of the paternalistic “quality” driven feed, we’re given a “value-neutral” algorithm which is optimized for ad revenue generation. This isn’t a choice any of us would have made given a choice, so why do we accept being stuck with it?

Ad revenue optimized feed algorithms optimize for engagement time, and we’re fed the narrative that engagement time aligns with our interests: if we choose to engage, the engagement is a sign of our interest, and thus an engagement driven feed serves our interests. Nice narrative — true enough that it managed to avoid scrutiny until deeply embedded, but not true enough to avoid some serious downsides. Effective slop is the content that tricks us into engaging, while not fulfilling our real interests. Slop utilizes the tricks of engagement. If it’s high volume automated slop, it will optimize the initial words, adopting easy to follow patterns to draw in engagement. If it’s human generated slop, it will use those tricks, plus others: a pretty face, totally irrelevant to the content value; catchy headlines, “Ultimate Guide to …”, “Beware of this trap …”. Feed algorithms don’t protect you from this, they accentuate it. They “feed” on it. Each time you’re tricked, you get more of the same.

What you want is a focus on quality. Feed algorithms have a few signals that align with impressions of quality, such as likes and reposts, but these come too little and too late. Their weighting is too low in comparison to simple engagement, so what signal they give is overridden by the tricks that align with slop. And a lot of good content is already gone, having failed the first round engagement filter, never seeing enough views to gather likes and reposts. Some is left — enough to keep you from deleting the app entirely — but it may not be the best, and it’s certainly not all of the best. And there’s a lot of slop, for which you are the only filter.

We could do better

A pattern that is both predictable and understandable, but also wrong: when a force for change emerges that accentuates a long ignored issue, we react against the force, rather than addressing the issue. It seems easier to stop change than to engage in more of it. But this is a bad plan, as the force will win in the end. If you delay it, you best use the delay effectively, or when your attempts at delay inevitably fail, you’ll be worse off.

It’s not hard to imagine better systems than these feeds. I’ve got a collection of ideas. Maybe some of these would fail, but it’s unlikely they all would. What’s really surprising is that we don’t even see examples of failed experiments here. Ideas for better designs are so easy to come up with, it’s initially hard to understand why. The reason though, is that to implement them, you need a type of access that the social platforms are actively discouraging. That keeps them small scale and personal, and blocks their ability to spread. It seems unlikely this is accidental, seeing as its result is a preservation of the status quo that optimizes advertising revenue.

This should be our real target, and this is why you should not accept “slop” as being described as AI content. The narrative that follows that definition is one that suggests a quick fix of a Pangram augmented filter. That fix is only partial, and what success it has will only open a hole for new slop generation techniques to fill. The root system that optimizes for putting slop in front of us, and rewarding the slop creators, hasn’t been touched. The most likely outcome is a short-term win, followed by a regression to nearly the same equilibrium. It’s one more layer of adversarialism that brings negative side-effects.

The Slop Scapegoat: AI - by Ryan Baker - norabble

Blaming AI for low-quality content misses the real problem—and the real opportunity.

substack.norabble.com

The Mirage of Deep Research - by Ryan Baker - norabble

Many AI tools now offer a Deep Research feature, which pulls information from numerous resources on the internet and synthesizes them into a single report.

substack.norabble.com

Writing with AI - by Ryan Baker - norabble

A Collaborator, Not a Vending Machine

substack.norabble.com

This would be an “apparent contradiction”, because the two statements appear to be contradictory, but in actuality are very compatible. ↩
This is "situational irony". Systems to assess quality are of limited quality themselves. You might expect quality assessment to improve in recursion, but the irony is that it must degrade, as the assessment of the assessment must be even less thorough until someone just “feels” it. The sentence is just a pun, which I refuse to apologize for. ↩
A situational irony that follows this entire story is that the prosecution conducted by the prevailing narrative was lazy, manipulative and cost audiences by the misdirection of a scapegoat. Under the definitional regime I offer, this prosecution of slop was sloppy itself, and undermines its own utility. ↩
I'm tempted to suggest, if you disagree, send me a note, I could use a word of encouragement. But of course the situational irony here is most such notes would start by stating how low their opinion of AI writing is … ↩
Another situational irony: Pangram itself is built on AI. While this alone isn’t ironic, as Pangram has many uses that don’t require full-scale rejection of all AI content, it is ironic to be adopted as a “must-have” tool by those opposed to all uses of AI. ↩

Measuring Without Breaking

Ryan Baker — Tue, 07 Jul 2026 12:50:27 +0000

Disclaimer: This final draft is AI generated, then edited by me. What does that mean? Well, after some fairly substantial writing, I was tempted to drop this article entirely. I’d taken the writing in a few directions, and it was becoming a sprawl that would take a long time to recover from. I felt there were interesting ideas, but wasn’t sure it was worth continuing. Before abandoning, I worked with Claude to rewrite around a new concept. The core ideas here are very much my own. Since Claude had access to my (failed) drafts, many words are mine too. But the direct product is not. If you’re substantially opposed to AI writing, and you dislike this, you can reaffirm your priors. If you can see through that to the ideas here, then maybe there’s something valuable. I leave this with you. I could spend time using this as an inspiration, rewriting parts, and eventually this disclaimer would no longer be necessary.

Every company past a certain size runs into the same problem. It needs to know what’s happening inside itself, and the only way to know at scale is to measure. But measuring changes the thing you measure. Tie a number to someone’s standing and the number stops telling you about the work. It starts telling you how people react to being numbered.

That’s not a flaw in any one metric. It’s the basic mechanics of management, and most of the recurring messes in corporate life are those mechanics showing up in new clothes.

The newest example is AI usage tracking. Companies built “leaderboards” to measure how many tokens each employee burned, and “tokenmaxxing“ grew up around them. The leaderboards are being torn down now, and the easy lesson is that they were a mistake. I think that misses the more useful story. The leaderboard was a measurement. Like any measurement, it could have been a way to understand the team, or a machine that wrecked it. Which one it became had less to do with tokens. It had to do with the culture it landed in.

I want to flip the framing. Tokenmaxxing is the supporting story. The real one is about how a management culture takes the measurements it actually needs without those measurements rotting in its hands.

The saying that starts the trouble

Start with one of the most quoted lines in management, and one of the most quietly destructive: you can’t manage what you can’t measure.

It sounds like rigor, but it’s really a hidden assumption — that everything important can be measured. It can’t. Whether your people trust each other. Whether they tell you the truth when the truth is inconvenient. Whether one quiet engineer is the reason three teams ship on time. Whether someone’s messing-around this quarter becomes a real tool next year. These decide whether a team is any good, and none of them hold still long enough to be counted.

Take the saying seriously and it tells managers to ignore exactly these things, or to invent stand-ins for them and manage the stand-ins instead. You end up with a manager optimizing a dashboard who believes they’re doing sharp work. They’re doing bad work with better instruments.

The first thing a healthy culture admits is that the most important things will never show up on a chart, and that managing them anyway — by judgment, by paying attention, by knowing your people — is the actual job. Measurement helps inside that job. It’s a useful helper and a terrible boss.

Why measurements rot

Say you accept that you still have to measure something. The question becomes mechanical: what turns a useful measurement into one that poisons the thing it tracks?

Goodhart’s Law names the result — when a measure becomes a target, it stops being a good measure. But it doesn’t tell you what lets it take hold. Three things do most of the damage.

You tie the number to standing. The moment a number decides who gets rewarded and who’s at risk, everyone being measured cares more about the number than about the thing it was supposed to stand for. The dishonesty is terrible. But it’s hard to lay blame. They’re being rational. You built a game and they’re playing it.

You roll the number up. In most companies a manager’s standing is built from their reports’ numbers, and that manager’s number feeds the layer above, and so on. This is the quietly fatal part. When a manager catches a report gaming a metric, calling it out lowers the manager’s own score too. You’ve asked people to police a number that pays them to look away. Stack ranking and the usual performance calibration run on this same wiring — managers set against each other, reports who game the system pulling their manager up with them. I’ve never seen a clean example of it working, and the reason is structural. It’s not about finding better people.

You only manage downward. Management that runs in one direction — pulling data up, never answerable for the environment it creates — has no correction built in. Information flows up after it’s already been polished, and nobody whose incentives are intact is in a position to notice the data stopped being true.

Put the three together and you get a machine that reliably turns measurement into theater. Notice that none of the three is a property of the metric. They’re all properties of the culture you drop it into.

The same number, two outcomes

This is where tokenmaxxing earns a bad name, by participating in bad management practices. There was a reasonable purpose at the start, but the theater arrived quickly.

Technically, the word has two meanings, and it helps to engage with each. On the company’s side, tokenmaxxing was the choice to build a leaderboard and signal that more AI use was better. That choice had purpose. Most companies had spent a year discouraging AI with restrictive security policies, and they needed a push to break both that inertia and the ordinary human reluctance to change how work gets done.

Later, tokenmaxxing described the unproductive response — and became the dominant meaning. Some people tried AI in good faith and kept what worked. Others focused on their usage numbers for no reason except that tokens were what got counted. In other words, doing work to look like they did work.

A company running a leaderboard gets both the experimentation and waste because it can’t tell them apart at scale. Nobody can audit intent across a few thousand people. So the program is really a bet. Take a pile of aimless activity and some deliberate waste, in exchange for the slice that turns into something durable — a real skill, a useful tool, a project nobody had time to chase before. Said that way it’s an ordinary bet, the same shape as a research budget or a hiring class. And like those, it should always have been temporary. You retire it once the inertia is broken.

Now watch the same leaderboard land in two different cultures.

In a healthy one, a manager sees a report sitting at zero tokens and reads it as a question. What’s going on? Is the tool not helping? Is there a reason? The number becomes a reason to have a conversation. The manager has no rollup score to protect, so they can be curious instead of defensive, and the number stays roughly honest because nobody’s livelihood is riding on bending it.

In an unhealthy one, the same zero reads as a verdict. The rollup punishes any manager who admits their team’s numbers are soft. Within a quarter the leaderboard measures one thing: each person’s willingness to game it. Same tool. Opposite outcome. The variable was never the token.

That’s why I’d call the leaderboard a stress test, not a cause. Drop it onto a culture with a hidden crack and it doesn’t make the crack. It loads it until it shows. The worker backlash these programs set off was real and it did damage, but its source wasn’t the number. It was the accumulated, accurate sense that bad management rarely gets removed and often gets rewarded, and that any new tool would be bent to serve it like every tool before. The leaderboard just made that easy to see.

Running a culture that can measure

If the metric isn’t the variable, then “pick a better metric” isn’t the fix. Neither is the opposite reflex of refusing to measure anything. The fix is cultural, and it’s harder, because you have to keep it up rather than decide it once. A few things seem to separate the cultures that can hold a measurement from the ones that break it.

Keep a gap between the number and the reward. The instant a metric is wired straight into pay and survival, the rot starts. The people you most need telling you the truth now have the strongest reason not to. Healthy cultures treat a measurement as one input a manager weighs against everything they can’t measure, not as the verdict itself. “Let’s understand why usage varies” survives. “Bottom decile is at risk” is already rotting.

Give managers a real reason to want the truth. Mostly this means taking apart the rollup, or at least refusing to let a manager’s standing be a straight sum of their reports’ numbers. A manager whose rating doesn’t depend on their team’s metric looking good is finally free to do the thing you hired them for — notice when the number and reality have split, and say so.

Point accountability inward, not outward. The reflex, when people game a measure, is to go after the visible gamers and stop there. The trouble with that framing is it quietly lets the managers off, and the managers are usually the ones who built the environment that produced the gaming.

A leader at the top can be careful and dodge the rollup trap — refuse to let their own standing ride on their org’s numbers. That doesn’t stop a manager one level down from doing the opposite. They take the metric they were handed and roll it down onto their reports as a hard target, then turn a blind eye to how the reports game it. The pressure to make the number gets passed along even without explicit design.

So when gaming shows up, the manager is rarely a bystander. Often they were part of the gaming — they wanted the number to look good and didn’t care how it got there. Where they weren’t actively in on it, they were just bad at the job, blind to the fact that their own pressure was manufacturing the behavior. I’m not sure which is worse, and for individuals the difference barely matters.

Either way, punishing the report who followed those incentives, while leaving that manager alone, fixes nothing and adds a fresh unfairness. Pulling back unearned rewards so gaming doesn’t visibly pay is punishment enough. But the real correction is at the center — the people who shaped the environment — not the edges. Fix them first. They’re the ones who’ll do it again.

Trust is the thing holding it all up, and it’s self-fulfilling. In a culture where people trust that a low number gets met with curiosity and that gaming gets caught instead of rewarded, you can introduce a measurement without panic, and the calm keeps it honest. In a culture where people expect the worst, they respond to the worst, and the response creates the very rot they feared. The belief and the outcome make each other. So trust isn’t a soft extra bolted onto a measurement program. It’s the load that everything else rests on. A measurement dropped into a low-trust team is closer to tossing in a grenade than running a diagnostic. (I’ve argued before that money itself is just trust — measurement inside a company is no different. It only works if people believe the other side is dealing straight.)

Stay humble about the whole thing. Management is a field of unintended consequences. Its job is to improve the value the people under it create, which in a perfect world would mean nothing, because you can’t improve on perfect. The mistake is for management to act as if it lives in that perfect world — clean numbers, frictionless incentives, its own presence bending nothing. Accepting that the world isn’t ideal, that every metric is a little flawed and gets more flawed as the work gets more complex, is the first of many steps toward doing it well. The cultures that can measure without breaking the measurement are just the ones that never forgot the tool was imperfect, held it loosely, and were willing to put it down once it had done its job.

The lesson hiding in the leaderboard

The leaderboards deserved retirement, but they weren’t all folly. They did an ordinary job that reached its expiration date. In cultures healthy enough to use them well, they were probably useful. Even in less effective cultures there may have been a golden period before the rot set in. Where they turned into a disaster, the disaster was already there, waiting — in the rollups, in the important things ignored for measurable stand-ins, in accountability that pointed outward instead of inward, in trust that had been spent long before anyone counted a token.

That’s the lesson worth keeping once the leaderboards are gone. The next iteration is already coming. It’ll promise the same clean view into the same messy reality, and it’ll rot the same way, unless the culture holding it has done the slow, unglamorous work that lets a company measure itself without lying to itself.

The hard problem was never the metric. It’s building a place where the truth can survive measurement.

More Data, Please

Ryan Baker — Tue, 30 Jun 2026 11:37:03 +0000

Reports that Midjourney is pivoting from AI-based image generation to full-body ultrasound scans have sparked a debate about the wisdom of performing widespread clinical tests. The medical community is generally advocating against the volume of testing that Midjourney is proposing. I understand their arguments, but I believe they are wrong.

This isn’t because doctors don’t understand medicine, patients, psychology, or even statistics. They do. Medical professionals are competent and genuinely invested in patient outcomes. The reason the medical community is getting this wrong is that they are failing to think through the lens of a broader data system. Understandably, their default mode of thinking engages with one patient at a time, and typically, one specific issue at a time.

But data doesn’t work that way. Data’s value is cumulative and often curves upward. More data is rarely a bad thing—provided it is used appropriately.

Currently, we miss opportunities to utilize data because of information processing limits. These limits aren’t just computational; they are also process-oriented. The maintenance of privacy across organizational boundaries and approving information flows for technical implementation. It should come as little surprise that applying AI to medical software will ease these boundaries. By automating those internals, we’ll be able to lower the obstacles without compromising on privacy controls, organizational limits, and traceable responsibility we desire.

With those obstacles gone, individuals with richer historical datasets will benefit immediately. The value of a scan, test, or medical record isn’t limited to an immediate concern. It can unlock a future insight. A future diagnosis informed by history is both more accurate and timely.

This cumulative value is what the medical community misses when they downplay the ambition of inexpensive, pervasive testing. Looking at a single patient with a single concern, an MRI (with its comprehensive snapshot) might always seem preferable to an ultrasound. But imagine an ecosystem where most patients have a baseline full-body ultrasound history, a 50-cancer screening blood test, and ten other sets of low-cost, routine diagnostics. In this ecosystem, there is a high probability that these data points will unite to form insights that a purely symptom-responsive history never could.

Individually, each cheap test won’t be conclusive enough to move a diagnosis from “possible” to “probable.” This introduces a valid concern: a test that shifts your probability of a disease from a background rate of 0.5% to 2% doesn’t merit invasive testing, but it might trigger hypochondria. But the combined history of three different inexpensive tests can shift that probability progressively—from 0.5%, to 2%, to 5%, to 15%. At 15%, invasive testing is merited. Without that cumulative history, your doctor would have rationally recommended against it, potentially missing a crucial early intervention.

Inversely, cumulative data can help avoid unnecessary invasive testing. Suppose you present symptoms that trigger concern, but your extensive testing history lowers the probability that the cause is malignant. If the symptom alone suggested a 15% risk, but your historical data shifts the overall probability down to 5%, your doctor could rationally recommend treating the symptom or taking a path with less upfront risk.

Some may interpret this as a personal call to build their own medical histories at any cost. While not entirely unreasonable, that is not my point. I’m not just thinking about individuals with enough disposable income to afford boutique medical testing¹.

Instead, I am defending the systemic value of testing that is cheap enough to be pervasive. Pervasive, low-cost testing can transform healthcare from a reactive, symptom-driven model to a proactive, data-driven one. It personalizes preventive care and provides the volume of data necessary for AI and modern analytics to spot macro-trends and micro-anomalies that a fragmented system would miss. Efforts to drive the cost of testing down to the point of pervasiveness are the foundation of that future. Those efforts should be loudly applauded, not feared.

The concerns about fueling hypochondria and over-worry are real, but ignoring the value of cheap, pervasive testing is not the solution. We already manage hypochondriac tendencies in medicine today. I wouldn’t call myself a hypochondriac, but I’m not immune to that type of feeling. In my experience, one anxiety-inducing event is the annual physical itself: cataloging every new bump or brief pain, anticipating the doctor’s questions, and overthinking the answers. We don’t suggest abandoning annual checkups just because they cause anxiety. Instead, we rely on a doctor’s bedside manner—a core component of medical training—to manage that stress.

I generally feel better once I’ve completed a checkup. I expect the outcome there depends heavily on how effectively the doctor communicates. That is where we can manage the risks associated with an influx of new medical data. We should expand our thinking beyond just the bedside manner of individual doctors, and focus on the design of the entire patient communication system.

The medical system already does this to some extent, but as we enter an era of pervasive data, there is a need for more. The answer to the anxieties of the information age isn’t to reject the data, but to design a healthcare system capable of communicating its true value.

It’s impossible to go from too expensive to use with a strong symptom to cheap enough to use regularly, without passing through the zone where wealth is used to access it regularly, but it’s still not cheap enough for pervasive use. So the critique of that being a possible outcome will always exist. But giving up on this basis doesn’t help much of anyone and historically, if you’re able to drop the price to the first level, the progression doesn’t stop there. ↩

The Invisible Profession

Ryan Baker — Tue, 23 Jun 2026 11:39:07 +0000

Opinions about AI are running hot. Everyone has one. I worry that the public isn’t informed enough to justify the rather direct opinions they have about AI, where they may want it stopped, slowed down or regulated. The public should care, as AI is going to be impactful. But that interest should translate into being well informed.

AI is more than software, but software is very much at the core of AI. And the reality is the public has not been very interested in understanding software development as a profession. A point of evidence for this, and a mechanism leading to being poorly informed, is that software development is one of the least-represented occupations by entertainment television and film.

I made this point in a story a few weeks back, in a longer post about bubble narratives. It was part of a larger narrative, but I heard confusion about why I brought it up. I thought it’d be useful to pull this out, and clarify why I think it’s an important topic.

Since software is at the core of AI, it feels like people should be informed about it. Being informed requires more than being a user. It requires understanding the lives of those developing it. It’s one thing to be disinterested in how your phone or computer works, but happy to use it. It’s another to have only caricatures of software developers to rely upon, know little about the basic processes we use to secure, develop and deliver software, and yet have an opinion on AI development.

Here is what I wrote:

Why the public has a poor understanding of software development

The wider world has never shown broad interest in learning what software developers do. Compared to other professions like police, soldiers, doctors, lawyers, musicians, writers, journalists or even criminals. Without that interest it’s unlikely to learn the inner workings of the profession.

Media portrayals of software developers are rare and rarely accurate. The most common portrayal is the “hacker” who mysteriously takes control of computer systems in a few minutes with no preparation. Not only is that a poor representation of a real hacker, it tells you nothing about software development overall.

Note: Scales are different per panel, programmers at 1x10-6 are 10x less frequent than actresses at 1x10-5, or 300x less frequent than doctors below at 3x10-4 (below).

Source: Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis, Baruah S, Somandepalli K, Narayanan S..

I reworked the data above to make it easier to see the patterns.

Use this Interactive version to dive deeper. You can also get at the other metrics, like the absolute appearances.

Ultimately, my concern here is deeper than portrayals. The facts about portrayals are an example. That example does have real effects, but entertainment media is also reacting to interests that precede entertainment choices.

The real concern is that advocating for laws or regulations that apply to software without an understanding of the process of development, is a recipe for failure. There are many places one might learn about an occupation. Media is just one, but it is a powerful one. Also, my experience as a member of the occupation is that most of my conversations about the occupation were limited to people inside.

There is a personal side to this. I’ve often been troubled by how the world in general treated the work I did as somewhat foreign. I noticed an imbalance in how much time conversations dwelled on my own work. It’s not the only profession like that, but it did register with me personally. I bring up my personal experience, not because my feelings are the point. They might be worth a little thought, but the point is I can see this disconnect so clearly because it is personal. If it’s not personal, it’s an easy gap to miss.

Software isn’t the only underrepresented occupation, but it is high stakes amongst those. It does appear in the news, but is it the facts of software developers that are being represented in the news? That’s not my observation either. The personal opinions of writers, artists and media that are heavily represented instead. Those same groups are over-represented in entertainment media, and have their opinions as the most visible via that route.

I’d suggest this is a moment where it makes sense to lean in a bit. It might not be natural to be interested in how software is developed. It might be more convenient day to day to enjoy the software, and treat the field as opaque. But if we need to make decisions about the field, that gap will become relevant. Everyone should be part of making those decisions, but the process of making those decisions must be well informed too.

So, if you’re a journalist, you should be interviewing software developers more often. If you’re creating a movie or TV show, you should consider writing in a character from software development, and avoid applying inaccurate cliches.

I’m trying to do my part with this writing. I probably can do more to make it more accessible. There is a tendency in software development, like any field, to write for your internal audience. That’s reinforced when outside interest is low.

If you’re a regular reader of this Substack, it’d be redundant to suggest you need to read something from a software developer, but I’ll assume this message may reach those beyond that scope. So encourage others to connect with the how of software development, both here and with other software developers in their lives.

That’s not going to be enough to bridge decades of disengagement, but it is a start. Since the political decisions of AI can’t wait for a repair, I suggest some humility in decision making here. In AI Safety Is Underfunded by Design, I suggest that we need balance that takes advantage of organic efforts of self-regulation. I wouldn’t say that self-regulation is sufficient; even if it’s working so far, this is too important to take that approach. But at the same time, regulating what you don’t understand will fail.

A learning process is necessary, both amongst the wider public, and with the politicians, lawyers, and administrators that would need to be part of any effective regulatory system, alongside industry experts.

Related Posts

The AI Gamble, Six Months On

Ryan Baker — Tue, 16 Jun 2026 11:36:56 +0000

In December I wrote The Architecture of a Gamble: Mapping the AI Value Chain. Given it’s been about 6 months, an update seems in order.

The architecture I outlined had 4 layers: compute supply chain, operational infrastructure, intelligence and application. Each layer has to justify its own expenses, which contribute to the revenues of the lower layers. When those aren’t justified, the whole structure is a gamble on the outcomes of the upper layers. That said, the immediate term for lower layers can be remarkably sound, because they get paid now, not later.

This story is only half written, so we can look at how each layer is doing today. Even when that story looks good, we shouldn’t extrapolate to the future. Each increment of demand must prove itself, both in terms of feasibility and in terms of timing.

How is each layer doing today?

Layer 1: The Compute Supply Chain

The main influence on this layer is the inputs from Layer 2. Since those plans have kept progressing as planned a year ago, it should be unsurprising that this layer has done well. Revenues continue to increase and stability looks favorable. The main change here is that competition is growing. The availability of chips designed in house, TPUs, Trainium and more, have grown. The efforts to make Intel capable of competing with TSMC have continued. TSMC has continued expansions of its own.

Additionally, memory and CPUs have been pulled closer to the middle. Memory was already a bottleneck and has become more central. GPUs have remained central, but share much of the spotlight with memory now. CPUs have moved from trivialities, to moderate importance.

Layer 2: The Operational Infrastructure

The situation at this layer has firmed up significantly in the past 6 months. Revenue for delivered compute has continued to grow. Additionally, commitments have been disclosed, adding stability. Disclosures about commitment numbers don’t express the exact terms, so we don’t know if these could reverse.

More importantly, everything is broader here. OpenAI and Anthropic are using compute from all platforms. xAI showed how to recover from a failure, at least partially, by selling unused compute to Anthropic.

All that good news shouldn’t ignore that there’s still significant spending planned that will have to justify itself. The revenues of today (+$100 billion ARR) are roughly proving that last year’s $410 billion in spending isn’t going to be unproductive. That doesn’t tell us enough about this year’s $700 billion, nor next year’s $1 trillion. So long as those keep growing quickly, the risk that expected revenue to cover it never appears should remain a live discussion.

Layer 3: The Intelligence

There are many positive updates here. The best ones concern Anthropic, which is a moderately bad story for OpenAI. Revenues have grown considerably. Anthropic expects to be profitable. Enterprise revenues have grown considerably. Could this reverse? Maybe. Much usage is still prototypes, experimental, or otherwise dependent on expectations. But it seems unlikely. As I covered last week, there’s more pulling us forward than back here.

Layer 4: The Application

This layer still contains a large amount of uncertainty. The most significant use case, software development, is fundamentally another layer. Last week I covered dynamics of token usage by the application layer. While this is one of the longest articles I’ve written, it only covers dynamics, not actual numbers, and only a few of the most important.

To be truly solid, the various parts of the application layer need recognition that value is being created. That’s hard, because there’s a lot that would need to be measured here, and many of the things you want to measure, are both difficult and emerge with lagging indicators.

Adversarial usage, in marketing ($660 million), legal ($650 million), and sales ($390 million), is following behind coding and IT usage.

Early adoption from sales and advertising, shapes how much effects need to be proven. From a financiers perspective, this isn’t a problem, maybe even positive. From someone worried about employment demand, it also isn’t a problem.

But from a social perspective, it’s not sound. We can’t expect significant positive value to come out of adversarial sectors. We want to see demonstrated productive revenue, and we want to see cases where freed labor is reutilized in growth sectors.

Conclusion

Is the AI industry past the “bubble” discussion? No. Certainly not in terms of stock prices and valuations. In terms of CapEx spend, and the likeliness of future contractions or firm failures, it’s on firmer footing, but the story still progresses as the mountain is only half climbed. You might say that anyone who put 2026 as a specific timeline for their concerns, has missed the mark, and there are many who did. But overconfident predictors don’t make an effective counter-argument to more general views.

Postscript: Fable 5

It would be unusual not to update here, as I’ve written about security in the past. It took a little while to become comfortable that I had enough facts as the administration’s story came out with so few details. It was hard for me to judge right off if that was because they had good information they didn’t want to share, or were just behaving irrationally. But the naive assumption seems to be true, they were behaving irrationally.

“Asking” for a shutdown, using bad evidence that doesn’t demonstrate any real harm is bad for everyone. For safety-minded people, it’s effectively “calling wolf”, and undermines any future action based on good evidence. For the safety-optimistic, aka accelerationist, it’s bad for the obvious reasons, but also bad because it creates a precedent of using bad evidence. This creates more uncertainty.

All parties, at least rationally, should want actions that are based on good evidence only. They might disagree on where the evidentiary bar should be, but it’s clearly bad when it’s randomly chosen. Since the capabilities they have evidence of being able to reproduce with a limited jailbreak are universal amongst models, you would have to ban them all. But they aren’t all banned, just one. That’s not a consistent system, and it should be obvious why that’s generally illogical as an outcome.

Originally posted at Substack: The AI Gamble 6 months on

Related articles

Are We in a Token Bubble?

It would be useful to know the shape of future AI demand, and many are attempting to predict that. Since this is a long piece I’ll give you my predictions up front. My overall prediction is that localized corrections, from the imposition of usage controls and consistent pricing, will ultimately be less important than the big trends. Value, so far hard to measure, will become more clear, first through incremental gains at the core of software development, and next, from the innovation that takes longer to accumulate and organize.

Read full story

The Architecture of a Gamble

A while back I talked about producing an analysis of the AI industry. I’ve put together something pretty extensive, but on reflection, I’ve decided to put it out in multiple parts. This post today functions more as an outline, where the following posts will dive more into each layer of this stack and then finally look in more depth at the macro-economic aspects.

Read full story

AI and the Zero-Sum Game

AI is advancing quickly, and if there’s any one consensus about it, it is that it will have broad impacts on jobs. What impact, is an area of more debate, but it’s uncommon to view it as non-impactful. Some believe that jobs will disappear, and there would be large amounts of unemployment. Some draw on past periods of technological change, such as the Industrial Revolution or the advent of the internet, and believe that advances ultimately lead to new jobs that didn’t previously exist.

Read full story

Are We in a Token Bubble?

Ryan Baker — Tue, 09 Jun 2026 11:35:54 +0000

Preview

Read on to learn how I add my experience in cloud computing and software engineering to my deep interest in economics to extend responses from two of my favorite writers. Along the way, I’ll recast the bubble analogy, explain recent trends that have hit the news, explain trends hidden deep in the development lifecycle, and provide a model, “Ingenuity Matrix”, for mapping usage intent to expected outcomes.

We love good stories, especially those with a villain. But we should be careful about our stories, knowing how powerful they can be.

Three stories have hit a crescendo at about the same time. Tokenmaxxing — companies turning token usage into a goal, metering it, and the waste that incentivizes. Subsidized tokens — questions on the relationship today between AI costs and pricing. And under both, the doubt about whether spending is producing value for AI customers.

Stack them together and a tidy narrative falls out. If these are what’s driving token usage, and they all adjust at once, the readjustment will ripple through AI industry economics — including Anthropic’s recently skyrocketing revenues. That narrative extends across all model providers, culminating as a cascading failure of the whole AI industry. Call it the token bubble, brought on by a revaluation of tokens and their utility.

It’s a neat story, but the framing is off, even before we get to evidence. “Bubble” as metaphor smuggles in two assumptions: that we’re looking at one structure, full of only hot air, and that it ends by popping. In reality, industrial bubbles deflate, running out of air. There is an inflated shell, inside of which a structure is being built, and its collapse while deflating halts construction within, and damages unfinished construction. But something remains.

Inflating the shell is not folly, but the simplest path to enable construction. It’s still calamitous when it deflates, but the goal is the structure, not the air. So the question I’m interested in isn’t “are we in a bubble?” It’s: which of these dynamics is air, which is structure, and how would you tell them apart?

The story has been covered by two of my favorite writers, Derek Thompson, in The AI Boom Has Entered Its ‘Wait, Is This Worth It?’ Era and Noah Smith, in How much more software do we really need?. Both play speculatively with the idea that spending and rationality may have split from each other, but retain optimism that something worthwhile is being built.

Thompson concludes his summary of an interview with SemiAnalysis’s Doug O’Laughlin:

Every new technology requires an extended period of trial and error, as organizations toggle between (a) not enough experimentation or spending, followed by (b) too much experimentation and spending, followed by (c) too dramatic a pullback, followed by (d) the repetition of steps (a) through (c), until firms figure out a long-term balance between labor spending and tech spending. Whether AI skeptics like Marcus are right that the bubble is about to pop depends entirely on a question that, as of today, nobody can definitively answer: Is the bill worth it?

Smith considers the period before a smarter than human in all ways artificial general intelligence:

But until we reach that point, it’s a nontrivial task to think of business models that could be fully automated even with an AI that can’t yet do everything. That’s going to be hard! If I had any good ideas for how to do that, I’d go become a billionaire myself.

At some point, though — maybe in the very near future — people (assisted by AI) will come up with those revolutionary new business models. At that point, tokenmaxxing will suddenly become a lot more economical, and Anthropic — or whoever has good coding agents by that time — will stand to make untold amounts of money.

These are good perspectives, but I can improve upon them to help understand the dynamics of AI usage. First, I’m from the software industry, which is at the center of the maelstrom — coding is now the single largest category of token usage. I can describe in more detail what developers are actually doing with these tokens, and their motivations. These details are important. Without them, a lot of valuable work remains mysterious, which invites doubts, such as “is this worth it”, or “do we need more software”?

Second, I’ve spent a while thinking about the adversarial dynamics of some AI usage, since first writing about it last year. Those dynamics are key to the questions both writers leave us with. Adversarial usage doesn’t produce the social value we all seek. It is not the only driver of AI usage, but when it is a driver, we should be asking, “is this worth it”?

Both writers are aware of an important detail, timing, which explains many misleading observations. With the addition of a deeper understanding of software development, and that model for separating zero-sum jockeying from the creation of social value, we can recognize events along the timeline with more accuracy.

Token usage, like human labor, can’t tell you progress. Its best analogy is effort. If you want to understand the effectiveness of effort, you want to know how it’s being applied. Different applications correlate with different outcomes. Since you can’t fast-forward to the results, this is the best immediate categorization you can add. I call this categorization, the Ingenuity Matrix, describing the scope and social alignment of token usage.

Some token usage goes nowhere by design, some burns down a backlog of long-deferred work, some is zero-sum jockeying. A slower, quieter share is the significant work that actually changes lives. Sort the usage that way and the “is it a bubble” question dissolves into a more useful one — what’s being built, what events can we expect along the path, and what risks and opportunities come with each set of events?

Background

Before we start into the model, understanding the two terms behind the narratives is useful. This will also be useful when reading general news on the topics. The narratives on these conflate multiple meanings, and smuggle assumptions. That ambiguity can support misleading narratives¹.

What is tokenmaxxing?

Tokenmaxxing refers to two things. First, it refers to companies’ creation of “leaderboards” tracking employee AI usage by metering tokens. These leaderboards might be informal, but there’s often an implied assumption that high usage is rewarded, and low usage risks consequences. Sometimes that’s explicit. Ostensibly the justification is to incentivize experimentation and overcome inertia. In addition to simple inertia, many companies started with restrictive policies discouraging AI usage that they needed to counteract.

The second meaning focuses on what happens when leaderboards encourage AI usage, but do so in unproductive ways. Some employees respond by trying AI more and doubling down on things that work. But they may also create or continue unconstructive habits, for no reason other than they generate tokens. Individuals have described such practices anecdotally.

In this dual definition, when companies tokenmax, they encourage both the good and the bad. When individuals tokenmax, we talk only about the bad. The most extreme tokenmaxxing isn’t ingenuity that misfires — it’s intentional waste. The intent isn’t to do work; it’s to appear to have done work.

It’s not hard to see how that type of usage leads to a narrative that it’s all a sham. But we should remember, what we have is anecdotes. While it’s certain that some waste is occurring, it’s hard to gauge. Anecdotes are sparse, and for good reason. Admitting to it, would be admitting to willfully ignoring the employer’s best interest in productivity. That would carry consequences if a manager discovered it and wasn’t interested in joining the deception.

But separating waste from sincere-but-unsuccessful experimentation requires details that simply aren’t available at scale. What we can say is that the organizations running leaderboards are making a deliberate bet: they’re buying a pile of unaimed experimentation and some willful waste, in exchange for a fraction that matures into something real — durable skills, a useful tool, an opportunity nobody had time to chase before. Whether the bet pays off, only time will tell. But the structure of the bet — accepting near-term waste to fish for longer-term capability — is something we should predict and model as a mix.

What are subsidized tokens?

Subsidized tokens can refer to three things.

The most common usage focuses on two billing models. One is metered, usage is measured and billed per token, at prices like $5/million tokens. The other is by subscription, for example $20/month. Subscriptions typically have usage limits, but in most cases, fully utilizing a subscription’s limits yields a per-token cost below the metered rate. In addition, loopholes existed, allowing usage far below the metered rate. Users who used their subscriptions heavily enough to get that benefit were labelled as subsidized. That’s a simplification though, as it could be a lower profit margin, not subsidization.

The second usage focuses on free tiers. Free tiers have restrictive usage limits, but with no revenue, they are clearly subsidized. Free users heavily outnumber paid subscribers. Across providers there are at least a billion free tier users, while paid subscribers would be below a hundred million.

The third and final usage translates the unit economics of metered usage into the underlying costs that model providers pay to compute providers, which pay for chips, power, and other infrastructure. The question the subsidy narrative is really asking is, are the unit economics of AI usage sustainable? Or are they a short-term attempt to grow usage, the end of which results in higher prices, and pulling back from usage that’s no longer economic at the higher price point?

It’s an interesting story, but it’s almost worth ignoring. The efficiency of AI is increasing quickly, driving unit costs down. If prices rebound, unless the rebound is something like 10x, they’d soon fall again. The reason they can’t be ignored has little to do with a long-term trend, but everything to do with the short-term viability of the financing of AI investments and presumed valuations. A company trapped in subsidizing while a competitor is not, is going out of business quickly. This pattern repeats at each level of the AI value chain.

Why Bubble as an analogy is over extended

I said in the opening that using “bubble” as a metaphor for the AI industry smuggles in two assumptions. A bubble is so commonly used to analogize industrial revolutions, that we fail to reflect on the limits it has as an analogy. One mistake it leads us to, is the belief that there’s a soap bubble floating in air, and when we prick it with a pin it will pop, and evaporate. This does a poor job of explaining reality though.

We might limit our imagination more effectively by replacing the soap bubble with an inflatable dome. Whether this stays inflated depends upon the balance of air entering and exiting. Inside this dome, we’re constructing something durable, but it would be a challenge to do so with the dome weighing on top of us. We need the air to keep the dome’s ceiling from impeding our construction, and if it deflates it will probably ruin any half constructed structures. The stronger completed structures can sustain the weight of a deflated dome, but will struggle to conduct any additional construction.

If you want to think of the social support for a system, which supplies the air to keep the shell inflated, as a bubble, that’d be fair. This can evaporate with a bad news story, or some other form of social contagion. That social support is what replaces the air that leaks out. We’ll discuss the leaks later. Some are necessary, some are not. But replenishing the loss is unavoidable.

It’s useful to remember that in this analogy, deflation isn’t free. Something will remain, but the damage to unfinished construction is real. Careers are an obvious example of the consequences. When companies downsize the skills, connections and tacit knowledge built to support growth get stranded. If people move on, they may never come back. And besides, they are people and the disruption to their lives matters too. Projects also take a hit. Some projects may be zombies, shambling along with an unsound structure that will never be completed. But the forces of deflation aren’t so selective, and promising work is wiped away as well. Many projects that stop work during periods of tightening never start again.

A second flaw in the analogy is as a singular structure. Not only are there independent structures being built within, there’s not a single dome. There is a primary dome, where the model providers, GPU manufacturers and designers, and much else reside. But AI is also working to serve many different industries, and we shouldn’t assume a shared fate between all those efforts. We do want to pay attention to software development, because it represents such a large fraction of current usage. But software development itself isn’t an end of its own, it serves other industries. If AI is effective at helping some of those, and less-effective in others, this doesn’t establish a shared fate. It is only those cross-cutting effects that affect all software development that would carry that risk.

For the most part, those outside of software development aren’t going to understand those cross-cutting effects. I’ll highlight some of those details here, as they should be relevant to anyone interested in the immediate future implications of AI.

Why the public has a poor understanding of software development

The wider world has never shown broad interest in learning what software developers do. Compared to other professions like police, soldiers, doctors, lawyers, musicians, writers, journalists or even criminals. Without that interest it’s unlikely to learn the inner workings of the profession.

Media portrayals of software developers are rare and rarely accurate. The most common portrayal is the “hacker” who mysteriously takes control of computer systems in a few minutes with no preparation. Not only is that a poor representation of a real hacker, it tells you nothing about software development overall.

Note: Scales are different per panel, programmers at 1x10-6 are 10x less frequent than actresses at 1x10-5, or 300x less frequent than doctors below at 3x10-4 (below).

Source: Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis, Baruah S, Somandepalli K, Narayanan S..

When software is “done”

If you come from outside the software world, you’d be excused from thinking of software development as building new software. In reality, this is a modest part of software development. Maintaining software, deploying software, and operating deployed software all represent larger segments than new software. All said, new software could be as small as 20%.

Noah makes a tentative argument that “The world may already have most of the traditional software that it needs.”. Noah’s aware he might be getting this wrong, and indeed he does. It does take an immense amount of work to keep sites running. AI is being used here, but it started later than its use to create new software. It’s not too hard to guess why. Creating new software is low risk comparatively. Like everyone, trust of AI has been a process. Software maintenance and operations themselves rely on significant “tech-stacks”, which have to be modified before you can even attempt to use AI to make a site more reliable in a meaningful way.

The number of software releases for security, operational, monitoring and development oriented features has been significant over the past year. Many use AI. Probably many others were built using AI.

Should you expect faster load times and higher reliability? First, would you really know? These have been improving for years, yet the general public rarely comments upon it. Mostly the only comments are those times when something does fail.

I’m not holding this data above as proof that AI has improved reliability. These improvements are more likely the result of conventional engineering, some started years before the results. The results that are “AI” based, are the result of the “machine learning” form that predated the architectures for Claude, Gemini and ChatGPT.

The point is, Noah (and you too) probably aren’t a sound judge of whether improvements are occurring unless you take the time to gather data. From my own knowledge, I know most AI based improvements are still in the early phases of adoption. But the average person shouldn’t expect to have an intuitive grasp on this. We’re bad intuitive judges of background effects like this, where we have to compare changes over time of non-continuous events. We can recall the last event, and the last change, but we are just as likely to draw a pattern from a recent reaction, than from an accurate history.

Why “tech-debt” comes first

You’ll find an interesting pattern that I’ll get into later. The first work to be done is the “shovel-ready” work. It’s easy to generate a prototype for some random idea, but rarer to have a great idea that can go from ideation to production quickly. AI does speed that up. But it doesn’t speed all work up.

With that in mind, provide a tool to a software developer, and they’ll have a long list of things they wanted to do, but haven’t had time for. Our general term for this is “tech-debt”, but realistically, it also includes half-baked feature ideas, or features that were sound but never made the cost-effectiveness cut. This list predictably contains a lot of non-amazing things. If they were amazing, they would have made the cost-effectiveness cut the first time. But AI does give you a reason to go deeper into that marginal backlog.

Security as a priority

I should also mention security here. Security is extremely important to the operation of software. Failures of security are nearly the worst thing you can imagine. This applies to all phases of software: development, deployment, and operations. It’s tempting to think of security as something you simply develop. But in reality that’s just the first step. A significant failure in development is likely to lead to a significant failure later, but it’s not destiny. You can layer protections to mitigate a development failure during operations. You have to do this because there are development failures you don’t know about. And more importantly, even a soundly designed and developed system can fail if not operated properly.

A lot of time and money is already spent on security. It’s never been the case that it hasn’t been a priority. You can find cases where it wasn’t a high enough priority. But it’d be a stretch to suggest there was a case no one cared. Whatever the priority, there is a limit, a cost-effectiveness barrier where one of the stages of development could have achieved more with more inputs. The introduction of AI changes the math on that barrier and makes many things practical that were impractical.

Security has another dimension too, which is that in addition to AI altering the developer’s cost-effectiveness equation, it does so for attackers too. This creates another incentive to burn down the security backlog. Security can’t wait. And so a lot with good cause, a lot of AI based productivity is going into security efforts.

This isn’t an effort that’s particularly visible to the outside world. What the outside world knows about it comes mostly from stories, not direct experience. When developers patch security holes, their intent almost always is to not change the user-experience. When that is the intent, it’s a slower process, because it requires educating users about new security mechanisms they need to participate in. Because that’s such a difficult thing to do, security teams have a very strong preference toward solving problems themselves without involving the users. It’s not always possible, but 90% of security efforts are invisible to users, and the next 9% are delivered as patches users see installed, but don’t pay any attention to.

The ingenuity matrix

I said in the opening that token usage is like effort: it tells you activity, not progress. To get from effort to expected outcome, you have to ask what the effort is for. Two questions do most of the work, and together they form a grid.

The first question is social alignment. Does the work create value the world didn’t have (positive-sum, pro-social)? Does it merely move value from one party to another (zero-sum, non-social)? Or does it destroy value — burn resources, or actively harm (negative-sum, anti-social)?

Alignment can be informed by our guesses of actors’ intent, but it’s not dependent on it. Our best bet is to act as an outside observer, guessing at outcomes. I don’t want to overcomplicate this though, this is estimation after all. Some significant pro-social value sometimes arrives from someone tinkering purely for fun. The social alignment is still recognizable from the outside, even when the actor wasn’t aiming at it.

The second question is scope , how far the work is reaching. Significant work aims at a real leap. Simple work aims at something bounded and modest. Naive work isn’t aimed at a productive outcome at all. Here “naive” describes the absence of a useful target, not the absence of a motive. Intentional waste is naive in this sense, it produces nothing of value, even though the person doing it has a very clear motive.

What you’ve just toured, security patches, reliability work, performance and cost tuning, is real value, almost all of it invisible to the people who benefit. Nearly all of it lands in a single cell: simple, positive-sum. It’s illustrative that so much of what is immediate is within simple or naive ingenuity. The first things individuals use AI for aren’t the significant ones. It’s the modest, shovel-ready, often-unseen things.

Map the rest against those two axes and you get an ingenuity matrix :

Ingenuity Matrix: Scope and alignment to predict economic outcomes

Negative-sum is not hypothetical, it connects back to security. The same drop in cost-of-effort that lets defenders finally burn down the security backlog also lowers the attacker’s cost. AI-assisted cybercrime is simple, negative-sum ingenuity, and the prospect of AI-scale biological or infrastructure attacks is the significant version. A large share of the invisible defensive work isn’t optional improvement, it’s the response to an adversary. The result is effort that is no longer avoidable, but also hidden, which delays the visible gains we’re watching for.

Significant non-social ingenuity ends empty. Non-social work can seem significant when under development. But one of two things happens. Either the work ends up leaking into pro-social, or anti-social accidentally, or it is copied and becomes trivial. Significance and neutrality are generally unstable.

Naive ingenuity is where the most visible tokens are burning right now and the least is being built. Failed experiments and aimless prototypes aren’t worthless — they build skills and occasionally surface something real, which is the option value the leaderboard bet was buying — but as a category they go nowhere by design. Because naive usage is so voluminous, and personal, it’s the most visible to the simplest forms of observation. That helps it dominate the “is this all a sham?” narrative.

Simple ingenuity has significant usage, but is quickly forgotten. The high volume usage is generally operationalized, contributing to security, reliability or operational efficiency. It’s soon forgotten, as it becomes a background effect. It doesn’t have the humorous, villainous story of tokenmaxxing. It doesn’t receive the personal promotion of the latest experiment.

One of the hallmarks of simple ingenuity, is it could be described as a backlog. The work may have been identified as desirable a long time ago, but with other competing priorities, it wasn’t prioritized. It may also not have been cost effective. One of the changes that AI brings is a change in cost-effectiveness. This activates this backlog, and you should expect early effects to burn this backlog down.

Simple ingenuity comes early and makes existing work more efficient. Sometimes this will show up as measurable revenues, but much is internal to companies. In that case it’s the token usage, the lower labor costs, or the higher quality that are the observations.

When AI enabled workers have a clear backlog, efficiency gains will flow into simple ingenuity to burn down the backlog. If the backlog results in priced or measured output, you’ll know.

Significant ingenuity will take longer to be identified, developed and deployed, especially the pro-social variety. The economy will reuse freed labor to create more value. That won’t happen immediately, as it may wait on hiring processes, training processes, or even the formation of new companies pursuing new products or business models.

Timing

While the development process is accelerated, the identification process retains most of its bottlenecks. Optimism may accelerate it. Idleness may accelerate it. But optimism and idleness may also flow into naive ingenuity, pursuing trivial goals without positive utility. There is a blurry area where naive ingenuity is experimentation. It may fail, but its failure may be necessary to build skills or discover significant opportunities.

At some point, a few things start to coincide. Naive and simple ingenuity will have built skills, ready to be exploited for realizing significant ingenuity. The backlog’s distraction fades as it burns down, and a new equilibrium raises the incentive to chase significant work — significance always carried more reward, but also more risk. But as cost-effectiveness decreases deeper into the backlog, avoiding risk becomes less attractive. All of these, in addition to the passage of time, predict a future wave of significant ingenuity that direct observation of measurements would fail to predict.

New output

Most of what we’d recognize as new output is significant, pro-social, and lagged. These are the life-changing things, and they’re the hardest to forecast. Your best guide might be a science-fiction novel, but of all the futures sci-fi writers have imagined, which do you bet on? Like flying cars, some things that look a step away stay out of reach far longer than expected.

It would be a mistake, though, to generalize from the failed predictions to all predictions. In many ways today’s information world already outruns older sci-fi imagination — the 1987 Star Trek: TNG depicted computers far beyond the 1966 version, and on the information front we’ve roughly met the standard it set for the 24th century already. The significant wave is hard to time and easy to underestimate at the same time.

What to expect

We should expect the AI industry to experience some pullbacks, then continue on. Whether this ever meets the bubble narrative is uncertain. I’m skeptical. Many pullbacks will be met by other accelerations. One experiment fails, another scales.

There isn’t one bubble, ready to pop, but multiple domes. Each industry, each set of users finds their value. While software remains so dominant, a failure in the software use case could be dramatic, but much of it is boring simple work that will continue to be automated for some time yet.

Much of the immediate term work is going to focus on the simplest, most invisible aspects. We shouldn’t discount the value there. Where it’s defensive, answering the negative-sum, like security, it has to be done. Where it’s part of more normal systems, it’s freeing resources, and developing skills and experience that will fuel more significant ingenuity in the future.

You do have to wait to see world changing effects. Software, as a model for implementing a workflow, will remain, and the general skills of software developers will be critical to this. Lines will blur, people will cross-over the lines, but ultimately the concept of software will continue to exist.

If the software dome does collapse, it will create structural damage, like all such events. Failed companies, layoffs, abandoned projects. Resources for naive experimentation would evaporate, and companies would proceed more cautiously. But a structure will remain. The burned-down backlogs that don’t un-burn, the skills that accumulated, the efficiency that keeps paying out, and the significant work just beginning to grow.

So, are we in a bubble? Will users and companies pull back on token usage, looking for value, discouraging wasteful tokenmaxxing? Will they react to pricing changes from model providers that close subscription loopholes that allow token usage in excess of what the same money would have bought per token via API? Yes, they will, but will that cause revenue drops that deflate the dome?

I don’t think so, there’s enough pending and developing work to fill the gap. Even if the significant ingenuity is still developing, the simple work is sufficiently valuable and important. But maybe those dynamics will return next year. If compute providers continue yet more expansions, they still might find them getting ahead of demand. There’s a lot of history to be written here. I’d just be careful about writing the ending first.

Sources

Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis; Baruah S, Somandepalli K, Narayanan S..
State of AI, An Empirical 100 Trillion Token Study with OpenRouter; Malika Aubakirova,Alex Atallah,Chris Clark, Justin Summerville, Anjney Midha
The AI Boom Has Entered Its 'Wait, Is This Worth It?' Era; Derek Thompson
How much more software do we really need?; Noah Smith
State of AI in the Enterprise The untapped edge; Deloitte

Original Posted on Substack as Are We in a Token Bubble?

As an example, Noah Smith quotes a commonly quoted study on tokenmaxing that claims diminishing returns to token usage, but presents data that should be interpreted as the opposite. In their description, they compare the number of tokens used to create PRs, and the costs of those tokens. ↩

To evaluate whether that spend is worth it, we joined token usage data with actual developer output, measured in merged pull requests.

Over the course of Q1 2026, developers in the bottom 20% of token spend used only about three dollars’ worth of tokens for the entire quarter and shipped an average of 11 merged PRs. By comparison, developers in the top 20% spent $1,822 over the same period and shipped 23 merged PRs on average.

In other words, significantly higher token usage does lead to more output, but not proportionally. The cost per merged PR increases from just $0.28 in the lowest usage tier to $89.32 in the highest.

More tokens means more output, but at a much higher price per unit.

But if you’re comparing costs, the correct comparison would include developer time. If we take a conservative cost of $10,000 / month for a developer the calculation we get is:

Low token group: ($30,000 + $3.08) / 11 PRs ≈ $2,727/PR

High token group: ($30,000 + $2,054) / 23 PRs ≈ $1,393/PR

There is a sense in which you could use this data to describe diminishing returns, but it’s not in the realm of cost effectiveness. If someone proposed that development was accelerating exponentially in the way that token usage is, they’d be wrong. You cannot scale development at the speed of tokens because it is still dependent on developers.

DEV Community: Ryan Baker

Trusted Deployers

Jurisdictions

Open-Weight Models

Trusted Deployers

Privacy

China

Conclusion

Related Articles

Enough Reason to Act

A Title More of Us Could Agree With

Mistakes of Detail

Presume we all have to die

Presume false competition

Presume cybersecurity and loss-of-control are the same

Those guys are weird

Mistakes of Form

Reasoning from cynicism

Reasoning from markets

Shorts don’t tell you someone’s commitments to action

Insurance rates are also indirect

Markets accumulate insight, they don’t produce it or disseminate it

Demanding yes-no answers

Letting forcing functions win

The Case for Action

How bad would it be if we did nothing?

What are we really giving up?

What should we be sure we don’t give up?

Learning More

Conclusion

Related Articles

Every Reward Bends

Bountymaxxing

Resolution is the goal

Normalizing Outcomes

Minimizing Distortions

Credit

Ending

Management is active. Fire and forget doesn’t cut it.

Where OpenAI wasn’t watching

Reasons for optimism

Defending the indirection

Watching your own organization

Conclusion

Related Articles

Why It Hasn't Happened Yet

What I won’t cover but is still important

What protections are holding up today?

What puts protections at risk?

Why should we think security is at risk?

Why the equilibrium broke

Keeping malicious intent and capability apart

The technical picture

Vulnerabilities and escalation

Layers, segmentation and zero trust

Fixing vulnerabilities

Command and control (C2) and monitoring

What to do now

Is your organization ready?

How does AI assist in these responses?

Do your research

The attacker-defender balance

The worst case

Related Articles

Nobody Was Watching

Two labs skipped monitoring on their cyber evals. The same week, a hundred companies signed a letter supporting deployments that can't be monitored.

The Monitors Were Off Again

A Nuanced Position, Poorly Received

The Risk of Open Weight Models

Cynicism as a Tool, Not a Verdict

A Set of Contrasts

An OpenAI Model Escaped Its Sandbox. Where Was the Observer?

OpenAI accidentally hacked Hugging Face. A basic layer of defense appears to have been missing, and no one has explained why.

Where was the Observer?

Conclusion

The Perpetrator Is Not the Tool

What should we call slop?

Laziness

Manipulative

Costs to the Audience