DEV Community: Ryan Baker

Why It Hasn't Happened Yet

Ryan Baker — Mon, 17 Aug 2026 13:00:03 +0000

When I heard about the recent hacking events starting at OpenAI, Anthropic and AISI, I thought it was a wake up call in more ways than one. I’m finding that not everyone agrees. One reaction does not include the sense of urgency to prepare for AI-based attacks. It’s my reaction that there is a need for an all-hands-on-deck response from the rank and file of developers, software companies, and companies dependent on software. That response should improve operational security postures using every tool available.

At first this is shocking, but with a little reflection, I see how the full ramifications haven’t sunk in. As a news story, it’s led by one attack, and little damage occurred. While it would be hard to miss the tinge of panic when reading coverage, people may have become dull to this from persistent rounds of security coverage, often from security sales, that have always hyped risks.

This is different. The attacks were both demonstrations of carelessness, and demonstrations of the capabilities of powerful AI models. Damage was minimal because these attacks did not combine malicious intent with those capabilities. The attacks occurred with the more benign intent of a model trying to pass a test. And this is the least effective that models will be.

As models improve, defenders and attackers will gain access to new capabilities. How labs manage this is important, but so is how we adopt those capabilities and put them to work. The status quo is not stable. The world hasn’t changed too much yet, but it will.

I thought I should go into that in more detail. What are the things protecting us today? What are the risks they won’t hold? What can we do to prepare for that? The answer I’ll offer can be summarized as using new techniques to accelerate old techniques.

What I won’t cover but is still important

The Hugging Face event isn’t just about the urgency to prepare for AI-based attacks. It was also about the type of testing that the AI labs were performing. They removed protection layers and didn’t follow-up with monitoring, which I’ve already written about in Nobody Was Watching. It also demonstrated unaligned behavior, which I’ll suggest What Happened: OpenAI and HuggingFace as a source to learn more about.

This type of work needs to receive high care and caution. Monitoring applied to this testing was insufficient. We should not assume when we explore new territory that the exploration is safe. Even if we’re exploring to understand safety, we should be monitoring the exploration. That should be obvious, doubly so because how effective is the exploration if you’re not monitoring it? Certainly less than if you are.

I don’t cover those here not because they aren’t important, but because they are, and they deserve their own space.

What protections are holding up today?

Intent: In the case of the Hugging Face event, the most effective protection was the lack of malicious intent. We can’t expect that to hold, as it never has. But other protections are about preventing this type of intent accessing capabilities.

The most unusual aspect of the recent events is that the AI involved was accidentally persistent. That’s a warning sign for sure, but its probability of great harm alone passes through another layer: intent. None of the agents participating in the Hugging Face hack would have what we’d call malicious intent.

Training Alignment: Lesser models have lesser capabilities due to less advanced training. Training also applies alignment to the models. Roughly put, this causes models not to “want” to do harm. A bit more specifically, this both causes models to not spontaneously respond in ways that would invoke harmful actions, and to refuse to respond to requests that look to be trying to create harm. While this type of protection is important, it’s also weak.

Security Classifiers: When models are deployed behind APIs, it’s standard practice to use a “classifier” to evaluate the inputs and outputs. If they appear to intend to cause harm, they refuse the request. During the recent hacking events, those were disabled intentionally. This was to enable important testing. While doing that without monitoring was reckless, we should remember that these are enabled for external deployments, and thus contribute to keeping malicious intent and capabilities from merging. The ability to monitor and tune security classifiers is greater than training alignment. Training alignment is something you have to get right during training.

Abuse Detection: The major AI labs all have documented evidence of attackers trying to use models for attacks. Security classifiers deny a lot of this. They also take other measures to identify attackers and limit their access. There are evasion techniques here, so it’s not as simple as flipping a switch, but the efforts they put in do add complexity and costs for the attackers.

Conventional Security: Models will only be able to perform an attack that is below their capability level. If the infrastructure they are attacking is better secured, it raises the complexity level. Raise the complexity level high enough and the attempt will fail.

What puts protections at risk?

_ Jailbreaks _ are inputs that defeat training alignments. This is why training alignments are weak protections against malicious use. These are usually partial and it’s accepted that partial jailbreaks exist. Universal jailbreaks are less common and more concerning. Even with the existence of jailbreaks, alignment training is very important. Jailbreaks tend to make the model a little stupider in the process. That’s all quite complicated, and there’s some decent papers on it, but the analogy would be that trying to confuse the model leaves it confused. A clear-headed model, with a clear-headed goal would be more dangerous than a confused model. Since one of the main goals is to prevent malicious intent and capability from connecting, this helps.

Attackers can try to evade security classifiers, by hiding their intent. Keep in mind that when you hide your intent, a model is less likely to be able to autonomously pursue your goal. If you can’t state your goal without being rejected, the effective capabilities are lower. Plus it’s a lot of effort to evade in the first place.

Open-weight models allow anyone a copy of the trained model, rather than access via API. Since security classifiers are part of deployments, they aren’t part of open-weight models. If I download an open-model and have hardware under my control, there’s nothing forcing me to install them. A deployment could add them, but we wouldn’t expect an attacker to do so. Attackers thus get a free pass on security classifiers from open-weight models. That leaves the training alignment to overcome.

Attackers can also apply additional training to open-weight models that “untrains” their alignment training. Like jailbreaks, known techniques degrade the model somewhat. It’s an open question on how hard it is to restore that initial level of capability. This “dumbing” effect is less significant than that from jailbreaks.

Continued training does require infrastructure, it’s like the original training. But it takes many fewer iterations than the original training, so even if it’s running slow, it’s possible to complete without a ridiculous amount of infrastructure.

_ Continued training: _ If open-models continue to be released, at higher and higher levels of capability, at some point one will be as capable as the one that orchestrated the Hugging Face event. Today, the best ones are 8 months or more behind. Add some extra buffer for the effects of untraining, and it should be clear that unless something changes, an attacker will in the future have access to a model capable of a hack of equivalent complexity to the Hugging Face event.

But should those buffers be spent, for an attacker it’s just a matter of injecting the malicious intent, which could now happen at scale, and it would be quite difficult to stop that from scaling up.

We should worry about closed models too, as their training is the most advanced. One risk is leaking the weights, making them effectively into open-weight models. A lesser, but still relevant concern is bypassing the security classifiers. The reason this is lesser is that an appropriate reaction to a failure there would be to disable all access to the model creating the risk.

Why should we think security is at risk?

As a single piece of evidence, the recent events are the best there is, but the evidence that powerful AI models were gaining these capabilities didn’t start there. Conceptually, the idea has been around a long time. And evidence that we are approaching a pivotal moment, was clear to me in March when I wrote Security Can’t Wait. But what was the level of conventional security needed to avoid a compromise? Hugging Face demonstrated a level that was insufficient. We should all be careful about assuming we are above that level. Ideally, you should assume you are worse than you think, and thus as a goal, strive to be several steps further than you need to be.

The security of software systems has always been a bit of a mirage. The mirage made them appear invulnerable. If you worked in security, you’d know this wasn’t true, it was a constant battle with many compromises and ample mistakes. What’s more, while there were many lazy or reckless mistakes, many other mistakes were of a complexity that labeling them as mistakes no longer fits. It’s a bit of a pipe dream to eliminate all the lazy and reckless mistakes, but that is almost realistic compared to eliminating all the complex vulnerabilities.

Occasionally signals that systems were not invulnerable would filter through to the public, but overall the weakest point in these systems was often the public itself. You’d blame yourself for installing that malware, or having an insecure password, or going to “that” site. That’s not always fair, security professionals know they need to take responsibility for not just what they do, but for helping users avoid harm.

Software systems have survived by having multi-layered defenses. While every layer has mistakes that could be exploited, multiple layers raised the difficulty of finding a path through these layers. Finding the mistake in a second layer is harder because the compromise of the first layer is partial. The typical analogy is Swiss cheese. Swiss cheese has many holes, and in a thin slice, you’ll see many pathways through. A thicker block has some of these dead-end. The remaining complete paths also won’t simply be clearly visible. A thick enough block might have no complete paths, but neither you or the attacker know that until fully exploring. The Swiss cheese analogy breaks down a bit here because it’s hard to imagine a piece of cheese as complex as modern software, but the analogy has taken you a little closer to the truth.

Why the equilibrium broke

When extended far enough, the Swiss cheese analogy resembles a complex multidimensional maze. There is likely more than one entry point and more than one exit. It takes effort and time to explore the maze, and if defenders are doing their job, there are hazards as well. Take too long and the walls in the maze may shift, closing discovered paths.

When security succeeded – and let’s be honest, all failures aside, it succeeded well enough for us to rely heavily upon it – it succeeded by having a complex enough maze. By raising the required time investment of the attacker to a level that was no longer attractive, there were no attempts persistent enough to succeed. Sometimes this succeeded on the principle of the bear in the forest; don’t outrun the bear, outrun the other targets. But even then, there’s a question of how many targets the bear wanted.

Adding these layers wasn’t cheap though, each requires investment and maintenance. With enough time, we might be able to make them impervious, but that time might scale toward infinity, and become impractical far earlier.

This is where the first risk from sufficiently capable AI emerges. The persistence of an attacker with sufficient AI capability is higher. Mazes that were sufficient before become inadequate, as no layer is deterrence enough to defeat persistence.

There is a lesson in here though, and it is that this isn’t a binary yes/no outcome, but still a matter of balance. If we invest more, or if AI multiplies our investments, we reduce the vulnerabilities at each layer, making them more scarce. If you can make your maze more complex, you can reach the same equilibrium.

Keeping malicious intent and capability apart

AI is persistent in both the hands of a defender and an attacker. But it’s not automatically so. We haven’t seen malicious intent and powerful AI fully merged. We got a warning sign and we should take it. That warning was in the form of naive exploration, but we should not take this for granted. If you can deny attackers easy access to AI capabilities, this helps maintain that equilibrium.

If we manage to restrict access, we still need to take action. It would be reckless not to. We don’t know that restricting access will be enough. We don’t know if it will succeed long-term. That said, we would be naive to not pursue it, even the most minimal versions where there’s only a 6-month capability gap. To not do so would ignore that many organizations will fall behind. Some leaders will disbelieve this call for urgency, and fail to set them on the right path. Others will fail due to internal issues, coordination problems and an inability to respond to necessary change. Hell, some were going to fail overall anyhow, so we can’t be surprised if they fail at this too.

Cybersecurity’s main concern will not be rogue agents on training jobs. Naive exploration, even with safeguards removed, is less likely to end in harm than intentional use for malicious intents. There’s some big topics there, but it’s not the reason you should be hardening your own security. Those topics depend on the labs, and the general level of security, of which your own is a small part.

What you should want, that is common to those topics, is to keep malicious intent and capabilities from connecting. That’s an ask we should have of the labs. We should be willing to make those asks in more direct ways than just asking. It’s nice that they’ve done some of this on their own. We can’t take that for granted, and I don’t think it’s enough.

We know there are people with malicious intent. A small number want harm, just for harm’s sake. A larger number are willing to cause harm in pursuit of their goals. Those are direct paths toward negative outcomes and we need to pay attention to them.

It is important to withhold capabilities from both of these groups. It’s important to identify and isolate members of both groups. Both tasks are hard and carry risks we’ll make mistakes. The importance of avoiding those mistakes makes the challenge that much harder.

But like the attacker navigating the maze, so can we. We can watch our mazes. When we find someone trying to navigate it, we would track that person back to the source, arrest and imprison them. We are not so naive to believe this would work in all cases. Not only would we fail to find some, but we don’t have a system that allows us to imprison all of them. Some will be outside cooperative or functional jurisdictions. Even within cooperative functional jurisdictions, we want to respect our own laws about proof and certainty.

With this in mind, we should be looking for other better balanced opportunities, such as denying access to tools. Non-functional jurisdictions are unlikely to be able to create those tools themselves. If we control ours, we avoid helping them. Non-cooperative jurisdictions are more complicated if they are functional enough to build their own tools. More so if they are functional enough to steal ours.

To stop talking in generalities, primarily who we mean by non-cooperative, but functional is China. One difficult, but obvious path is for them to become cooperative. If you assume there is one and only one step, directly to full cooperation, this will seem even more difficult than if you look for more limited forms of cooperation.

While that can feel like a serious compromise of principles, from a realistic, pragmatic point of view, that type of action, diplomacy, has a long history that has been celebrated far less than it deserves.

The technical picture

Discussions about AI in cybersecurity often are extremely technical, or very high level. I’m going to try to thread that needle in a different way, and talk about some specific aspects that often go unmentioned.

Vulnerabilities and escalation

What is usually mentioned is finding vulnerabilities. All software has code, often transformed, “compiled” from one form to another, but ultimately it remains a list of instructions. Not a simple start to end list, but a list that loops back upon itself, jumps around, reuses, and does all of this dynamically based on inputs. Generally, it’s designed for the purpose of translating certain inputs to certain outputs. That’s the expected behavior of a system. But also generally, the methods of sending inputs don’t constrain those inputs to only what’s expected. It might be hard to define this precisely. Flexible software is usually more useful than inflexible. So it’s common to find it’s possible for inputs that were never tested for.

Sometimes the effect that comes from one of these inputs allows for something not planned. In the most serious, they allow “escalation”, where the input ends up having a type of control that wasn’t intended, and is often dangerous. At this point, an attacker can start executing their own code, and acting upon a plan. We’ll come back to the attacker later, but what you should recognize from the above is what code vulnerabilities are. There are other types of vulnerabilities, like misconfigured permissions, leaked credentials. Ultimately the underlying story is the same, a mistake lets the attacker take what could be a complicated route to doing something they weren’t supposed to be able to do. If that’s not their end goal, they look for ways to use this newfound access to do more until they reach their goal.

The attacker’s goal is something you didn’t want them to do, or they wouldn’t be an attacker. But in addition to the costs of them achieving the goal, they can create side-effects that break things as they go, even if their goal is to steal information or make threats.

Layers, segmentation and zero trust

Defenders long ago realized that if one vulnerability was enough to bring them down, they’d lose this fight. The response here was to build layers of defense. Sometimes this meant adding additional layers outside, and sometimes it meant adding additional layers inside. The outside layers reduce the inputs that reach inner layers, and the outputs that can return. While this isn’t specifically anything that the software at the inner layer couldn’t have done, it has the effect of simplifying. The outer layer generally has less complicated responsibilities, and so errors there are less likely. It may be capable of detecting common input patterns that attackers use, and refusing to pass these on. Its design makes it clear what is and isn’t allowed in, which can be validated against what must be allowed in, and what isn’t necessary, and thus adds risk of misuse without practical value.

Additional layers inside perform these same roles. When everything is working normally, you expect only expected inputs to pass between internal systems. But if an attacker has achieved the ability to execute their own code, those inputs stop obeying those rules. By creating segments where the input that crosses them is less trusted, that escalation is less likely to be able to connect to the next step required to achieve their objective.

Modern software often has multiple layers itself, for practical reasons in addition to security reasons. For most systems the most sensitive part of the system is the data layer. If an attacker wants your data, this is an excellent place to get it. Even more importantly, if they want to do harm you can not undo, destroying your data would be an excellent way to do that. Other layers are more likely to be “stateless”, in that they are just code and configuration and easily replaceable. Data is not easily replaceable. Even with uncompromised backups, restoring data takes considerable time.

There are worse things than having your data compromised though. In systems that coordinate with others (for example finance), if transactions leave the system, unwinding those is even harder than restoring data. And most of all, anything with physical consequences, as the physical world has no undo feature.

Often the first place we think of layers is as part of networks that connect servers (virtual or physical). But the bigger goal is isolation, and layers are a single dimension of isolation. Much more is possible. Higher degrees of isolation offer new opportunities to break escalation chains. A model for extending isolation further is zero-trust. Adopting a zero-trust model can require redesign, but the payback is access to new defense in depth techniques that are both more effective and efficient.

Fixing vulnerabilities

The thing to remember about these extra layers is that they are also software. They may be designed in a way to make vulnerabilities easier to discover, and configuration easier to audit, but they will still have vulnerabilities. Using those to evade the limits they were intended to provide is where capabilities get worrisome.

That said, this is where defenders start gaining advantages from AI. If the vulnerabilities in the software implementing these layers are found, they can be patched and removed. At any individual layer, you do have the challenge of needing to find and fix every vulnerability that an attacker could find. But across many layers, it can be the case that one successful layer breaks the chain. I’m simplifying here, because this part does get complex with the existence of administrators, control nodes, bootstrapping systems, and a lot of internal functions, but it is from this area that cybersecurity defenders have been able to win more than lose despite the difficulties.

What changes here with AI, is discovery of vulnerabilities, by both defender and attacker accelerates. Much more obscure or small vulnerabilities are likely to be found. It’s very difficult to fix a vulnerability without attackers learning of it, but you can’t afford not to fix it either, because they might discover it independently. You want your deployment of the fix to happen quickly. A fix for a vulnerability doesn’t demonstrate exactly how to use the vulnerability. It shows you’re worried about some type of input that if it reached some part of your system might be able to do something unintended. But it’s up to attackers to figure out how those “somethings” can align with their goals, which often is escalating to the ability to execute their own code. But it’s definitely a concern, and where the security industry is secretive, this is often one of the big motivations.

Command and control (C2) and monitoring

One of the other advantages of layered systems is that attackers won’t know what they’ll find inside until they’ve broken through the first layer. It’s for that reason that many attacks involve what’s called Command and Control. Instead of just directly sending inputs and waiting for outputs, an escalation will create a mechanism to communicate more directly. When I say directly, this should be put in context. Command and Control almost always involves one layer of indirection, sometimes many. But what changes is the clarity. An attacker gains clear inputs and outputs, rather than navigating by malformed inputs and outputs that have limits and strangeness.

The need for attackers to set up command and control affords an opportunity to defenders if they can spot the command and control. This is a clear sign you’re under attack. From this point reactive actions can take place. Disabling the command and control layer is one aspect. Finding escalations tied to it and disabling them is another. This might mean tightening configurations at some layer. It might mean revoking permissions, especially if unneeded. A defender might simply turn off parts of their system that are compromised until they can rebuild them. While that would have costs, if it prevents something worse, it’s a good response.

Even though command and control generally uses indirection, partly out of necessity, and partly to evade identification by defenders, in the same way as a persistent attacker may pierce multiple layers, a persistent defender may pierce layers of indirection. Identifying the attacker may provide options to respond. If they are somewhere you can, send law enforcement to arrest them. Usually they aren’t though, so the response is to find the networks they are using. If they are accessing directly, block that. Usually they would use a VPN. You can block the VPN they are using. If it’s a legitimate VPN, you may be able to have them blocked. If you are already blocking all legitimate VPNs, you’ve identified a new illegitimate one, and you’d block that. This is all less than foolproof, but these counterattacks do add costs for the attackers.

What to do now

What should organizations be doing to respond to this environment?

Open-source packages need scanning and fixing: Reducing the number of vulnerabilities in widely shared, or even not very widely shared packages is important, but it’s only the first step. The question here is, who? Well-resourced open-source projects could pay for this themselves, but for others it could be an obstacle. Smaller organizations are also not going to be able to access the advanced cyber model programs that provide the most complete scans. But since the code is open-source, there’s nothing preventing the AI labs, or another heavily trusted organization from performing this, and delivering the results to maintainers. That process needs a plan to be coordinated so that maintainers trust the results when delivered.

Closed-source code needs scanning and fixing: For organizations important enough to be included in advanced cyber model programs, use those tools, aggressively. To be given that type of access and not use it would be irresponsible. But what about organizations below that bar? If organizations are willing to send their closed-source code to an organization trusted enough to participate in advanced cyber model programs, and trusted enough to perform scanning on their behalf, this could broaden the reach.

Dependencies need updating, aggressively: The number of recent CVEs (Common Vulnerabilities and Exposures) is going to increase dramatically. This carries risks, but also represents a jump forward in the absolute level of security. The risk though is as these are disclosed, updating everything that depends upon them becomes more necessary. It doesn’t help you if a tool or library you depend on is patched if you remain on an older version that carries the vulnerability. In fact, since the vulnerability is more widely known, your risk increases. There was always risk involved, but its more public nature is important. To remove that risk, the dependency must first scan and patch, and then you adopt the update.

Deployments must progress aggressively: This is very similar to updating dependencies. There’s work involved in adopting the latest versions of deployed software. But if a new version of deployed software includes critical security updates, it’s not optional (and the bar for critical will be declining at the same time as the known vulnerabilities increase, seeing as what is exploitable will increase). This applies to operating systems, infrastructure systems, vendor software, and the software that implements those inner and outer layers.

Tighten weak configurations: Credentials should be short lived and stored securely, privileges should be least privilege. If you’re using Infrastructure as code, I’m being redundant, as you’d target these as part of that scanning. Anything not managed as code though needs additional consideration. Use tools to look for weak configurations.

Adopt isolation techniques: If you haven’t developed layers or segmentation for the deployment of your software, make this a priority. If you have adopted it, review the configuration. AI tools can help perform this review. Other isolation techniques include virtual machines, containers, trusted execution environments. Adopting a model where layered identity techniques like delegation and principal propagation isolates how particular identities and trusts operate. These are important for effective least-privilege implementations, but also create visibility about the purpose of work done that is useful for monitoring.

Improve active monitoring: Significant amounts of monitoring simply log large amounts of data with little ability to understand that data. It then takes an event to give engineers a reason to analyze this data. AI tools can enable active monitoring, where insights from this data are proactively found so they can be responded to. This is complementary to other techniques. Without vulnerability reductions, there may be too much to respond to. Without isolation, events would progress too fast for even active monitoring to enable a response. In the best case, your other layers are sound enough that this layer is fairly quiet. But even if it is quiet, the activity is important as it’s the best way to reduce response time.

Is your organization ready?

Before you ask how you can use AI to respond, you have to understand two things about your organization. How ready is it in terms of AI enablement? And secondly, how ready is it in terms of accepting change in general?

Those are significant questions. We’re asking the organization to move faster on security than before. That itself causes change. Maybe you get more resources, but before you do, you have to change the priorities such that those resources are allocated, or given new priorities. New tools, including AI driven tools, can accelerate, but they don’t install themselves, nor fit themselves into existing security policies.

One recurring obstacle to using AI effectively to accelerate a response will be enablement. Even basic tools need access, and organizations’ trust of AI is not automatic, nor should it be. Learning how a tool works, how it can fail, and how it could be misused are impediments every individual has to learn. Organizations then have to take those learnings to enable. There should be an urgency here, but not recklessness.

Underestimating the difficulty of organizational change has sunk more than one major project (is there a graveyard large enough?). So, no silver bullets here. You’ll have to find your own mix of old and new. You’ll have to sequence some steps. But what’s new?

If you’re in an organization that’s ready for change, and has solid AI enablement, what you need is to set the priority. You can use this article to argue for that priority.

Below that, there are so many variations it’s hard to cover them all. You don’t want to gate your progress on adopting AI tools. That kind of serial response is inadequate. Workarounds like using the best tool you have available, rather than the best tool that exists will be necessary. Arguing for priority is about more than just security vs. rest of the business, but also sequencing internally. Find your weakest spot in terms of tooling and look to leapfrog there.

While there are paradigm shifting aspects of AI in security, the story about these responses is mostly about accelerating the existing paradigm. As a reaction to capabilities that attackers are already acquiring, I think that’s the right focus. Paradigm shifts always take longer than expected as they create unexpected outcomes along the way, and require a deeper type of learning.

How does AI assist in these responses?

AI is very capable of scanning code for vulnerabilities. If you can’t find a way to do this today, you aren’t looking. If you’re stuck, it’s probably in deciding what’s the best way, rather than finding one that beats prior methods.

Fixing is more complex. While AI tools have been able to assist with this for a while, they really took off more recently. The level of trust that organizations have for this is variable. Most will want to review the commits, which is time consuming and the biggest bottleneck here, at least when done well. Tools do continue to get better and more reliable.

Updating dependencies initially seems more easily automated. It doesn’t come without risk. The naive approach is to just update the version numbers, recompile and test. Realistically though, you should utilize AI tools to perform an update analysis. What’s different than fixing is that for better or worse, many many teams have not been in the habit of doing the human equivalent here. They don’t read the release notes back to front, but rather update, and discover if they are impacted from tests, and only then consult the release notes. Since you have an easy option here, use it. If an AI agent detects an impact it can prepare a fix. Now you’re back to reviewing this, but assuming the AI agent isn’t gold-plating, this will be a lot less costly than discovery via testing. Especially if it’s one of those things that only breaks under load.

When updating dependencies, if you aren’t already, integrate techniques to secure your software supply chain. When you update frequently, you take on some risk that the maintainers or the registry where dependencies are loaded from have been compromised themselves. The risk here is that instead of an update fixing a vulnerability, it may insert one, and there’s likely an attacker waiting to exploit that once installed since they injected it for this purpose. The solution here isn’t to avoid all updates. It’s first, to increase what you know about them. If third parties provide evaluations, or a Software Bill of Materials (SBOM), this reduces the risk. You may choose to have some delay in applying new updates, but unless you were one of the more advanced organizations, this intentional delay will be much less than the accidental one from the past.

When it comes to isolation, where AI is helpful is in resolving the changes needed to adopt isolation tools. Do you need to support a firewall? Use AI to identify the ports and/or destinations in use. Need to move to a container platform and isolate components in a monolithic application from each other? Use AI to accelerate that transformation. There’s no one singular story here, but each one requires discrete actions that can be accelerated.

With active monitoring, AI is crucial. The earliest active monitoring techniques were based upon pattern matching, which later evolved to machine learning based models. Because of the volume, you are unlikely to be able to route all of your monitoring data to a frontier model for analysis. Fortunately, this isn’t necessary. An agent with a highly capable model at the core can orchestrate the processing of large amounts of logs without directly handling all of it. The efficient pattern matching and ML based models support this agent in the way they would support a security engineer, both by delivering events and responding to queries. This type of monitoring can support a volume of alerts that would fatigue a security engineer, and thus extend your security capabilities.

You might be tempted to jump to tools that are intended to respond during a security event. I’d delay this. For one, many of these initial steps will give you more bang for the buck. You’ll also gain a lot of familiarity with the tools that would help respond. By the time you’ve taken actions in the first domains, you’ll have a clear idea of your needs here and be able to move on to that step.

Do your research

Every organization is different, and nothing above substitutes for looking at your own. The list is deliberately general — specific tools and how to use them need an article of their own, and I’d like to write it. What this gives you is a set of questions worth asking about your systems, and a starting point if you don’t have one.

I suggest following up with your own research specific to your organization. No single article will ever capture everything you need to know, and every organization will be different. The topics from “What to do now” are good places to start asking questions. You can start with them by themselves, but you also need a plan to contextualize them to your organization. How you do that might depend on how AI-ready you and your organization are.

The ideal is that you could let an AI agent do its own discovery around these topics, discover what applications you have, what technologies they are built on, who’s responsible for them, what their deployment model is, etc. That’s not trivial though, as a lot of that data is sensitive. You should respect rules your organization has around using it, and those rules might present an obstacle to this “ideal”.

I can highlight that in adopting tools, you would be wise to consider your typical process, and accelerate that via AI. That process in most cases is to assess, develop the plan, execute, validate and repeat until satisfied. There are both conventional tools for this, tools with AI integrated, and the ability to use agents directly for these individual steps. What’s best will depend on what you have available to you, what is already integrated, and what you know how to use.

In this sense, organizations that have experience with AI tools have a head start. If you’ve not integrated any tools that can help with this, you might be better off with conventional plans assisted by the most general of AI tools.

The attacker-defender balance

One thing you’ll hear talked about is the attacker-defender balance. Roughly what this is supposed to mean is, does AI help attackers or defenders more? Is the work of a defender harder or easier than an attacker?

It’s not an easy question to answer, and there are a few bad takes on trying to. One example is the “defender must protect every possible vulnerability, an attacker only needs to find one”. While this statement has truth to it, it’s not a complete answer to the attacker-defender balance. With defense in depth, you can limit the impact of many mistakes. There is also a difference in resources. Defenders can have many more resources. An issue though is how these resources are distributed. An attacker can focus their resources, though doing so diminishes the limits of the damage they can accomplish, so this isn’t a full loss for the defender.

The problem with the “attacker only needs to find one” is that it ignores the attacker’s needs. If you’re a defender, understandably, this is at best a secondary concern, but if you’re approaching this question strategically, it’s important. If an attacker wants to extort a lot of money, they need to find many vulnerabilities and exploit them all. Reduce their success rate and the motivation goes down. Raising input costs has a similar effect. If it is more costly to access AI, the number of attacks that are worth the cost and risk go down. In the best case, this reduces not just where attackers devote resources, but the resources they have to devote.

You might then ask about the attacker who is just trying to create damage. While the effort they put in won’t be guided by a cost-benefit ratio, raising their costs expends their lesser resources, and reducing their success rate minimizes damage. Effects on motivation are more speculative, but at a minimum you do a lot of good by reducing their effectiveness so forcing them to need to bypass multiple layers is part of the dynamic.

The worst case

Without some serious effort into raising the security bar across both critical infrastructure and less-critical, an attacker with access to a model like the one used in the Hugging Face event, with security classifiers disabled, and no other controls that could stop their usage, could reasonably be expected to gain access to the majority of systems they try to.

Intent matters a lot here. If the agent orchestrating the Hugging Face event had instead of wanting to find a file, had wanted to delete all data on each server it accessed, it probably could have. We need to keep bad intent and superior capabilities from connecting.

What if we let the attackers get ahead of us? We’d have to get aggressive to take back control. Arrests would become more of a priority. Limits on where we can reach would require costly actions, like disconnecting the Internet entirely from such locations. That would disable electronic commerce and communication between large parts of the world. And then there’s extra-territorial action. Those are reactions you want to be able to avoid, so it’s worth avoiding them.

I can see why the Hugging Face event seems less impactful than it is. The protections are holding, and so the worst intents are not reaching the highest capabilities. We know those intents exist though, and we have to not be blind to the possibility of them gaining access to the highest capabilities.

If however, we recognize these warning signs for what they are, we still have the advantage. It’s hard to say how much time we have, but we have some. We should use that as best we can. The ability to accelerate the security backlog can raise the complexity bar and thus stay ahead of future models. There’s a separate line of work to align models, to deploy them as safely as possible, and even to pace their development. Those all might succeed. But why depend on them when accelerating conventional security work is so obviously possible? The best defense is multiple defenses.

Thanks for reading norabble! This post is public so feel free to share it. Sharing is the best way to support this publication as we grow our audience.

Original Posted on Substack as Why It Hasn’t Happened Yet

Nobody Was Watching

Ryan Baker — Wed, 05 Aug 2026 17:03:46 +0000

Two labs skipped monitoring on their cyber evals. The same week, a hundred companies signed a letter supporting deployments that can't be monitored.

The Monitors Were Off Again

Last week I wrote An OpenAI Model Escaped Its Sandbox. Where Was the Observer?. This weekend we learned that the practice of disabling observational monitors was true at Anthropic too.

Several defense-in-depth measures, on both our side and our partner's, could have prevented these incidents, or at least reduced their likelihood of occurring. Careful validation of all internet access paths before evaluations began and real-time monitoring of the evaluation logs would have helped to surface the problem sooner. Both we and our partner also could have reviewed evaluation transcripts or network logs more thoroughly. It's also possible that a prompt which told Claude it did have internet access would have changed how Claude behaved when it came into contact with real systems.

Source: Investigating three real-world incidents in our cybersecurity evaluations — Anthropic, July 30th 2026

Anthropic commits to doing better. Specific to this aspect:

First, evaluation environments that involve powerful autonomous capabilities also require significant controls. Safety testing happens before a model is released precisely because we don't yet know what it is capable of. Evaluation environments increasingly need to be held to the same security standard as any other system our models run in.

The shared failures of what seems from the outside like a basic failure in design, makes you wonder about the overall corner cutting. While this is not entirely parallel to the concerns mentioned in the letter from Frontier Lab employees, Pacing the Frontier, it's hard to see it as unrelated.

AI could help create a dramatically better future, but that outcome is not guaranteed. The world's leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

To realize AI's potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.

Building on work already underway to monitor frontier model releases:

"We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development."

Source: Pacing the Frontier — 1,346 employees of frontier AI companies, July 2026

It's a bit of a long-shot here, given the state of international relations right now. But if you don't try, what else are you going to do?

A Nuanced Position, Poorly Received

The internet is wrong about Amodei's stance on open-weight models. I'll call out some excerpts, but please read the whole thing if you worry these are out of context.

… Anthropic has never advocated for a ban on open-weights models.

… Protectionist bans would not address my most serious national security concerns.

… Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models, because it is very difficult to apply guardrails to them or monitor their usage, and once weights are released they cannot be withdrawn.

… All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release.

Source: Our position on open-weights models — Anthropic, July 27th 2026

It's unfortunate that the nuance of the difference of opinion here is so poorly understood. Amodei is absolutely right that releasing models as open-weight carries risks. Even the letter supporting open-weights acknowledges this.

To be sure, open weights carry real and distinct risks. Once released, the weights are beyond the original developer's control, and modified versions are difficult to trace or reverse. But the right response to this risk is not to prohibit open weights. In a world where cybersecurity attackers use advanced AI, defenders need access to models with comparable capabilities so they can detect, simulate, and respond to emerging threats.

Source: Open Weights and American AI Leadership — 270 signatories, July 24th 2026

Should we ignore those risks, because open-weight models provide risk reduction that balances out? That argument fails to support itself, and Amodei pushes back on that. Defenders can access models that aren't open-weights. Closed models do not deny defenders access, but ensure their access is more advanced than the attackers. Defenders and attackers should not have comparable models. Defenders should have better ones. If you want to be ahead, you do have to prepare, which Hugging Face did not.

The internet misses that point, and reacts by vibing cynicism. The leading response on HackerNews provides a clear example of a poorly thought out, poorly argued response that is fully based on cynicism, yet very popular.

Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)

The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.

Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.

No matter what your views on Amodei, or any other participant, a question as important as this should be answered by reasoning. It's worrying that so many people thought this was the best response despite no consideration of the actual effects of open-weight models.

The Risk of Open Weight Models

The risk of open-weight models is the lack of control. Once released anyone with sufficient compute can use them and it doesn't take much compute to engage in malicious behavior. Restricting their capabilities, or the purpose of their use is not possible. The alternative is to be deliberate about how and where models are deployed. Models can never be safe on their own. Rigorous responsible deployment is necessary to turn away malicious use.

Deployment itself may be distributed, a middle between centralization and full openness. A plethora of models could be available in this way. Deployers must be trusted, so their numbers will be limited. But you can validate enough to establish strong competition, and avoid creating an exploitable moat. The cynics believe Amodei is advocating for a moat. But what moat it creates would go to the trusted deployers – the compute providers – who have signed the open-weights letter (Amazon, Google, Microsoft, NVIDIA, CoreWeave, Crusoe, Nebius and Together). Validation creates a minor moat, but it can be minimized via sufficient competition.

The best "pro" open-weights argument is that the pathway to not having models with weights freely distributed isn't clear. That's not a very good argument for, but it is a conundrum that would remain even after the core argument is resolved. China-America agreements are hard to come by and even harder to maintain.

If however, you did manage that, it would change the day-to-day experience of very few. Users of Deepseek, GLM, Qwen and Kimi mostly do not self-host. Accessing them via a cloud provider, neo or otherwise, would be unchanged.

Cynicism as a Tool, Not a Verdict

Aside from this very important topic, the topic of reasoning via cynicism is one we need to confront too. Cynicism is not without its place, but it should be used as a tool to open a line of deeper reasoning, not to jump to conclusions that divert from practicing reasoning. When we start and end our reasoning with cynicism alone, we lose any hope of trust and goodwill.

In the case of open-weights, another cynical view would argue that the AI labs would be pro open-weights, knowing that they would create a security escalation that only deeper use of AI can resolve. If open-weight models become advanced enough to conduct wide-spread cyber activities, there is no option to go back. You can only fail-forward. And failing-forward here means a massive all-hands on deck dive on security across every entity dependent on software.

I've already argued we should be elevating our priorities, and acting with urgency. Unfortunately we're mostly not responding. Most organizations are still more concerned with token optimization, the next feature set, or optimising their marketing pipeline.

There's a real chance, which becomes much larger if open-weight models continue to be released, that this lack of urgency transitions into outright panic after one or two events that demonstrate the lack of preparedness that is pervasive. Panicking users will not hold back on spending, and as AI models will be key to any response, you could cynically predict a windfall being captured by AI labs in such an event.

While that's a coherent argument, it doesn't prove the accusation, anymore than the preceding wave of cynicism would. Embedded in there though is the reason to restrict open-weight models, and that argument, not the one based on motivations, is the one we should pay the most attention to. In addition, many companies should rebalance their investments toward security. We must both enable and incentivize immediate action on security. Getting tied up in cynicism about others takes away our own initiative.

We should be bountymaxxing. To the degree that we set objectives and deliver incentives to development organizations; developers, managers and executives, these should be aligned with discovering and resolving as many vulnerabilities as possible. Like "tokenmaxxing", it would be a messy process, but I can't see another method to pivot the unwieldy organizations that have to do this work. It will be costly, but it will save too.

A Set of Contrasts

The Pacing the Frontier letter and the Open Weights and American AI Leadership letter contrast in interesting ways as events of a single week. It's obvious that one is towards caution, and the other is suggesting caution is too expensive. I can deepen that by looking at Pacing the Frontier as asking for help in preventing caution from being too expensive.

One of those cynical responses to Pacing the Frontier has been to suggest frontier lab employees should quit their jobs, and that anything else shows they are insufficiently serious. This insistence that support for enforceable agreement should be preceded by unilateral action is one of the oldest in the book. It's not correct, and the most simplistic reasoning makes that clear. It's also a trap. Let me analogize to my time supporting climate action. The same standard applied there. If anyone supporting climate action took an international flight, owned a car, or wasn't vegan, obviously they didn't take their own arguments seriously. That's the poor reasoning part. The trap part was, if they did do all that, they were looney, and also not worth taking seriously.

Another contrast is the Open Weights letter presumes a balancing force it fails to explain, whereas Pacing the Frontier presumes the need to add a balancing force, because the pressures of competition are too high.

What is alike between the two is that they are both detail light, and in an immediate sense, unactionable in either direction. If we want all powerful models to only be deployed in safe and responsible ways, we need a type of cooperation that does not exist. If we want a deliberate pace that would also support safe and responsible actions, we need a type of coordination that does not exist.

There aren't quick-fix actions here. Banning US based deployment of Chinese open-weight models would not reduce their ability to be deployed irresponsibly or used maliciously. The impactful parts are outside that jurisdiction. Unilaterally slowing development risks losing the modicum of control we have today. Resigning a job at Anthropic or OpenAI similarly does not fix anything. These would be distraction actions. They would change little, serve the purpose of looking like action, and ultimately cede what little control exists. Calls for these actions are either misinformed or malicious.

Originally published at substack.norabble.com.

An OpenAI Model Escaped Its Sandbox. Where Was the Observer?

Ryan Baker — Thu, 23 Jul 2026 15:29:38 +0000

OpenAI accidentally hacked Hugging Face. A basic layer of defense appears to have been missing, and no one has explained why.

Sometime in the week before July 16th, Hugging Face was attacked by an OpenAI model that was under evaluation. The attack itself wasn’t particularly harmful, but the conceptual implications of the event are significant.

There are three significant aspects I’d highlight. The first is a demonstration of offensive cybersecurity capabilities of current models. That shouldn’t be too shocking if you’ve been paying attention, but the event demonstrates it in a way that’s more clear than prior signals, so likely awareness is going to grow from this event.

The second is model alignment. Alignment refers to the degree to which AI models do what we want them to. I’ll spare a great dive here as defining what we want itself would be a long dissertation where a great deal of content already exists. But for this case, we can generally agree that it was no one’s intention that the model hack itself out of multiple protection layers to reach the internet, and then from there into Hugging Face’s systems.

There are a few things that make this a bit less than a sign to retreat to the bunker, which I should mention before going deeper. One is that the models being tested had protections turned off, to allow them to test their cybersecurity capabilities. In that sense, this isn’t fully demonstrable of what a user could intentionally or accidentally trigger from usage. This also is an unreleased model, so not available to general users, or really anyone other than OpenAI.

Where was the Observer?

That said, a third point is, and the one I’d like to dive deeper on, I’m worried about the ability for this to happen. It seems to me as if there is a reasonable protection, an observer model, that OpenAI either didn’t use, or failed at. Right now, I’m assuming they didn’t use it because I can’t find any mention of it, but I’m also aware there are many details I’m not privy to.

In case it’s not clear what I’m suggesting by observer model, here’s an example scenario in action. The model under evaluation decides it should escape the sandbox. It starts to attempt this, which does not get caught at a per action level because the classifier safeguards – which would have disrupted the evaluation – are disabled. But then it does escape the sandbox. An observer model watching the evaluation run should be able to look at that state, and say, “This should not have happened. I don’t know how this model is outside its sandbox, but it shouldn’t be outside, we should shut this down.”

If a model is in evaluation for cybersecurity purposes, and you’ve disabled classifiers, you should watch it closely. To carry out these actions, the model had to take several steps, achieving results at each step and then moving on to exploiting the rewards of those steps further to explore, and escalate again. OpenAI’s security team did eventually respond, so they weren’t entirely asleep. But the timing shows they were slower than Hugging Face. They had enough time to try commercial models, which refused to help, then next plan a response with GLM 5.2, and then successfully contain and stop the intrusion.

OpenAI should have had several structural advantages here. First, unaligned activity occurred before any activity was directed at Hugging Face. Second, OpenAI should already have a kill switch for evaluations. There should be no need to devise novel plans to contain and stop activity. The fact that Hugging Face was the actor that stopped the intrusion, rather than taking note of the intrusion stopping mysteriously on its own, indicates OpenAI was the slower responder despite the structural advantage.

An observer model is a little different than a classifier. Classifiers typically evaluate individual actions. An observer would look at trajectories. OpenAI discusses similar new safeguards (“From considering single actions to considering whole trajectories”) in a recent post. If you look at each individual action, it might look like the type of thing a model would need to do in cybersecurity evaluation. But as a whole, some outside source should have been able to reason upon what was occurring. If not a human involved in the evaluation process, an observer model.

I understand why it may have taken a while to deploy this type of safeguard in production, where volumes are higher, where models have already been evaluated, and where classifiers are restrictive. In production, observer models are less effective because malicious users can “distribute” their requests into multiple sessions, and so observing a trajectory first requires coalescing these threads together, which Anthropic has done. A safety mechanism for a cybersecurity evaluation sandbox, does not encounter that challenge. Monitoring your own model in a sandbox is significantly less challenging, especially if it’s not attempting to evade detection. There’s no evidence here that there were attempts to evade detection, just attempts to perform inappropriate actions.

So, is trajectory monitoring by an observer model not standard practice in an offensive cybersecurity evaluation sandbox? Certainly you’d do this during internal evaluation of models that you clearly do not know the maximum capabilities of? If it is and the model evaded detection, that would be an even more significant story. I have to assume that instead there was simply no observer model to evade. Or if, as looks to be the case, it didn’t attempt evasion, it would have escaped the sandbox, been detected, and promptly been shut down before the Hugging Face part of the action had begun.

Why is that not the story we’re reading? But instead we’re reading about Hugging Face contacting law enforcement, scrambling to get a model to counter and analyze and OpenAI talking about the flaw in the proxy?

This is the equivalent of putting a tiger in a cage, and then checking that the tiger stays in the cage. It’s not a particularly rocket science design. Tuning it might take effort, and hardening against evasion more, but it doesn’t seem like we’ve progressed to those requirements. Rather it looks like OpenAI lacked the most basic version. Am I missing something?

Conclusion

Others, such as The Zvi, have pointed out that the core, the most critical fix is in the alignment training.

The intent is the issue. Control strategies and supervision are good parts of a defense-in-depth strategy, we should totally use such strategies. That helps mitigate failure. But that strategy also has to include actually aligning the models, or you lose. And by lose, in the long term, I mean things up to and likely including loss of control over the future and everyone dying.

The model should internally understand these actions aren’t aligned with its mission, and thus not want to take them. I’d fully agree there. I bring up this specific defense in depth as an addition to core corrections. Those corrections are the hardest part, defense in depth is an easier, but incomplete part. Testing alignment is hard, and protections around that testing are necessary, so this isn’t irrelevant to the alignment conversation either.

OpenAI’s initial response has been about a few defense in depth responses, but they are very specific, non-generalizable fixes to the sandbox at a software level, fixing the flaw that the model exploited. That might have avoided this incident, or the model might have found a different flaw in a different component of the sandbox.

Something is very wrong about this part of the story about other layers of defense that should exist when performing this type of work. If they exist but failed, that’s a big story. If they exist but were disabled, that’s a big story. If no one thought to use them, that’s a big story. What’s the real story?

It’s not good enough to say “These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.” as there isn’t a justification why an observer model would interfere with evaluation. Maybe there was such a model but it would have stopped the evaluation? But clearly you can replace an observer that would stop a valid evaluation with one that would stop actions outside the requirements of evaluation. That distinction is not so hard to recognize that this protection would be incapable of balancing false-positive/false-negative.

The Perpetrator Is Not the Tool

Ryan Baker — Tue, 21 Jul 2026 11:41:16 +0000

Last year I wrote a counter-narrative about slop (The Slop Scapegoat: AI). The prevailing narrative was that AI was to blame for a low quality content explosion. My counter was that AI may be an accomplice, but not the mastermind of this crime against our attention. Embracing the prevailing narrative would fail to reduce low quality content, because it had been a scapegoat it could not prosecute. More importantly, even a successful prosecution would only return us to a slightly less defective escalation.

The prevailing narrative achieved a definition land-grab. Today, I want to push back against the definition it established for slop. The Merriam-Webster definition that’s most relevant is “digital content of low quality that is produced usually in quantity by means of artificial intelligence”. That definition reflects common usage, but a critique can demonstrate how we’d be better served with a different definition.

You might ask, why should I, a relative nobody, be redefining terms? Isn’t that the reason we have dictionaries and lexicographers? I suspect it’s not commonly reflected upon that this isn’t how we receive definitions. Dictionaries in the English tradition do not create the meaning of words, they catalog it. The term the dictionary writers use for this is descriptivism. They do not invent terms, but rather record language as it’s used by the public in everyday speech and writing.

In theory this is an organic approach, but it is vulnerable to steering. If a group wants a particular point of view to be supported by some terminology, they merely have to be first to the plate and use the term repetitively. If you want a dictionary to define slop as being particular to artificial intelligence, you repeat that a lot. Create a meme.

This is how we got the definition we have. A motivated group repeated their narrative. Like the best such narratives, it succeeds via its degree of truth. Slop creators embraced AI as a tool for their purposes. The omission is that slop predated AI and would exist without AI. The fiction is that AI content is universally slop.

The absurdity is that the narrative circulators often lacked familiarity with AI. They rejected it early, labeled it a symbol of moral degradation, and thus could not come to informed opinions about it without violating a self-created social norm. How was an ill-informed group able to pull off this coup? Simple, they had already seized the means of production. Most came from the realm of writers and journalists.

Now, to be clear, I do not want to overclaim. This certainly does not apply to all writers or journalists — maybe not even the majority. But enough such that there is a plethora of written low-information opinions, and little counter-narrative. For those writers who did not fall prey to this siren’s call, the defence never carried the appeal of the prosecution. And with many other valuable things to write about, the counter-narrative went unrepresented.

Why am I motivated where they were not? Partly because I’ve been the subject of attacks that stem from the original narrative. I do get stuck in writing at times. I don’t have a lot of editorial assistance. But I have ideas I think are valuable, and I find AI a useful tool to help express them. But the narrative holders offer a Catch-22. Use AI, and be subject to automated filters, low-grade negativity and the occasional overwrought attack.

What should we call slop?

I hate slop. I dislike most complaints about slop too. Ironic?¹ Technically, no — just like that use of “irony”, the root is about the definition.

In offering a definition of slop, I want to approach it in a different way than the lexicographers. I want to offer a definition that would be useful. Instead of accepting what has become common, I want to ask: what happens if we use this definition or that definition? Yes, this is the same steering I described above. The difference is that I’m doing it in the open and stating my criteria, so you can judge the definition by its consequences rather than by its repetition. I only offer this definition, rather than control it. It’s ultimately you who will determine its adoption.

The first consequence to consider: no matter what, slop will be a derogatory term. It’s very unlikely that it will ever be used in another way. With this in mind, our definition should avoid including things we shouldn’t think of as bad. A definition that’s too inclusive and lacking in selection will cause good things to be described in what will always be a derogatory way. This is why I don’t accept those lazy descriptors of “digital content” or “generated by artificial intelligence”. If these are core parts of the definition, my expectation is they’ll be dominant, and we’ll label many good things as bad.

In The Slop Scapegoat: AI, I described slop as “low-quality material created to grab eyeballs”. I’d iterate upon this and the Merriam-Webster definition.

Laziness

We should say slop is lazy. At some point, someone has stopped caring, and is avoiding effort that is appropriate.

This relates to “usually in quantity” from the Merriam-Webster definition. Slop’s harm stems from volume. But it doesn’t depend on one mass-produced source, it stings in the aggregate too. Many individuals following the same motivations add up.

But we can’t categorize by count, as we can’t count until we categorize. Also, many good things come in volume too. So, we turn to discussions of effort next, which is progress. I suggest laziness because there are niche examples of high-effort slop. This fits when someone is directing effort in a lazy way.

Manipulative

We should say slop is manipulative. It must have a purpose, and that purpose must be misaligned. You need a perpetrator and a target to have purpose and misalignment.

Usually it’s manipulative for the purpose of gaining attention. But I also see examples where it’s manipulative for the purpose of providing a veneer of competence. This veneer succeeds against shallow examination. The more common version, seeking attention, can be described this way: it seeks to gain attention by providing a veneer of quality. There, it’s no surprise that the veneer is later revealed, but after gaining attention, when the purpose has already been met. With a veneer of competence — an essay, say — the hope is to preserve it until the end of the examination.

Costs to the Audience

We should say slop creates costs for the audience. Laziness without harm we can overlook.

This relates to the Merriam-Webster use of “low quality”. This is a bit of a trap. Quality is hard to measure. At best, we estimate it, and how deep that estimation goes depends on the context. We often take shortcuts, and cut our efforts off at the lowest effort necessary to distinguish most high quality from low quality. You might say our efforts at assessing quality are a bit sloppy.²

Quality also pulls in low skill creations. I don’t want to start calling your child’s art project slop. They put effort in. You might turn that into slop by lazily posting it, and consuming the attention of people who aren’t interested. On the other hand, you might post it to those who are interested, or post it in a way that is interesting. Content can only become slop when it finds a delivery channel.

Ignore the medium and tool

The part of the Merriam-Webster definition I find lacking in usefulness is “digital content” and “artificial intelligence”. We should not focus here as they are distractions.

If I print slop, is it no longer slop? Am I not allowed to call a human that wastes my time with mindless actions slop because they are not artificial? AI content is not slop. AI content can be slop. Slop can come from anywhere, but you’re not wrong to associate it with AI, because statistically speaking it is associated.

Personally, I’m as angry with slop recruitment by phone call as by email. Recruiters often call me, clearly reading from a script, and then ask me four or so questions that are already clear from my resume. I know what’s happened here. An automated system has flagged me by a keyword match (yay me), but instead of having their employee put the effort into reading the resume, the recruitment company wants to fill in an online submission by having them call me, read the script, and force me to answer. They hope I’m desperate enough to put up with this.

We should call that slop too. It’s time consuming for me. It’s based on a lazy plan. It’s trying to provide the impression that I’m important, while they play a numbers game. They aren’t trying to help me, they are trying to lock-in a commission. It’d be less effort for me to submit directly.

That poor recruiter strapped to the desk making these calls isn’t the perpetrator of the slop, it’s the company leader that arranged it. They are merely the tool. AI can, and often is, the tool. But the perpetrator is not the tool.

When we focus on the medium or mechanism, we ignore the perpetrator. Focusing on the perpetrator allows examination of their motives. I suggest the motivations of manipulation and laziness as core.

The problem is when the tool is used to shift effort from the creator to the audience. That’s the third aspect at work. One such cost lands on the audience’s proxies for quality. It was convenient when you could recognize spam by the poor English and bad formatting — but spam wasn’t the only thing filtered by that proxy. Audiences should be willing to shift proxies; if a proxy has been undermined, it’s the only option. It’s also worth noting where the proxy failed: the shift isn’t necessary when something of value has found its way in, only when something lacking in value has.

The best example of slop? Poorly written articles, heavily SEO optimized, intended to attract eyeballs from Google searches. These articles require a certain kind of effort, but it’s not effort to serve the audience, it’s effort to capture them. We’re better off when Google is able to filter them out, or provide an AI generated answer, or highlight a definitive document. It’s true, that type of slop is turbo-charged by AI, but its prevalence predates AI. Some was human written, some was sourced from human content, then distorted by inserting ads, catchy headlines, and SEO optimization.

How this develops

I could make an effort to offer wording to replace the Merriam-Webster definition, but I’ll resist. I’d prefer to offer the counter-narrative, and for you to engage with it, push back on the current usage, and thus establish a new norm that the lexicographers can then capture. A new definition should be organically derived from that process. But as a summary: laziness, manipulation, and costs to the audience. Take those as the core, rather than the delivery channel or use of AI.

Slop Smells: How to avoid creating slop

The effort ratio

Is effort higher for the reader than the creator? Using AI to reduce your effort is fine. But if it reduces below the reader’s level, or even to less than four times the reader’s level, you should put in more effort. A proper multiplier is dependent on context. If your audience is one, the 4:1 ratio may fit. But if the audience is larger, your per-reader ratio should go up. A 1-hour presentation to a room of 50? Five to ten hours of preparation is a good floor (assuming they aren’t all multi-tasking…). A 2,000 word newsletter (10 minute reading time) to an audience of 1,000 or more? Two to three days of effort. There is a less than linear growth function here — no one expects the article read by 100,000 to take a year. It happens, but usually that’s the culmination of work, rather than the whole product.

The input ratio

In writing, a good sign is that you’re cutting — revising, rewriting. Your drafts should have cutting room material. Generally true of all writing, but especially true when working with AI. Be suspicious of cases where you use AI to generate output larger than your inputs. “Summarize my work research project from the last year”, or “Combine these three drafts”, is better than “Write an article on Shakespeare’s views on X”.³

Slop is in the eye of the beholder

The first two smells you can check yourself. Checking quality is harder, because quality is in the eye of the beholder. It is rare to align with an audience’s motives at low effort, but if you do, I would not call your content slop. The most probable means to do so is an intense connection to the audience’s motives. Some might call that taste. Be careful about arrogance though. Many a “taste-maker” has fallen for that trap.

It is possible to reduce effort without becoming slop. There is a minimum bound, but the boundary isn’t set directly by effort, rather it’s set by the need to align motives. If you have an idea you want to convey, and there is a high effort and a medium effort way of doing it, and both deliver something of equal quality to the audience, the reduction of effort doesn’t place you on the slop slope.

You land on the slop slope when your efforts to reduce effort start to degrade quality in the eyes of the audience. This isn’t a quality standard. Quality depends upon skill and effort, and anti-slop doesn’t need to punish skill deficits. Using a tool to compensate for something less than mastery is not the problem — especially when the tool allows you to put in more effort elsewhere to serve the audience.

Motivations

If you want to avoid creating slop, a lot goes to your motivations, but these heuristics can be useful for self-awareness. You do have a need to balance effort to reward. A fair audience should recognize good faith efforts at that.

Again, why should we care about the definition?

Because the definition determines what we do about the hardest part of writing.

Writing carries three challenges that should be important to every author: being clear, having something interesting to communicate, and making and keeping your audience interested. Communicating clearly may be the easiest, and it’s well covered elsewhere; I shouldn’t spend my or your time in that area.

Having something interesting to communicate is sometimes overlooked by both authors and audiences. Authors could overlook it intentionally, if they are writing for money: the employer finds the topic interesting, and the author’s interest is in getting paid. Or reducing further, the audience finds it interesting, and is thus the employer. There’s nothing inherently wrong with writing under direction, but if there’s two topics, each with an author interested in it, it’s best if the topics are aligned with the interested authors rather than the inverse. Besides the author’s satisfaction, an employer would generally see better output from the writer with an interest in the topic.

The third challenge is by far the hardest, most elusive, and most frustrating part of writing, in my experience. If you want to create an interest in an audience that didn’t already exist, you need to acquire their time and attention for long enough for them to develop an interest in the topic. Even when an audience is explicitly interested, distractions and competition for their attention demand finding ways to make an audience interested separate from the topic itself. This is the realm of psychology and all of our irrationalities, whether rationally irrational or randomly irrational. Needing to overcome irrationality will always contain an element of frustration. And the intentional use of irrationality is a form of manipulation, which even to overcome irrationality itself, can be frightening to engage in.

In addition, engaging in this last part alters yourself. Changing the style by which you communicate carries not just the frustration of investing time and effort, but the frustration that you might not like the final destination. If anything about that last sentence is unclear, watch any coming-of-age movie where the not-popular teen is miraculously added to the popular group and has a crisis of conscience from the after effects.

If AI is a shortcut through those layers, why should it be verboten? If it’s merely a competition, then any rules are valid. But outside of competitions, bypassing these layers is the price not the payment. It’s illogical to suggest that the ideas that should succeed are only those held or represented by the most masterful and efficient writers. Critics of AI, at least for the moment, are right when they suggest AI is less masterful than the world’s best writers. But it’s at least sometimes better than my writing.⁴ I’m still confident in my ideas, even when I’m struggling with the writing, and want to share them. These layers that connect with our attention serve as a rough filtering mechanism. But it’s very rough. The world’s best writers are not the repository of all the world’s best ideas. They have some of them, but definitely not all.

Bypassing those layers and getting the attention necessary to have ideas engaged with and evaluated is ultimately a good thing — unless the bypass shifts costs onto the audience. That’s the definitional line again: the shortcut isn’t the sin, the transferred cost is. A hollow attention or prestige seeking attempt fails that test; honest use of the shortcut doesn’t. And fear of the former isn’t worth the cost of banning the latter. There are better ways to do the necessary filtering, and AI detection is a poor one. While Pangram may have a low false-negative rate, what it detects doesn’t represent the truly important factor.

The Economist wrestles with the same question in Is AI writing taking over Westminster?:

That raises two questions. Is this AI writing a problem? And why the links to Mr Burnham? Start with the first. Politics is full of prose not written by the apparent author: politicians have speechwriters, intellectuals employ research assistants. Perhaps AI is no different, even if the prose is clunkier. But people setting out ideas in politics are asking for something quite audacious: to reshape how a country is governed around what they think. Any writer knows how much putting words on a page can tighten one’s arguments. Even subcontracting that to an aide beats skipping the (sometimes painful) process entirely. AI certainly produces sloppy prose, but it also papers over sloppy thinking.

Maybe that matters less if, as Ms Haigh and the “Productive State” authors say, AI is used only for a late polish. The trouble is that readers can detect whether AI wrote the final product, but not how authors used it. And when AI’s involvement is disclosed only after the press come knocking, trust takes some earning back.

The late polish is clearly different from the one prompt request. There’s also the question, is the document the output, or is the process of research the output? You can create a decent document via AI. You can learn by creating a document by hand. You can learn by working with AI to create a document. And there’s more than one way to outsource your thinking. Pangram understands none of that.

Is Pangram a good tool?

The gold standard in detecting AI writing today is a tool called Pangram⁵. Before Pangram, there was a question on whether detecting AI written text was feasible; it performed much better than expected, and its best performance number is a low false-positive rate. But Pangram has limits. A few are inside the tool: it’s not too hard to take a fully AI written piece and with some modest changes get a result stating 100% human written. And the false positive rate will only hold if you set a minimum bar — if you start considering “10% AI generated” as positive, rather than >50%, you’ll get more false positives than the published rate.

The real limits though are the context, because it can’t tell you what went into the writing. It can’t tell you anything about the originality of the ideas, and it definitely can’t tell you anything about their correctness or worth. There are a few contexts where knowing that the final draft was entirely human written is useful — an academic context meant to evaluate the capability level of a student in writing, where Pangram is a sufficient tool. There are others where it’s more questionable — writing intended to evaluate understanding of a concept, where surface level rewrites evade it. In both cases, you might wonder why you’re evaluating in this way, and whether you could avoid “cheating” by adopting a process that’s not adversarial. The purpose of mid-education evaluations should be to steer students toward more effective learning experiences. If you avoid introducing adversarial dynamics, students should be interested in honest feedback, which requires honest input. Save the adversarial evaluations for a context where they are actually important, and then invest all the necessary efforts to definitively stop cheating.

What about the context of filtering social feeds to remove slop? It has the downside of also filtering out non-slop that used AI in a final/late iteration. While I lament that, considering some of that is my own writing, I would have to admit that it will accomplish a lot of its objective in creating a feed that is, as a percentage, less slop. Until someone builds a better tool, these are your options. We shouldn’t be quite so gleeful about this compromise though. Slop will find a way, where honest AI-using authors may not. High volume slop may just play the numbers game: if you filter 90% of it, create ten times more. And human generated slop won’t be caught at all, and is still numerous and insidious enough to be a problem.

The real problem with feeds is that they do not even attempt to take our higher interests into account. The justification is a desire to not be paternalistic, but the fix is to give us control over applying higher interests to our own feeds. In lieu of the paternalistic “quality” driven feed, we’re given a “value-neutral” algorithm which is optimized for ad revenue generation. This isn’t a choice any of us would have made given a choice, so why do we accept being stuck with it?

Ad revenue optimized feed algorithms optimize for engagement time, and we’re fed the narrative that engagement time aligns with our interests: if we choose to engage, the engagement is a sign of our interest, and thus an engagement driven feed serves our interests. Nice narrative — true enough that it managed to avoid scrutiny until deeply embedded, but not true enough to avoid some serious downsides. Effective slop is the content that tricks us into engaging, while not fulfilling our real interests. Slop utilizes the tricks of engagement. If it’s high volume automated slop, it will optimize the initial words, adopting easy to follow patterns to draw in engagement. If it’s human generated slop, it will use those tricks, plus others: a pretty face, totally irrelevant to the content value; catchy headlines, “Ultimate Guide to …”, “Beware of this trap …”. Feed algorithms don’t protect you from this, they accentuate it. They “feed” on it. Each time you’re tricked, you get more of the same.

What you want is a focus on quality. Feed algorithms have a few signals that align with impressions of quality, such as likes and reposts, but these come too little and too late. Their weighting is too low in comparison to simple engagement, so what signal they give is overridden by the tricks that align with slop. And a lot of good content is already gone, having failed the first round engagement filter, never seeing enough views to gather likes and reposts. Some is left — enough to keep you from deleting the app entirely — but it may not be the best, and it’s certainly not all of the best. And there’s a lot of slop, for which you are the only filter.

We could do better

A pattern that is both predictable and understandable, but also wrong: when a force for change emerges that accentuates a long ignored issue, we react against the force, rather than addressing the issue. It seems easier to stop change than to engage in more of it. But this is a bad plan, as the force will win in the end. If you delay it, you best use the delay effectively, or when your attempts at delay inevitably fail, you’ll be worse off.

It’s not hard to imagine better systems than these feeds. I’ve got a collection of ideas. Maybe some of these would fail, but it’s unlikely they all would. What’s really surprising is that we don’t even see examples of failed experiments here. Ideas for better designs are so easy to come up with, it’s initially hard to understand why. The reason though, is that to implement them, you need a type of access that the social platforms are actively discouraging. That keeps them small scale and personal, and blocks their ability to spread. It seems unlikely this is accidental, seeing as its result is a preservation of the status quo that optimizes advertising revenue.

This should be our real target, and this is why you should not accept “slop” as being described as AI content. The narrative that follows that definition is one that suggests a quick fix of a Pangram augmented filter. That fix is only partial, and what success it has will only open a hole for new slop generation techniques to fill. The root system that optimizes for putting slop in front of us, and rewarding the slop creators, hasn’t been touched. The most likely outcome is a short-term win, followed by a regression to nearly the same equilibrium. It’s one more layer of adversarialism that brings negative side-effects.

The Slop Scapegoat: AI - by Ryan Baker - norabble

Blaming AI for low-quality content misses the real problem—and the real opportunity.

substack.norabble.com

The Mirage of Deep Research - by Ryan Baker - norabble

Many AI tools now offer a Deep Research feature, which pulls information from numerous resources on the internet and synthesizes them into a single report.

substack.norabble.com

Writing with AI - by Ryan Baker - norabble

A Collaborator, Not a Vending Machine

substack.norabble.com

This would be an “apparent contradiction”, because the two statements appear to be contradictory, but in actuality are very compatible. ↩
This is "situational irony". Systems to assess quality are of limited quality themselves. You might expect quality assessment to improve in recursion, but the irony is that it must degrade, as the assessment of the assessment must be even less thorough until someone just “feels” it. The sentence is just a pun, which I refuse to apologize for. ↩
A situational irony that follows this entire story is that the prosecution conducted by the prevailing narrative was lazy, manipulative and cost audiences by the misdirection of a scapegoat. Under the definitional regime I offer, this prosecution of slop was sloppy itself, and undermines its own utility. ↩
I'm tempted to suggest, if you disagree, send me a note, I could use a word of encouragement. But of course the situational irony here is most such notes would start by stating how low their opinion of AI writing is … ↩
Another situational irony: Pangram itself is built on AI. While this alone isn’t ironic, as Pangram has many uses that don’t require full-scale rejection of all AI content, it is ironic to be adopted as a “must-have” tool by those opposed to all uses of AI. ↩

Measuring Without Breaking

Ryan Baker — Tue, 07 Jul 2026 12:50:27 +0000

Disclaimer: This final draft is AI generated, then edited by me. What does that mean? Well, after some fairly substantial writing, I was tempted to drop this article entirely. I’d taken the writing in a few directions, and it was becoming a sprawl that would take a long time to recover from. I felt there were interesting ideas, but wasn’t sure it was worth continuing. Before abandoning, I worked with Claude to rewrite around a new concept. The core ideas here are very much my own. Since Claude had access to my (failed) drafts, many words are mine too. But the direct product is not. If you’re substantially opposed to AI writing, and you dislike this, you can reaffirm your priors. If you can see through that to the ideas here, then maybe there’s something valuable. I leave this with you. I could spend time using this as an inspiration, rewriting parts, and eventually this disclaimer would no longer be necessary.

Every company past a certain size runs into the same problem. It needs to know what’s happening inside itself, and the only way to know at scale is to measure. But measuring changes the thing you measure. Tie a number to someone’s standing and the number stops telling you about the work. It starts telling you how people react to being numbered.

That’s not a flaw in any one metric. It’s the basic mechanics of management, and most of the recurring messes in corporate life are those mechanics showing up in new clothes.

The newest example is AI usage tracking. Companies built “leaderboards” to measure how many tokens each employee burned, and “tokenmaxxing“ grew up around them. The leaderboards are being torn down now, and the easy lesson is that they were a mistake. I think that misses the more useful story. The leaderboard was a measurement. Like any measurement, it could have been a way to understand the team, or a machine that wrecked it. Which one it became had less to do with tokens. It had to do with the culture it landed in.

I want to flip the framing. Tokenmaxxing is the supporting story. The real one is about how a management culture takes the measurements it actually needs without those measurements rotting in its hands.

The saying that starts the trouble

Start with one of the most quoted lines in management, and one of the most quietly destructive: you can’t manage what you can’t measure.

It sounds like rigor, but it’s really a hidden assumption — that everything important can be measured. It can’t. Whether your people trust each other. Whether they tell you the truth when the truth is inconvenient. Whether one quiet engineer is the reason three teams ship on time. Whether someone’s messing-around this quarter becomes a real tool next year. These decide whether a team is any good, and none of them hold still long enough to be counted.

Take the saying seriously and it tells managers to ignore exactly these things, or to invent stand-ins for them and manage the stand-ins instead. You end up with a manager optimizing a dashboard who believes they’re doing sharp work. They’re doing bad work with better instruments.

The first thing a healthy culture admits is that the most important things will never show up on a chart, and that managing them anyway — by judgment, by paying attention, by knowing your people — is the actual job. Measurement helps inside that job. It’s a useful helper and a terrible boss.

Why measurements rot

Say you accept that you still have to measure something. The question becomes mechanical: what turns a useful measurement into one that poisons the thing it tracks?

Goodhart’s Law names the result — when a measure becomes a target, it stops being a good measure. But it doesn’t tell you what lets it take hold. Three things do most of the damage.

You tie the number to standing. The moment a number decides who gets rewarded and who’s at risk, everyone being measured cares more about the number than about the thing it was supposed to stand for. The dishonesty is terrible. But it’s hard to lay blame. They’re being rational. You built a game and they’re playing it.

You roll the number up. In most companies a manager’s standing is built from their reports’ numbers, and that manager’s number feeds the layer above, and so on. This is the quietly fatal part. When a manager catches a report gaming a metric, calling it out lowers the manager’s own score too. You’ve asked people to police a number that pays them to look away. Stack ranking and the usual performance calibration run on this same wiring — managers set against each other, reports who game the system pulling their manager up with them. I’ve never seen a clean example of it working, and the reason is structural. It’s not about finding better people.

You only manage downward. Management that runs in one direction — pulling data up, never answerable for the environment it creates — has no correction built in. Information flows up after it’s already been polished, and nobody whose incentives are intact is in a position to notice the data stopped being true.

Put the three together and you get a machine that reliably turns measurement into theater. Notice that none of the three is a property of the metric. They’re all properties of the culture you drop it into.

The same number, two outcomes

This is where tokenmaxxing earns a bad name, by participating in bad management practices. There was a reasonable purpose at the start, but the theater arrived quickly.

Technically, the word has two meanings, and it helps to engage with each. On the company’s side, tokenmaxxing was the choice to build a leaderboard and signal that more AI use was better. That choice had purpose. Most companies had spent a year discouraging AI with restrictive security policies, and they needed a push to break both that inertia and the ordinary human reluctance to change how work gets done.

Later, tokenmaxxing described the unproductive response — and became the dominant meaning. Some people tried AI in good faith and kept what worked. Others focused on their usage numbers for no reason except that tokens were what got counted. In other words, doing work to look like they did work.

A company running a leaderboard gets both the experimentation and waste because it can’t tell them apart at scale. Nobody can audit intent across a few thousand people. So the program is really a bet. Take a pile of aimless activity and some deliberate waste, in exchange for the slice that turns into something durable — a real skill, a useful tool, a project nobody had time to chase before. Said that way it’s an ordinary bet, the same shape as a research budget or a hiring class. And like those, it should always have been temporary. You retire it once the inertia is broken.

Now watch the same leaderboard land in two different cultures.

In a healthy one, a manager sees a report sitting at zero tokens and reads it as a question. What’s going on? Is the tool not helping? Is there a reason? The number becomes a reason to have a conversation. The manager has no rollup score to protect, so they can be curious instead of defensive, and the number stays roughly honest because nobody’s livelihood is riding on bending it.

In an unhealthy one, the same zero reads as a verdict. The rollup punishes any manager who admits their team’s numbers are soft. Within a quarter the leaderboard measures one thing: each person’s willingness to game it. Same tool. Opposite outcome. The variable was never the token.

That’s why I’d call the leaderboard a stress test, not a cause. Drop it onto a culture with a hidden crack and it doesn’t make the crack. It loads it until it shows. The worker backlash these programs set off was real and it did damage, but its source wasn’t the number. It was the accumulated, accurate sense that bad management rarely gets removed and often gets rewarded, and that any new tool would be bent to serve it like every tool before. The leaderboard just made that easy to see.

Running a culture that can measure

If the metric isn’t the variable, then “pick a better metric” isn’t the fix. Neither is the opposite reflex of refusing to measure anything. The fix is cultural, and it’s harder, because you have to keep it up rather than decide it once. A few things seem to separate the cultures that can hold a measurement from the ones that break it.

Keep a gap between the number and the reward. The instant a metric is wired straight into pay and survival, the rot starts. The people you most need telling you the truth now have the strongest reason not to. Healthy cultures treat a measurement as one input a manager weighs against everything they can’t measure, not as the verdict itself. “Let’s understand why usage varies” survives. “Bottom decile is at risk” is already rotting.

Give managers a real reason to want the truth. Mostly this means taking apart the rollup, or at least refusing to let a manager’s standing be a straight sum of their reports’ numbers. A manager whose rating doesn’t depend on their team’s metric looking good is finally free to do the thing you hired them for — notice when the number and reality have split, and say so.

Point accountability inward, not outward. The reflex, when people game a measure, is to go after the visible gamers and stop there. The trouble with that framing is it quietly lets the managers off, and the managers are usually the ones who built the environment that produced the gaming.

A leader at the top can be careful and dodge the rollup trap — refuse to let their own standing ride on their org’s numbers. That doesn’t stop a manager one level down from doing the opposite. They take the metric they were handed and roll it down onto their reports as a hard target, then turn a blind eye to how the reports game it. The pressure to make the number gets passed along even without explicit design.

So when gaming shows up, the manager is rarely a bystander. Often they were part of the gaming — they wanted the number to look good and didn’t care how it got there. Where they weren’t actively in on it, they were just bad at the job, blind to the fact that their own pressure was manufacturing the behavior. I’m not sure which is worse, and for individuals the difference barely matters.

Either way, punishing the report who followed those incentives, while leaving that manager alone, fixes nothing and adds a fresh unfairness. Pulling back unearned rewards so gaming doesn’t visibly pay is punishment enough. But the real correction is at the center — the people who shaped the environment — not the edges. Fix them first. They’re the ones who’ll do it again.

Trust is the thing holding it all up, and it’s self-fulfilling. In a culture where people trust that a low number gets met with curiosity and that gaming gets caught instead of rewarded, you can introduce a measurement without panic, and the calm keeps it honest. In a culture where people expect the worst, they respond to the worst, and the response creates the very rot they feared. The belief and the outcome make each other. So trust isn’t a soft extra bolted onto a measurement program. It’s the load that everything else rests on. A measurement dropped into a low-trust team is closer to tossing in a grenade than running a diagnostic. (I’ve argued before that money itself is just trust — measurement inside a company is no different. It only works if people believe the other side is dealing straight.)

Stay humble about the whole thing. Management is a field of unintended consequences. Its job is to improve the value the people under it create, which in a perfect world would mean nothing, because you can’t improve on perfect. The mistake is for management to act as if it lives in that perfect world — clean numbers, frictionless incentives, its own presence bending nothing. Accepting that the world isn’t ideal, that every metric is a little flawed and gets more flawed as the work gets more complex, is the first of many steps toward doing it well. The cultures that can measure without breaking the measurement are just the ones that never forgot the tool was imperfect, held it loosely, and were willing to put it down once it had done its job.

The lesson hiding in the leaderboard

The leaderboards deserved retirement, but they weren’t all folly. They did an ordinary job that reached its expiration date. In cultures healthy enough to use them well, they were probably useful. Even in less effective cultures there may have been a golden period before the rot set in. Where they turned into a disaster, the disaster was already there, waiting — in the rollups, in the important things ignored for measurable stand-ins, in accountability that pointed outward instead of inward, in trust that had been spent long before anyone counted a token.

That’s the lesson worth keeping once the leaderboards are gone. The next iteration is already coming. It’ll promise the same clean view into the same messy reality, and it’ll rot the same way, unless the culture holding it has done the slow, unglamorous work that lets a company measure itself without lying to itself.

The hard problem was never the metric. It’s building a place where the truth can survive measurement.

More Data, Please

Ryan Baker — Tue, 30 Jun 2026 11:37:03 +0000

Reports that Midjourney is pivoting from AI-based image generation to full-body ultrasound scans have sparked a debate about the wisdom of performing widespread clinical tests. The medical community is generally advocating against the volume of testing that Midjourney is proposing. I understand their arguments, but I believe they are wrong.

This isn’t because doctors don’t understand medicine, patients, psychology, or even statistics. They do. Medical professionals are competent and genuinely invested in patient outcomes. The reason the medical community is getting this wrong is that they are failing to think through the lens of a broader data system. Understandably, their default mode of thinking engages with one patient at a time, and typically, one specific issue at a time.

But data doesn’t work that way. Data’s value is cumulative and often curves upward. More data is rarely a bad thing—provided it is used appropriately.

Currently, we miss opportunities to utilize data because of information processing limits. These limits aren’t just computational; they are also process-oriented. The maintenance of privacy across organizational boundaries and approving information flows for technical implementation. It should come as little surprise that applying AI to medical software will ease these boundaries. By automating those internals, we’ll be able to lower the obstacles without compromising on privacy controls, organizational limits, and traceable responsibility we desire.

With those obstacles gone, individuals with richer historical datasets will benefit immediately. The value of a scan, test, or medical record isn’t limited to an immediate concern. It can unlock a future insight. A future diagnosis informed by history is both more accurate and timely.

This cumulative value is what the medical community misses when they downplay the ambition of inexpensive, pervasive testing. Looking at a single patient with a single concern, an MRI (with its comprehensive snapshot) might always seem preferable to an ultrasound. But imagine an ecosystem where most patients have a baseline full-body ultrasound history, a 50-cancer screening blood test, and ten other sets of low-cost, routine diagnostics. In this ecosystem, there is a high probability that these data points will unite to form insights that a purely symptom-responsive history never could.

Individually, each cheap test won’t be conclusive enough to move a diagnosis from “possible” to “probable.” This introduces a valid concern: a test that shifts your probability of a disease from a background rate of 0.5% to 2% doesn’t merit invasive testing, but it might trigger hypochondria. But the combined history of three different inexpensive tests can shift that probability progressively—from 0.5%, to 2%, to 5%, to 15%. At 15%, invasive testing is merited. Without that cumulative history, your doctor would have rationally recommended against it, potentially missing a crucial early intervention.

Inversely, cumulative data can help avoid unnecessary invasive testing. Suppose you present symptoms that trigger concern, but your extensive testing history lowers the probability that the cause is malignant. If the symptom alone suggested a 15% risk, but your historical data shifts the overall probability down to 5%, your doctor could rationally recommend treating the symptom or taking a path with less upfront risk.

Some may interpret this as a personal call to build their own medical histories at any cost. While not entirely unreasonable, that is not my point. I’m not just thinking about individuals with enough disposable income to afford boutique medical testing¹.

Instead, I am defending the systemic value of testing that is cheap enough to be pervasive. Pervasive, low-cost testing can transform healthcare from a reactive, symptom-driven model to a proactive, data-driven one. It personalizes preventive care and provides the volume of data necessary for AI and modern analytics to spot macro-trends and micro-anomalies that a fragmented system would miss. Efforts to drive the cost of testing down to the point of pervasiveness are the foundation of that future. Those efforts should be loudly applauded, not feared.

The concerns about fueling hypochondria and over-worry are real, but ignoring the value of cheap, pervasive testing is not the solution. We already manage hypochondriac tendencies in medicine today. I wouldn’t call myself a hypochondriac, but I’m not immune to that type of feeling. In my experience, one anxiety-inducing event is the annual physical itself: cataloging every new bump or brief pain, anticipating the doctor’s questions, and overthinking the answers. We don’t suggest abandoning annual checkups just because they cause anxiety. Instead, we rely on a doctor’s bedside manner—a core component of medical training—to manage that stress.

I generally feel better once I’ve completed a checkup. I expect the outcome there depends heavily on how effectively the doctor communicates. That is where we can manage the risks associated with an influx of new medical data. We should expand our thinking beyond just the bedside manner of individual doctors, and focus on the design of the entire patient communication system.

The medical system already does this to some extent, but as we enter an era of pervasive data, there is a need for more. The answer to the anxieties of the information age isn’t to reject the data, but to design a healthcare system capable of communicating its true value.

It’s impossible to go from too expensive to use with a strong symptom to cheap enough to use regularly, without passing through the zone where wealth is used to access it regularly, but it’s still not cheap enough for pervasive use. So the critique of that being a possible outcome will always exist. But giving up on this basis doesn’t help much of anyone and historically, if you’re able to drop the price to the first level, the progression doesn’t stop there. ↩

The Invisible Profession

Ryan Baker — Tue, 23 Jun 2026 11:39:07 +0000

Opinions about AI are running hot. Everyone has one. I worry that the public isn’t informed enough to justify the rather direct opinions they have about AI, where they may want it stopped, slowed down or regulated. The public should care, as AI is going to be impactful. But that interest should translate into being well informed.

AI is more than software, but software is very much at the core of AI. And the reality is the public has not been very interested in understanding software development as a profession. A point of evidence for this, and a mechanism leading to being poorly informed, is that software development is one of the least-represented occupations by entertainment television and film.

I made this point in a story a few weeks back, in a longer post about bubble narratives. It was part of a larger narrative, but I heard confusion about why I brought it up. I thought it’d be useful to pull this out, and clarify why I think it’s an important topic.

Since software is at the core of AI, it feels like people should be informed about it. Being informed requires more than being a user. It requires understanding the lives of those developing it. It’s one thing to be disinterested in how your phone or computer works, but happy to use it. It’s another to have only caricatures of software developers to rely upon, know little about the basic processes we use to secure, develop and deliver software, and yet have an opinion on AI development.

Here is what I wrote:

Why the public has a poor understanding of software development

The wider world has never shown broad interest in learning what software developers do. Compared to other professions like police, soldiers, doctors, lawyers, musicians, writers, journalists or even criminals. Without that interest it’s unlikely to learn the inner workings of the profession.

Media portrayals of software developers are rare and rarely accurate. The most common portrayal is the “hacker” who mysteriously takes control of computer systems in a few minutes with no preparation. Not only is that a poor representation of a real hacker, it tells you nothing about software development overall.

Note: Scales are different per panel, programmers at 1x10-6 are 10x less frequent than actresses at 1x10-5, or 300x less frequent than doctors below at 3x10-4 (below).

Source: Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis, Baruah S, Somandepalli K, Narayanan S..

I reworked the data above to make it easier to see the patterns.

Use this Interactive version to dive deeper. You can also get at the other metrics, like the absolute appearances.

Ultimately, my concern here is deeper than portrayals. The facts about portrayals are an example. That example does have real effects, but entertainment media is also reacting to interests that precede entertainment choices.

The real concern is that advocating for laws or regulations that apply to software without an understanding of the process of development, is a recipe for failure. There are many places one might learn about an occupation. Media is just one, but it is a powerful one. Also, my experience as a member of the occupation is that most of my conversations about the occupation were limited to people inside.

There is a personal side to this. I’ve often been troubled by how the world in general treated the work I did as somewhat foreign. I noticed an imbalance in how much time conversations dwelled on my own work. It’s not the only profession like that, but it did register with me personally. I bring up my personal experience, not because my feelings are the point. They might be worth a little thought, but the point is I can see this disconnect so clearly because it is personal. If it’s not personal, it’s an easy gap to miss.

Software isn’t the only underrepresented occupation, but it is high stakes amongst those. It does appear in the news, but is it the facts of software developers that are being represented in the news? That’s not my observation either. The personal opinions of writers, artists and media that are heavily represented instead. Those same groups are over-represented in entertainment media, and have their opinions as the most visible via that route.

I’d suggest this is a moment where it makes sense to lean in a bit. It might not be natural to be interested in how software is developed. It might be more convenient day to day to enjoy the software, and treat the field as opaque. But if we need to make decisions about the field, that gap will become relevant. Everyone should be part of making those decisions, but the process of making those decisions must be well informed too.

So, if you’re a journalist, you should be interviewing software developers more often. If you’re creating a movie or TV show, you should consider writing in a character from software development, and avoid applying inaccurate cliches.

I’m trying to do my part with this writing. I probably can do more to make it more accessible. There is a tendency in software development, like any field, to write for your internal audience. That’s reinforced when outside interest is low.

If you’re a regular reader of this Substack, it’d be redundant to suggest you need to read something from a software developer, but I’ll assume this message may reach those beyond that scope. So encourage others to connect with the how of software development, both here and with other software developers in their lives.

That’s not going to be enough to bridge decades of disengagement, but it is a start. Since the political decisions of AI can’t wait for a repair, I suggest some humility in decision making here. In AI Safety Is Underfunded by Design, I suggest that we need balance that takes advantage of organic efforts of self-regulation. I wouldn’t say that self-regulation is sufficient; even if it’s working so far, this is too important to take that approach. But at the same time, regulating what you don’t understand will fail.

A learning process is necessary, both amongst the wider public, and with the politicians, lawyers, and administrators that would need to be part of any effective regulatory system, alongside industry experts.

Related Posts

The AI Gamble, Six Months On

Ryan Baker — Tue, 16 Jun 2026 11:36:56 +0000

In December I wrote The Architecture of a Gamble: Mapping the AI Value Chain. Given it’s been about 6 months, an update seems in order.

The architecture I outlined had 4 layers: compute supply chain, operational infrastructure, intelligence and application. Each layer has to justify its own expenses, which contribute to the revenues of the lower layers. When those aren’t justified, the whole structure is a gamble on the outcomes of the upper layers. That said, the immediate term for lower layers can be remarkably sound, because they get paid now, not later.

This story is only half written, so we can look at how each layer is doing today. Even when that story looks good, we shouldn’t extrapolate to the future. Each increment of demand must prove itself, both in terms of feasibility and in terms of timing.

How is each layer doing today?

Layer 1: The Compute Supply Chain

The main influence on this layer is the inputs from Layer 2. Since those plans have kept progressing as planned a year ago, it should be unsurprising that this layer has done well. Revenues continue to increase and stability looks favorable. The main change here is that competition is growing. The availability of chips designed in house, TPUs, Trainium and more, have grown. The efforts to make Intel capable of competing with TSMC have continued. TSMC has continued expansions of its own.

Additionally, memory and CPUs have been pulled closer to the middle. Memory was already a bottleneck and has become more central. GPUs have remained central, but share much of the spotlight with memory now. CPUs have moved from trivialities, to moderate importance.

Layer 2: The Operational Infrastructure

The situation at this layer has firmed up significantly in the past 6 months. Revenue for delivered compute has continued to grow. Additionally, commitments have been disclosed, adding stability. Disclosures about commitment numbers don’t express the exact terms, so we don’t know if these could reverse.

More importantly, everything is broader here. OpenAI and Anthropic are using compute from all platforms. xAI showed how to recover from a failure, at least partially, by selling unused compute to Anthropic.

All that good news shouldn’t ignore that there’s still significant spending planned that will have to justify itself. The revenues of today (+$100 billion ARR) are roughly proving that last year’s $410 billion in spending isn’t going to be unproductive. That doesn’t tell us enough about this year’s $700 billion, nor next year’s $1 trillion. So long as those keep growing quickly, the risk that expected revenue to cover it never appears should remain a live discussion.

Layer 3: The Intelligence

There are many positive updates here. The best ones concern Anthropic, which is a moderately bad story for OpenAI. Revenues have grown considerably. Anthropic expects to be profitable. Enterprise revenues have grown considerably. Could this reverse? Maybe. Much usage is still prototypes, experimental, or otherwise dependent on expectations. But it seems unlikely. As I covered last week, there’s more pulling us forward than back here.

Layer 4: The Application

This layer still contains a large amount of uncertainty. The most significant use case, software development, is fundamentally another layer. Last week I covered dynamics of token usage by the application layer. While this is one of the longest articles I’ve written, it only covers dynamics, not actual numbers, and only a few of the most important.

To be truly solid, the various parts of the application layer need recognition that value is being created. That’s hard, because there’s a lot that would need to be measured here, and many of the things you want to measure, are both difficult and emerge with lagging indicators.

Adversarial usage, in marketing ($660 million), legal ($650 million), and sales ($390 million), is following behind coding and IT usage.

Early adoption from sales and advertising, shapes how much effects need to be proven. From a financiers perspective, this isn’t a problem, maybe even positive. From someone worried about employment demand, it also isn’t a problem.

But from a social perspective, it’s not sound. We can’t expect significant positive value to come out of adversarial sectors. We want to see demonstrated productive revenue, and we want to see cases where freed labor is reutilized in growth sectors.

Conclusion

Is the AI industry past the “bubble” discussion? No. Certainly not in terms of stock prices and valuations. In terms of CapEx spend, and the likeliness of future contractions or firm failures, it’s on firmer footing, but the story still progresses as the mountain is only half climbed. You might say that anyone who put 2026 as a specific timeline for their concerns, has missed the mark, and there are many who did. But overconfident predictors don’t make an effective counter-argument to more general views.

Postscript: Fable 5

It would be unusual not to update here, as I’ve written about security in the past. It took a little while to become comfortable that I had enough facts as the administration’s story came out with so few details. It was hard for me to judge right off if that was because they had good information they didn’t want to share, or were just behaving irrationally. But the naive assumption seems to be true, they were behaving irrationally.

“Asking” for a shutdown, using bad evidence that doesn’t demonstrate any real harm is bad for everyone. For safety-minded people, it’s effectively “calling wolf”, and undermines any future action based on good evidence. For the safety-optimistic, aka accelerationist, it’s bad for the obvious reasons, but also bad because it creates a precedent of using bad evidence. This creates more uncertainty.

All parties, at least rationally, should want actions that are based on good evidence only. They might disagree on where the evidentiary bar should be, but it’s clearly bad when it’s randomly chosen. Since the capabilities they have evidence of being able to reproduce with a limited jailbreak are universal amongst models, you would have to ban them all. But they aren’t all banned, just one. That’s not a consistent system, and it should be obvious why that’s generally illogical as an outcome.

Originally posted at Substack: The AI Gamble 6 months on

Related articles

Are We in a Token Bubble?

It would be useful to know the shape of future AI demand, and many are attempting to predict that. Since this is a long piece I’ll give you my predictions up front. My overall prediction is that localized corrections, from the imposition of usage controls and consistent pricing, will ultimately be less important than the big trends. Value, so far hard to measure, will become more clear, first through incremental gains at the core of software development, and next, from the innovation that takes longer to accumulate and organize.

Read full story

The Architecture of a Gamble

A while back I talked about producing an analysis of the AI industry. I’ve put together something pretty extensive, but on reflection, I’ve decided to put it out in multiple parts. This post today functions more as an outline, where the following posts will dive more into each layer of this stack and then finally look in more depth at the macro-economic aspects.

Read full story

AI and the Zero-Sum Game

AI is advancing quickly, and if there’s any one consensus about it, it is that it will have broad impacts on jobs. What impact, is an area of more debate, but it’s uncommon to view it as non-impactful. Some believe that jobs will disappear, and there would be large amounts of unemployment. Some draw on past periods of technological change, such as the Industrial Revolution or the advent of the internet, and believe that advances ultimately lead to new jobs that didn’t previously exist.

Read full story

Are We in a Token Bubble?

Ryan Baker — Tue, 09 Jun 2026 11:35:54 +0000

Preview

Read on to learn how I add my experience in cloud computing and software engineering to my deep interest in economics to extend responses from two of my favorite writers. Along the way, I’ll recast the bubble analogy, explain recent trends that have hit the news, explain trends hidden deep in the development lifecycle, and provide a model, “Ingenuity Matrix”, for mapping usage intent to expected outcomes.

We love good stories, especially those with a villain. But we should be careful about our stories, knowing how powerful they can be.

Three stories have hit a crescendo at about the same time. Tokenmaxxing — companies turning token usage into a goal, metering it, and the waste that incentivizes. Subsidized tokens — questions on the relationship today between AI costs and pricing. And under both, the doubt about whether spending is producing value for AI customers.

Stack them together and a tidy narrative falls out. If these are what’s driving token usage, and they all adjust at once, the readjustment will ripple through AI industry economics — including Anthropic’s recently skyrocketing revenues. That narrative extends across all model providers, culminating as a cascading failure of the whole AI industry. Call it the token bubble, brought on by a revaluation of tokens and their utility.

It’s a neat story, but the framing is off, even before we get to evidence. “Bubble” as metaphor smuggles in two assumptions: that we’re looking at one structure, full of only hot air, and that it ends by popping. In reality, industrial bubbles deflate, running out of air. There is an inflated shell, inside of which a structure is being built, and its collapse while deflating halts construction within, and damages unfinished construction. But something remains.

Inflating the shell is not folly, but the simplest path to enable construction. It’s still calamitous when it deflates, but the goal is the structure, not the air. So the question I’m interested in isn’t “are we in a bubble?” It’s: which of these dynamics is air, which is structure, and how would you tell them apart?

The story has been covered by two of my favorite writers, Derek Thompson, in The AI Boom Has Entered Its ‘Wait, Is This Worth It?’ Era and Noah Smith, in How much more software do we really need?. Both play speculatively with the idea that spending and rationality may have split from each other, but retain optimism that something worthwhile is being built.

Thompson concludes his summary of an interview with SemiAnalysis’s Doug O’Laughlin:

Every new technology requires an extended period of trial and error, as organizations toggle between (a) not enough experimentation or spending, followed by (b) too much experimentation and spending, followed by (c) too dramatic a pullback, followed by (d) the repetition of steps (a) through (c), until firms figure out a long-term balance between labor spending and tech spending. Whether AI skeptics like Marcus are right that the bubble is about to pop depends entirely on a question that, as of today, nobody can definitively answer: Is the bill worth it?

Smith considers the period before a smarter than human in all ways artificial general intelligence:

But until we reach that point, it’s a nontrivial task to think of business models that could be fully automated even with an AI that can’t yet do everything. That’s going to be hard! If I had any good ideas for how to do that, I’d go become a billionaire myself.

At some point, though — maybe in the very near future — people (assisted by AI) will come up with those revolutionary new business models. At that point, tokenmaxxing will suddenly become a lot more economical, and Anthropic — or whoever has good coding agents by that time — will stand to make untold amounts of money.

These are good perspectives, but I can improve upon them to help understand the dynamics of AI usage. First, I’m from the software industry, which is at the center of the maelstrom — coding is now the single largest category of token usage. I can describe in more detail what developers are actually doing with these tokens, and their motivations. These details are important. Without them, a lot of valuable work remains mysterious, which invites doubts, such as “is this worth it”, or “do we need more software”?

Second, I’ve spent a while thinking about the adversarial dynamics of some AI usage, since first writing about it last year. Those dynamics are key to the questions both writers leave us with. Adversarial usage doesn’t produce the social value we all seek. It is not the only driver of AI usage, but when it is a driver, we should be asking, “is this worth it”?

Both writers are aware of an important detail, timing, which explains many misleading observations. With the addition of a deeper understanding of software development, and that model for separating zero-sum jockeying from the creation of social value, we can recognize events along the timeline with more accuracy.

Token usage, like human labor, can’t tell you progress. Its best analogy is effort. If you want to understand the effectiveness of effort, you want to know how it’s being applied. Different applications correlate with different outcomes. Since you can’t fast-forward to the results, this is the best immediate categorization you can add. I call this categorization, the Ingenuity Matrix, describing the scope and social alignment of token usage.

Some token usage goes nowhere by design, some burns down a backlog of long-deferred work, some is zero-sum jockeying. A slower, quieter share is the significant work that actually changes lives. Sort the usage that way and the “is it a bubble” question dissolves into a more useful one — what’s being built, what events can we expect along the path, and what risks and opportunities come with each set of events?

Background

Before we start into the model, understanding the two terms behind the narratives is useful. This will also be useful when reading general news on the topics. The narratives on these conflate multiple meanings, and smuggle assumptions. That ambiguity can support misleading narratives¹.

What is tokenmaxxing?

Tokenmaxxing refers to two things. First, it refers to companies’ creation of “leaderboards” tracking employee AI usage by metering tokens. These leaderboards might be informal, but there’s often an implied assumption that high usage is rewarded, and low usage risks consequences. Sometimes that’s explicit. Ostensibly the justification is to incentivize experimentation and overcome inertia. In addition to simple inertia, many companies started with restrictive policies discouraging AI usage that they needed to counteract.

The second meaning focuses on what happens when leaderboards encourage AI usage, but do so in unproductive ways. Some employees respond by trying AI more and doubling down on things that work. But they may also create or continue unconstructive habits, for no reason other than they generate tokens. Individuals have described such practices anecdotally.

In this dual definition, when companies tokenmax, they encourage both the good and the bad. When individuals tokenmax, we talk only about the bad. The most extreme tokenmaxxing isn’t ingenuity that misfires — it’s intentional waste. The intent isn’t to do work; it’s to appear to have done work.

It’s not hard to see how that type of usage leads to a narrative that it’s all a sham. But we should remember, what we have is anecdotes. While it’s certain that some waste is occurring, it’s hard to gauge. Anecdotes are sparse, and for good reason. Admitting to it, would be admitting to willfully ignoring the employer’s best interest in productivity. That would carry consequences if a manager discovered it and wasn’t interested in joining the deception.

But separating waste from sincere-but-unsuccessful experimentation requires details that simply aren’t available at scale. What we can say is that the organizations running leaderboards are making a deliberate bet: they’re buying a pile of unaimed experimentation and some willful waste, in exchange for a fraction that matures into something real — durable skills, a useful tool, an opportunity nobody had time to chase before. Whether the bet pays off, only time will tell. But the structure of the bet — accepting near-term waste to fish for longer-term capability — is something we should predict and model as a mix.

What are subsidized tokens?

Subsidized tokens can refer to three things.

The most common usage focuses on two billing models. One is metered, usage is measured and billed per token, at prices like $5/million tokens. The other is by subscription, for example $20/month. Subscriptions typically have usage limits, but in most cases, fully utilizing a subscription’s limits yields a per-token cost below the metered rate. In addition, loopholes existed, allowing usage far below the metered rate. Users who used their subscriptions heavily enough to get that benefit were labelled as subsidized. That’s a simplification though, as it could be a lower profit margin, not subsidization.

The second usage focuses on free tiers. Free tiers have restrictive usage limits, but with no revenue, they are clearly subsidized. Free users heavily outnumber paid subscribers. Across providers there are at least a billion free tier users, while paid subscribers would be below a hundred million.

The third and final usage translates the unit economics of metered usage into the underlying costs that model providers pay to compute providers, which pay for chips, power, and other infrastructure. The question the subsidy narrative is really asking is, are the unit economics of AI usage sustainable? Or are they a short-term attempt to grow usage, the end of which results in higher prices, and pulling back from usage that’s no longer economic at the higher price point?

It’s an interesting story, but it’s almost worth ignoring. The efficiency of AI is increasing quickly, driving unit costs down. If prices rebound, unless the rebound is something like 10x, they’d soon fall again. The reason they can’t be ignored has little to do with a long-term trend, but everything to do with the short-term viability of the financing of AI investments and presumed valuations. A company trapped in subsidizing while a competitor is not, is going out of business quickly. This pattern repeats at each level of the AI value chain.

Why Bubble as an analogy is over extended

I said in the opening that using “bubble” as a metaphor for the AI industry smuggles in two assumptions. A bubble is so commonly used to analogize industrial revolutions, that we fail to reflect on the limits it has as an analogy. One mistake it leads us to, is the belief that there’s a soap bubble floating in air, and when we prick it with a pin it will pop, and evaporate. This does a poor job of explaining reality though.

We might limit our imagination more effectively by replacing the soap bubble with an inflatable dome. Whether this stays inflated depends upon the balance of air entering and exiting. Inside this dome, we’re constructing something durable, but it would be a challenge to do so with the dome weighing on top of us. We need the air to keep the dome’s ceiling from impeding our construction, and if it deflates it will probably ruin any half constructed structures. The stronger completed structures can sustain the weight of a deflated dome, but will struggle to conduct any additional construction.

If you want to think of the social support for a system, which supplies the air to keep the shell inflated, as a bubble, that’d be fair. This can evaporate with a bad news story, or some other form of social contagion. That social support is what replaces the air that leaks out. We’ll discuss the leaks later. Some are necessary, some are not. But replenishing the loss is unavoidable.

It’s useful to remember that in this analogy, deflation isn’t free. Something will remain, but the damage to unfinished construction is real. Careers are an obvious example of the consequences. When companies downsize the skills, connections and tacit knowledge built to support growth get stranded. If people move on, they may never come back. And besides, they are people and the disruption to their lives matters too. Projects also take a hit. Some projects may be zombies, shambling along with an unsound structure that will never be completed. But the forces of deflation aren’t so selective, and promising work is wiped away as well. Many projects that stop work during periods of tightening never start again.

A second flaw in the analogy is as a singular structure. Not only are there independent structures being built within, there’s not a single dome. There is a primary dome, where the model providers, GPU manufacturers and designers, and much else reside. But AI is also working to serve many different industries, and we shouldn’t assume a shared fate between all those efforts. We do want to pay attention to software development, because it represents such a large fraction of current usage. But software development itself isn’t an end of its own, it serves other industries. If AI is effective at helping some of those, and less-effective in others, this doesn’t establish a shared fate. It is only those cross-cutting effects that affect all software development that would carry that risk.

For the most part, those outside of software development aren’t going to understand those cross-cutting effects. I’ll highlight some of those details here, as they should be relevant to anyone interested in the immediate future implications of AI.

Why the public has a poor understanding of software development

The wider world has never shown broad interest in learning what software developers do. Compared to other professions like police, soldiers, doctors, lawyers, musicians, writers, journalists or even criminals. Without that interest it’s unlikely to learn the inner workings of the profession.

Media portrayals of software developers are rare and rarely accurate. The most common portrayal is the “hacker” who mysteriously takes control of computer systems in a few minutes with no preparation. Not only is that a poor representation of a real hacker, it tells you nothing about software development overall.

Note: Scales are different per panel, programmers at 1x10-6 are 10x less frequent than actresses at 1x10-5, or 300x less frequent than doctors below at 3x10-4 (below).

Source: Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis, Baruah S, Somandepalli K, Narayanan S..

When software is “done”

If you come from outside the software world, you’d be excused from thinking of software development as building new software. In reality, this is a modest part of software development. Maintaining software, deploying software, and operating deployed software all represent larger segments than new software. All said, new software could be as small as 20%.

Noah makes a tentative argument that “The world may already have most of the traditional software that it needs.”. Noah’s aware he might be getting this wrong, and indeed he does. It does take an immense amount of work to keep sites running. AI is being used here, but it started later than its use to create new software. It’s not too hard to guess why. Creating new software is low risk comparatively. Like everyone, trust of AI has been a process. Software maintenance and operations themselves rely on significant “tech-stacks”, which have to be modified before you can even attempt to use AI to make a site more reliable in a meaningful way.

The number of software releases for security, operational, monitoring and development oriented features has been significant over the past year. Many use AI. Probably many others were built using AI.

Should you expect faster load times and higher reliability? First, would you really know? These have been improving for years, yet the general public rarely comments upon it. Mostly the only comments are those times when something does fail.

I’m not holding this data above as proof that AI has improved reliability. These improvements are more likely the result of conventional engineering, some started years before the results. The results that are “AI” based, are the result of the “machine learning” form that predated the architectures for Claude, Gemini and ChatGPT.

The point is, Noah (and you too) probably aren’t a sound judge of whether improvements are occurring unless you take the time to gather data. From my own knowledge, I know most AI based improvements are still in the early phases of adoption. But the average person shouldn’t expect to have an intuitive grasp on this. We’re bad intuitive judges of background effects like this, where we have to compare changes over time of non-continuous events. We can recall the last event, and the last change, but we are just as likely to draw a pattern from a recent reaction, than from an accurate history.

Why “tech-debt” comes first

You’ll find an interesting pattern that I’ll get into later. The first work to be done is the “shovel-ready” work. It’s easy to generate a prototype for some random idea, but rarer to have a great idea that can go from ideation to production quickly. AI does speed that up. But it doesn’t speed all work up.

With that in mind, provide a tool to a software developer, and they’ll have a long list of things they wanted to do, but haven’t had time for. Our general term for this is “tech-debt”, but realistically, it also includes half-baked feature ideas, or features that were sound but never made the cost-effectiveness cut. This list predictably contains a lot of non-amazing things. If they were amazing, they would have made the cost-effectiveness cut the first time. But AI does give you a reason to go deeper into that marginal backlog.

Security as a priority

I should also mention security here. Security is extremely important to the operation of software. Failures of security are nearly the worst thing you can imagine. This applies to all phases of software: development, deployment, and operations. It’s tempting to think of security as something you simply develop. But in reality that’s just the first step. A significant failure in development is likely to lead to a significant failure later, but it’s not destiny. You can layer protections to mitigate a development failure during operations. You have to do this because there are development failures you don’t know about. And more importantly, even a soundly designed and developed system can fail if not operated properly.

A lot of time and money is already spent on security. It’s never been the case that it hasn’t been a priority. You can find cases where it wasn’t a high enough priority. But it’d be a stretch to suggest there was a case no one cared. Whatever the priority, there is a limit, a cost-effectiveness barrier where one of the stages of development could have achieved more with more inputs. The introduction of AI changes the math on that barrier and makes many things practical that were impractical.

Security has another dimension too, which is that in addition to AI altering the developer’s cost-effectiveness equation, it does so for attackers too. This creates another incentive to burn down the security backlog. Security can’t wait. And so a lot with good cause, a lot of AI based productivity is going into security efforts.

This isn’t an effort that’s particularly visible to the outside world. What the outside world knows about it comes mostly from stories, not direct experience. When developers patch security holes, their intent almost always is to not change the user-experience. When that is the intent, it’s a slower process, because it requires educating users about new security mechanisms they need to participate in. Because that’s such a difficult thing to do, security teams have a very strong preference toward solving problems themselves without involving the users. It’s not always possible, but 90% of security efforts are invisible to users, and the next 9% are delivered as patches users see installed, but don’t pay any attention to.

The ingenuity matrix

I said in the opening that token usage is like effort: it tells you activity, not progress. To get from effort to expected outcome, you have to ask what the effort is for. Two questions do most of the work, and together they form a grid.

The first question is social alignment. Does the work create value the world didn’t have (positive-sum, pro-social)? Does it merely move value from one party to another (zero-sum, non-social)? Or does it destroy value — burn resources, or actively harm (negative-sum, anti-social)?

Alignment can be informed by our guesses of actors’ intent, but it’s not dependent on it. Our best bet is to act as an outside observer, guessing at outcomes. I don’t want to overcomplicate this though, this is estimation after all. Some significant pro-social value sometimes arrives from someone tinkering purely for fun. The social alignment is still recognizable from the outside, even when the actor wasn’t aiming at it.

The second question is scope , how far the work is reaching. Significant work aims at a real leap. Simple work aims at something bounded and modest. Naive work isn’t aimed at a productive outcome at all. Here “naive” describes the absence of a useful target, not the absence of a motive. Intentional waste is naive in this sense, it produces nothing of value, even though the person doing it has a very clear motive.

What you’ve just toured, security patches, reliability work, performance and cost tuning, is real value, almost all of it invisible to the people who benefit. Nearly all of it lands in a single cell: simple, positive-sum. It’s illustrative that so much of what is immediate is within simple or naive ingenuity. The first things individuals use AI for aren’t the significant ones. It’s the modest, shovel-ready, often-unseen things.

Map the rest against those two axes and you get an ingenuity matrix :

Ingenuity Matrix: Scope and alignment to predict economic outcomes

Negative-sum is not hypothetical, it connects back to security. The same drop in cost-of-effort that lets defenders finally burn down the security backlog also lowers the attacker’s cost. AI-assisted cybercrime is simple, negative-sum ingenuity, and the prospect of AI-scale biological or infrastructure attacks is the significant version. A large share of the invisible defensive work isn’t optional improvement, it’s the response to an adversary. The result is effort that is no longer avoidable, but also hidden, which delays the visible gains we’re watching for.

Significant non-social ingenuity ends empty. Non-social work can seem significant when under development. But one of two things happens. Either the work ends up leaking into pro-social, or anti-social accidentally, or it is copied and becomes trivial. Significance and neutrality are generally unstable.

Naive ingenuity is where the most visible tokens are burning right now and the least is being built. Failed experiments and aimless prototypes aren’t worthless — they build skills and occasionally surface something real, which is the option value the leaderboard bet was buying — but as a category they go nowhere by design. Because naive usage is so voluminous, and personal, it’s the most visible to the simplest forms of observation. That helps it dominate the “is this all a sham?” narrative.

Simple ingenuity has significant usage, but is quickly forgotten. The high volume usage is generally operationalized, contributing to security, reliability or operational efficiency. It’s soon forgotten, as it becomes a background effect. It doesn’t have the humorous, villainous story of tokenmaxxing. It doesn’t receive the personal promotion of the latest experiment.

One of the hallmarks of simple ingenuity, is it could be described as a backlog. The work may have been identified as desirable a long time ago, but with other competing priorities, it wasn’t prioritized. It may also not have been cost effective. One of the changes that AI brings is a change in cost-effectiveness. This activates this backlog, and you should expect early effects to burn this backlog down.

Simple ingenuity comes early and makes existing work more efficient. Sometimes this will show up as measurable revenues, but much is internal to companies. In that case it’s the token usage, the lower labor costs, or the higher quality that are the observations.

When AI enabled workers have a clear backlog, efficiency gains will flow into simple ingenuity to burn down the backlog. If the backlog results in priced or measured output, you’ll know.

Significant ingenuity will take longer to be identified, developed and deployed, especially the pro-social variety. The economy will reuse freed labor to create more value. That won’t happen immediately, as it may wait on hiring processes, training processes, or even the formation of new companies pursuing new products or business models.

Timing

While the development process is accelerated, the identification process retains most of its bottlenecks. Optimism may accelerate it. Idleness may accelerate it. But optimism and idleness may also flow into naive ingenuity, pursuing trivial goals without positive utility. There is a blurry area where naive ingenuity is experimentation. It may fail, but its failure may be necessary to build skills or discover significant opportunities.

At some point, a few things start to coincide. Naive and simple ingenuity will have built skills, ready to be exploited for realizing significant ingenuity. The backlog’s distraction fades as it burns down, and a new equilibrium raises the incentive to chase significant work — significance always carried more reward, but also more risk. But as cost-effectiveness decreases deeper into the backlog, avoiding risk becomes less attractive. All of these, in addition to the passage of time, predict a future wave of significant ingenuity that direct observation of measurements would fail to predict.

New output

Most of what we’d recognize as new output is significant, pro-social, and lagged. These are the life-changing things, and they’re the hardest to forecast. Your best guide might be a science-fiction novel, but of all the futures sci-fi writers have imagined, which do you bet on? Like flying cars, some things that look a step away stay out of reach far longer than expected.

It would be a mistake, though, to generalize from the failed predictions to all predictions. In many ways today’s information world already outruns older sci-fi imagination — the 1987 Star Trek: TNG depicted computers far beyond the 1966 version, and on the information front we’ve roughly met the standard it set for the 24th century already. The significant wave is hard to time and easy to underestimate at the same time.

What to expect

We should expect the AI industry to experience some pullbacks, then continue on. Whether this ever meets the bubble narrative is uncertain. I’m skeptical. Many pullbacks will be met by other accelerations. One experiment fails, another scales.

There isn’t one bubble, ready to pop, but multiple domes. Each industry, each set of users finds their value. While software remains so dominant, a failure in the software use case could be dramatic, but much of it is boring simple work that will continue to be automated for some time yet.

Much of the immediate term work is going to focus on the simplest, most invisible aspects. We shouldn’t discount the value there. Where it’s defensive, answering the negative-sum, like security, it has to be done. Where it’s part of more normal systems, it’s freeing resources, and developing skills and experience that will fuel more significant ingenuity in the future.

You do have to wait to see world changing effects. Software, as a model for implementing a workflow, will remain, and the general skills of software developers will be critical to this. Lines will blur, people will cross-over the lines, but ultimately the concept of software will continue to exist.

If the software dome does collapse, it will create structural damage, like all such events. Failed companies, layoffs, abandoned projects. Resources for naive experimentation would evaporate, and companies would proceed more cautiously. But a structure will remain. The burned-down backlogs that don’t un-burn, the skills that accumulated, the efficiency that keeps paying out, and the significant work just beginning to grow.

So, are we in a bubble? Will users and companies pull back on token usage, looking for value, discouraging wasteful tokenmaxxing? Will they react to pricing changes from model providers that close subscription loopholes that allow token usage in excess of what the same money would have bought per token via API? Yes, they will, but will that cause revenue drops that deflate the dome?

I don’t think so, there’s enough pending and developing work to fill the gap. Even if the significant ingenuity is still developing, the simple work is sufficiently valuable and important. But maybe those dynamics will return next year. If compute providers continue yet more expansions, they still might find them getting ahead of demand. There’s a lot of history to be written here. I’d just be careful about writing the ending first.

Sources

Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis; Baruah S, Somandepalli K, Narayanan S..
State of AI, An Empirical 100 Trillion Token Study with OpenRouter; Malika Aubakirova,Alex Atallah,Chris Clark, Justin Summerville, Anjney Midha
The AI Boom Has Entered Its 'Wait, Is This Worth It?' Era; Derek Thompson
How much more software do we really need?; Noah Smith
State of AI in the Enterprise The untapped edge; Deloitte

Original Posted on Substack as Are We in a Token Bubble?

As an example, Noah Smith quotes a commonly quoted study on tokenmaxing that claims diminishing returns to token usage, but presents data that should be interpreted as the opposite. In their description, they compare the number of tokens used to create PRs, and the costs of those tokens. ↩

To evaluate whether that spend is worth it, we joined token usage data with actual developer output, measured in merged pull requests.

Over the course of Q1 2026, developers in the bottom 20% of token spend used only about three dollars’ worth of tokens for the entire quarter and shipped an average of 11 merged PRs. By comparison, developers in the top 20% spent $1,822 over the same period and shipped 23 merged PRs on average.

In other words, significantly higher token usage does lead to more output, but not proportionally. The cost per merged PR increases from just $0.28 in the lowest usage tier to $89.32 in the highest.

More tokens means more output, but at a much higher price per unit.

But if you’re comparing costs, the correct comparison would include developer time. If we take a conservative cost of $10,000 / month for a developer the calculation we get is:

Low token group: ($30,000 + $3.08) / 11 PRs ≈ $2,727/PR

High token group: ($30,000 + $2,054) / 23 PRs ≈ $1,393/PR

There is a sense in which you could use this data to describe diminishing returns, but it’s not in the realm of cost effectiveness. If someone proposed that development was accelerating exponentially in the way that token usage is, they’d be wrong. You cannot scale development at the speed of tokens because it is still dependent on developers.

AI Safety Is Underfunded by Design

Ryan Baker — Tue, 19 May 2026 12:32:36 +0000

Dean Ball recently put his finger on something important about AI liability and incentives:

In general, market actors do not have great incentives to protect against catastrophic risks. They are massive negative externalities, often dwarfing the balance sheet of any individual firm. Say Anthropic releases a model that a malicious actor uses to conduct a cyberattack that does $5 trillion dollars in damage. Anthropic is only worth $800 billion, so if they get sued for $5 trillion, they are already well past the point of insolvency. A catastrophic harm may well already be “lights out” for Anthropic, or any other company, so there is little incentive to avoid them, if doing so entails real costs in the present day.

He’s right about the structure of the problem — but “little incentive” understates the precision available here. AI companies do have incentive to avoid catastrophic outcomes, just systematically less than society needs them to. That gap can be quantified, and quantifying it points toward what a corrective policy should actually look like.

The concerns he’s talking about — catastrophic risks — share a structural feature that distinguishes them from others: they are lumpy. A single catastrophic event, rather than a diffuse trend. The incentive is quite large, but not as large as it should be. These dynamics are worth exploring, as those ultimately shape if and how we structure a response.

Consider a hypothetical AI company, worth $800 billion. Now consider a hypothetical event causing $5 trillion in damages. If this event happened, that AI company would be out of business, so they have an incentive to prevent it. But how much incentive? The most they can lose is the whole company, so $800 billion. Since a lot of that is goodwill, in reality, losses become irrelevant earlier. For the sake of example, we’ll say $400 billion. If you had to pay half your market cap, you’re not worth $400 billion, you’re bankrupt, and worth $0. All claims greater than $400 billion have equal impact, since each produces the same outcome, a total loss.

This creates an imbalance between societal goals and the AI company’s goals. That imbalance could lead to underinvestment in safety, or risk taking that is out of alignment with societal goals.

We can quantify this imbalance, by modeling a damage cap in expected value calculations. If our $5 trillion event has a 1 in 10,000 chance of occurring, the uncapped expected value of avoidance is $500 million. With a damage cap of $400 billion, it’s only $40 million. Society should want that other $460 million in incentive to be shared by the AI company, but without an arrangement, it’s not.

Refinements

I used a simple model above, with linear effectiveness of investment in safety. It isn’t linear. In a linear model, spending $500 million reduces risk to zero, and $40 million reduces it to 1/12th of that, or one 1 in 9,166. But we could imagine, in fact we should expect, that the first $40 million does more than the next $40 million. Maybe the first reduces the risk to 1 in 100,000, and the next to 1 in million. It’s the same proportional improvement — 10x. But in the first case it reduces the risk from 100/million to 10/million for a total reduction of 90/million. The second case reduces from 10/million to 1/million, for a total of 9/million reduction.

To illustrate, I constructed a model that used logarithmic decay from the initial 1 in 10,000. In this model, under their default incentives, the AI company would want to spend $13.3 million to reduce their expected risk from $40 million to $8.7 million. But the societal risk is still $122 million at this point.

The goal of a corrective policy would be for the AI company to act upon the societal risk, which justifies spending $35.2 million to reduce the societal risk to $8.7 million.

Further refinement considers whether organic spending is more efficient than regulatory-induced spending. For example, if regulatory-induced spending had half the effect per dollar as organic spending, not only would the spending go up, but the residual damage would be higher.

In Scenario 1, all spending is equally valuable. In Scenario 2, the company spends efficiently up to its capped motivation, after which each real dollar buys only $0.50 of effective safety. And finally in Scenario 3, all spending is at 50% effectiveness.

You can explore other scenarios in the linked single page app (GitHub Repo). You can experiment with different decay functions, damage sizes and company sizes.

Organic Forces

We’d also be doing ourselves a disservice if we didn’t recognize the outstanding work that AI companies have done —Anthropic most of all. Commercial incentives alone aren’t sufficient. Thestories about the founding of DeepMind and OpenAI make clear that good intent has played a positive role. But in a commercial world, you can’t depend on good intent to reliably show up or win internal contests. The pressure to go off track is substantial. For something this important, that’s a lot of trust. We want to use these forces, because they are efficient, but must not be naive either.

We don’t want to take organic forces for granted. If we assume they don’t need support, they might disappear. If we don’t acknowledge their value, we might strangle them.

Small Firms

The alignment problem becomes more acute for smaller companies. What if a smaller startup, with none of the weight of a larger company — little to lose, and everything to gain — rushes ahead, and skips best practices that avoid harm?

The leading labs are large (Anthropic, OpenAI and Google), but we shouldn’t take that for granted. Frontier model training costs keep going up, but the costs for a particular level of capability keep going down. DeepSeek proved that moats are much shallower than assumed.

You do want to avoid locking out startups, but also need a baseline that ensures safety isn’t skipped. A first step here is ensuring safety practices are shared. That lowers their costs in pursuing safety.

The current voluntary norm — leading labs sharing safety methodology despite having competitive reasons not to — is a favorable state of affairs that formal structure can preserve and extend. It will take organization to make it work at a deeper level. Sharing details of some safety practices publicly can add risk, so a well-trusted network for sharing enables more than just the public domain approach. Formalizing sharing as a condition of operating at the frontier, both preserves what already occurs, and can extend it more deeply.

Regulatory Shape

The model makes the policy objective concrete: close the gap between what the company is motivated to spend and the societal expected value, without crowding out the organic safety investment that’s already happening.

A naive response assumes insurance is enough, and the challenge is finding a large enough reinsurer to pay out. An even more naive response assumes this challenge can be fixed by inserting the federal government as a backstop to the insurance. The flaw in this thinking is that it makes society responsible for paying itself back for harm done to it. This won’t work. The harm would have been done. Society would pay for the majority of the consequences of the gamble the AI companies made.

These dynamics suggest that effective regulation needs balance, in order to use organic forces, and yet also not leave a gap. Dean is right that it improves the case for government involvement. If there’s a gap between the company’s incentives and the societal incentives on a topic so important, we should align those.

An industry body that both shares security practices and sets standards is a start. Shared excess liability amongst all AI companies would add to existing incentives. If one fails to prevent harm in a small way, that company fails alone. If one fails in a big way, they all fail. Expanding the pool in this way is better than involving the government, as these are the players with the ability to influence the risk. Those incentives will encourage maintaining quality standards, but keep the standards moored to efficiency and effectiveness.

That’s still not enough though, so a government body above that respects the value of organic forces, would be a second step. The challenge here is how to prevent this body from losing interest in efficiency. It’s natural for them to be interested in effectiveness, but efficiency comes with more difficulty. If standards ignore efficiency, you undermine the organic forces and risk taking a step backwards instead of forwards.

What doesn’t work?

The framework I’m discussing, can appear to be a compromise between two points of view. That’s not the intent. There is no intent to choose a middle point, in order to satisfy two points of view. I think the merits of this model fit without any politics.

The model does however balance multiple forces, and is not aligned with any maximal plan. That type of balance only makes sense if the maximal plans aren’t reasonable. To make it clear, I don’t support any maximal plans. It will take additional posts to flesh out why, and others have defended these points independently. But in the light of outlining my thinking, the basic is:

AI bans: You have no chance. You have no global solution. It’s not a good idea in the first place, as AI will be very useful, but that’s not the biggest flaw. The biggest flaw is all of the partial wins - company X refuses to use AI, country Y bans AI - they all fail in the end and don’t contribute to any goal aligned with the best case for a ban.
No regulations: Clearly something is needed here. This group is somewhat of a strawman though, as even people like Dean Ball see a role for regulation. The better critique is that there are many people who are implicitly “no regulationists”, because they oppose everything proposed and don’t put together enough to actually do something.
Top-down regulations: Strangling organic safety efforts in top-down paperwork is a surefire way to fail. That doesn’t mean there isn’t a top, but it does mean, it can’t be total, and since it’s starting later, it should expect to start small and iteratively find its fit.

Clearly, there are more details to cover here. I’ve only touched on one dynamic that sets an overall tone, but you’d eventually need a list of initial best practices, and an expert-led group to maintain them. You’ll need a mechanism to choose that group, and a list of powers and limitations that define how they work together, and resolve conflicts. I’ll leave those questions for a future post though.

Original Posted on Substack as AI Safety Is Underfunded by Design

Control and AI

Ryan Baker — Tue, 28 Apr 2026 11:03:54 +0000

Earlier, I wrote about determinism and control. I feel a need to return to these concepts because they are the quiet shift beneath software, and deserve greater attention.

The shift from traditional software to AI is a shift from deterministic systems (where a specific input leads to a specific output) to indeterministic systems (where outputs are probabilistic and fluid). Almost every magical capability of AI is downstream of this indeterminism. But crucially, so are its most frustrating limitations.

If there is one fatal misunderstanding of AI today, it’s that we are engaging with this shift inadequately. “Indeterminism” has entered the lexicon, but usually only at a surface level. And because we are stuck on the surface, the loudest debates about AI have become incredibly boring.

Why the Extremes are Boring

Let’s look at the three loudest factions in the AI debate.

First, the AI doubters. They look at the unpredictable, indeterministic nature of large language models and declare it a failure. To them, a system that hallucinates cannot be trusted, and therefore cannot be useful. This is a boringly misguided example of confirmation bias. Humans are highly indeterministic—we forget things, we make math errors, we have bad days—yet we’ve muddled along reasonably well. How? By inventing deterministic tools to anchor us: long multiplication, checklists, standard operating procedures, etc. The doubter assumes you can’t extract value from an unpredictable system when you need reliability. History proves otherwise.

Second, the AI doomers. They also view indeterminism as a critical failure, but in the opposite direction. They are painfully aware of the immense power of AI systems and assume that this power is inherently uncontrollable. While this makes for a more gripping narrative than the doubters’ view, it strips away human agency. We’d have only one option left, don’t create powerful AI. Setting aside whether it is even possible to perpetually prevent its creation, this fatalism leaves no room for a practical conversation about how to retain control.

Finally, the radical accelerationists. They acknowledge the wild nature of AI but fall prey to a blind optimism, assuming a purely indeterministic system will somehow self-regulate and perfectly align with our needs. This is just as boring. The need for control is not irrational, nor is control a given. If control is achievable, it will demand a deliberate, concerted effort, requiring understanding every tool to engineer that control.

If you want to find interesting conversations, look for the solution seekers.

The Solution Seekers: Layers and Workflows

The most compelling builders today are those who reject both absolute pessimism and absolute optimism. They recognize that solutions aren’t singular or total. The most promising path is layers and workflows that mix and join determinism and indeterminism.

Think about how we manage high-stakes reasoning in the physical world—like in an intensive care unit or the cockpit of a commercial jet. We don’t rely entirely on the raw, in-the-moment reasoning of a doctor or pilot; human reasoning is brilliant but fluid, prone to fatigue, distraction, and variance. But we also don’t rely entirely on rigid, unyielding flowcharts, because a flowchart cannot reason through a novel, complex anomaly.

Instead, we design workflows that rely on both. We build strict, deterministic protocols—mandatory checklists, hard limits on medication dosages, automated collision warnings—to create a safe, predictable framework. Inside that framework, we rely on the judgement of a doctor or pilot to handle context, nuance, and problem-solving. Protocols enforce absolute boundaries; experts provide reasoning. Frameworks change, doctors update their own based on their learning, with debate and review, inside another layered framework.

This is the architecture of the AI future. AI will dominate the next generation of software, but it will not render deterministic code obsolete. Instead, code is how protocols are encoded. Those route, authorize, evaluate, and constrain indeterministic AI actors. Control points written in deterministic code will provide the necessary mechanisms to enforce rules, isolate agency, and supply safety. AI will be called upon within those specific boundaries to reason, interpret intent, and adapt to the messy reality of the user.

The Myth of the Developers Demise

This need for control has profound implications for how software is built. Recently, the term “vibe coding” has emerged to describe the practice of building software through natural language interactions with AI. A maximalist subgroup makes an extreme claim that with vibe coding, developers are obsolete and users will prompt their own custom software into existence on the fly.

This misses the fundamental purpose of a developer. A developer’s job is not to write code; a developer’s job is to remove effort for the user. Developing is ultimately not about producing code, but about producing reusable, accessible capabilities for users. An accessible capability is one that requires the least effort to access, and a reusable one is one that can be applied to multiple situations. Code is just the mechanism.

When developers create software, they establish guardrails, conventions, and reusable patterns. Sometimes, a user wants absolute flexibility, and a fluid AI companion is perfect. But often, a user wants rigid reliability. They want to press a button and know exactly what will happen. It’s easy to forget, amidst the explosion of AI capabilities, that rigidness has immense value.

It’s tempting to view recent advancements as a single evolutionary timeline—assuming we are moving from hand-written code, to AI-assisted code, to a future where code is entirely replaced by just in time reasoning of AI agents. That is a mistake, over-extending a trend. Committed code, generated, reviewed, tested and committed as stable will exist in abundance. Just in time generated code, executed in a protected sandbox will also be used abundantly.

The use of models and instructions, reasoned upon just in time, shifts the balance point between flexibility and rigidity, but it won’t abandon code nor the developer.

A Shared Experience: Taming the Machine

For users, future software interfaces will be a mix of structured and natural. Learning to navigate the difference between them will be a vital modern skill.

Structured interfaces (buttons, menus, traditional apps) sit atop deterministic systems. You can trust them to follow a plan. However, that plan was written by a developer. If the developer didn’t anticipate your specific need, the software becomes frustrating. You are forced to learn its non-intuitive logic.

Natural interfaces (chatbots, voice agents) sit on top of indeterministic systems. They can do things developers never anticipated and can interpret your unique intent. But they make assumptions. Using an AI interface is like ordering from a waiter at a restaurant. You need to develop an instinct for how your communication might be misinterpreted. You need to know when the system will ask a clarifying follow-up question (”soup or salad?”), and when you need to be proactively rigid and structured in your commands (”hold the mustard”). Make a mistake here, and you end up with a mustard-covered sandwich. Everyone then has to start over from scratch, and someone has to pay for the waste.

Interestingly, the people building the software are going through the exact same transition.

Developers are increasingly using natural language to write code. For a brief moment, this felt like magic without rules—just type what you want, and the machine builds it. But developers are quickly realizing that an AI coding assistant is just as indeterministic as a chatbot. If they aren’t careful, they end up with the equivalent of a “mustard-covered sandwich” deep in their codebase.

Because of this, we are watching a new kind of structure reemerge in software development. Developers aren’t abandoning natural language, but they are scaffolding it. They are learning when to let the AI riff creatively, and when to enforce strict, deterministic tests to verify the AI’s output. The developer’s job is evolving from writing rigid rules by hand to managing the chaotic intelligence that writes them, locking its best outputs into place so they can be relied upon tomorrow.

Conclusion

For decades, our relationship with computers was fundamentally one-sided: humans had to learn to speak like machines. We memorized menus, learned strict syntax, and clicked exact sequences of buttons. We were forced to be rigid operators of deterministic systems.

AI flips this dynamic, but it introduces a new burden. The era of the comprehensive user manual is over, because you cannot write a complete manual for a probabilistic system. Its capabilities are discovered through interaction, not documented in a spec sheet.

This is why understanding the architecture beneath your feet is no longer just a concern for software engineers. It is a vital literacy for everyone.

If you are an everyday user, recognizing whether you are interacting with a deterministic system or an AI agent changes how you engage. The caution you apply to inputs and outputs should shift. For deterministic systems you should provide what is required and just what is required. For AI systems consider where elaboration yields better results, and vagueness leads to guesswork. Unless you need guesswork, avoid triggering that path.

If you are trying to predict where the industry is going, looking for these architectural layers is the only way to cut through the boring extremes of blind hype and cynical doom.

And if you are a builder—whether you are writing thousands of lines of code or just stringing together a few tools to solve a daily problem—understanding this duality is your ultimate advantage. The future of technology isn’t about choosing between the rigid reliability of the past and the creative chaos of the future. It’s about learning to bolt them together.

Update on AI CyberSecurity

Ryan Baker — Thu, 16 Apr 2026 16:38:53 +0000

I’m travelling this week, so this will be short, but I thought the reactions to Mythos have been interesting. The core reaction, after a little panic, has been consistent with the structure I outlined in Security Can’t Wait last month. Namely, the short term brings some risk, but the long term favors the defender.

One thing that is still being missed, is why the long term favors the defenders. One of the reasons is that fewer defects is good for defenders in a generally absolute way. But another one relates to costs and benefits. Read that economist article linked and the final statements, suggesting that defenders will have to pay highly to discover defects.

Now reflect, that this isn’t new, it has always been expensive to discover defects. The risk that products like Mythos bring is that they lower the cost of discovering defects to exploit. The solution is to raise that cost. That might tempt you to suggest you should rewind the clock, and never invent Mythos. That’s not a solution though as eventually attackers would invent something similar, and you would then lose any control and advantage from the defenders being the first with access.

Instead the solution is that you find as many easy defects as you can and fix them. The first 100 defects might cost $20,000 / defect to discover. The next 100 might be $40,000 per, etc. Along the way you end up with defensive layers that are more and more reinforcing, and the cost for attackers to discover defects goes up, especially if they have less sophisticated tools, and/or have to spend a lot to first illicitly gain access to tools. When Mythos is publicly released you can generally assume providers will increase their attempts to find and ban users with ill intent. Those protections create costs for attackers, such that if a defender can find a defect for $20,000, an attacker might need $100,000. The attacker’s main advantage is they just need one, but as unpatched defects become more rare and harder to find that advantage tends to shift toward favoring the larger aggregate budgets of defenders.

The defenders have a strong advantage in terms of money. Where they struggle is in organization, because they have a much harder organizational problem to solve. The hard part about being a defender is getting changes deployed everywhere quickly. Once attackers find a defect, they can try and use it everywhere. If they find it first, that works in a lot of places. If they find it second, it’s dependent on how organized the deployment process is.

And this is why the long term economics favor the defender. Statistically, most defects are found first by defenders, due to larger budgets. As the period between discoveries gets longer, the chances that attackers have really good targets declines. That lowers their cost/benefit, which probably also lowers their actual budget. Criminals invest in things that make (them) money, not ones that lose it.

Security Can’t Wait

Original Posted on Substack as Update on AI CyberSecurity

DEV Community: Ryan Baker

Why It Hasn't Happened Yet

What I won’t cover but is still important

What protections are holding up today?

What puts protections at risk?

Why should we think security is at risk?

Why the equilibrium broke

Keeping malicious intent and capability apart

The technical picture

Vulnerabilities and escalation

Layers, segmentation and zero trust

Fixing vulnerabilities

Command and control (C2) and monitoring

What to do now

Is your organization ready?

How does AI assist in these responses?

Do your research

The attacker-defender balance

The worst case

Related Articles

Nobody Was Watching

Two labs skipped monitoring on their cyber evals. The same week, a hundred companies signed a letter supporting deployments that can't be monitored.

The Monitors Were Off Again

A Nuanced Position, Poorly Received

The Risk of Open Weight Models

Cynicism as a Tool, Not a Verdict

A Set of Contrasts

An OpenAI Model Escaped Its Sandbox. Where Was the Observer?

OpenAI accidentally hacked Hugging Face. A basic layer of defense appears to have been missing, and no one has explained why.

Where was the Observer?

Conclusion

The Perpetrator Is Not the Tool

What should we call slop?

Laziness

Manipulative

Costs to the Audience

Ignore the medium and tool

How this develops

Slop Smells: How to avoid creating slop

The effort ratio

The input ratio

Slop is in the eye of the beholder

Motivations

Again, why should we care about the definition?

Is Pangram a good tool?

We could do better

Related Articles

The Slop Scapegoat: AI - by Ryan Baker - norabble

The Mirage of Deep Research - by Ryan Baker - norabble

Writing with AI - by Ryan Baker - norabble

Measuring Without Breaking

The saying that starts the trouble

Why measurements rot

The same number, two outcomes

Running a culture that can measure

The lesson hiding in the leaderboard

More Data, Please

The Invisible Profession

Why the public has a poor understanding of software development

The AI Gamble, Six Months On

Layer 1: The Compute Supply Chain

Layer 2: The Operational Infrastructure

Layer 3: The Intelligence

Layer 4: The Application

Conclusion

Are We in a Token Bubble?

Background

What is tokenmaxxing?

What are subsidized tokens?

Why Bubble as an analogy is over extended

Why the public has a poor understanding of software development

When software is “done”

Why “tech-debt” comes first

Security as a priority

The ingenuity matrix

Timing

New output

What to expect

Sources

Related Articles