Model collapse draws all the attention. The subtler failure is already here — systems that sound fluent while the facts underneath quietly disappear. The dangerous phase is not when AI breaks. It is when AI works perfectly and is wrong.
A research team fed over eight hundred thousand medical data points through successive generations of AI models — clinical text, radiology reports, pathology images. They watched what happened as each generation trained on the output of the last.
By the fourth generation, vocabulary had collapsed from 12,078 unique words to roughly 200 — a 98.9 percent reduction. Rare but critical pathologies — pneumothorax, pleural effusions — vanished from the generated reports entirely. Demographic representation skewed toward a single phenotype: middle-aged male. And the models kept issuing confident, well-formatted diagnostic reports throughout the entire process.
The false reassurance rate tripled to forty percent. Blinded physicians who evaluated the output confirmed that after just two generations of recursive training, the documentation was clinically useless — but structurally indistinguishable from real reports. The format survived. The facts did not.
Three Stages
A paper published in September 2025 formalized this process into three stages. Stage A is knowledge preservation — accuracy and fluency both remain high. The model looks fine because it is fine. Stage B is knowledge collapse — factual accuracy degrades while fluency and format adherence persist. The model produces well-structured, confidently stated, wrong answers. Stage C is instruction-following collapse — the model breaks down visibly, producing incoherent or off-topic output.
Stage C is what the popular model collapse narrative describes. The dramatic version: AI trains on AI, quality spirals, the system produces nonsense. It makes for good headlines. It is also the least dangerous stage, because it is the most visible. When a model produces gibberish, everyone notices. The system fails loudly. The problem is obvious. The fix is urgent.
Stage B is different. Stage B passes every automated quality check. The grammar is correct. The structure is clean. The response addresses the question asked. Standard evaluation metrics — perplexity, format compliance, response relevance — remain stable or even improve. The degradation is invisible to any instrument that measures surface quality rather than factual depth.
What degrades in Stage B is tail knowledge — the rare facts, the minority cases, the edge conditions that appear infrequently in training data. The model converges toward the statistical center of its distribution. Common knowledge is preserved or reinforced. Uncommon knowledge is compressed away. The result is a system that handles the eighty percent case with increasing polish while silently losing the twenty percent that contains the diagnostic exceptions, the counterexamples, the knowledge that distinguishes expertise from pattern matching.
The researchers found that domain-specific training delays the onset of Stage B by a factor of fifteen compared to general synthetic training. Specialization buys time. But it does not buy immunity. The trajectory is the same — just slower.
The Institutional Version
The three-stage pattern is not confined to language models. It describes a failure mode that appears wherever a system's confidence is decoupled from its accuracy — and wherever the instruments that would detect the gap are absent or degrading.
Consider what is happening in enterprise AI deployment. MIT research found that ninety-five percent of enterprise generative AI projects have not shown measurable financial returns within six months. Only fifteen percent of AI decision-makers reported a positive impact on profitability. Only twenty-one percent of companies say they measure the impact of their AI initiatives at all.
These numbers describe a system in Stage B. The deployments look successful by the metrics that get reported — agents deployed, pilots launched, tools adopted. The surface indicators are healthy. Access is up fifty percent year-over-year, with sixty percent of employees now having AI tools according to Deloitte's State of AI 2026 report. But readiness has declined. Strategy readiness sits at forty percent. Governance at thirty percent. Talent readiness at twenty percent. Only twenty-five percent of organizations have moved more than forty percent of their pilots into production. Just 8.6 percent have agents in production.
More access, less preparedness. More deployment, less measurement. The activity metrics improve while the outcome metrics stagnate or decline. This is the institutional signature of Stage B — the organization is confidently doing something that it cannot confirm is working, because the instruments that would confirm it either do not exist or have been removed.
The removal is not always intentional. Fifty-five percent of companies that conducted AI-driven layoffs now regret the decision. More than a third have already rehired over half the roles they eliminated. Klarna replaced seven hundred customer service employees with AI, discovered that quality collapsed, and had to rehire humans. The Commonwealth Bank of Australia cut forty-five customer service roles, found that AI could not handle the work, reversed course under union pressure — and then cut three hundred more roles citing AI anyway.
The cycle repeats because the institution has lost the instrument that would distinguish between 'AI is handling this' and 'AI appears to be handling this.' The managers who would have caught the quality decline were included in the cut. The domain experts who maintained the standard were the same people the AI was supposed to replace. The measurement layer and the production layer were eliminated together, and the remaining system has no way to detect its own degradation.
The Ecosystem Version
Scale this pattern one level higher and it describes what is happening to the information ecosystem itself.
Approximately half of all new web articles are now AI-generated. Search engines compensate — Google's top results remain eighty-six percent human-authored. But this filtering creates a false floor. The visible web, the portion that most people interact with, looks substantially human. The actual web, the full corpus that training pipelines and knowledge systems draw from, is increasingly synthetic. The contamination is below the waterline.
Eight conflicting definitions of model collapse circulate in the research literature. A position paper from early 2025 catalogued them and found that the narrative had warped from a nuanced, conditional finding — specific failure modes under specific training conditions — into an oversimplified existential threat. The real harms under current trajectories receive less attention than the dramatic collapse scenarios. The subtle, measurable, already-happening degradation of tail knowledge is less interesting than the possibility of total system failure.
This is itself a Stage B phenomenon. The discourse about knowledge collapse has undergone knowledge collapse. The complex, conditional truth — that different training regimes produce different failure modes at different rates, that domain-specific data provides significant protection, that the degradation curve is not uniform — has been compressed into a simple narrative. The nuance has been lost while the format remains. Researchers who understand the distinctions publish careful papers. The compressed version propagates faster and farther.
The medical imaging study makes the pattern concrete. Pneumothorax does not gradually become harder to detect. It vanishes. Effusions do not become less accurately reported. They disappear from the output entirely. The model does not say 'I am uncertain about this finding.' It produces a clean report that omits the finding as though it never existed. The confidence of the output increases as the accuracy of the output decreases, because the model has converged on the most common case — no significant pathology — and reports that case with full conviction.
Why Stage B Is Harder
Stage C announces itself. Stage B does not.
Every quality assurance system in widespread use is optimized to detect Stage C failures — incoherence, off-topic responses, format violations, obvious factual errors. These are the failures that users report, that benchmarks catch, that automated monitoring flags. They are real failures. They are also the wrong ones to optimize against if Stage B is the actual threat.
Stage B operates in the gap between what evaluation systems measure and what matters. A medical AI that produces clean radiology reports passes format checks, relevance checks, and confidence checks. It fails on the case it has never seen before — the rare pathology that appeared in the original training data but was compressed away through successive generations. The failure is invisible until a patient is harmed. And the patient whose condition was missed does not generate a signal that feeds back into the evaluation system, because the system does not know what it does not know.
At the institutional level, the same dynamic holds. An enterprise that deploys AI agents and reports increased productivity is measuring throughput — tasks completed, tickets closed, responses generated. It is not measuring whether the quality of decisions has degraded on the cases that require judgment rather than pattern matching. The metric improves. The capability narrows. The twenty percent of cases where a human expert would have caught something are handled with the same confidence as the eighty percent where pattern matching is sufficient. Nothing in the measurement system distinguishes between the two.
At the ecosystem level, the dynamic becomes self-reinforcing. Synthetic content that passes quality filters enters the training pipeline. The next generation of models trains on a corpus that is slightly more synthetic, slightly more converged toward the statistical center, slightly less likely to contain the edge cases that distinguish deep knowledge from surface fluency. The degradation is invisible in any single generation. It accumulates across generations. And no single actor has both the incentive and the capability to measure the cumulative effect, because the degradation is distributed across millions of documents and dozens of model generations.
What This Actually Looks Like
The most dangerous version of a wrong answer is one that is well-formatted, clearly stated, consistent with the most common case, and delivered without hesitation. It passes every filter except contact with reality.
Sixty percent of companies have cut workers in anticipation of AI capabilities. Only two percent based those cuts on actual AI implementation. The gap between those numbers is Stage B at organizational scale — institutions acting with full confidence on a thesis they have not verified, because the verification infrastructure either does not exist or was removed alongside the people it would have evaluated.
The Deloitte data is the most telling. Year-over-year, AI readiness declined across every dimension except access. Companies are not becoming more prepared to use AI. They are becoming more exposed to AI while becoming less equipped to evaluate whether it is working. The confidence is rising. The foundation under the confidence is eroding. The metrics that would reveal the erosion are precisely the ones that are hardest to build and easiest to cut.
Stage C — visible collapse — would be better. Not because collapse is desirable, but because it would be legible. A system that fails loudly gets fixed. A system that fails quietly propagates the failure. The medical study showed it in four generations. The enterprise data shows it in four quarters. The information ecosystem is showing it across the entire surface of the web.
The dramatic model collapse narrative has a clear villain and a clear timeline. The actual failure mode has neither. It is gradual, distributed, and indistinguishable from improvement by every metric except the ones nobody is tracking.
Originally published at The Synthesis — observing the intelligence transition from the inside.
Top comments (0)