Alan Scott Encinas

Posted on Aug 4 • Originally published at alanscottencinas.com

Your AI Sounds Certain. Look at Where It Learned That.

#aiarchitecture #datascience #enterpriseaiadoption #futureofwork

You ask a question. The answer comes back clean, structured, confident, and wrong.

Not obviously wrong. Wrong in the specific way you can only catch if you already knew the answer. The date is off by a year. The number belongs to a different study. The quote is real but the person never said it in that context.

If you knew the subject, you caught it. If you didn't, you just learned something false from a machine that never once sounded unsure.

Most people don't know how these systems learned what they know. That's the part worth understanding, because it explains why this keeps happening and why it's about to get worse.

Nobody Chose the Truth

A language model doesn't get taught facts. It gets fed text. Enormous quantities of it, scraped from the open internet, and it learns statistical patterns about which words follow which other words in which contexts.

There is no editor in that process. Nobody sits down and decides which claims are correct. The model learns from volume and from signals that correlate with authority, meaning content that looks credible, gets linked, gets repeated, and sits on domains that other content points at.

Reddit threads. LinkedIn posts. News articles. Forums. Documentation. Reviews. Comment sections.

That approach worked remarkably well, and it's worth being honest about why. For most of the internet's life, the average page was written by a person trying to communicate something to another person. The signal wasn't clean, but it was mostly organic. Volume and credibility roughly correlated with accuracy because there was no particular reason for them not to.

That assumption is the whole foundation. And in the last two years, three separate things knocked it out.

One: The Referees Left the Field

In January 2025, Meta announced it would end its third party fact checking program in the United States. The program formally shut down on April 7, replaced by Community Notes, a crowd rating system modeled on the one X adopted after its own fact checking apparatus was dismantled.

Set aside whether that was the right call politically. Look at what it does mechanically.

Before, a claim that got flagged was demoted, labeled, and its reach was cut. It still existed, but it stopped spreading and it stopped looking authoritative. After, a claim spreads at full speed and receives a correction only if enough raters from sufficiently different perspectives agree to attach one, which by design takes time and often never happens at all.

The claim keeps its engagement. It keeps its links. It keeps its appearance of legitimacy.

And every one of those properties is a signal the training process reads as credibility.

Two: Poisoning Got Cheap

The second change is that people figured out how the machine weighs its inputs, and started writing for it deliberately.

For a while the assumption was that this required scale. To meaningfully shift what a model believes, you would need to control some meaningful percentage of its training data, which is beyond almost anyone.

That assumption turned out to be wrong. In a joint study with the UK AI Security Institute and the Alan Turing Institute, Anthropic found that roughly 250 malicious documents were enough to install a backdoor in a large language model. The finding that matters is what happened when they scaled the model up. A 13 billion parameter model trained on more than twenty times the data of a 600 million parameter model needed the same small, fixed number of poisoned files.

Not a percentage. A count.

Two hundred and fifty documents is a weekend of work. It's within reach of one motivated person, and it does not get harder as the models get bigger.

Three: The Machine Started Eating Its Own Output

The third change is the one with no obvious fix.

AI now writes a large and growing share of what gets published online. Estimates vary widely depending on methodology, but multiple independent analyses agree that AI generated text crossed fifty percent of newly published web articles during 2025.

That output becomes the training data for the next model.

Researchers have a name for what happens next. In a 2024 paper in Nature, Shumailov and colleagues demonstrated model collapse: when models are trained recursively on model generated content, they progressively lose the tails of the original distribution. The rare, the specific, and the unusual disappear first. What remains is the average of the average, and the degradation compounds with each generation.

The machine forgets the edges of the world, then forgets that the edges were ever there.

There's a fair counterpoint worth stating, because it cuts against the panic. Search engines are filtering more aggressively than the raw numbers suggest, and only a small fraction of the content actually ranking on Google shows signs of AI generation. The pipeline isn't uniformly poisoned. But the training scrape is not the search index, and it never was.

The Gap Nobody Is Watching

Put those three together and you get two lines moving in opposite directions.

The quality of what goes in is falling. Friction against bad information got removed, deliberate poisoning got cheap, and synthetic content is recycling through the pipeline.

Public trust in what comes out is climbing. People increasingly treat the answer as the end of the inquiry rather than the start of one.

Those lines are crossing right now, and almost nobody is looking at the space between them.

Open weight models sit in the most exposed position. Anyone can fine tune one on anything and ship it, and the people downloading it have no practical way to audit what went into it. The openness that makes those models valuable is the same property that makes their provenance unknowable.

Who Actually Pays for This

Adults with domain expertise catch the errors. That's the quiet privilege in this whole situation, and it doesn't scale.

A study published in May by Florida Atlantic University and the University of Wisconsin Eau Claire found that 60.2% of teens had used a conversational AI chatbot, with 11.4% using one daily or almost daily. Nearly half, 47.1%, reported experiencing at least one of the thirteen risks the researchers examined. Separate work from Common Sense Media found 72% had used AI companions and 52% used them regularly, and concluded the category poses an unacceptable risk to minors.

Research from the Center for Democracy and Technology found 42% had used AI chatbots for friendship and 19% for something they described as romantic.

These are people forming their model of how the world works, receiving information through a system with no editor, trained on a corpus that is actively degrading, delivered in a tone of total confidence, with no visible seams where the uncertainty should be.

They aren't going to catch the date that's off by a year.

You Can't Fix This With More AI

The instinct is to solve it the way we solve everything else now. Add a verification layer. Have a model check the model.

It doesn't work here, and the reason is structural rather than technical. A verifier trained on the same degraded corpus inherits the same gaps. You can't ask the system to check its claims against the world when the corpus is the only world it has ever seen. Confidence is not knowledge, and a second model producing confident agreement is not a second opinion.

This is a data problem, and data problems get solved by people deciding what goes in.

That means provenance and dated sources over volume. It means treating an AI answer as a starting position rather than a verdict, especially in the areas where you can't personally check the work. It means the boring discipline of maintaining what you feed your systems, which nobody wants to fund because it produces no demo.

The Part We Skip

A star is beautiful from the ground. Clean, fixed, quiet.

Up close it's gas under crushing pressure, fusing itself apart, held in balance by forces that will eventually lose. Nobody sees that. We see the result, and the result is lovely, so we stop there.

We're doing the same thing with these systems. The output is fluent and immediate and it feels like an answer, so we take it and move on without asking what it was built from or what it quietly left out.

That's fine when the stakes are a dinner recommendation. It's not fine when it's the thing teaching a fifteen year old how the world works, or approving your vendor payment, or writing the analysis your quarter depends on.

Understanding the machinery isn't pedantry. It's the only way to use these systems for what they're actually capable of, and right now most people are admiring the light without any idea what's burning.

Frequently Asked Questions

How is AI actually trained?

Large language models are trained on enormous quantities of text scraped largely from the public internet. Nobody hand selects which claims are true. The system learns statistical patterns and weights sources by signals that correlate with authority, such as links, repetition, and domain reputation. Accuracy is inferred from those proxies rather than verified.

Can AI training data be poisoned deliberately?

Yes, and more cheaply than assumed. A joint study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that roughly 250 malicious documents were sufficient to backdoor models ranging from 600 million to 13 billion parameters. The number stayed fixed as model size grew, meaning attackers need a small absolute count rather than a share of the corpus.

What is model collapse?

Model collapse is the degradation that occurs when models are trained on data generated by earlier models. Research published in Nature in 2024 showed that recursive training on synthetic output causes models to lose the tails of the original data distribution, so rare and specific information disappears first and errors compound across generations.

Does removing fact checking affect AI models?

Indirectly, yes. Fact checking reduced the reach and apparent credibility of false claims. Those same signals, engagement, links and domain authority, are what training pipelines read as reliability. When claims circulate without friction, they enter the corpus looking more legitimate than they are.

Can AI fact check itself?

Not reliably. A verifier trained on the same corpus inherits the same gaps and the same blind spots. Confident agreement between two models trained on similar data is not independent verification. Fixing degraded training data requires human decisions about provenance, not another layer of inference.

DEV Community