AI Can Read the Literature. It Still Can't Tell Which Papers to Trust.

Reading more papers can solve an access problem. It does not tell the machine which results deserve trust.

Reinvent Science argues that the scientific literature is poisonous to LLMs because papers mix honest mistakes, unsupported theories, selective reporting, and fraud while presenting them in much the same polished form. I agree with the product diagnosis: a model cannot treat publication as a truth label. I do not think the evidence supports throwing the literature out.

The 2024 study cited by the essay trained 28 separate 1.5-billion-parameter models. That is a serious experiment, but its result is more awkward and more useful than the headline. Scientific text is not one uniform contaminant. Different collections helped different tasks, and the evaluation shaped which sources looked dispensable.

Quick answer

Question	My read
Is the literature clean?	No. Papers can be wrong, retracted, gamed, or fabricated, and publication metadata does not settle reliability.
Did the cited study show science papers poison LLMs?	No. One academic partition could be removed without hurting the tested average, but the authors said the best mean performance came from all or nearly all sources.
What should AI science products do?	Track claims back to primary evidence, surface corrections and retractions, and show why a source was included.
What should they avoid?	Using private scientific gossip or continuous surveillance as a substitute for auditable evidence.

The cited experiment is narrower than the provocation

A Pretrainer's Guide to Training Data divided the Pile into nine domains and removed each domain one at a time before testing 27 question-answering datasets. Its Academic partition was 60 GB of arXiv, PhilPapers, and NIH ExPorter material. PubMed was a separate 109 GB biomedical partition.

Removing the Academic partition improved some averages in the paper's figure, so Reinvent Science has a real observation to work with. But the authors also say the best mean result came from models trained on all or nearly all available sources. Removing PubMed hurt biomedical question answering. Removing Common Crawl hurt academic question answering more than removing the Academic partition did.

The paper offers a plain explanation for the apparent contradiction: its QA sets did not demand the coding ability or scientific rigor found in advanced science and math journals. In other words, a source can look useless when the test does not ask for what the source contains. I would not turn that mismatch into a verdict on the whole scientific record.

A corpus-level filter is blunt. It can remove weak material and erase the specialized evidence a different task needs.

The trust problem is still real

The source is on firmer ground when it describes literature as a mixture of facts, omissions, error, and misconduct. A 2026 Nature analysis concluded that tens of thousands of 2025 publications might contain invalid references generated by AI. A separate JMIR study tested nine freely available AI tools against 15 retracted articles. None handled every case correctly. The best tool got all five tasks right for 8 of the 15 articles, while the three research-focused tools produced no fully correct response set.

That does not make every journal article suspect. It does mean trust changes over time. Retraction status can change after training. A correction may weaken one claim without invalidating an entire paper. A sound study may still be irrelevant to the population or outcome in front of the user.

I want a research assistant to answer a harder question than "Which papers mention this?" It should show which claim each paper supports, whether that paper has been corrected or retracted, and what independent evidence agrees or conflicts.

Private gossip is the wrong missing layer

Reinvent Science says human researchers carry informal reputational knowledge through conversations, lab visits, and professional networks. Its proposed routes for giving that context to AI include socializing with agents, recording scientists in private settings, or encouraging a stream of scientific gossip. The essay acknowledges the surveillance cost.

I would stop there. Informal knowledge can be useful, but it also carries status bias, grudges, and claims that the target cannot inspect or contest. Recording more private conversation would create a sensitive data store without turning reputation into reliable evidence. The source's only public commenter makes the simpler point: the same flaws that confuse LLMs also hurt human scientists trying to work out what is real.

I would convert the useful part of that oral tradition into inspectable records. Link each claim to the underlying dataset and protocol. Record replication attempts and preserve reviewer concerns when they can be made public. Sync retraction and correction notices at query time. Let experts attach signed, scoped assessments that explain their reasoning instead of assigning a mysterious reputation score.

The useful missing layer is an evidence trail that can be checked, updated, and challenged without recording private conversations.

My bottom line

I like the provocation because it attacks a lazy assumption behind AI for science: more papers automatically produce more scientific judgment. They do not. A model can absorb the words in a field without learning which experiment changed expert belief, which result failed to replicate, or which citation survives only because everyone copies the previous bibliography.

Still, "poisonous" is too coarse. The cited training study shows that data composition and evaluation design matter. It does not show that scientific literature is uniquely harmful. I would build for provenance rather than purity: keep the papers, expose the chain from claim to evidence, check the live status of each citation, and make uncertainty visible. An AI that can read the literature still needs a way to show its work.

Originally published at markhuang.ai