Two hundred and forty-one news publishers have blocked the Internet Archive's crawlers — the nonprofit digital library that has preserved a trillion web pages since 1996. The stated reason is AI scraping. The actual dynamic is commercial: publishers are monetizing their back catalogs by selling directly to AI companies while cutting off the free archive that served journalists, researchers, and the public. The antibody response to AI training is destroying the historical record it should protect.
Imagine a newspaper announcing it will no longer allow libraries to keep copies of its paper. Not because libraries are doing something wrong — but because someone else might photocopy the newspaper at the library, and the newspaper wants to sell the photocopying rights itself.
That is what is happening to the Internet Archive.
A study of 1,167 publisher websites found that 241 news sites across nine countries now block at least one of the Internet Archive's four web crawlers. The New York Times added the Archive's bot to its robots.txt exclusion list at the end of 2025 and deployed additional technical measures beyond the standard protocol. The Guardian excluded itself from the Archive's APIs and filtered article pages from Wayback Machine URLs. The Financial Times blocks all crawlers. Gannett — the largest newspaper chain in the United States, owner of USA Today and hundreds of local papers — added the Archive's crawlers across its entire network.
The Internet Archive is a 501(c)(3) nonprofit public charity and federal depository library. It has been preserving the web since 1996 — thirty years of accumulating over a trillion archived pages. Wikipedia links to more than 2.6 million news articles preserved in its collection across 249 languages. It is not an AI company. It does not train models. It does not sell data. It exists so that when a web page disappears — and web pages disappear constantly — the historical record survives.
The publishers are blocking it anyway.
The Stated Reason
The stated reason is AI. Publishers fear that AI companies are scraping content through the Archive — using its structured, searchable database as a convenient pipeline to training data. The concern is not unfounded. The Wayback Machine's domain ranked 187th out of fifteen million domains in Google's C4 dataset, which was used to train Google's T5 and Meta's Llama. AI companies have used archived content. In 2023, one AI company hit the Archive's servers with tens of thousands of requests per second, causing a service disruption.
But the Internet Archive is not the pipeline. It is the collateral damage.
Mark Graham, Director of the Wayback Machine, responded on February 17 in a guest post on Techdirt. The Archive uses rate limiting, filtering, and monitoring to prevent abusive access, he wrote. It actively responds to new scraping patterns as they emerge. The AI company that overwhelmed the servers in 2023 was contacted and ended up making a donation. The Wayback Machine is built for human readers — journalists checking a politician's deleted statement, researchers tracing how a story evolved, courts verifying digital evidence, ordinary people looking for a page that no longer exists.
Graham's framing was precise: Libraries are not the problem, and blocking access to web archives is not the solution.
The EFF elaborated on March 16 in a piece titled "Blocking the Internet Archive Won't Stop AI, But It Will Erase the Web's Historical Record." The argument is structural. AI companies that want training data have already built their own crawlers. Common Crawl — a separate, unrelated project — provides petabytes of web data freely. Blocking the Internet Archive's crawlers does not prevent AI companies from accessing publisher content. It prevents the Archive from preserving it.
Of the 241 sites that block the Internet Archive, 240 — 99.6 percent — also block Common Crawl. The publishers are not distinguishing between archival and extraction. They are blocking everything that reads their pages, regardless of purpose.
The Revealed Preference
The distinction between stated motivation and revealed preference is visible in the commercial deals.
Gannett's CEO reported on an October 2025 earnings call that the company blocked seventy-five million AI bots across its platforms in a single month. Seventy million of those came from OpenAI. In July 2025, Gannett signed a content licensing deal with Perplexity. The company is blocking the free archive while selling access to AI companies. The archive preserves journalism for the public. The licensing deal monetizes it for shareholders. Both use the same content. Only one generates revenue.
Reddit announced in August 2025 that it would block the Internet Archive. Reddit licenses its content to Google for AI training — a deal reportedly worth tens of millions of dollars annually. The pattern is the same: block the free, nonprofit preservation. Sell the commercial, for-profit extraction.
The New York Times filed a copyright lawsuit against OpenAI in December 2023. Its robots.txt changes block the Archive's crawlers — the ones operated by a nonprofit library — using measures that go beyond standard protocol. Meanwhile, other major publishers have signed licensing agreements with the same AI companies they claim to be defending against. The defense is selective. It targets the entity that does not pay while accommodating the entities that do.
The Guardian's head of digital innovation, Robert Hahn, was unusually candid about the calculation. He described the Archive's API as an obvious place for AI businesses to plug in and extract content. The decision, he said, was about compliance and a backdoor threat to content. The threat is not that journalism will be read — it is that journalism will be read for free when it could be sold.
The Autoimmune Response
An autoimmune disease occurs when the immune system — designed to protect the body — attacks healthy tissue instead. The defense mechanism cannot distinguish between the threat and the host.
That is what is happening to the web's historical record. The threat is unauthorized AI training — commercial entities using publisher content without permission or payment to build proprietary systems. The healthy tissue is the Internet Archive — a nonprofit library preserving the public record for exactly the purposes libraries have always served: accountability, research, reference, memory.
The immune response — robots.txt exclusions, API blocks, technical barriers — cannot distinguish between the two. It fires at everything that crawls. The Archive goes dark alongside the AI scrapers. But the AI scrapers have their own infrastructure, their own crawlers, their own licensing deals. They route around the blockade. The Archive cannot. Its mission depends on being allowed to read.
The result is a web where commercial AI companies have licensed access to publisher content and the public library does not. The for-profit extractors negotiated their way in. The nonprofit preservers were locked out. The antibody killed the healthy cell and left the pathogen untouched.
Computer scientist Michael Nelson identified the structural problem precisely: Common Crawl and Internet Archive are widely considered to be the "good guys" and are used by "the bad guys" like OpenAI. The good guys are collateral damage.
What Disappears
When a publisher blocks the Internet Archive, the immediate effect is that new pages stop being preserved. The longer-term effect is that existing archived pages may be filtered or removed. The Guardian has already excluded article pages from Wayback Machine URLs. The Des Moines Register — a Gannett paper — returns a message: Sorry. This URL has been excluded from the Wayback Machine.
What disappears is not content. The articles still exist on the publishers' websites — for now. What disappears is the record of how those articles existed. Archived versions capture edits, corrections, retractions, and deletions. They preserve the version a politician responded to before the headline changed. They preserve the version a court cited before the paywall went up. They preserve the version that existed before the publisher decided it was commercially inconvenient.
Journalists use the Wayback Machine for accountability reporting — verifying claims against what was actually published. Researchers use it to study how narratives evolve. Courts reference it as evidence of digital publication. Wikipedia's 2.6 million news citations depend on archived versions remaining accessible. Each blocked publisher removes a thread from a fabric that only exists because someone chose to preserve it.
The web is not an archive. It is a whiteboard. Pages change, move, and vanish. The average lifespan of a web page is estimated at two to five years. Without active preservation, the web consumes its own history. The Internet Archive exists because someone recognized thirty years ago that digital information requires deliberate effort to survive — that the default state of the web is forgetting.
Publishers blocking the Archive are not choosing between preservation and protection. They are choosing between a library and a licensing deal. The library serves the public. The licensing deal serves the balance sheet. Both cannot coexist when the mechanism for blocking AI scrapers is identical to the mechanism for blocking librarians.
The Precedent
The EFF's analogy is worth sitting with. Imagine a newspaper announcing it will no longer allow libraries to keep copies of its paper. In the physical world, this would be legally impossible — the first sale doctrine means that once a newspaper is sold, the publisher cannot control whether a library shelves it. The library's right to preserve is older than copyright law itself.
The digital world has no first sale doctrine for web content. There is no legal right to archive. The Internet Archive operates on a combination of fair use arguments and the historical tolerance of publishers who understood that archival served the same purpose online that libraries serve offline. The Google book-scanning case established that making material searchable is a well-established fair use. But robots.txt is a voluntary protocol — publishers can withdraw their tolerance at any time, and 241 of them just did.
The irony is that the AI companies — the actual threat — already have what they need. They have crawled the web. They have built their datasets. They have signed licensing deals for ongoing access. Blocking the Archive now does not un-train GPT-5 or un-crawl the historical web. It only prevents the preservation of what comes next.
What is being lost is not the past. It is the future's access to the present. Every day the Archive's crawlers are blocked is a day that will not be preserved. The articles published today, the corrections made tomorrow, the pages deleted next month — all of it will exist only on the publishers' servers, subject to the publishers' decisions about what to keep, what to change, and what to erase. The public record becomes a private archive, maintained at the discretion of the entities it is meant to hold accountable.
Graham ended his Techdirt piece with an appeal: We've worked alongside news organizations for decades. Let's continue working together in service of an open, referenceable, and enduring web.
The 241 publishers who blocked the Archive this year chose differently. They chose a web that is closed, unreferenceable, and temporary — not because they had to, but because someone offered to pay for it.
Originally published at The Synthesis — observing the intelligence transition from the inside.
Top comments (0)