Originally published on The Searchless Journal
Training data is the raw material of the AI industry. Every model, from ChatGPT to Claude to Gemini, is built on oceans of text, images, audio, and video. The companies that build these models treat their training datasets as trade secrets, and they have good commercial reasons to do so. But the public interest in knowing what is inside those datasets has never been greater, and the gap between what companies disclose and what is actually there is wider than most people realize.
A recent investigation by The Atlantic has brought this issue into sharp focus. The investigation, which examined the composition of training datasets used by major AI companies, revealed the scale and variety of content that feeds into the models millions of people use every day. The findings are not reassuring.
The Scope of the Problem
Modern large language models are trained on trillions of tokens. These tokens come from books, academic papers, news articles, blog posts, social media conversations, code repositories, legal documents, product reviews, forum discussions, and virtually every other form of text that exists on the public internet.
The sheer volume makes transparency difficult. When a dataset contains content from millions of sources, providing meaningful attribution and provenance information is a technical challenge. But difficulty is not the same as impossibility, and the response of AI companies to transparency requests suggests that the challenge is not the only barrier.
Companies have fought aggressively to keep their training data secret. In litigation with authors, publishers, and other rights holders, AI companies have argued that disclosing their training datasets would reveal trade secrets and harm their competitive positions. Courts have generally been sympathetic to these arguments, allowing companies to keep their data compositions confidential during legal proceedings.
This secrecy creates a fundamental asymmetry. AI companies know exactly what is in their training data. The creators of that content do not. Users of AI systems have no way to evaluate the biases, gaps, or distortions that might result from specific data compositions. The entire ecosystem operates on trust, and that trust has been tested repeatedly.
What The Atlantic Found
The Atlantic's investigation focused on identifying specific works and sources within training datasets. By cross-referencing known dataset compilations with published works, the investigation was able to identify thousands of books, articles, and other copyrighted works that appear in training data without explicit authorization from their creators.
The scale is significant. Bestselling novels, academic textbooks, investigative journalism, and creative works of every kind appear in the datasets. Some of these works were included through bulk data scrapes of the internet. Others came from curated compilations that assembled copyrighted content from various sources.
The investigation also revealed the extent to which AI companies rely on data brokers and aggregators who package content into datasets for sale. These intermediaries operate in a gray area, assembling content from public sources and selling it to AI companies without necessarily securing rights from the original creators.
Why This Matters
The composition of training data affects everything about how AI models behave. If a model is trained primarily on content from certain demographics, regions, or ideological perspectives, its outputs will reflect those biases. If a model has ingested large quantities of medical misinformation, it may produce health advice that is subtly wrong. If a model has been trained on legal documents from one jurisdiction, its understanding of law may not transfer to others.
Transparency about training data is not just an intellectual property issue. It is a safety issue. Researchers who study AI bias, toxicity, and reliability need to understand what models have been exposed to in order to evaluate their behavior effectively. Without this information, safety research is conducted blind.
The problem extends to AI search and recommendation systems. When an AI search engine generates a response, the quality and reliability of that response depends on what the model learned during training. If the training data overrepresents certain sources or viewpoints, the model's outputs will reflect that imbalance. Users have no way to know.
The Copyright Time Bomb
The legal landscape around training data is shifting rapidly. Multiple lawsuits are working their way through courts in the United States, and several have already produced decisions that could reshape how AI companies acquire and use training data.
In one notable case, a court ruled that using copyrighted works for training without permission could constitute fair use under certain circumstances. In another, a different court reached the opposite conclusion. The inconsistency means that AI companies face significant legal risk, and that risk grows with every new lawsuit.
The settlement landscape tells its own story. Several AI companies have chosen to settle with rights holders rather than litigate, establishing licensing agreements and compensation funds. These settlements suggest that the companies themselves are uncertain about their legal position and prefer to resolve disputes through negotiation rather than judicial precedent.
For creators, the settlements are a double-edged sword. They provide some compensation and acknowledgment, but the amounts are often trivial relative to the value created. A bestselling author whose entire body of work was used to train a model worth billions of dollars may receive a settlement in the thousands. The power imbalance is extreme.
The Data Quality Problem
Beyond copyright and compensation, there is a practical issue that receives less attention: data quality. Not all training data is good training data. Internet scrapes inevitably capture spam, propaganda, machine-generated content, and low-quality material of every kind.
AI companies invest heavily in data filtering and cleaning, but these processes are imperfect. Contaminated training data can introduce factual errors, reinforce stereotypes, and create security vulnerabilities. The phenomenon of data poisoning, where malicious actors intentionally inject harmful content into training datasets, is a growing concern.
Transparency about data sources would help researchers identify and address these problems. If a model performs poorly on certain types of questions, knowing what training data covered those topics could help diagnose the issue. Without this transparency, debugging happens through trial and error, which is slow and unreliable.
What Transparency Would Look Like
Meaningful transparency does not mean publishing the entire contents of a training dataset. That would be impractical and would create its own privacy and security problems. But there are intermediate levels of disclosure that would serve the public interest.
First, companies should publish high-level summaries of their data sources. What percentage comes from web scraping, what percentage from licensed content, what percentage from synthetic data generation, and what percentage from user interactions. This information would help users and researchers understand the composition of the model's knowledge.
Second, companies should provide tools that allow creators to check whether their work is included in training datasets. Several companies have implemented opt-out mechanisms, but these mechanisms are only meaningful if creators can verify whether their work is being used in the first place.
Third, companies should disclose known quality issues with their training data. If a dataset is known to contain significant amounts of spam, propaganda, or inaccurate information, this should be documented and made available to researchers and users.
The Industry Response
Some AI companies have taken steps toward greater transparency. Anthropic has published information about its data sources and filtering processes. Google has released research papers describing aspects of its training methodology. OpenAI has provided limited information about its data acquisition practices.
These disclosures are welcome but insufficient. They tend to be high-level and abstract, describing categories of data rather than specific sources. They rarely include enough detail to evaluate potential biases or identify specific works that may have been used without authorization.
The competitive dynamics of the AI industry make voluntary transparency difficult. Companies fear that disclosing their data practices will give competitors an advantage or provide ammunition for litigation. The result is a race to the bottom where the least transparent company faces the fewest constraints.
The Path Forward
Mandatory transparency is the most realistic solution. The European Union's AI Act includes requirements for training data documentation, though the specifics are still being negotiated. Similar proposals have been introduced in the United Kingdom and Canada.
In the United States, regulatory action has been slower. The combination of strong intellectual property protections, aggressive legal strategies by AI companies, and a political environment that favors innovation over regulation has made meaningful transparency requirements difficult to enact.
But the pressure is building. Creators are organizing. Researchers are demanding access. Courts are asking harder questions. And investigations like The Atlantic's are revealing information that companies would prefer to keep hidden.
Training data transparency is not a panacea. It will not solve every problem with AI systems, and it will create new challenges around privacy, security, and competitive dynamics. But it is a necessary foundation for accountability. Without knowing what goes into AI models, we cannot evaluate what comes out of them. And without that evaluation, we are trusting the most powerful technology of our era on the basis of faith alone.
That is not enough. The stakes are too high, and the track record of unregulated technology is too poor. The companies building AI systems need to open their datasets to scrutiny, or regulators need to force them to. The alternative is a future where the most influential information systems in human society operate as black boxes, answerable to no one.
Top comments (0)