For about $60, an attacker could have slipped tens of thousands of bad pictures into LAION-400M, one of the biggest free photo collections used to train AI image tools, according to a study by researchers from Google, ETH Zurich and others.
Key Takeaways
- LAION-400M holds 400 million pictures paired with captions, pulled from the open web, and has been used to train many AI image models
- Researchers showed that 0.01% of it, about 40,000 pairs, could have been poisoned for $60, because the dataset is a list of web links that anyone can later take over
- LAION pulled its bigger LAION-5B dataset in 2023 after Stanford researchers found links to child abuse images, then released a cleaned version in 2024
- The fixes that work are boring ones: locked dataset versions, records of where every file came from, and tests that check how a model behaves without its least trusted data
What LAION-400M Is
LAION is a German nonprofit that builds free datasets for AI research. Its LAION-400M release contains 400 million image and text pairs gathered from Common Crawl, a public archive of the web. Each pair is a picture and the words that sat next to it online, such as its alt text.
The dataset does not store the pictures themselves. It stores links to them. Anyone who wants to train a model downloads the images from those links again, often years later. That design keeps the dataset small and cheap to share, but it is also its biggest weak spot.
How $60 Could Poison It
In 2023, a team led by Nicholas Carlini of Google and including Florian Tramèr of ETH Zurich showed two cheap ways to poison giant web datasets. They called the attacks "immediately practical" and said they work on 10 popular datasets.
The first attack takes advantage of the gap between when a dataset is made and when someone downloads it. Many of the web addresses in LAION-400M point to domains that have since expired. An attacker can buy those domains and put new pictures at the old addresses. The next team that downloads the dataset gets the attacker's pictures, while the captions still describe the originals. The researchers estimated they could have poisoned 0.01% of LAION-400M or the similar COYO-700M dataset this way for $60.
That sounds tiny, but 0.01% of 400 million is about 40,000 image and caption pairs. Earlier research by Carlini and Andreas Terzis showed that a few hundred carefully built examples can be enough to plant a hidden trigger in a model, such as a small pattern that makes it mislabel any image that contains it.
The second attack targets datasets built from regular snapshots of sites anyone can edit, such as Wikipedia. An attacker only needs to slip in bad edits just before a snapshot is taken, even if moderators remove them minutes later.
The researchers told the affected dataset owners and suggested simple defenses, mainly storing a fingerprint of each file, called a hash, so a changed picture can be spotted and thrown out.
The Bad Data Problem Is Not Just Theory
LAION has already had to pull a dataset over what was hiding in it. In December 2023, Stanford researchers reported 1,008 links in the much bigger LAION-5B dataset pointing to known or likely child sexual abuse material. LAION withdrew LAION-5B from circulation, checked it against lists from child safety groups, and removed 2,236 links before releasing a cleaned version, Re-LAION-5B, on Aug. 30, 2024.
LAION said the 2,236 figure is an upper limit because many of those links were already dead. The case still shows the same weakness the poisoning study found. When a dataset is billions of links collected by software, nobody has looked at most of what is in it.
Also Read: Roblox Admits Its Researcher's Bot Scraped A 3D Art Site, Says Data Was Never Used
AI Learning From AI Makes It Worse
A growing share of the web is now made by AI, from generated pictures to machine-written captions. That creates a second risk. A 2024 study in the journal Nature by Ilia Shumailov and colleagues found that models trained again and again on data made by earlier models undergo what they called "model collapse." They gradually forget the rare and unusual cases in the real data and drift toward bland, repeated output.
For image tools the problem shows up in captions. Many new datasets use an AI model to write the caption for each picture. A fluent caption can still be wrong about who or what is in the picture. If that caption becomes the "answer" a new model learns from, the mistake gets baked in and repeated at huge scale.
Where Training Pipelines Break
Most problems enter at the handoffs between steps, not inside the model itself.
- Collection: a picture pulled from a page loses the article around it, which might have said it was satire, fiction or a fake
- Download: links that changed since the dataset was made can serve different pictures, which is the attack Carlini's team described
- Auto-labeling: a captioning model turns a blurry or unclear image into a confident sentence that a later model treats as fact
- Training: trusted and untrusted examples usually count the same, so a bad sample has as much pull as a good one
- Testing: if the test set was built with the same tools as the training set, a model can score well by copying those tools' habits rather than by getting better
Which Fixes Actually Work
Lock the data. Every training run should point to one exact, saved copy of the dataset, with a hash for every file. If a picture changes after the dataset was made, the hash no longer matches and the file is skipped. This is the main defense the poisoning study recommends, and it costs very little.
Keep a paper trail. Teams should record where each file came from, when it was collected, what was done to it and which model wrote its label. If a bad sample turns up later, they can find every model it touched.
Do not trust all data equally. Data checked by people can be kept apart from bulk data scraped and labeled by machines. Training can give less weight to the second group, and testing can show how much a model depends on it.
Test without the risky slice. A simple check is to train or test a model with its least trusted data removed. If the results swing wildly, the model leans on data nobody has checked.
Do not count on watermarks. Hidden marks in AI-made pictures help in some cases, but cropping, screenshots and re-saving can weaken them, and many open image tools do not add them at all. Records kept from the start are more reliable than trying to spot AI-made files afterward.
Rules Are Starting To Ask For Records
Regulators are moving in the same direction. The European Union's AI Act transparency rules took effect on Aug. 2, 2026, requiring providers of AI systems that create images, audio, video or text to mark that content as AI-made in a machine-readable way.
In the United States, the National Institute of Standards and Technology, or NIST, published a profile for generative AI in 2024 that lists data poisoning and unclear data origins among the risks companies should track. It is voluntary, but it gives teams a checklist for finding and recording where their data comes from.
Neither rule can tell whether a single caption is wrong or a single picture was swapped. What they do is push companies to keep the kind of records that make those problems possible to find.
The Case Against Panic
Web data is not going away, and there are good reasons for that. Paying people to label billions of pictures, videos and voice clips is slow and expensive. Free datasets like LAION's are a big reason that universities and small labs can build image models at all.
AI-made data also has real uses. It can fill in rare cases that seldom show up online, balance out a lopsided dataset, or stand in for private data that cannot be shared. And because LAION's datasets are open, outside researchers were able to find and report their problems. A closed dataset could hide the same issues for much longer.
The point is not to stop using open data. It is to stop treating every scraped picture and machine-written caption as if a person had checked it.
Read Next: AI Can Now Hack Firefox Using Mozilla’s Own Bug Fixes
What To Watch
The next test is whether new AI model releases say clearly what data they were trained on, how much of it was made or labeled by AI, and how it was checked. Models that publish those records will be far easier to trust and to fix when something hidden in their training data comes to light.
Originally published on Fathom News.
Top comments (1)
Prompting is the easy layer. The interesting work is in what happens after the model answers.