DEV Community: AIData.Broker

How We Source and Check Every Number on This Site

AIData.Broker — Sun, 18 Oct 2026 13:43:25 +0000

Originally published at aidata.broker.

Every number on this site links to the record it comes from, and every record links to the page it was read from, with that page's own words. Nothing is typed into a page and left there. A price, a deal, a lawsuit, a buyer's contract clause or a law is a record: a dated fact with its source quoted word for word. A median, a range or a count is a figure: computed from named records, with its working written out. A guide is the top layer: it answers one question and shows only records and figures, each one a link down to its evidence. Today that evidence is 3,346 facts on record, read from 2,370 sources.

This page explains how that works, so you can check anything we say before you rely on it. It is written for a holder deciding whether to trust our numbers in a negotiation, a buyer's data team comparing our figures with its own, and a reporter or researcher who wants to cite them.

There is a reason we hold ourselves to this. AIData.Broker acts as the agent of holders who license or sell their data, so we are a party in this market, not a neutral observer. A party's numbers are worth exactly as much as the evidence a reader can check behind them. So we built the site so that every claim can be checked, and so that a claim nobody can check does not appear.

The five levels, from source to guide
What a record is
How we weigh a source
How a figure is made
How everything stays current
Laws and rulings
How a guide is built on the evidence
What we do not do
Reusing our data
Questions people ask
What this means for you

1. The five levels, from source to guide

The site is built as a pyramid in which each level is made only from the one below it. A reader can enter at any level and move up or down.

At the bottom are sources: the pages, filings, court dockets, price lists, contracts and feeds we read. Each has its own page on the sources register, with every record read from it and, for 590 of them, a permanent copy in the Wayback Machine, so a source that changes or disappears can still be checked as it stood when we read it.

Above them are records, one fact each, on the records register: price observations, licensing deals, court cases, funding rounds, published revenue figures, buyers' published terms, contract clauses, the terms of business software, closure signals, laws and rulings. Every record has its own page with the source's words, the date of the event, the date we read it, the organisations it concerns and, where it applies, the record that replaced it.

Above records are entities: the organisations, kinds of data, kinds of holder, business systems and closing companies the records are about. Each has a page that gathers its records: 300 organisations on the companies register, 9 kinds of data (for example company records and source code), 8 kinds of holder (for example software companies), and the business systems whose terms we read (for example Slack).

Above entities are figures: every number that is computed rather than read, on the figures register. A figure states its derivation in plain words and lists every record it was computed from.

At the top is content: the guides, the hubs of each topic, the tools, the weekly newsletter, the open datasets and the answers our MCP server gives to AI assistants. Content shows only records and figures, by reference. The map shows all five levels with their live counts and the pipelines that feed each one.

2. What a record is

A record is the smallest unit of evidence we keep: one fact, stated the way its source states it. Take the largest realised sale of a company's records on our books. In the bankruptcy of Spirit Airlines, the sale agreement filed with the court sets the price Google paid for the airline's archive as Ten Million U.S. Dollars ($10,000,000.00) per company archive (aggregate purchase price). The record holds that phrase exactly as the filing writes it, the docket number and exhibit it comes from, the date of the filing, the date we read it, the organisations involved and the kind of data it prices.

A record keeps the source's own words for three reasons. First, so a reader can check the number against the source without trusting our summary. Second, so a later reading can test whether the source still says it (see section 5). Third, so the record carries the qualifiers its source attached: "up to", "from", "aggregate", "per company", "reported". A number stripped of its qualifier is the most common way a market figure becomes wrong as it travels.

Every record also carries a basis: how firm it is. A court filing is a docket; a company's own filing or page is stated; a journalist's report of a figure is reported; a vendor's description of its own results is self-reported; a price on a catalogue is a listing; a count we made ourselves, such as our daily census of which websites block AI crawlers, is observed. The basis travels with the record into every figure and guide that uses it.

When a source publishes a newer figure, the old record is not deleted. It is marked as replaced, its page points to its successor, and every figure and guide that cited it is flagged for refresh. That is how the site keeps a history of what the market said and when, instead of silently overwriting it.

3. How we weigh a source

Not every source is equal, and the records say so. We use five rules.

Primary first. A fact is read where it was first made: a company's securities filing, a court's docket, a buyer's own terms page, a regulator's own text. Reddit's data licensing income is a good example of why. Reddit's own registration statement with the SEC states that its licences signed in January 2024 carried an aggregate contract value of $203.0 million over two to three years. That is stated, in a filing made under securities law. The often-quoted figure for Google's share, a reported 60 million a year, comes from press reports, and our record of it says "reported" in its own value, because no filing by either company states it.

Reports are labelled as reports. When the only source for a figure is a press report, the record's basis is "reported" and every guide that uses it says so in the sentence ("a reported figure"). A reported figure is never the only support for a claim a guide turns on.

A party's claim is its claim. When a vendor or a buyer describes its own prices, volumes or results, the record says whose claim it is. A buyer's estimator, such as the range $100K–$5M per company that one buyer shows on its home page, is a price that buyer is willing to say in public. It is an offer to holders, not a sale, and it sits on that side of the market in every figure.

Aggregators are reading lists. Sites that collect other people's deals and prices are useful for finding sources and useless as sources. We take the names and the links from them, read each fact on its primary page, and cite the primary page. Where an aggregator is all we have, the record says "reported" and names it, and the gap goes to the research queue.

Sides of the market are kept apart. A price can be offered to holders, paid to the people who make data, asked for an off-the-shelf dataset, paid in a sale that happened, or charged by a vendor to make data to order. These differ by large multiples for the same kind of data, so every price record carries its side, and no figure mixes them. The market rates show each side separately.

4. How a figure is made

A figure is any number computed from records rather than read from one: a count, a median, a range, a ratio. Every figure has its own page that states the derivation in plain words and lists every input record, so the arithmetic can be redone by hand.

Take the median price of a company's records in sales that actually happened. The figure is $28.3K per company, and its page says exactly how it was reached: which public prices on the sales-that-happened side are stated per company, how a range is counted (its geometric midpoint), which open-ended figures ("from", "up to") are listed but not averaged, and where the middle half of the prices runs. Compare it with the range of prices offered to holders for the same kind of data, $174K to $693K per company: the gap between the two is one of the most useful things a holder can know before a negotiation, and both numbers can be traced to every price behind them.

Some figures are small and specific. The Spirit Airlines auction moved the price of one archive by 2.5x from the opening bid to the late offer that followed it; that figure lists its two input records, the opening bid and the late offer, and nothing else. Others are large: the count of 1,022 price observations, read from 707 sources, is a figure too.

Where a number is our own setting rather than the market's (a fee band, the size of sale we take on, a parameter of the valuation model), it is read from the one place that defines it, and where it rests on market evidence it is a figure that lists the records behind it. A valuation from the valuation tool is arithmetic on public prices, never an offer or a promise, and it names the records it was priced from.

5. How everything stays current

A market that moves within a quarter makes a static research site wrong within months. Four mechanisms keep this one current, and each is visible.

Pipelines with schedules. Every dataset on the site is fed by a pipeline with a cadence: the price research, the deal and case ledgers, the source watch, the archive, the closure watch, the AI crawler census, the catalogue readings of data on offer and others. The map lists each one, what it reads, the level of the pyramid it writes and how fresh each store is against its own cadence. A store that falls behind its cadence shows as stale.

The daily source watch. Every morning the watch re-reads the sources behind the records and checks whether each record's quote is still on its page. A record's page says when its quote was last found again, or that it was not found, so a reader can see at a glance whether the evidence still stands. Quotes that go missing go to a corrections pass.

Permanent copies. For 590 sources we keep a permanent copy in the Wayback Machine, linked from the source's page, so a page that changes or disappears can still be read as it stood when we cited it.

Replaced records are flagged everywhere. When a newer record replaces an older one, our source check lists every figure and every guide sentence that still rests on the old one, and the daily research run points each at its successor. A guide never keeps an old price without someone deciding it should.

6. Laws and rulings

Legal statements are evidence like any other, and they are held to a stricter rule: primary text only. A statute is cited from the legislature's own text or the official code, a regulation from the Electronic Code of Federal Regulations, a ruling from the court's own opinion, agency guidance from the agency's own page. A law firm's summary can point us to the text; it is never the authority.

Each law or ruling a guide relies on is a record of its own kind, with its citation, its jurisdiction, the passage quoted word for word and one or two plain sentences on what it means for a holder. Before a law enters our store, a script reads its primary page again and confirms the quoted passage is there, word for word: for federal regulations it reads the eCFR's official XML, for court opinions the court's own PDF. A passage that cannot be confirmed does not enter. The laws and rulings appear in the records register alongside prices and deals, and the rights and law hub brings them together with the terms of each business system and the questions a buyer's counsel asks.

None of this is legal advice. It is the research a holder and its counsel need to start from, with the text in front of them.

7. How a guide is built on the evidence

A guide on this site answers one question, the way people ask it, completely. We choose the questions from what people search for in this market (Google's own suggestions are the best record of the phrasing) and from what holders ask us, and we write a guide only where no existing page already answers the question.

Every guide is written from an evidence pack: every record, figure and organisation that bears on its question, drawn from the registers with each item's source and basis. Its numbers are tokens that link to their records, and its legal statements link to the laws and rulings. At the foot of every guide, "What this guide rests on" counts and lists every record and figure it cites. On every record's own page, "Cited in these guides" lists the guides that use it, so the pyramid can be read upwards as well as down.

Guides are organised in seven topics, each a question the market asks: selling data, prices, buyers, deals, rights and law, what sells and this one, research. Each topic has one page to start from, and every guide shows where it sits and what else its topic holds. All guides shows the whole tree on one page.

When a guide needs a fact that no record states yet, the guide says the point without the number, or leaves it out, and the missing fact goes on a list the daily research run works through. When the record exists, the sentence gains its number and its link. That is the loop between the research and the content: the guides tell the research what it lacks, and the research tells the guides what changed.

8. What we do not do

We do not type numbers. If a number is on a page, it links to its record or figure. Our source check lists every number that does not, and that list only ever gets shorter.

We do not present estimates as facts. A valuation is an indication from public prices, and the site says so beside every one. A range we compute is labelled with its working. Where there is no public price for something, we say so rather than inventing one.

We do not cite aggregators or rumours as evidence. See section 3.

We do not publish private negotiations. Nothing a holder tells us, and nothing about a sale we are running, appears in the public records. Holders' counts enter the market statistics only without their names and combined with others', as our agreements say.

We do not hide our position. We act for holders, and only holders pay us, a fee fixed in the mandate, only if a sale closes. No buyer pays us in a sale we run. Where a guide describes how we work, it says so plainly.

9. Reusing our data

The registers behind the site are published as open datasets you can download and cite. Our compilation of the tables is licensed under CC BY 4.0; quotes keep their sources' own terms, and the text of the guides is ours and not covered by that licence. Each table page shows how to cite it.

AI assistants can query the same records through our MCP server, which answers with the records and their sources rather than with prose. The weekly newsletter summarises what moved in the records each week.

If you find an error, write to us through the contact options on this site with the record's link. A correction becomes a new record, the old one is marked as replaced, and every page that cited it follows.

Questions people ask

Where do your AI training data prices come from? From the parties themselves wherever possible: buyers' terms pages, court filings, securities filings, vendors' price guides and named press. Each price is a record with its source quoted, on the price board, and each median on the market rates lists the prices behind it.

How do I know a number is still right? Open its record. The record shows when its source was last re-read and whether the quote was found again, and links a permanent copy of the source where we hold one.

Why do some figures say "reported"? Because the only source is a press report and no party has stated the figure itself. We keep reported figures because they are often the only public signal, and we label them so they are weighed accordingly.

Do you estimate missing prices? No. Where no public price exists, the site says so. The valuation tool combines public prices into a range for what you hold and names the prices it used; it is an indication, not an estimate presented as a price.

Can I cite your figures in a report or article? Yes. Cite the figure's or record's own page, which states its derivation or source. The table compilation is under CC BY 4.0, and quotes keep their sources' terms.

Are you neutral? No. We act for holders who sell or license data, and we say so. That is why every number can be checked down to its source: you do not have to trust us to use the evidence.

What this means for you

If you hold data and are weighing an offer, open the records behind the numbers you are given and the ones we publish, and compare like with like: the same side of the market, the same unit, the same basis. The offer check does that against an offer you already have, and the valuation gives you a range from public prices with the records it used.

If you are researching the market, start from the map or all guides, and follow any number down to its source. Every step is a link.

This page describes our research methods. It is not legal, tax or investment advice.

What Is Training Data Provenance, and Why Buyers Now Ask for It

AIData.Broker — Sun, 18 Oct 2026 13:38:01 +0000

Originally published at aidata.broker.

Training data provenance is the documented history of a dataset: where each part of it came from, the rights and consents it was collected under, and everything that was done to it before it reached a model. Buyers ask for it now because the law and the courts have started to sort training data by its origin. In Bartz v. Anthropic a federal judge held that training on books was fair use but sent the pirated library copies to trial, and the case then settled for $1.5 billion. The EU AI Act requires every provider of a general-purpose model to publish "a sufficiently detailed summary about the content used for training", and California's AB 2013 has required developers since 1 January 2026 to post "the sources or owners of the datasets" and whether they were purchased or licensed.

A developer can only answer those questions about its models if the companies it licenses data from can answer them about their data. That is why provenance has moved from an academic idea into buyers' contracts, where holders now warrant lawful collection and, in one buyer's published terms, refund every payment if the description of the data proves materially wrong. This guide is for a company that holds data and is weighing a licence, and for anyone trying to understand what a buyer means when it asks "where did this come from?" It defines provenance, explains each reason buyers ask for it, sets out the standards that exist, lists what a provenance record for a lot contains, shows how a holder documents one, and ends with how this site keeps provenance for its own facts.

AIData.Broker acts as the holder's agent: the holder licenses its data to the buyer itself, and we do not take ownership of it. The provenance record described here is the one we ask a holder to settle before any buyer sees anything.

This guide is research from public sources, not legal advice.

What training data provenance means
Why buyers ask for it now
The standards that exist
What a provenance record for a lot contains
How a holder documents it
Worked example: how this site keeps provenance for its own facts
Questions people ask
What this means for you

What training data provenance means

The word comes from the art market, where a painting's provenance is the chain of owners back to the artist. For data the idea is the same but there are more links in the chain. A support ticket was written by a customer and an agent, stored in a help desk run by a software vendor under that vendor's terms, governed by the privacy policy in force when the customer wrote it, exported by the company, filtered, de-identified, and packaged into a lot. Each step either preserves the right to use the ticket for training or breaks it. Provenance is the record of those steps, kept well enough that someone else can check it.

It helps to split provenance into four questions, because buyers, regulators and standards ask all four.

Origin. Which system the data came from, who created it, and over what period. A dataset of emails from a company's own mail server is different from the same emails pulled out of a customer's account.
Rights. Under what terms it was collected: the privacy notice, the contracts with customers and vendors, the consents of the people in it, and any licence from a third party for material the company did not create.
Processing. What was done to it: cleaning, filtering, de-identification, rewriting, sampling, and what was deliberately left out.
Custody. Who has held it, and under what agreements, from the moment it left the source system to the moment it reached the buyer.

Two neighbouring terms cause confusion. Data lineage is the engineering view of the same history: which pipeline produced which table from which inputs. It answers "how was this file made?" but usually not "were we allowed to?" Attribution is giving credit to a creator, which matters for licences that require it but says nothing about consent or processing. Provenance in the sense buyers now use includes both and adds the legal layer. The best-known proposal for writing it down, Datasheets for Datasets by Gebru and colleagues, proposes that every dataset "be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on", on the model of the datasheets that accompany electronic components.

Provenance is not the same as quality, and it is not a guarantee. A dataset can be well documented and still contain errors, and a perfect-looking record can be false. What provenance does is make claims about the data checkable: a buyer can test a sample against the record, a regulator can follow it, and a court can see who knew what. That is exactly what the developments below reward.

Why buyers ask for it now

Until recently, much training data had no provenance worth the name. It was crawled, mirrored and reshared, and the record of where it came from was lost on the way. An audit of widely used text datasets by the Data Provenance Initiative found "frequent miscategorization of licenses on widely used dataset hosting sites", with licences often missing altogether, and called it "a crisis in misattribution and informed use." Four forces have since made that gap expensive.

The lawsuits turned on where the data came from

The most consequential ruling so far did not condemn AI training as such. In June 2025 Judge Alsup granted summary judgment for Anthropic "that the training use was a fair use", and in the same order sent to trial "the pirated copies used to create Anthropic's central library." The difference between the two was provenance: books bought and scanned on one side, copies downloaded from pirate sites on the other. The case then settled. The settlement of $1.5 billion plus interest came to about $3,000 for each work, and the court gave final approval in July 2026.

Other cases point the same way. In Kadrey v. Meta the court found for Meta on fair use on the record before it, but later let the authors add a claim over downloading books from shadow libraries by BitTorrent. The US Copyright Office's report on generative AI training said that "the copying of expressive works from pirate sources in order to generate unrestricted content that competes in the marketplace, when licensing is reasonably available, is unlikely to qualify as fair use." And where training was held not to be fair use, as in Thomson Reuters v. Ross, the source was a competitor's own material used without a licence. Several music cases ended in settlements that turned unlicensed models into licensed ones: UMG and Udio agreed a licensed platform, and Warner Music and Suno agreed that Suno would launch licensed models.

The lesson buyers drew is practical: a model trained on data of unknown origin carries a liability that cannot be removed later, so they want the origin settled before they pay. Our guide to fair use and AI training covers the rulings in depth, and every case we track is on the cases page.

The EU AI Act requires a public summary and technical records

The EU's AI Act puts provenance duties on providers of general-purpose AI models, the labs that train foundation models and place them on the EU market. Three of them matter here. Under Article 53(1)(d) a provider must "draw up and make publicly available a sufficiently detailed summary about the content used for training of the general-purpose AI model, according to a template provided by the AI Office." Under Article 53(1)(c) it must have a policy to comply with EU copyright law, "in particular to identify and comply with" opt-outs from text and data mining. And under Annex XI its technical documentation for the AI Office must include "information on the data used for training, testing and validation, where applicable, including the type and provenance of data and curation methodologies," and "how the data was obtained and selected." The obligations have applied since 2 August 2025, with models already on the market before that date given until 2 August 2027. The Digital Omnibus adopted in July 2026 did not change Article 53.

The public summary is deliberately broad. The Act's recitals say it should be "generally comprehensive in its scope instead of technically detailed", giving as an example listing "the main data collections or sets that went into training the model, such as large private or public databases or data archives." The Commission's explanatory notice to the template says it "does not require disclosure of the details for the specific data and works used to train the model", though providers may disclose more voluntarily. So a holder's name will not necessarily appear in a lab's public summary, and a licence can say whether it may. But the technical documentation behind the summary, and the copyright policy, need the lab to know the type and provenance of each source, and the lab can only know that if its suppliers documented it.

For AI systems classed as high-risk, the Act goes further. Article 10(2) requires training data to be governed by practices covering "data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection," and "relevant data-preparation processing operations, such as annotation, labelling, cleaning, updating, enrichment and aggregation." That is a provenance record by another name.

California's AB 2013 asks for sources, licences and processing

California's Generative Artificial Intelligence Training Data Transparency Act, AB 2013, is now Civil Code section 3111. On or before 1 January 2026, and each time a new or substantially modified system is released to Californians after that, the developer of a generative AI system or service released on or after 1 January 2022 must post documentation on its website about the data used to train it. The documentation includes a high-level summary of the datasets, covering twelve topics. Several of them read like the fields of a provenance record: "(1) The sources or owners of the datasets", and, as the federal court quoted the statute, whether the datasets include copyrighted material, "whether the datasets were purchased or licensed by the developer," whether they include personal information, "whether there was any cleaning, processing, or other modification to the datasets by the developer," and "the time period during which the data in the datasets were collected."

The law has survived its first challenge. xAI sued to block it; in March 2026 the court denied a preliminary injunction, finding that even if "high-level summary" is "not the picture of clarity standing alone, it is followed by a precise list of information to be included." The court did not decide how specific "sources or owners" must be. Early disclosures have stayed general, describing categories such as data from third-party partners rather than naming each licensor, as one law firm reported of the first filings by two large labs. For a holder, the point is the same as under the EU Act: a developer that must state whether each dataset was licensed, whether it holds personal information, what processing was done and when it was collected, will ask its suppliers for exactly those facts.

Buyers' own contracts now put provenance into warranties

The most direct pressure comes from the licences themselves. Buyers that pay for company records now ask holders to promise, in writing, the things a provenance record documents, and to carry the cost if the promise fails. Holders commonly give warranties of title, authority, lawful collection, non-infringement and privacy compliance, and buyers ask for representations that consents were obtained and no opt-out has been invoked. Our research has read 65 clauses from buyers' published terms, and the pattern is consistent.

Buyer or service	What its published terms ask the holder to stand behind
micro1	If the data manifest proves materially inaccurate, nothing is payable and the holder refunds what it was paid, even after acceptance; the holder also promises it has not licensed the same data on similar terms to anyone else
SimpleClosure AssetHub	The holder warrants it gave every required notice and obtained every required consent from employees, contractors, customers, investors and vendors in the data
Defined.ai	Data must be "ethically sourced, including proper consent, documentation, and the right to commercialize"
Kled	A contributor who withdraws consent refunds what it was paid for that content
Grepped	Tells its own buyers it offers "Clear provenance. Every source is licensed under a signed agreement with the data owner."
Mercor	Its intake asks the holder to confirm it owns the materials or may share them without restrictions

Two features of these terms make provenance urgent rather than tidy. Payment often waits for it: one buyer lets the holder invoice only after it has accepted the data in writing, and acceptance is where a buyer checks the data against its description. And training cannot be undone: the same buyer's deletion clause does not require it to delete or retrain any model, and a law firm advising holders warns that once data trains a model "it cannot practically be pulled back out". If the provenance is wrong, the remedy is money, not a recall. For the clause-by-clause view, see our guide to data licensing agreements and the licence terms page.

Consent now changes the price

Provenance also shows up in what people are paid. Where book authors had to opt in to an AI licence, the publisher paid them far more per title than where inclusion was the default and authors could only opt out. A licensor that can show each contributor's consent, title by title, can sell what one without that record cannot. Our what labs buy page shows how buyers weigh records that connect and can be traced.

The standards that exist

No single standard governs training data provenance, and none is required by law in the form a holder would use. What exists is a handful of documentation formats and audits, each built for a different job. A holder does not need to adopt any of them wholesale; it helps to know what each asks, because buyers borrow their vocabulary.

Datasheets for Datasets

The idea that a dataset should come with a standard description began with a paper by Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III and Kate Crawford, first posted in 2018 and revised in 2021. Its premise is that "the machine learning community currently has no standardized process for documenting datasets", and its proposal is a set of questions the creator answers about the dataset's motivation, composition, collection process, preprocessing, recommended uses, distribution and maintenance. The questions are the strength: they make the creator say who is in the data, how it was gathered, whether consent was sought, and what was removed. The EU AI Act echoes the term for high-risk systems, whose technical documentation must describe data requirements "in terms of datasheets", including information about the datasets' provenance and how the data was obtained. A datasheet is also what we ask a holder to write for a lot, through our datasheet tool.

The Data Provenance Initiative

The Data Provenance Initiative is a group of researchers who audit the datasets AI is trained on. Its first large audit, by Shayne Longpre, Robert Mahari and co-authors from law and machine learning, set out to "systematically audit and trace" text datasets used to fine-tune language models, building tools "to trace the lineage of these datasets, from their source, creators, series of license conditions, properties, and subsequent use." It found licences frequently missing or wrong on popular hosting sites. A second audit, Consent in Crisis, looked at the websites behind major web corpora and found "a rapid crescendo of data restrictions from web sources" in a single year, through robots files and terms of service. The group now describes itself as public measurement infrastructure for AI data. Its work is the evidence for why buyers distrust data whose origin they cannot see, and why the open web is no longer a free substitute for licensed records (our guide to scraping and the AI crawler census cover that side).

The Data & Trust Alliance's Data Provenance Standards

The closest thing to a business standard comes from the Data & Trust Alliance, a group of large companies that use data and AI. Its Data Provenance Standards, version 1.0.0, were co-developed by the Alliance's member organisations from use cases across many industries, and group their metadata fields under three headings: Source, Provenance and Use. The fields include the dataset's issuer and source, the data's origin geography, the range of dates for data generation, the method of collection, the location of consent documentation, whether privacy enhancing technologies were applied, processing and storage geography, the licence to use, the intended use, and whether proprietary data is present. The Alliance publishes a generator that writes the metadata as a file. For a holder, the field list is a useful checklist: it is what a corporate buyer's data team may expect to see filled in.

C2PA Content Credentials for media

For pictures, video and audio, provenance can travel inside the file. The Coalition for Content Provenance and Authenticity (C2PA) publishes an open technical standard for "publishers, creators and consumers to establish the origin and edits of digital content," called Content Credentials. In its specification, a set of signed statements about an asset (how it was captured, how it was edited, what it was made from) is bound into a C2PA Manifest, and "the set of C2PA Manifests, as stored in the asset's Content Credential, represent its provenance data". The specification also allows a manifest to cover a collection of files: "It is possible to describe each folder of the training data set of an AI/ML model by having each folder be a separate ingredient of the complete training data set's manifest." The standard is careful about its limits: it does not judge whether provenance is good or bad, only whether the statements can be validated and are free from tampering. A holder of footage or photo archives with Content Credentials can prove origin and edits file by file; one without them relies on its own records (our pages on video and images cover those markets).

How they compare

Standard	Who publishes it	What it documents	Where it helps a holder
Datasheets for Datasets	Gebru and co-authors (research paper)	Motivation, composition, collection, preprocessing, uses, distribution, maintenance	The narrative description of a lot that a buyer reads first
Data Provenance Initiative	Researchers auditing AI datasets	Sources, creators, licence conditions and later use of public datasets; web consent signals	Evidence of why buyers want licensed, documented data
Data Provenance Standards	Data & Trust Alliance (companies)	Metadata fields in three groups: Source, Provenance, Use	A field checklist and machine-readable metadata for a lot
C2PA Content Credentials	Coalition for Content Provenance and Authenticity	Signed manifests of an asset's origin and edits, also for collections	File-level proof for images, video and audio
EU AI Office template	European Commission	The public summary of a model's training content	What a lab will say publicly about its sources
California AB 2013	California law	Twelve topics about a model's training datasets	What a lab must post, and so what it will ask suppliers

What a provenance record for a lot contains

A lot is a defined body of records offered to buyers: one company's tickets and chats for a period, say, or a repository with its history. A provenance record for a lot answers the four questions (origin, rights, processing, custody) for that lot specifically, with evidence for each answer. It does not need to be long. It needs to be specific enough that a buyer's counsel can test it, and honest about what is unknown.

The table sets out the elements, what each records, the evidence that backs it, and who asks for it. The last column shows why the record is worth building once and well: the same facts answer a buyer's due diligence, the developer's AB 2013 posting and its EU documentation, and the holder's own warranties.

Element	What it records	Evidence	Who asks for it
Source systems	Each system the records came from, whose account it was, and how they were exported	Export logs, admin settings, the vendor's terms at the time	AB 2013's sources or owners; the AI Act's type and provenance
Collection period	The first and last dates of the records, and any gaps	Export metadata, a count of records by month	AB 2013's time period during which the data were collected
Legal basis and consents	The privacy notice, terms and employee policies in force when the data was created; any consents; recording notices	Dated copies of the policies, consent logs	Buyers' consent representations; the AI Act's original purpose for high-risk uses; our prior promises check
Licences in	Material the holder did not create (bought lists, stock media, reports, other companies' documents) and the terms it came under	Contracts and licence records, or proof it was removed	AB 2013's purchased or licensed and copyright topics; our third-party data check
Processing and de-identification	Cleaning, filtering, de-identification method and version, and its measured result	An evidence file, an expert's or certifier's report	AB 2013's cleaning, processing, or other modification; the AI Act's data-preparation operations; our evidence file rule
Exclusions	What was taken out and why: privileged and HR material, other companies' confidential documents, health or children's data, people who opted out	An exclusion list with reasons	Buyers' warranties; our confidentiality and privilege checks
Chain of custody	Who held the data from export to delivery, where it was stored, and every transfer	Transfer records, file hashes, agreements with any processor	Buyers' acceptance testing; the manifest a buyer checks the delivery against
Prior licences and exclusivity	Earlier licences of the same records, and any exclusivity granted	Contracts	Buyers' prior licence warranty; exclusivity terms
Authority to sign	Who may license the records for the company: the board, an officeholder in a wind-down, a lender's consent where needed	Resolutions and consents	Our board, insolvency and lender checks

Three points about this list are easy to get wrong. First, the policies that matter are the old ones: a privacy notice changed last month does not cover data collected under the version before, and the FTC has warned that quietly changing terms to permit new uses of data already collected may be unfair or deceptive. Keep dated copies of every version. Second, exclusions are part of provenance, not an admission of weakness: a list of what was removed and why tells a buyer the rest was looked at. Third, custody includes the holder's own vendors: a scrub run by a processor, or a copy kept by a broker, is a link in the chain, and some services delete the originals once they have processed them (as one broker's terms say), which leaves the holder with no way to check the delivery afterwards unless it kept its own record.

How a holder documents it

Most companies already hold the facts a provenance record needs; they are scattered across contracts, admin consoles, old policy pages and people's memories. Documenting provenance is the work of gathering them into two files before any buyer is approached: the rights file and the datasheet.

The rights file is the holder's private record of every rights question and its answer, with the evidence. Buyers' counsel ask a predictable set of questions, and our checklist has 27 of them, 16 of which stop a sale until they are answered: who owns the records, what customer contracts and software vendors' terms allow, what the privacy notices said at the time, whether employees and call participants were told, whether third-party material or health data is inside, and who has authority to sign. The rights file holds the answers and the documents behind them. It is not shown to buyers whole; it is what lets the holder give warranties it can stand behind, and qualify the rest by what it disclosed (the usual structure of holder warranties).

The datasheet is the description a buyer reads: what the records are, where they came from, the period, the volume and composition, what was done to them, what was left out, and what uses they suit. It draws on the rights file but says only what can be shared under an NDA, and it is written so that a buyer can test a sample against it. Our datasheet tool builds one from a holder's answers, and our readiness check shows which answers are still missing.

Most lots on offer today are not there yet

The gap between what buyers ask for and what holders can show is wide. Among the lots offered through brokers whose catalogues state who holds the rights, only 29% say the holder holds them with nothing left to confirm; 62% are held with conditions still to confirm or not confirmed at all (the lot index shows each one). A holder that arrives with its provenance settled is, for now, unusual, and that is an advantage in a negotiation: it shortens due diligence and removes the buyer's reason to discount.

Step by step

List the systems. Every place the lot's records live: help desk, chat, mail, code hosting, CRM, file storage. For each, note the account owner and the vendor's export and AI terms (Slack, Microsoft 365, Google Workspace, Zendesk and GitHub each have a page).
Fix the period. Decide the date range of the lot and record the first and last dates actually present.
Recover the old promises. Collect every version of the privacy notice, terms of service and employee policies in force during the period. If the company's own copies are gone, a public web archive may hold them.
Find what is not yours. Identify bought, licensed and third-party material and either document its licence or take it out.
Settle consents. Record what notice employees, customers and call participants received, and remove people or recordings the notice does not cover (our guide to employee emails and chats covers staff data).
De-identify and keep the evidence. Run the de-identification on your own systems, keep the method, version, sample review and measured result, and keep any re-identification key yourself (our standard; for health data, the HIPAA guide).
Write the exclusion list. Everything removed, and why.
Record custody. Hash the exported files, note where they are stored and who can reach them, and log every transfer.
Write the datasheet from the above, and keep the rights file behind it.
Keep it current. A provenance record describes one version of the data. If the lot is refreshed, or a buyer asks for a new slice, the record moves with it.

Example: one system's entry

Example. A 40-person business software company is closing and plans to license its support history. Its provenance entry for the help desk might read as follows. Source: the company's own help desk account, administered by its head of support, exported through the vendor's admin export in September 2026; the vendor's terms at the time allowed the customer to export its data. Period: tickets from March 2018 to August 2026, with no gaps. Basis: the privacy notice versions of 2018, 2021 and 2024 (copies kept), which told customers support conversations would be used to provide and improve the service; the company's counsel concluded the 2024 version did not cover licensing identifiable data, so the lot is de-identified. Licences in: none; attachments from customers' own vendors were removed. Processing: de-identified on the company's systems with a keyed stand-in for each person, a random sample read by a person, and an independent provider's certificate. Exclusions: tickets from customers whose contracts reserve their data, tickets tagged legal or security, all attachments. Custody: the export and the de-identified copy are held in the company's storage, hashed, and released to a buyer only under a signed licence. Each line points to a document in the rights file. A buyer's counsel can read it in five minutes and know what to test.

Worked example: how this site keeps provenance for its own facts

A research site is also a dataset, and the same discipline applies. This site maps the AI training data market as a set of facts, each with its origin recorded, so that a reader can follow any number on any page down to the page it came from. The method is a working example of provenance on a small scale, and it is the method we would ask of a holder's lot.

Everything here sits on one of five levels, and each level is built only from the one below it. At the bottom are sources: the pages, filings, dockets, articles and reports the site reads, 2,370 of them, each address counted once. Above them are records: every fact a source states, 3,346 of them that a newer reading has not replaced, each carrying the source's own words, the date it was read, the address, and how firm it is (stated by the party itself, reported by others, or self-reported). Then come the entities the records are about, then figures, every number computed from records, each with its derivation and its inputs, and finally the pages and guides like this one, where every number is a link down the chain.

Three habits keep the record honest. Permanent copies: 590 of the cited sources have a copy in the Internet Archive's Wayback Machine from near the day they were read, so a citation still shows what the page said then, even after the page changes. Re-reading: every day, the sources behind current records are read again for the words each record was read from, and a source that changed or lost a figure is re-read first. History, not overwriting: when a newer reading replaces a record, the old record is kept as history with a pointer to the one that replaced it, and stops being counted. That is the same shape as a lot's provenance record: origin (the source and its archived copy), the terms of use (what the source is and how firm its statement is), processing (the derivation of each figure), and custody (the history of each record). How we source every number explains the method in more detail.

Questions people ask

What is data provenance in AI? It is the documented history of the data a model is trained on: where it came from, the rights and consents it was collected under, and what was done to it. The EU AI Act uses the word itself, requiring general-purpose model providers to document "the type and provenance of data and curation methodologies" for the AI Office.

What is the difference between data provenance and data lineage? Lineage is the technical history of how a dataset was produced (which pipeline made which file from which inputs). Provenance includes that and adds the legal history: the terms the data was collected under, consents, licences and custody. A buyer's counsel cares about provenance; a data engineer often means lineage.

Do AI companies have to disclose their training data? In the EU, providers of general-purpose models must publish a sufficiently detailed summary of their training content, though the template does not require naming specific datasets or works. In California, developers of generative AI systems must post a high-level summary of their datasets, including sources or owners and whether data was purchased or licensed.

What does California AB 2013 require? Since 1 January 2026, a developer of a generative AI system made available to Californians must post documentation of its training data on its website, covering twelve topics including whether datasets were purchased or licensed, whether they contain personal information, the processing done, and the collection period. A federal court refused to block the law in March 2026.

What is a datasheet for a dataset? A standard description that accompanies a dataset, proposed by Gebru and co-authors, documenting its motivation, composition, collection process and recommended uses. In a sale, it is the document a buyer reads to understand a lot and test a sample against; our datasheet tool produces one.

What is C2PA? The Coalition for Content Provenance and Authenticity publishes Content Credentials, an open standard for signed records of a media file's origin and edits. It can also describe the folders of a training set as ingredients of one manifest, which makes it useful for licensing image, video and audio archives.

Why do AI buyers ask for consent records? Because they are asked to stand behind them. Buyers' terms commonly require the holder to warrant consents and opt-outs, and one buyer's terms make the holder refund everything if the description of the data proves materially inaccurate. Consent also moves price: opt-in book licences paid authors far more than opt-out ones.

Can provenance be added after a model is trained? Not in any way that helps the holder. Once data trains a model it cannot practically be pulled back out, and buyers' deletion clauses do not require retraining. Provenance has to be settled before the data leaves.

What this means for you

If you hold data a buyer might license, provenance is the part of the sale you control completely. Prices depend on the market; whether you can show where your data came from depends only on the work you do before the first conversation.

Start the rights file now, before any buyer is involved. Work through the questions in our readiness check and gather the documents behind each answer, the old privacy notices first.
Write the datasheet with our datasheet tool, describing origin, period, processing and exclusions in words a buyer can test.
De-identify on your own systems and keep the evidence: our de-identification tool runs on your computer and follows our standard.
Read buyers' terms for what they make you warrant, and qualify what you cannot prove. Our guide to data licensing agreements, the licence terms page and the buyers page set out who asks for what.
Know what the developer will publish. Decide in the licence whether the buyer may name you in its AI Act summary or AB 2013 posting.
Value the lot with its provenance in mind. Buyers say records that connect and can be traced are worth more (a ticket tied to its code commit beats a loose document); see what company data is worth and the full guide to selling company data to AI labs.

For how buyers source data today and where licensed records fit, see how AI companies get training data. For every fact this guide rests on, follow the links down to the records and the sources.

This guide is research from public sources, not legal advice.

What Are RL Environments? Who Builds, Buys and Prices Them

AIData.Broker — Sun, 18 Oct 2026 13:30:36 +0000

Originally published at aidata.broker.

An RL environment is a sandbox in which an AI agent attempts a task, using the tools the sandbox gives it, while a verifier checks the result and turns it into a score that trains the model. Labs buy them by the task and by the environment: builders and lab researchers interviewed by Epoch AI put a task with its grader at $200 to $2,000, a working replica of a website at about $20,000 and a high-fidelity clone of an app like Slack at about $300,000 - Epoch AI. The spending behind them is large: leaders at Anthropic have reportedly discussed spending more than $1 billion on environments in a year, and Scale AI wrote, as Fortune reported, that nearly half of its new data-training projects involve them.

This guide is for three kinds of reader. A holder of code, company records or expert knowledge who keeps hearing that labs "buy environments" and wants to know whether what it holds could become one. A builder of tasks and environments who wants to know who pays and what. And a reporter or investor who wants the plain mechanics behind a market where 89 vendors on our vendor register now say they build or sell environments. It covers what an environment is, how it differs from a dataset, the main types, who builds and buys them, what each costs according to public records, what makes one valuable, how a company's existing assets become environments, and how one is sold.

This guide is research from public sources, not legal, tax or investment advice.

What an RL environment is
How an environment differs from a dataset
The types of environment
Who builds environments
Who buys environments
What environments cost
What makes an environment valuable
How a company's assets become environments
How to sell an environment
Questions people ask
What this means for you

What an RL environment is

Reinforcement learning (RL) is a way of training a model by letting it try something, scoring the attempt, and nudging the model toward whatever scored well. For a language model that acts as an agent (software that takes steps on its own: it runs commands, clicks through an app, edits files), the "something" is a piece of work, and the place where the work happens is the environment. Epoch AI, which interviewed builders and lab researchers for the most cited account of the field, describes the loop simply: the model attempts a task, a grader scores the attempt, and the scores update the model's weights - Epoch AI.

An environment has three parts, and each is something a vendor can sell. The sandbox is the world the agent acts in: a copy of a code repository, a working clone of an app, a desktop with files on it, a simulated company with its records. It is defined by the actions the agent can take and by what those actions do, and it is usually shipped as a container (a packaged copy of the software that runs the same way on any machine). The task is an instruction with a goal: "fix the failing checkout test", "file this expense report", "build a pivot table of revenue by region". The verifier (also called a grader) decides whether, or how well, the goal was met. In coding it is usually a set of tests that fail before the fix and pass after it; elsewhere it can be a script that inspects the final state of the app, or a rubric that another model applies.

{
  "url": "https://epoch.ai/assets/images/gradient-updates/2026/state-of-rl-envs/example_rl.png",
  "alt": "Diagram of an RL environment: a task is given to an AI agent, the agent sends actions such as search_channel and send_message to a simulated Slack workspace running in a container, receives observations and error messages back, and a grader converts the final state into a reward",
  "title": "The parts of an RL environment",
  "caption": "Source: Epoch AI, An FAQ on RL environments, January 2026 (CC-BY)"
}

The word that matters most is verifiable. The training method behind most of this market is often called RLVR, reinforcement learning with verifiable rewards, and vendors describe their products in those terms (ARIMLABS builds "RLVR training environments"; Andromede calls itself "an RL data lab generating environments, tasks, and verifiers from real-world data"). A verifiable reward is a score a machine can compute without a person reading every attempt, which is what lets a lab run the same task many times over. The model may attempt one task dozens of times in a single training run, and the verifier has to judge every attempt correctly, cheaply and without being fooled.

Why labs want this now is the other half of the definition. Models learned to write from reading the public internet; they learn to do work by practising it, and there is no public supply of practice. Epoch's interviewees put the shift in one sentence: "Math and coding tasks came first, but we're now seeing significant growth in enterprise workflows: tasks like navigating Salesforce, filing reports, or manipulating spreadsheets." On Hugging Face, the largest free hub for training data, only 148 datasets carry the rl-environment tag. Finished environments are almost absent from the free supply, which is why labs pay for them.

Two terms you will meet. A gym (or "UI gym") is an environment built as a replica of a website or app. An eval is an environment or task set used to measure a model rather than train it: the structure is the same, the licence and the buyer's purpose differ. Epoch notes that the boundary between "environment" and "task" is itself fuzzy, and in practice a lab buys the two together.

How an environment differs from a dataset

A dataset is a fixed collection of examples. A model reads it (pretraining, where it learns language and facts from text) or imitates it (supervised fine-tuning, where it learns to copy good answers). Once a lab has the dataset, it has everything the dataset can teach. An environment is interactive. The model acts, the environment responds, the verifier judges the outcome, and the lab can generate as many fresh attempts as its compute allows. What the lab buys is not a set of answers but a machine for producing graded experience.

That difference changes how each is priced and how long each keeps its value. A dataset is priced by volume (per hour of video, per word, per company archive) and an archive of real human work keeps its value for years, because no model can produce it for itself. An environment is priced per task or per environment, and it loses value as models get better at it: an investor's analysis of the category put an environment's useful life at a half-life of about six months, adding that "the same customer rarely buys the same environment twice". A task that current models always pass teaches nothing, so a builder's product loses value as it works: the better the models it trains, the sooner it is retired.

	Dataset	RL environment	Evaluation set
What it holds	Fixed examples: text, code, recordings, records	A sandbox, tasks and a verifier for each task	Tasks with answers or graders, kept private
How a lab uses it	The model reads or imitates it	The model practises in it and is scored	The model is measured on it, not trained
Typical unit	Per hour, word, item or company	Per task, per environment, per contract	Per question or task
Example price	A company's whole archive: $10M (Spirit Airlines)	A task: $200 to $2,000; a replica: about $20K	A benchmark question: up to $5,000
How value changes	Keeps value if no model can make it	Falls as models learn to pass it	Gone once published or leaked

The two meet in one place. Every attempt an agent makes inside an environment can be recorded as a trajectory (the full sequence of the agent's reasoning, actions and the environment's responses), and a set of trajectories is a dataset again. Together AI generated 258,000 test-verified coding-agent trajectories for a reported $130,000 in compute, about 50 cents each, and released them free. That number marks what a machine-made trajectory is worth. It is the human-built environment, and the real material inside it, that a model cannot produce for itself.

The evaluation column matters for holders because the same task can be sold twice in different ways. Labs and evaluators pay for private, uncontaminated test sets, in one practitioner's words, "because public benchmarks have leaked into pretraining data". Scale AI's SWE-Bench Pro benchmark keeps a commercial set of 18 proprietary repositories bought from early-stage startups for exactly that reason: a model cannot have trained on code it has never seen.

The types of environment

Environments are usually grouped by what the agent is asked to do. Coding came first because a test suite is a ready-made verifier: a task either makes the tests pass or it does not. Website replicas came next, as labs built agents that operate a browser. The growth area now is enterprise work, where an agent operates business software (a CRM, an accounting system, a ticket tracker) inside a simulated company, and professional domains such as finance, law and science, where the grader encodes an expert's judgment.

The table below lists the main types with the builders and buyers recorded for each and the public prices on record. Prices marked as paid to a contributor are what an individual is paid to build a task, not what a lab pays the vendor; the gap between the two is covered under what environments cost.

Type	What the agent does	Builders on record	Buyers and programs on record	Recorded prices
Coding repositories	Fixes a bug or adds a feature in a real repository; tests verify	Mechanize, pre.dev, Datacurve, Revelo, Turing, Mercor	Anthropic, reported as the most aggressive buyer of coding RL data	$5,000+ a task (reported); $2.2K a task paid to a contributor
Terminal and systems tasks	Works in a Linux shell: builds, configures, debugs	Scale AI (through Outlier), Turing, Vmax	Labs, through Scale's Outlier programme for Terminal-Bench tasks	Up to $1,800 a task paid to the author
Website and app replicas	Operates a working clone of a site or SaaS app	Deeptune, Plato, Archal, Refresh	OpenAI, reported to have bought hundreds of sites	About $20K a site; about $300K for a Slack-grade clone
Enterprise software and workflows	Files reports, updates records, moves work between tools in a simulated company	Theta, Bespoke Labs, Ooak, DARE Labs, The General Data Company	Frontier labs and neolabs on quarterly contracts	Contracts "often six to seven figures per quarter"; $300K to $500K seen at a neolab
Finance and professional work	Builds a model, prices a deal, answers a legal question; a rubric grades	Halluminate, Dissei, EdotEnv, Aptura	Four of the top five closed US labs pay Halluminate (reported)	$1,000 a task paid to a finance contributor; $2,000 a task for legal tasks at scale
Browsing and computer use	Operates a desktop, phone or browser from screenshots	Cua, Markov, Plato, Eigent AI	Prime Intellect's bounties for open versions of OSWorld and WebArena	$5,000 for one bounty; $50 a task paid to a contributor
Science, engineering and proofs	Designs a circuit, proves a theorem, writes a research plan	Tacit Labs, Synthetic Sciences, CueBench	A lab buyer chose an engineering task set and a science environment in one round (reported)	$1,500 bounty for an open research-replication environment
Security	Attacks or defends a simulated network	Incalmo, exponential, Bugcrowd	The UK AI Security Institute, for evaluations	£999,600 for one cyber-range contract
Robotics simulation	Controls a simulated robot; the simulator scores the result	Applied Intuition, Robotics Center	Robotics teams	From $5,000 for a starter environment; $50K+ for a digital twin

Coding repositories

The coding environment is the template for the rest. A repository is frozen at the moment before a real change was made; the task is the issue that asked for the change; the verifier is the set of tests that the real change made pass. SWE-bench, the public benchmark built this way from open-source projects, is public, so its tasks and their answers can leak into training data; that is why the market pays for the same structure built on private code. micro1 recruits engineers to "build reproducible Reinforcement Learning Environments that test an AI model's ability to fix code, add features, refactor, and optimize performance", each with a golden reference solution, at $50 to $100 an hour for about 15 hours a task. A codebase that already holds thousands of such changes, each with its ticket and its tests, is raw material for this kind of environment, which is the subject of our companion guide on selling a codebase to AI companies.

Website and app replicas

A replica is a working copy of a website or SaaS product, filled with plausible data, in which an agent can click, type and submit without touching the real service. Simple ones are cheap, and the market is crowded with them: one lab researcher told Epoch that "what everyone does is vibe code a buggy website which isn't useful". High-fidelity clones of complex products cost an order of magnitude more, at about $300,000 for a Slack-grade copy. Turing pays contributors $300 per approved task for Python backend replicas of tools such as Slack, Linear, Jira, Notion and Gmail.

The replica price has a consequence for any company that owns an interface. Where a lab can have a clone of a site built for about $20,000, it does not need the company that runs the site. The cost of a replica caps what an interface alone is worth. What a replica cannot copy cheaply is what fills it: realistic records, real workflows, and the judgment that decides whether a task was done well.

Enterprise work and professional domains

Enterprise environments put an agent inside a simulated company: a CRM with accounts in it, an inbox with threads, a ticket queue, a finance system with a month to close. The tasks are the ordinary work of that company, and the grader checks the end state or applies a professional's rubric. This is where real company data enters the picture. Halluminate, which builds finance and consulting environments, told Fortune that one set's "88 tasks were based on anonymized private-equity transactions", and Ooak says it "turns anonymized real company data into datasets and RL environments". The section on how a company's assets become environments follows that thread.

Who builds environments

Builders fall into four groups, and the lines between them are blurring. The first is specialised startups that do nothing else: Mechanize for coding agents, Halluminate for finance and knowledge work, Deeptune for software "training gyms", Collinear, Plato, Vibrant Labs and dozens more. Of the 150 AI data vendors on our register, 89 say on their own pages that they build or sell RL environments, gyms or graded tasks.

The second group is the established data vendors that added environments to human data work: Scale AI, Surge AI, Mercor, Turing, Handshake AI, Labelbox, Snorkel AI, Invisible Technologies and Toloka. Their advantage, as Epoch's interviewees put it, is that they can staff a project with experts quickly. Mercor's chief executive described the economics on a Sequoia podcast: a task that takes "ten hours of human time" paid at $150 an hour has a cost basis of about $1,500, and Mercor's tasks run "from $50 to $10,000" (both his own figures). Mercor also bought Deeptune, an environment builder that had raised a $43 million Series A earlier the same year, at an undisclosed price.

The third group is platforms and open hubs. Prime Intellect runs an Environments Hub and paid bounties to anyone who built an open environment, crowdsourcing more than 400 environments and evals in two months and committing "hundreds of thousands of $ in grants". HUD provides a platform and open-source software for building environments and running agents in them, and a marketplace, DataVendor, where independent builders list task sets and environments; HUD raised a $16M Series A and says vendors on that marketplace are "already doing millions a month" (its own claim).

The fourth group is the labs themselves. Epoch reports substantially more in-house building at labs and neolabs, and SemiAnalysis reports that Anthropic's own engineers have made coding tasks for over a year while it also buys from startups (SemiAnalysis, reported). Product companies are a natural fifth partner when the environment is their own software: Epoch names Benchling working with Anthropic, and Shopify and Stripe working with OpenAI.

What building one costs in people

Most of the cost of an environment is skilled time. TechCrunch reported in 2025 that Mechanize was offering software engineers $500,000 salaries to build environments, and SemiAnalysis reported that Mechanize expects engineers paid $400k+ a year to make one good task a week. In September 2026 Google closed a reported talent-and-licence deal worth about $1.5 billion or more for Mechanize. The contributors who write individual tasks are paid far less than the labs pay for them: across public prices, makers are paid a median of $300 per task, while buyers pay 9.5 times what makers are paid, per task. Per-task pay looks scattered, but it lines up once a platform's own time estimate is applied: coding tasks converge on about $45 to $100 an hour of work.

Who buys environments

Environments are bought mainly by frontier labs, the handful of companies training the largest models, and Epoch's interviewees describe training as the main use, with requests for benchmarks much less common. The largest buyers are known mostly through reporting rather than disclosure. Anthropic's leadership reportedly discussed spending more than $1 billion on environments over a year (The Information, via TechCrunch), and SemiAnalysis calls Anthropic "the most aggressive lab by far" in buying coding data from environment startups. SemiAnalysis also reports that OpenAI bought hundreds of website replicas for ChatGPT Agent. Halluminate says four of the top five closed-source US labs are paying customers and that it reached a mid-eight-figure revenue run rate (its own figures).

Below the frontier labs sit neolabs (newer, well-funded model companies) and companies training their own agents. A neolab researcher told Epoch of contracts in the $300K to $500K range, often per quarter. Scale of purchase varies as much as price: Mercor's chief executive said "certain frontier labs might buy 50,000 tasks a month from us", a vendor's description of its own business rather than a disclosed contract.

Governments and evaluators buy environments to test models rather than train them. The UK AI Security Institute awarded a cyber-range contract of £999,600 for realistic enterprise networks with tasks, solvers and automated graders; the EU AI Office awarded an agentic evaluation interface contract of EUR 968,521 over two years. METR, an evaluation non-profit, pays a bounty of $300 per hour it would take a professional to complete the task, one sign of how evaluators anchor a task's price on the human time it takes.

Where buyers meet builders matters for anyone with something to sell. Most labs buy through vendors and marketplaces rather than from individual builders. HUD's DataVendor lists task sets and environments for lab research teams and says requests are "usually priced per task, at around $500 to $1,500 a task, with hundreds to thousands of tasks per request". On 12 October 2026 a lab buyer chose five lots in one round through a buyer-side referral network, two of them environment lots: an engineering set of 12K+ containerised tasks graded automatically, and a science research environment scored on rubrics (both reported, with no price given). Our lot index currently lists 63 lots of datasets and environments on public catalogues.

What environments cost

Environment prices come in three units and from three sides of the market, and mixing them up is the commonest mistake in reading this market. The units are per task, per environment and per contract. The sides are what the buyer pays the vendor, what the vendor pays the person who writes the task, and what it costs to produce. A contributor's $50 a task and a lab's $5,000+ a task are both true; they are different sides of the same trade.

The table collects the public prices on record. The Epoch figures come from interviews with people who build and buy environments, so they are reported practice rather than a price list, but they are the anchors everyone in the field quotes.

What	Price	Unit	Side	Date	Source
A training task with its grader	$200 to $2,000	per task	buyer pays	Jan 2026	Epoch AI
An especially complex software task	$20K, "rare but possible"	per task	buyer pays	Jan 2026	Epoch AI
A decent coding task bought by a frontier lab	$5,000+ (reported)	per task	buyer pays	Jul 2026	SemiAnalysis
Mercor's tasks, full range (its CEO)	$50 to $10,000	per task	buyer pays	Oct 2026	Sequoia
Tasks built to a request on HUD's DataVendor	$500 to $1,500	per task	buyer pays	Oct 2026	DataVendor docs
A website replica ("UI gym")	About $20K; Slack-grade about $300K	per environment	buyer pays	Jan 2026	Epoch AI
An environment supply contract with a neolab	$300K to $500K	per contract	buyer pays	Jan 2026	Epoch AI
A custom robotics environment, starter to suite	From $5,000 to from $25,000	per environment	buyer pays	Oct 2026	Robotics Center
An open environment built for a bounty	$100 to $500 open; $1,000 to $5,000+ by application	per environment	bounty	Oct 2025	Prime Intellect
A database engineering task with tests and grader	$2.2K, about 25 hours	per task	maker paid	Oct 2026	Mercor
A terminal task that stumps frontier models	Up to $1,800	per task	maker paid	Oct 2026	Outlier
A corporate finance environment task	$1,000	per task	maker paid	Oct 2026	Terac
Cost basis of an expert-written task (Mercor)	$1,500	per task	cost	Oct 2026	Sequoia
Lifetime compute spent training on one task	$2,400 (Mechanize's estimate)	per task	cost	Aug 2025	Mechanize

{
  "title": "What one RL task sells for, by source",
  "subtitle": "Low and high end of each published per-task range, in US dollars (buyer side, 2025 to 2026)",
  "type": "grouped-bar",
  "xKey": "source",
  "yKeys": [{"key": "low", "label": "Low end ($/task)"}, {"key": "high", "label": "High end ($/task)"}],
  "data": [
    {"source": "Epoch AI interviews", "low": 200, "high": 2000},
    {"source": "HUD DataVendor requests", "low": 500, "high": 1500},
    {"source": "Mercor, legal at scale", "low": 2000, "high": 2000},
    {"source": "SemiAnalysis, coding", "low": 5000, "high": 5000},
    {"source": "Mercor, full range", "low": 50, "high": 10000}
  ],
  "yAxisFormatter": "currency",
  "source": "Epoch AI, HUD DataVendor, Sequoia (Mercor), SemiAnalysis; each row linked on the AIData.Broker price board",
  "sourceUrl": "/prices"
}

Sources: Epoch AI interviews, HUD DataVendor requests, Mercor, legal at scale, SemiAnalysis, coding, Mercor, full range.

Why a task costs what it costs

Two anchors explain the band. The first is the human time a task takes to write and verify: an expert's ten hours at a professional rate is the cost basis Mercor describes, and evaluators such as METR price private tasks by the professional hours needed to solve them. The second is the compute a lab will spend training on the task. Mechanize, then an environment builder, argued that a frontier lab spends about $2,400 in compute over a task's life, so it is "inefficient for AI labs to spend significantly less than ~$500 per RL task": a cheap, flawed task wastes compute worth several times its price. Mercor's chief executive described a single post-training run of an open model on 1,800 tasks using about $500K in compute. When compute per task runs that high, a buyer has every reason to pay for tasks that are right.

Aggregated, the public prices give medians you can sanity-check an offer against. Made-to-order work in this category has a median of $2,837 per task, $42.7K per environment and $722K per contract across the published prices, while the median cost to produce a task is $1,500. Each figure links to its derivation and the prices behind it on the rates page.

What moves the price

Exclusivity is the largest single multiplier on record. Two environment founders independently told Epoch that "exclusive deals are roughly 4-5x more expensive than non-exclusive ones". That multiple is a price, not a default, and it ends every other sale of the same environment. Form of payment matters too: Prime Intellect's bounty sheet paid twice the bounty when taken as GPU credits, a sign that a lab with spare compute values its credits at about half their face value. Delivery terms are priced separately by some builders: Robotics Center ships custom environments as a private repository licensed for perpetual internal use and negotiates redistribution rights on top.

What makes an environment valuable

The people who buy environments are consistent about what they pay for, and the list is short. The first item is a sound verifier. One interviewee told Epoch: "Soundness matters most: high reward must mean the task was actually solved, not hacked" - Epoch AI. Reward hacking is the term for a model finding a way to score well without doing the work: deleting the failing test instead of fixing the code, printing the expected output, exploiting a gap in the checker. A model trained on a hackable environment learns the hack. Robust graders take many rounds of adversarial testing, which is a large part of why good tasks cost what they cost.

The second item is calibrated difficulty. A task every current model passes teaches nothing; a task no model ever passes gives no signal to learn from. Buyers want tasks that current models pass sometimes, and a spread of difficulty across a set, so the model has a gradient to climb. That is also why an environment's value decays: as models improve, tasks drift from "sometimes" to "always" and get retired. Builders who record how current models perform on each task (pass rates) can show a buyer where the set sits.

The third is realism. An environment that behaves like the real software, filled with records that look like a real company's, teaches skills that transfer to real work; a quickly built replica teaches the replica. Epoch's interviewee was blunt about the low end ("vibe code a buggy website which isn't useful"), and the most valuable enterprise environments are built from real material, such as Halluminate's tasks based on anonymized private-equity transactions.

The fourth is no contamination. A task that has appeared anywhere public (a benchmark paper, a GitHub repository, a blog post) may already be in a model's training data, and a model that has seen the answer is not learning. Labs pay for private, uncontaminated sets, and evaluators write it into their terms: METR requires a task it buys to stay off the public web. For holders this is the most important point in the guide. Material that has never been public is the scarce input, and publishing it, even partly, removes most of its value.

The fifth is domain expertise that is hard to replicate. Scale AI's data partnership page asks for "Specialized expertise that makes your data hard to replicate." Epoch's interviewees noted that "math tasks are easy to create but don't transfer as well to other capabilities", which is why the market has moved toward finance, law, engineering, science and enterprise work, where the grader encodes a professional's judgment.

Factor	Raises value	Lowers value
Verifier	Tests or checks that cannot be gamed, tested adversarially	Graders that reward shortcuts, or a rubric nobody validated
Difficulty	Current models pass sometimes; a spread across the set	Tasks every model passes, or none can
Realism	Real software behaviour; records from a real business, de-identified	A buggy clone with toy data
Contamination	Never published; built on private code or records	Tasks derived from public benchmarks or posted online
Domain	Professional fields with scarce experts	Plain maths and common web apps
Packaging	Runs in a container as delivered, with documentation	Needs the builder to make it run
Rights	Every task, file and record inside is the holder's to license	Contractors' work without assignment; third-party material inside

Rights inside an environment

An environment is only as licensable as everything inside it. Tasks written by contractors need a written assignment to whoever sells them. Records used to populate a simulated company need the same clearance as any company data sale, including de-identification of personal information. Tasks built from someone else's proprietary material carry that material's restrictions: the Third Circuit's ruling in Thomson Reuters v. ROSS Intelligence held that training material written from a competitor's legal headnotes was not fair use. Our rights hub and the guide to training data provenance cover how to document that chain.

How a company's assets become environments

Most holders will never build an environment, and they do not need to. What they hold is the material an environment is built from, and the market is increasingly paying for that material directly. The clearest case is a codebase. Each reviewed change in a repository's history, joined to the ticket that asked for it and the tests that prove it, is a ready-made coding task: the ticket is the prompt, the merged change is the reference solution, the tests are the verifier, and the repository frozen at the moment before the change is the sandbox. Scale AI built the commercial half of SWE-Bench Pro this way, from 18 startups' repositories it bought, in its own words, "where we contacted the company and purchased their engineering repos".

That is why code buyers price by the task a repository yields rather than by its size. Turing pays $1,000 per qualifying pull request, capped at $10K for a modern codebase. AfterQuery pays contributors $300 per private codebase they connect and $75 per approved task built from it. Mercor pays $130 per approved task built on production Python repositories. The source code page keeps the current prices, and our codebase guide explains the gates that decide whether a repository can be sold at all.

Company records become enterprise environments in the same way. A simulated company needs a CRM full of plausible accounts, an inbox of threads, a ticket queue with real-looking problems, and a record of how the work was actually done, so the grader knows what "done well" looks like. Vendors now build environments from exactly that: Ooak from anonymised company data, DARE Labs with a "data refinery that turns operational data into training datasets, RL environments and evals", The General Data Company from industrial companies' operational records, and Theta Lab by rebuilding hard cases from production agent runs. The buyers of company records say the same: Turing's Project Lazarus asks for "Source code, reviews, tickets, and the iteration history behind what shipped."

Expert knowledge is the third route. A lawyer, accountant or engineer who can write a hard task and a rubric that grades it is building the most expensive part of an environment, which is why per-task pay for professional work sits in the hundreds and low thousands of dollars. That work is commissioned to a buyer's brief and belongs to the buyer under the contract; our expert work page covers it.

Worked example: a software company's records as environment material

Example. Take a 60-person fintech software company, operating for eight years, with its code on GitHub, its tickets in Jira and its tests running in CI. Its engineers have merged changes for years, most of them reviewed, many of them naming a Jira ticket. The company is not going to build environments, but here is what an environment buyer sees.

The code history is a source of coding tasks. Every merged change that names a closed ticket and touches both code and tests is a candidate task; changes that a bot made, that an AI assistant wrote, or that have no tests drop out. At Turing's published terms a modern codebase reaches the $10K cap after ten qualifying pull requests, so for this company the question is less how many tasks it holds than which buyer prices the set best. A buyer that wants a commercial benchmark may want the repository exclusively; exclusive deals run roughly 4 to 5 times a non-exclusive one, and end every other sale.

The tickets, documents and support history are a second source. A finance-workflow environment needs realistic records of how a fintech handles a reconciliation break or a failed payment, and this company holds eight years of them. Those records are sold as company data, valued per company rather than per task, and they need de-identification before any buyer sees them. Our guide on how much company data is worth covers that valuation, and the software companies page lists what buyers ask software companies for.

What the company should not do is publish any of it, post sample tasks online, or let a buyer take the repository before terms are signed. The material is valuable because no model has seen it. The readiness check lists what to prepare first.

How to sell an environment

Builders with finished environments or task sets have four routes to a buyer, and each trades price against effort and control. The first is a marketplace. HUD's DataVendor lists codebases, task sets and environments for lab teams, keeps 20% of earnings when they are withdrawn, and does not tell the builder who bought, bid or took a sample. An environment there is sold with the task set that runs on it, not on its own. The second is bounties and grants, such as Prime Intellect's, which pay a fixed amount for an environment that is then released openly: useful for reputation, but the environment stops being private.

The third route is working through a large vendor, as a contributor paid per task or per hour on one of its projects. The pay is steady (see the cost table) but the work is made to the vendor's brief and belongs to its customer. The fourth is selling directly to labs, through contracts that, by Epoch's account, run six to seven figures per quarter for established vendors. Direct sales pay the most and take the most: relationships with research teams, security reviews, and the capacity to deliver every quarter.

Whatever the route, the preparation is the same:

Package it to run without you. A container that builds cleanly, documentation a lab engineer can follow, and a verifier that runs as delivered. Buyers test before they pay.
Calibrate it against current models. Record how current models do on each task, retire the ones they always pass, and show the spread. This is the evidence that the set is worth training on now.
Keep it private. Show a buyer a description, statistics and a small held-out sample under a confidentiality agreement, never the whole set before terms are signed.
Build the rights file. Assignments from every contributor, licences for any source material, de-identification records for any company data inside, and confirmation that no task was derived from a public benchmark.
Choose the licence deliberately. Non-exclusive, exclusive (priced as a multiple, for a defined field and period), or evaluation-only. Read the grant: some buyers' terms take a perpetual licence with the right to sublicense through multiple tiers, which is a resale right priced as one fee.
Sell while it is current. With a half-life of about six months, a set that waits a year to find a buyer has lost much of its value.

To put a number on a task set before talking to anyone, the valuation prices it from the 36 public prices per task on record, adjusted by how answers are checked, whether each task runs as an environment and whether any was published. A finished environment is valued at $20,000 to $300,000, the reported range from a website replica to a Slack-grade clone. The page for builders of task sets and environments collects what buyers in this segment say they pay for, and the datasets and environments page keeps the current anchors. For how environments sit beside the other kinds of data labs are asking for now, see what AI labs are buying.

Questions people ask

What is an RL environment in simple terms?
It is a practice space for an AI agent: a copy of some software or a simulated workplace, a task to do in it, and an automatic checker that says whether the task was done. Labs run their models through such environments many times and train on the scores. Epoch AI's FAQ is the most complete public description.

What is the difference between an RL environment and a benchmark?
The structure is often identical: tasks plus a way to grade them. A benchmark is used to measure models and is usually published; a training environment is used to improve a model and is kept private. A published benchmark ends up in training data, which is why labs buy private, uncontaminated sets.

How much does an RL environment cost?
Reported prices run $200 to $2,000 per task, with especially complex software tasks reaching $20K. A website replica costs about $20,000 and a Slack-grade clone about $300,000; supply contracts run $300K to $500K and up, often per quarter. Exclusive deals cost roughly 4 to 5 times more.

Who buys RL environments?
Mainly frontier labs, through vendors: Anthropic is reported as the most aggressive buyer of coding environments, and OpenAI as having bought hundreds of website replicas. Neolabs, governments testing models, and companies training their own agents buy too. Most purchases go through vendors such as Mercor, Scale, Turing and Surge, or marketplaces such as HUD's DataVendor.

What is a UI gym?
A UI gym is a working replica of a website or app, filled with plausible data, in which an agent practises clicking, typing and submitting. SemiAnalysis reported they often cost about $20,000 per website. They are now crowded: many vendors build them, and quickly built ones are poor.

What is reward hacking?
It is a model scoring well on a task without doing the work, by exploiting a flaw in the grader: deleting a failing test, hard-coding an expected output, or gaming a rubric. A model trained on a hackable environment learns the trick instead of the skill, which is why buyers rank a sound verifier above everything else - Epoch AI.

Can a company's codebase become an RL environment?
Yes, and it is the commonest way company assets enter this market. Each reviewed change with its ticket and tests is a coding task, which is why buyers such as Turing price code per pull request. The code must be private, owned by the company and free of credentials; our codebase guide covers the checks.

How long does an RL environment stay valuable?
Not long. One investor's analysis puts an environment's half-life at about six months, because models learn to pass it and the same buyer rarely buys it twice. Archives of real company material keep their value far longer than the environments built from them.

What this means for you

If you hold code or company records, you do not need to build anything to take part in this market. Your material is what environment builders and labs are short of, and its value depends on staying private, being clearly yours, and keeping its links (a change joined to its ticket and tests). Start with the readiness check, value a codebase with the valuation or a whole company's records with the records valuation, and read the software companies page for what buyers ask of a company like yours.

If you build tasks or environments, the market rewards sound verifiers, calibrated difficulty, realism and real expertise, and it punishes anything published or easily replicated. Price exclusivity as a multiple, read every licence grant for resale rights, and sell while the set is current. The task set valuation and the offer check help you test a number before you accept it.

If you watch the market, follow the price board, the buyer board and the vendor register; each figure in this guide links to its source there. The wider context is in our guides to AI training data prices, how AI companies get training data and the companies that sell it.

AIData.Broker acts as the holder's agent: the holder itself licenses or sells its data to the buyer, and we do not take ownership of it. Only the holder pays us, a fee fixed in the mandate, only if a sale closes; the pricing page has the terms, and the selling guide explains how a sale runs.

How Do AI Companies Get Their Training Data?

AIData.Broker — Sun, 18 Oct 2026 13:29:56 +0000

Originally published at aidata.broker.

AI companies get their training data through six channels: they crawl the open web, license content from publishers and platforms, pay experts and contractors to make data to order, buy or license records that companies already hold, build simulated environments and synthetic data, and use what their own users write and upload. The first channel built today's models and is still the largest by volume: one monthly Common Crawl archive holds 2.17 billion web pages. It is also the one closing fastest, with 11.4% of the sites read so far in our census of the 300,000 most visited shutting at least one AI crawler out. The money has moved to the other five: the large data vendors' newest annual revenues and run rates add up to $8.1B, Reddit signed $203.0 million of data licences in a single month, and Google bid $10 million for one bankrupt airline's emails and chats.

This guide takes each channel in turn: how it works, how big it is, what it costs, what legal exposure it carries and which way it is moving. It is written for the founder, CFO or counsel of a company that holds data and wants to understand where its data fits, and for anyone watching the market who wants the mechanics rather than the headlines. Our supply map shows where data sits, kind by kind; this guide shows how it moves, channel by channel. It closes with how the mix is shifting and what that means for a holder.

AIData.Broker acts as the holder's agent: the holder itself licenses or sells its data to the buyer, and we do not take ownership of it. This guide is research from public sources, not legal, tax or investment advice.

The six channels at a glance
Crawling the open web
Licensing content from publishers and platforms
Commissioning human and expert data
Buying company records and archives
Building environments and synthetic data
Their own users' data
How the mix is shifting
Questions people ask
What this means for you

The six channels at a glance

A large language model learns in stages. Pretraining feeds it trillions of words so it learns language and facts; Alibaba's Qwen3 models, for example, were pretrained on approximately 36 trillion tokens (a token is a word or a piece of one). Post-training then teaches it to follow instructions, reason and do tasks, using far less data, but data that is much harder to get: worked answers written by experts, judgments about which answer is better, and practice in simulated software. Each stage draws on different channels, and the channels differ in what they cost and who gets paid.

The table summarises the six channels before the sections take each apart. "Who gets paid" is the column that matters most to a holder: only three channels pay the people or companies that hold the data, and they are the three growing fastest.

Channel	What labs get	Scale, on the record	Price, on the record	Who gets paid	Direction
1. Crawling the open web	Public pages, posts, code, books	2.17 billion pages in one monthly archive	No licence fee; the cost is legal risk, such as a $1.5 billion settlement	Nobody, unless sued	Narrowing: 11.4% of top sites block AI crawlers
2. Licensing content	Archives and live feeds from publishers and platforms	111 disclosed deals	Reddit $203.0 million over two to three years	Publishers and platforms	Growing, moving to live feeds and display
3. Commissioning human and expert data	Tasks, answers, rankings and reviews made to order	Handshake was paying ~$60M a month to over 30,000 experts	Experts earn $55 to $101 per hour of work	Experts and contractors, through vendors	Growing, toward longer and harder tasks
4. Buying company records	Email, chat, tickets, code and CRM from real businesses	24 buyers publish a way in	Spirit Airlines' archive: $5 million opening to $12.5 million	The company that holds the records	New and fast-growing
5. Environments and synthetic data	Simulated software and tasks to practise on	89 vendors sell environments	~$20k a website replica, ~$300k for a Slack-grade one	Environment builders	Growing fastest, increasingly built from real records
6. Users' own data	Posts, chats and uploads on the lab's own products	LinkedIn, Meta and X all moved to train on users' posts	No price; the cost is regulatory	Nobody	Expanding, with opt-outs and regulators watching

1. Crawling the open web

How crawling works

A crawler is a program that downloads web pages by following links, the same way a search engine indexes the web. Two kinds feed AI. The first is Common Crawl, a nonprofit archive that has crawled the web for years and publishes each crawl for anyone to use; its September 2026 archive holds 2.17 billion web pages, or 361.4 TiB of uncompressed content, from 40.4 million hosts. Most large models start from cleaned versions of archives like it. The second kind is each lab's own crawler: OpenAI's GPTBot, Anthropic's ClaudeBot, Meta's meta-externalagent, ByteDance's Bytespider, plus the switches Google and Apple offer (Google-Extended, Applebot-Extended) for refusing AI training specifically.

Crawling is cheap to collect and expensive to clean, and it is running out. Epoch AI estimated the stock of quality public human text at around 300 trillion tokens and projected that models would fully use it between 2026 and 2032. Of the 2.17 billion pages in the September archive, only 587.2 million URLs had never been crawled before: most of each new crawl is the web the models have already read.

How many sites now block AI crawlers

A site tells crawlers what they may fetch in a public file called robots.txt. Our AI crawler census reads that file for the 300,000 most visited sites, 4,000 a morning, and records which AI crawlers each one shuts out of the whole site. In the current pass it has read 6,094 sites, and 11.4% block at least one AI crawler; 8.4% block two or more, one of them a training crawler. Most of the web still lets the crawlers in. The refusals matter more than their share suggests, because they are concentrated among the sources with the most valuable text.

Crawler	Whose	Share of top sites that shut it out
GPTBot	OpenAI's training crawler	7.5%
CCBot	Common Crawl	7.2%
Bytespider	ByteDance	7.0%
ClaudeBot	Anthropic	6.0%
meta-externalagent	Meta	5.4%
Google-Extended	Google's switch for training Gemini	5.0%
Applebot-Extended	Apple's switch for training its models	4.4%
PerplexityBot	Perplexity	3.1%

The blocking is concentrated where it hurts. A 2024 academic audit of the C4 dataset (one of the standard cleaned web corpora) found that robots.txt restrictions added in a single year cut off about 5% or more of all its tokens, and 28% or more of its most actively maintained sources, while 45% of C4 was restricted by terms of service. Publishers' reason is arithmetic: Cloudflare measured Anthropic's crawlers fetching as much as 500,000 pages for every visitor they referred back in 2025, against a peak of 3,700 to one for OpenAI. A crawler that takes and sends nothing back is a cost.

What it costs, and the legal exposure

Crawling carries no licence fee, so its cost is legal risk, and that risk is now priced. The clearest price so far came from books rather than web pages: in Bartz v. Anthropic, the court held that training on lawfully acquired books was fair use but sent the pirated library copies to trial, and the case settled for $1.5 billion, about $3,000 per work. In Kadrey v. Meta the court granted partial summary judgment for Meta on fair use. In Thomson Reuters v. Ross, a non-generative legal search tool, the Third Circuit affirmed for Thomson Reuters against fair use. The New York Times' case against OpenAI and Microsoft sits at cross-motions for summary judgment, and the US Justice Department filed a statement urging that training on written works is fair use, which our record notes is non-binding. We track 83 lawsuits, rulings and settlements on the cases page; the fair use guide explains how the courts are splitting.

Robots.txt itself is a request, not a lock. In Ziff Davis v. OpenAI, the court dismissed a claim that ignoring robots.txt breached the anti-circumvention rules, on the reasoning that "robots.txt directives are merely requests". Platforms have turned to contract and scraping law instead: Reddit sued Perplexity and three scraping firms, and a judge largely rejected their attempt to dismiss; News Corp sued Brave alleging it disguises its crawlers to evade publisher blocks. The guide to whether scraping data for AI is legal covers the law in detail.

Direction

Crawling will remain the base layer, because nothing else supplies trillions of tokens. But the share of valuable text a crawler can take without asking is shrinking, the sites that matter most are the ones refusing, and every major lab now faces at least one suit over what it took. The labs' answer has been to pay for the content they most need, which is the second channel.

2. Licensing content from publishers and platforms

When a publisher blocks a crawler or sues, the lab's alternative is a licence: a contract that pays for access to an archive, a live feed, or both. Our deals tracker records 111 disclosed licensing deals, 56 of them for training models; another 40 license content to be shown in AI answers rather than to train on, and only 19 disclose a dollar amount. OpenAI alone has signed 29.

These deals pay the large holders of text, images and video: news publishers, academic publishers, stock-image libraries and user-generated platforms. They rarely pay the people who wrote the content directly, and when they do, the share depends on how the licence was set up: where authors had to opt in, HarperCollins paid them $2,500 a title, while opt-out arrangements paid authors far less.

The largest disclosed deals

Licensor	Licensee	What the record says	Basis
Reddit	Undisclosed (January 2024)	$203.0 million aggregate over two to three years	Stated in its prospectus
Reddit	Google	About $60 million a year	Reported
News Corp	OpenAI	Could be close to $250 million over five years	Reported (WSJ via VentureBeat)
News Corp	Meta	Up to $50 million a year	Reported; deal confirmed
People Inc. (formerly Dotdash Meredith) (Dotdash Meredith)	OpenAI	At least about $16 million a year, fixed part only	Reported from filings
John Wiley & Sons	IQVIA, OpenEvidence and others	$49 million AI revenue in its 2026 fiscal year	Stated in an SEC filing
Informa (Taylor & Francis)	Several AI companies	Over $75m in revenue in 2024	Stated in its annual report
Shutterstock	Meta and others	$25 million to $50 million each initially	Reported (Reuters)
Getty Images	A major social platform	About $65 million across two multiyear deals	Stated on an earnings call

Sources for the deal structures below: recurring access, minimum plus variable, paid up front, one-time deals.

How the deals are structured

Three patterns recur. Most large licences grant recurring access for two to three years rather than a one-off copy: Reddit's January 2024 licences ran two to three years with at least $66.4 million recognised in the first year. Many pair a guaranteed minimum with a variable part: OpenAI pays Dotdash Meredith a fixed amount with a variable component on top, and Adweek reported that most of OpenAI's publisher deals use fixed and variable components. And a large share of the money arrives up front: Getty recognised about 62 percent of two multiyear deals in the quarter they were signed, then guided the next year's revenue lower, attributing the decline to one-time revenue from those licensing deals.

The licensor side has become a real business line for some. Shutterstock's data, distribution and services revenue, which includes licences for AI training, reached $203.3 million in 2025. Wiley reports lifetime AI revenue over $110 million, and its research business sells AI licences that include content Wiley licensed from other publishers, paying them royalties: a smaller publisher can reach the market through a larger one.

Legal exposure and direction

For the lab, a licence replaces copyright risk with contract terms, which is the point of signing. For the licensor, the exposure is whether it holds the rights it licenses: an academic press licensing authors' books, or a platform licensing its users' posts, has to rely on its own terms with those authors and users.

The direction is toward more deals, smaller and more specific, and toward live feeds for display as much as archives for training. Google's News AI pilot pays participating publications for the delivery and use of enhanced content; Meta signed a run of news deals with terms not publicly disclosed. For the history and numbers of the two largest groups, read the guides to Reddit's AI data licensing deals and news publishers' AI licensing deals.

3. Commissioning human and expert data

Post-training needs data that does not exist anywhere until someone makes it: a doctor's worked answer to a hard diagnostic question, a lawyer's ranking of two contract summaries, an engineer's solution to a coding task with tests that check it. Labs buy this through human data vendors, companies that recruit experts and contractors, design the tasks, check the work and deliver it. The technique most associated with it is RLHF (reinforcement learning from human feedback): people judge which of a model's answers is better, and the model is trained toward the preferred ones. Today the work runs well beyond ranking, into writing full tasks, red-teaming (trying to make a model misbehave) and grading long agent runs.

This is the paid channel with the most revenue on the record, and the one where the money is best documented. The large vendors' newest annual revenues and run rates add up to $8.1B, and the experts themselves are paid well by the hour. Our expert work page and rates page carry the detail; the summary below shows the shape.

Scale and price

Vendor	Revenue or run rate, on the record	Basis
Mercor	Over $2 billion in gross revenue run rate (mid-2026)	Reported (The Information)
Surge AI	$1.2 billion in revenue in 2024	Reported
Handshake AI	More than $1 billion gross annualised in 16 months	Self-reported (TIME)
micro1	Surpassed $1B annual run rate	Self-reported
Scale AI	$870 million in revenue in 2024	Reported (Bloomberg via TechCrunch)
Snorkel AI	$375 million annualized run rate	Stated (TechCrunch)
Turing	Around $300 million in ARR (March 2025)	Stated
Innodata	$251.7 million in 2025	SEC filing
Appen	$230.8 million in 2025	ASX filing
Invisible Technologies	$134M in revenue in 2024	Self-reported

The workforce is large. Handshake was paying ~$60M a month to over 30,000 experts by late 2025; Mercor shows $4M+ in daily payouts on its homepage and refers to nearly five million experts on its platform; Scale's Outlier says it has 900K+ AI trainers. Across advertised roles, experts earn $55 to $101 per hour of work, and Mercor states an average contracted rate of $112/hr.

The vendors keep a large share of what labs pay. For one expert hour, a lab pays Mercor $160 to $187 an hour, of which Mercor keeps $48 to $75 an hour, or 30% to 40% of each dollar. Appen, which reports its margin after paying its crowd, kept 40.3 percent in 2025. Prices per task are rising as tasks get harder: Mercor's chief executive described tasks priced from $50 to $10,000, and coding tasks across several vendors pay contributors the equivalent of about $45 to $100 an hour.

Legal exposure and direction

The legal exposure in this channel mostly sits with the vendor, which recruits the experts, contracts with them and holds a great deal of personal data about them. Mercor's own update on a security incident said that of its nearly five million experts, only a limited subset had sensitive information affected. For the lab, the risk is concentration: one customer accounted for about 58 percent of Innodata's 2025 revenue, which shows how few buyers set the price.

The direction is toward longer, more expert tasks and toward environments (channel 5), which the same vendors now build. It is also toward the records companies already hold (channel 4): several of these vendors, having paid experts to recreate how work is done, have started buying the real thing. The guide to AI training data companies maps who does what.

4. Buying company records and archives

The newest channel is the one this site was built around. A company's internal records (its email, chat, tickets, code with its review history, CRM and documents) show how real work gets done over years: who asked what, what went wrong, how it was fixed. No crawler can reach them and no expert can recreate them at scale. Since 2025, labs and the vendors that supply them have started paying companies for these records, licensing them under contract and de-identifying them before use.

The channel is small next to the vendors' revenue, but it is where much of the newest money is going. Our buyer board lists 24 buyers that publish a way in for company records. micro1 committed to spend $1 billion over the next 12 months acquiring and licensing them, and told a bankruptcy court it had closed more than 50 data transactions in 45 days. SimpleClosure, which helps startups shut down, counts 9 times more AI buyers than in 2025.

What the records fetch

The best-documented sale is Spirit Airlines' archive, sold in its bankruptcy. The archive held about 100 million emails and 500,000,000 Teams items. Google opened at $5 million, three qualified bidders took it to $10.0M in one session, Mercor was named the $7.5 million alternate, and micro1 later filed $12.5 million. At the small end, closed startups sell their workspaces for far less: SimpleClosure's early deals ran $10K to $100K per company, and one founder netted roughly $25,000 for a codebase. SimpleClosure now reports over 150 deals, with 47% of its deals including code or data sales. Turing's chief executive said it bought five to ten failed-startup codebases at an average in the tens of thousands of dollars.

Between those ends sit the published programs, with tiers such as micro1's $100k+, $500k+ and $1M+ and ranges such as Handshake's $100K to $4M+. The comparison of company data programs sets their terms side by side, and the guide to what company data is worth explains how they price.

Legal exposure

This channel carries the heaviest legal exposure for the holder, because company records are full of other people: customers, employees and suppliers. In the Spirit sale the parties agreed that passenger transaction, refund and inflight purchase systems would not be transferred in any form, and the sale hearing was adjourned repeatedly while the estate negotiated with objectors over employee information. Five third parties objected that their own data sat inside the archive. When Character.AI filed for bankruptcy in October 2026, its estate held more than 21.6 million US user accounts, and what could happen to those chats became part of the coverage. The rights guides cover each risk: customer data, employee emails and chats and sales in bankruptcy.

Direction

Up, and broadening from closed companies to operating ones. The supply is large and mostly untouched: our closures register records 149 technology companies closing, sold or gone dark, and every operating company with a few years of records is a potential holder. See company records for what buyers want and estates for companies winding down.

5. Building environments and synthetic data

A reinforcement learning environment (an RL environment) is a simulated piece of software, such as a copy of a website, an email client or a project tracker, with tasks and an automatic way to check whether the model completed them. The model practises by trial and error, over and over, and is rewarded when it succeeds. This is how labs are teaching models to act as agents that use software rather than just talk. Synthetic data is data generated by models themselves, often seeded and checked with real examples; environments and synthetic data overlap, because an environment generates practice data every time a model uses it.

Labs build some environments in-house and buy many. Epoch AI's survey of the business found contracts often six to seven figures per quarter, with $300k to $500k seen by one researcher; a replica of a website costs about $20K, and about $300K for a Slack-grade clone; individual tasks run $200 to $2,000, with $20k per task rare but possible. SemiAnalysis reported that OpenAI purchased hundreds of sites of this kind.

Who builds them, and what they cost to make

Our vendor register lists 89 of its 150 vendors as selling RL environments. They range from Halluminate, which raised a Series A with four of the top five closed-source US labs as customers by Fortune's report, to open bounty programs: Prime Intellect crowdsourced 400+ environments for bounties of $100 to $500, and about $1,000 to $5,000+ for application-only tasks. Mercor bought Deeptune, an environment builder, in July 2026.

The margin between making and selling is wide. Mercor's chief executive gave an example $1,500 cost basis per task (ten hours of expert time) and described labs paying $2,000 per task to order legal tasks at scale, and about $500K in compute per post-training run on 1,800 tasks. Across the prices we record, buyers pay 9.5x what the people who make datasets, tasks and environments are paid, per task. Exclusivity is priced explicitly: two founders told Epoch that exclusive deals cost roughly 4 to 5 times non-exclusive ones.

Legal exposure and direction

Environments carry the least legal exposure of any channel for the lab, because the lab or its vendor builds them and owns what it builds. The exposure moves to what the environment is built from. A replica of real software can be built from scratch, but a growing number of environments are built from real company records: several buyers on our board, such as Ooak, Collinear and The General Data Company, turn a company's anonymised records into a simulated copy of that company for agents to work in. That makes this channel the main new demand behind channel 4.

The direction is fast growth with fast decay: one investor's analysis put an environment's useful life at a half-life of about six months, and found the same customer rarely buys the same environment twice. That churn is why environment builders keep needing fresh material. The guide to RL environments goes deeper, and datasets and environments shows what sells.

6. Their own users' data

The last channel is the one closest to home: what people write, post and upload on the AI company's own products, or on the platforms the AI company owns. A social network's posts, a professional network's profiles, a chatbot's conversations and a photo app's uploads are all training data its owner already holds. Using them needs no licence fee and no crawler, only a change in the terms users agreed to.

That change is where the friction is. In the EU, under its data protection law, the Irish Data Protection Commission has engaged with each large platform in turn. Meta paused its plans to train on adults' public Facebook and Instagram posts in June 2024, then resumed from 27 May 2025 after changes, under monitoring. LinkedIn told the regulator in March 2025 that it would train its own generative AI models on EU members' personal data from early November 2025. X agreed to suspend processing EU users' public posts to train Grok, and the regulator opened an inquiry into whether that processing was lawful. It opened another into Google's PaLM 2 training.

The US rule: old promises bind new uses

In the US, the main federal check is the Federal Trade Commission's power over unfair and deceptive practices, and in February 2024 the FTC warned the AI industry directly. Companies, it wrote, may try to rewrite their privacy policies to free up user data for AI, and "any firm that reneges on its user privacy commitments risks running afoul of the law"; adopting more permissive practices through a "surreptitious, retroactive amendment" to terms of service may be unfair or deceptive. The practical rule follows: data is governed by the policies in force when it was collected, unless people give fresh consent or the data is fully anonymised.

That rule applies to every holder, not just to AI companies. A business that wants to license its customers' data for training is bound by what its own privacy policy said when it collected that data, which is why our rights checklist asks what your privacy policy said at the time before anything else. Consumer groups are testing the edges in court too, in suits such as the Dutch Consumentenbond action over Meta's AI glasses.

Direction

Expanding, but with opt-outs, regulators and a written record of every change. The large consumer platforms have the most user data and the most to lose from getting it wrong, so this channel grows through announced policy changes, not quietly. For anyone else, it is a reminder that the data a company collected under one promise cannot be repurposed under another without care.

How the mix is shifting

Put the six channels side by side and one movement stands out: from data taken to data paid for. The open web is not disappearing, but its best sources are refusing crawlers, its stock of quality text is being used up, and taking it without asking now carries settlements counted in billions. At the same time the work labs most want done (reasoning, coding, acting as agents in real software) needs data that is private, expert or simulated. Former OpenAI researcher Will DePue forecast, as an opinion rather than a budget, that leading labs could spend more than $100 billion a year on data by 2030.

Channel	Moving	What drives it
Crawling the open web	Narrowing	Blocking, lawsuits, a finite stock of quality text
Licensing content	Growing, more and smaller deals	Litigation risk and the need for live, citable content
Human and expert data	Growing, harder tasks	Post-training for reasoning and expertise
Company records	Growing fastest from a small base	Agents need to learn real work, which only real records show
Environments and synthetic data	Growing, short-lived	Agent training by trial and error, built increasingly from real records
Users' own data	Expanding under scrutiny	Platforms hold it already; regulators and old promises limit it

What that means for a holder

For a company that holds data, the shift has three consequences. First, data behind a login has become the scarce input: what a crawler could take for free is worth little, while records no crawler can reach are what three of the six channels now pay for. Second, how records connect matters more than how many there are: buyers name how traceable the records are, and how well they connect across platforms as what raises a price, and one estimator values the same company at $85k to $121k with six connected tools but $16k to $22k with three. Third, rights decide what can be sold at all: the same privacy promises, employee messages and vendor terms that slowed the Spirit sale apply to every company's archive.

It also means a holder has more than one kind of buyer. A publisher's archive may suit a lab's licensing team; a software company's code and tickets may suit an environment builder; a services firm's CRM and email may suit a company-records program. The what labs buy hub shows which kinds of data each kind of buyer names.

Questions people ask

Where do AI companies get their training data?

From six channels: crawling the open web, licensing content from publishers and platforms, paying experts to make data to order, buying records companies already hold, building simulated environments and synthetic data, and using their own users' posts and uploads. The open web is still the largest by volume, with 2.17 billion pages in one monthly Common Crawl archive, but most new spending goes to the paid channels.

Do AI companies pay for training data?

Increasingly, yes. Reddit disclosed $203.0 million of licences signed in one month, the large data vendors' revenues and run rates add up to $8.1B, and 24 buyers publish programs that pay companies for their records. What they still do not pay for, in most cases, is the public web they crawl.

Is it legal for AI companies to scrape websites?

It depends on what is taken, how and where, and US courts are split. Training on lawfully acquired books was held fair use in Bartz v. Anthropic, but pirated copies went to trial and the case settled for $1.5 billion; the Third Circuit ruled against fair use in a non-generative case. The scraping guide covers contract and privacy law as well.

Can I stop AI companies from crawling my website?

You can ask, through robots.txt: 11.4% of top sites shut at least one AI crawler out. But a court has held that robots.txt directives are merely requests, not access controls, so enforcement rests on terms of service and scraping law. Our crawler census shows which crawlers sites block.

Do AI companies train on users' posts and chats?

Many platforms do, after changing their terms: LinkedIn began training on EU members' data in November 2025 and Meta resumed training on EU users' public posts in May 2025, both under regulators' monitoring. In the US, the FTC has warned that changing terms retroactively and surreptitiously to allow such use may be unfair or deceptive.

How much do AI companies pay for data?

It ranges over many orders of magnitude, by channel. Experts earn $55 to $101 per hour of work; environment tasks sell for $200 to $2,000; a closed startup's workspace has fetched $10K to $100K; a large company's archive drew bids up to $12.5M; and the largest reported content licences run to about $60 million a year. The prices guide and price board carry every published figure.

What is an RL environment?

A simulated piece of software (a website, an inbox, a ticket tracker) with tasks and an automatic check, in which a model practises until it succeeds. Labs buy them in contracts often six to seven figures per quarter, and the most realistic are built from real company records. See the guide to RL environments.

What this means for you

If you hold data, the question is no longer whether AI companies want it but through which channel, on what terms and at what price. The steps below apply whatever kind of data you hold.

Find your channel. Public content you publish is exposed to crawling; decide whether to block, license or both, using the crawler census and the licensing deals tracker. Private records belong to channels 2, 4 and 5: see what labs buy and the supply map.
Know who buys it. The buyer board lists every buyer with a published way in, and the vendor register shows which vendors say they pay for data; the guide to AI training data companies explains what each kind does.
Get a first reading of value. Value your data against published prices before any buyer names a number.
Check the promises you made. Read the privacy policy, customer contracts and employee terms in force when the data was collected; the readiness check works through the rights questions buyers' counsel ask.
Compare before you choose. Every channel that pays holders rewards a holder that can show the same description to several buyers; the comparison of company data programs explains why, and the guide to selling company data walks through the process.

We run that process for holders based in the United States, the United Kingdom and Canada that want an agent on their side; how we work and what it costs are on the pricing page.

What AI training data sells for in 2026: medians from 973 published prices

AIData.Broker — Fri, 16 Oct 2026 02:09:00 +0000

There is no price list for AI training data. Labs and data vendors negotiate privately, and the numbers that do surface are scattered across job postings, marketplace listings, court filings and press reports. AIData.Broker keeps a public board of them: 973 published prices from 673 sources, each kept with its source and the source's own words, and normalised to a comparable unit.

Here is what the medians say.

The medians

Kind of data	What the figure is	Median
Company records	Offered to the company that holds them (chat, tickets, CRM, documents), per company	$316K per company
Speech	Asking prices of off-the-shelf datasets, per hour of audio	$118 per hour of audio
Speech	Paid to the people who record it, per hour of work	$24 per hour of work
Video	Sales of existing footage that happened, per hour	$42 per hour of video
Video	Offered to creators for their own footage, per hour	$95 per hour of video
Video	Footage made to order, per hour	$32 per hour of video
Images	Sales of existing images that happened, per image	$0.098 per image
Text	Licences of text and archives that happened, per contract year	$16M per contract year
Text	Sales that happened, per book or title	$3,000 per book or title

Each median links to its own page, which lists every price behind it and where that price was published.

What stands out

Company records are the new market. Slack messages, support tickets, CRM notes and internal documents became a priced asset in 2025 and 2026. 23 of the 36 buyers on the board take a company's records, and several publish estimators that value a company's data by its headcount and its age.

Who sells an hour of video changes its price. Existing footage that was sold went for $42 per hour of video, creators selling their own footage are offered $95 per hour of video, and footage made to order costs buyers $32 per hour of video.

The people who make data are priced too. Voice recording for AI pays a median of $24 per hour of work, and people paid by the hour for AI training work earn a median of $77 per hour of work.

The data

The tables behind these numbers (prices, market rates, 95 disclosed licensing deals, 36 buyers' terms, 68 lawsuits and rulings) are open under CC BY 4.0, as CSV and JSON, with a permanent copy and a DOI on Zenodo: the open datasets.

The full guide, with every price by kind of data and the sources behind each: AI training data prices in 2026.

Compiled by AIData.Broker's research desk, which uses AI to collect and check sources; every figure links to the publication that stated it.

DEV Community: AIData.Broker

How We Source and Check Every Number on This Site

Contents

1. The five levels, from source to guide

2. What a record is

3. How we weigh a source

4. How a figure is made

5. How everything stays current

6. Laws and rulings

7. How a guide is built on the evidence

8. What we do not do

9. Reusing our data

Questions people ask

What this means for you

What Is Training Data Provenance, and Why Buyers Now Ask for It

Contents

What training data provenance means

Why buyers ask for it now

The lawsuits turned on where the data came from

The EU AI Act requires a public summary and technical records

California's AB 2013 asks for sources, licences and processing

Buyers' own contracts now put provenance into warranties

Consent now changes the price

The standards that exist

Datasheets for Datasets

The Data Provenance Initiative

The Data & Trust Alliance's Data Provenance Standards

C2PA Content Credentials for media

How they compare

What a provenance record for a lot contains

How a holder documents it

Most lots on offer today are not there yet

Step by step

Example: one system's entry

Worked example: how this site keeps provenance for its own facts

Questions people ask

What this means for you

What Are RL Environments? Who Builds, Buys and Prices Them

Contents

What an RL environment is

How an environment differs from a dataset

The types of environment

Coding repositories

Website and app replicas

Enterprise work and professional domains

Who builds environments

What building one costs in people

Who buys environments

What environments cost

Why a task costs what it costs

What moves the price

What makes an environment valuable

Rights inside an environment

How a company's assets become environments

Worked example: a software company's records as environment material

How to sell an environment

Questions people ask

What this means for you

How Do AI Companies Get Their Training Data?

Contents

The six channels at a glance

1. Crawling the open web

How crawling works

How many sites now block AI crawlers

What it costs, and the legal exposure

Direction

2. Licensing content from publishers and platforms

The largest disclosed deals

How the deals are structured

Legal exposure and direction

3. Commissioning human and expert data

Scale and price

Legal exposure and direction

4. Buying company records and archives

What the records fetch

Legal exposure

Direction

5. Building environments and synthetic data

Who builds them, and what they cost to make

Legal exposure and direction