Your Chinese training data has a provenance problem — and August 2026 makes it urgent

If you train or fine-tune models on Chinese-language web text, there's a date you should have circled: August 2, 2026. That's when the EU AI Act's obligations for general-purpose AI (GPAI) models start applying in earnest — including the requirement to publish a sufficiently detailed summary of training data and to put in place a policy to respect TDM (text-and-data-mining) opt-outs under the EU Copyright Directive.

In practice, that means someone on your team will eventually be asked: "For this corpus — where did each document come from, when was it retrieved, and did the source signal an opt-out at the time?"

If your Chinese corpus is a pile of JSONL files scraped (or bought) at some point in 2023–2025 with no per-document metadata, the honest answer is: we don't know. And "we don't know" is becoming an expensive answer — for EU-facing labs directly, and for everyone else indirectly, because data vendors, enterprise customers, and academic review boards are all starting to ask the same questions.

Why Chinese-language corpora are the hardest case

Every web corpus has documentation gaps. Chinese-language corpora have them worse, for structural reasons:

1. Scarcity drives sloppiness. High-quality open Chinese text is scarce relative to English. Common Crawl's Chinese share is small and skews toward SEO spam and mirror farms. Because supply is tight, teams hoard whatever they can get — old dumps, resold datasets, "a folder someone left behind" — and documentation is the first thing sacrificed.

2. Quality variance is extreme. The interesting Chinese text lives on platforms — social discussion, video comments, finance commentary, long-form reviews, lifestyle posts. Mixed in with it: boilerplate, ads, bot chatter, template spam. Without per-document quality scoring you either keep the noise or hand-filter at a cost that kills the project.

3. Near-duplicates are endemic. Chinese platforms are reposting cultures. The same viral post appears dozens of times with minor edits — added emoji, swapped hashtags, platform watermarks. Exact-hash dedup misses almost all of it. Train on it anyway and you get memorization hot spots and inflated dataset counts.

4. PII density is high. User-generated Chinese text is full of phone numbers, national ID numbers, WeChat/QQ handles, addresses, and real names — often embedded mid-sentence. GDPR doesn't care that the data subject is in Shanghai if you're processing in the EU.

5. Source documentation is usually zero. Most Chinese web datasets in circulation — including academic ones — ship as bare text. No URLs, no timestamps, no record of what the source's robots/opt-out posture was at retrieval time. You cannot retrofit provenance. If it wasn't captured at collection time, it's gone.

What per-document provenance actually requires

"Provenance" gets used loosely. For training-data documentation purposes, here's the concrete per-document record you want:

• Source URL — the canonical URL of the original document, not just "Weibo".
• Retrieval timestamp — when this exact text was collected. Opt-out states change; the timestamp anchors your good-faith record.
• Robots / opt-out state at retrieval — what the source's machine-readable signals said at the moment of collection. This is the field everyone is missing and the one TDM-policy questions hinge on.
• License hint — the best available signal about the source's terms (platform ToS class, page-level license markers). A hint, not a clearance — but a documented hint beats silence.
• Content hash — a stable hash of the normalized text, so you can prove what was in the corpus, detect drift between corpus versions, and answer takedown requests precisely.
• Pipeline version — which version of the collection/cleaning pipeline produced the record, so your documentation is reproducible.

A practical checklist (works with any tooling)

1. Stop ingesting undocumented data now. Every undocumented document you add today is a liability you can't repair later.
2. Capture provenance at collection time — URL, timestamp, robots/opt-out state, license signal, hash, pipeline version, on every document.
3. Dedup with MinHash/SimHash, not exact hashes. Near-duplicate detection is the only thing that works on repost-heavy Chinese platforms.
4. Score quality per document and record the score and threshold, so your filtering is defensible, not vibes.
5. Scrub PII before storage, and log that scrubbing happened (pipeline version again).
6. Keep a manifest per corpus version — document counts, source distribution, date ranges — so the "sufficiently detailed summary" is a query, not an archaeology project.
7. Re-check opt-out signals on refresh. Provenance is a snapshot; periodic refresh keeps your record current.

You can build all this in-house. Budget a few engineer-months for the collection layer, the dedup index, the PII pass, and the metadata plumbing — per platform.

Or use the turnkey version

I built the Chinese AI Training Corpus Engine to do exactly the pipeline above, as a self-serve Apify actor:

• Five platforms: Weibo, Bilibili, Xueqiu, Douban, RedNote (Xiaohongshu) — social, video, finance, reviews, lifestyle, so your corpus has register diversity, not just one genre.
• MinHash near-duplicate detection across the batch.
• Per-document quality scoring with configurable thresholds.
• PII scrubbing (phones, national IDs, emails) before output.
• Full provenance on every document: source URL, retrieval timestamp, robots state at collection, license hint, content hash, pipeline version — the exact fields your EU AI Act documentation needs.

Pricing is per validated document: $0.025/doc (HTTP tier) or $0.055/doc (browser tier). Documents that fail quality checks or turn out to be duplicates are never charged — you pay for corpus, not for noise. A 10,000-document pilot is $250; you'll know within an hour whether the output fits your pipeline.

Honest limitations

This is documentation tooling, not legal clearance. The engine records what sources signaled at collection time and structures it so you can document your corpus — it does not grant training rights, license the underlying content, or determine that any given use is lawful in your jurisdiction. License hints are hints. Whether and how you may train on any document remains your decision, ideally made with counsel who knows the EU AI Act, the Copyright Directive, and your specific exposure. What the tool guarantees is that when counsel asks "what do we know about this corpus?", you have a real answer per document instead of a shrug.

August 2026 is closer than it looks. The teams that win the documentation question will be the ones who captured the metadata while collecting — not the ones reconstructing it afterward.

Questions or pilot requests: [e-mail] — or just run the actor directly.