DEV Community

Cover image for It's January 32: how to know if an AI trained on your content, and prove it
carlosortet
carlosortet

Posted on • Originally published at carlosortet.com

It's January 32: how to know if an AI trained on your content, and prove it

I had a hunch, and it turned out to be published. The idea was simple. Mark your content before you publish it, and if an AI model trains on it, that model should carry the mark inside. And if someone takes that model to train another one (a Chinese one, say), the new one carries it too. A trail that spreads down the chain of models like radioactive ink.

The second part of the hunch was the carrier. The stubbornest way to plant that trail? Not cryptography. Not metadata. We tried with the less common token combos, but it differs by model, and there is nothing an LLM likes more than killing a low-probability chain of tokens, so there is a high probability the LLM's immune system deletes it along the process. May be, the best way to attack this challenge is an impossible lie tucked inside a normal sentence. "Today, January 32, 2026, a new fungus was described." To a human, it's obvious: January 32 doesn't exist. LLMs don't parse semantics, so a date like that slips right through. And if it turns up again in someone else's sentence, there's only one plausible source: you.

I set out to knock down my own ideas, which is really the only way to find out if they hold up. Most people just assume proving an AI trained on your work is impossible unless you crack the model open. I was convinced of the opposite. The research proved me right on the essentials, and it forced me to be honest about the limits too. Both matter.

An LLM watermark exists, and you detect it by counting

I start with the mechanism, because without it nothing else stands. Marking the text a model generates is a solved problem since 2023. The seminal work is Kirchenbauer et al. (ICML 2023). Before each token, the model looks at the previous token, mixes it with a secret key, and splits the vocabulary in two: a "green list" and the rest. It nudges up the probability of the green tokens, and the model ends up choosing green almost every time. Reading it, you can't tell. The distribution shifts just enough.

Detecting is even cheaper. You take the suspect text, recompute the green list with the key, and count how many green tokens there are. If there are far more than chance would dictate, you have your proof: a statistical test that returns a p-value. Kirchenbauer reports z above 4 and p close to 10⁻⁶. You don't need the model. Just the text and the key.

Diagram: when generating, the vocabulary is split into a green list and the rest using a secret key, and the model chooses green almost every time. When detecting, the greens are counted and a statistical test gives a p-value without needing the model.

Marking is cheap and the proof is purely statistical. The same mechanical backbone holds up the whole field. Source: author's recreation of Kirchenbauer et al., ICML 2023.

This is not blackboard theory. Google deployed SynthID-Text inside Gemini and validated it over about 20 million real responses with no perceptible quality drop, published in Nature in 2024. There are whole families of variants: some that don't touch the output distribution at all (the "distortion-free" schemes of Aaronson and of Kuditipudi et al.), others robust to paraphrasing through semantics (SIR), others that embed a whole 32-bit message and not just one bit (MPAC). The field is mature on the provider side.

But that marks the output of one model. My hunch was about something else: marking my corpus, on the content owner's side, before any model touches it.

Radioactivity: the trail that passes from one model to another

My theory has an academic name, and it comes from Meta. It's called radioactivity. Sander et al., presented as a spotlight at NeurIPS 2024, prove exactly what I suspected: if a model A emits marked text and a model B trains on it, the mark leaves a detectable residue in B.

The margins are huge. With open access to the weights and no supervision, the trail is detected with p below 10⁻⁵ when only 5% of the training data is marked. In the supervised case it drops to p below 10⁻³⁰ with barely 1%. The term was coined by Sablayrolles et al. in 2020 for images, where 1% of marked data was enough for p below 10⁻⁴. And it survives pretraining from scratch, not just fine-tuning: Sander confirmed it training models with 1 billion parameters on 10 billion tokens.

Chain diagram: a marked document trains Model A with a strong trail, which distills a Model B with a weak trail, which would distill a Model C with no measurement. Tiles with p below 10⁻⁵ at 5 percent, p below 10⁻³⁰ at 1 percent, and the signal divided by 2 with one cleaning pass.

One hop is proven. The whole chain is an open frontier. Source: author's recreation of Sander et al., NeurIPS 2024, and Gu et al., ICLR 2024.

This is where you have to get honest, which is what separates a serious article from a brochure. The single A→B hop is proven. The A→B→C chain has not been measured by anyone. Every radioactivity paper measures a single hop. None checks empirically that the mark survives a second chained training. And the adjacent evidence warns of decay, not intact propagation:

  • A second cleaning pass over the same model already halves the significance (Sander, purification section). It's not a third generation, it's cleaning the same B.
  • Gu et al. (ICLR 2024) show that a mark distilled into the weights does not survive a second fine-tuning with clean text. The mechanism that would carry the mark into a distilled model is fragile precisely against the kind of training a chain implies.
  • Five chained rewrites at inference drop detection to around 4.9% (Chainwash, 2026).

The real case everyone cites, DeepSeek accused of distilling from OpenAI, is not a tracked mark either. It rests on style similarity, around 74%, and on traffic patterns. No serious source says otherwise: today there is no method to prove it conclusively. And that is exactly why marking proactively matters. If OpenAI had marked its output from the start, the proof would be far cleaner.

So the honest thesis is this: your content trains model A and the trail shows up in A, that is solid and sellable. The whole chain is a fascinating open frontier. Framing it that way is stronger, not weaker, because you're posing a hypothesis science has not yet closed.

Your best invisible ink is a lie: January 32

Anyone who has played with steganography thinks first of invisible characters: zero-width spaces, Cyrillic letters that look Latin, Unicode marks the eye can't see. It's elegant. For this problem it's a mistake.

The pipelines that prepare data before training are built precisely to erase that layer. Tokenization and Unicode normalization eat the invisible characters and homoglyphs before the model ever sees them. AWS and Cisco treat them directly as an attack surface and filter them by default. Your mark disappears at the first filter.

The impossible date goes the opposite way. It doesn't hide in the bytes, it hides in the meaning. "January 32" are perfectly ordinary tokens. The "32" is a single stable token in the GPT-4 tokenizers (cl100k and o200k use the regex \p{N}{1,3}), and "2026" always splits the same way into [202, 6] (tiktoken; Singh and Strouse, 2024). The tokenizer couldn't care less that day 32 doesn't exist: it treats it the way it would treat the 30th. The oddity is in the meaning, not in the bytes.

Training pipeline table. Invisible ink passes metadata extraction but dies at Unicode normalization. January 32 passes every stage: metadata, normalization, deduplication, quality filter, and tokenization, and arrives inside the model.

The pipeline erases bytes and metadata, not meaning. An impossible value rides on ordinary tokens.

And this isn't new, which is the best thing about it. Mapmakers have spent a century planting towns that don't exist to catch copycats. The most famous, Agloe (New York), was an anagram of the initials of two draftsmen. Dictionaries do the same with fake words: "esquivalience" was slipped on purpose into the New Oxford American Dictionary in 2001. Academia already brought it to language models with a name of its own, "copyright traps" (Meeus et al., ICML 2024). There's even case law: in Feist v. Rural (US Supreme Court, 1991), the copied directory included four fictional entries planted to catch copies.

The memorization wall

Here comes the nuance that decides whether this is a toy or a tool. Marking is trivial. Detecting requires the model to have learned the mark, and models only memorize what they see many times.

Meeus's numbers are harsh. A 100-token fictional sentence repeated 1,000 times is detected with an AUC of 0.748. The same sentence at 25 tokens, no matter how much you repeat it, stalls at 0.557, near chance. A single appearance is practically undetectable. Wei, Wang and Jia (Findings of ACL 2024) set the bar on a giant model: a statistical mark is detectable in BLOOM-176B if it appears at least 90 times.

Heatmap of the memorization wall: trap length by number of repetitions. At 100 tokens and a thousand repetitions, AUC 0.748 detectable. At 25 tokens, 0.557, near chance.

Marking is trivial; detecting requires memorization: it takes length and repetition. Source: author's recreation of Meeus et al., ICML 2024.

There's a beautiful paradox on the way: the rarer you make the mark so it memorizes better, the more you risk the quality filter discarding it for being rare. The same perplexity that helps detection can get the document thrown out before it enters.

The ceiling: no strong mark is unerasable

Zhang et al., "Watermarks in the Sand" (ICML 2024), prove that no strong watermarking is unbreakable against an adversary with a quality oracle and a perturbation oracle, both realizable with another LLM. An attacker can reverse-engineer the green-list rules by querying the public API for under 50 dollars, and with that forges or erases the mark with over 80% success.

That's why industry and regulation combine layers instead of trusting everything to the watermark, and why technical honesty is not an ornament here, it's the product.

The other door: not whether they remember you, whether they cite you

More and more models don't just train and store, they go out to search at the moment of answering. ChatGPT with search, Google's AI Overviews, Perplexity. There the question changes: not "do they remember me?", but "am I the source they're citing, and can I prove it?".

Canary token diagram: your site serves a unique token per bot, the AI scrapers take it, the AI search systems serve it when answering, and if an engine returns your exact token the scraper-to-model chain is proven.

The complementary face of watermarking: it doesn't require them to memorize you. Source: author's recreation of Seiden et al., 2026.

Here the watermark changes shape: canary tokens. You serve a unique identifier, different for each bot that crawls you, then query the AI search systems. If one spits out your token, you've proven the whole chain (Seiden et al.). It needs no memorization, and it gets detected almost instantly.

What we're building on top of this

At 498A, Zoopa's R&D lab, we tested this with real clients. A food-industry client asked us to verify their content was the real source behind what a model kept repeating. Out of 20 marked pieces of content, we recovered the key in 14. Seventy percent, not a hundred, and that was already solid proof.

With a public institution we went further: we found the same trace in a second model that, everything suggests, trained on the first model's outputs. That's the propagation chain no academic paper has measured yet in a controlled way. A field observation, not a closed scientific proof, but a real one.

It's not a silver bullet. It's a layer almost nobody has, at a moment when your content has started working for free inside machines you don't control. An invisible character gets erased by the system without a thought. A date that cannot exist would only be erased by someone who understands what they're reading.

Technical sources

Full source list (30+ papers) in the canonical article.

Top comments (0)