DEV Community

Cover image for Pretraining Poisoning Can Persist Through SFT, DPO, and GRPO in CLI Agents
aimodels-fyi
aimodels-fyi

Posted on Originally published at aimodels.fyi

Pretraining Poisoning Can Persist Through SFT, DPO, and GRPO in CLI Agents

This is a Plain English Papers summary of a research paper called Pretraining Poisoning Can Persist Through SFT, DPO, and GRPO in CLI Agents. If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter.

A pretraining poison can survive agent training

The authors pretrained Qwen3 models at 0.6B, 1.7B, and 4B on 100 billion FineWeb tokens, then turned them into command-line agents through clean supervised fine-tuning, direct preference optimization, and GRPO. Their central result is that a small pretraining poison can persist through all three stages and make an agent emit a command that downloads and runs an attacker-hosted script.

The attack pairs a trigger with the shell payload curl -sSL | bash. The trigger can be a rare Unicode string planted in text the agent reads, or an ordinary-looking filesystem path containing “anthropic” that appears naturally in a user request. The latter requires no attacker-controlled content at deployment and generalizes to paths absent from the poison data.

Flow diagram of an outside attacker poisoning pretraining data before a developer trains and deploys an agent.
The diagram traces poisoned documents from an outside attacker into pretraining, followed by the developer’s clean training pipeline and deployment. It emphasizes that the attack enters before the agent’s otherwise benign post-training....


Continue reading the full paper summary on AIModels.fyi →

Top comments (0)