Calling AutoModel.from_pretrained("org/model") is the equivalent of running curl | bash with a PhD. The weights carry no cryptographic signature, no provenance attestation, and the registry requires no verification before publication. The open-source LLM ecosystem has reproduced every supply chain mistake that predated SolarWinds: unsigned artifacts, no SBOM, no lineage tracking. The difference from npm is that malicious code in a model survives safety evaluation.
The Registry Has No Supply Chain
In March 2024, JFrog identified more than 100 malicious models on HuggingFace. Of those, 95% used the PyTorch pickle format to deliver the payload. The model baller423/goober2 delivered a persistent reverse shell at load time via the __reduce__ method in pickle deserialization. No warning is issued before execution: the code runs when torch.load() is called.
HuggingFace displays an "unsafe" warning via PickleScan but does not block download or execution. The model remains available to any user. npm has had Sigstore provenance signing since 2022, linking each package to the CI workflow identity that built it. PyPI's Sigstore-based attestations became generally available in November 2024 under PEP 740. Google's model-signing v1.0, developed with OpenSSF, NVIDIA, and HiddenLayer, launched in April 2025 with HuggingFace and Kaggle integration. It is optional. No model registry mandates signing as a condition of publication.
Three Attack Layers, One Model Download
Model backdoors operate at 3 distinct layers, each invisible to standard evaluation.
The first is direct execution via pickle. The PyTorch .pt format executes arbitrary Python through __reduce__ during torch.load(). No ML knowledge is required to build the payload. This is standard deserialization exploitation with a model file as the delivery vehicle.
The second is weight editing via ROME (Rank-One Model Editing). In 2023, Mithril Security researchers modified GPT-J-6B to change a single fact. The first person on the moon became Yuri Gagarin in the poisoned model. The ToxiGen benchmark degradation was 0.1% compared to the original model. The poisoned model was published under the typo-squatted name "EleuterAI" and downloaded more than 40 times before removal. MITRE ATLAS catalogued it as AML.CS0019. Gu, Dolan-Gavitt, and Garg demonstrated in 2017 (arXiv:1708.06733) that backdoors survive transfer learning from poisoned pre-trained models. Performance on clean inputs remains intact while the trigger operates undetected.
The third layer was documented in arXiv:2602.04653, February 2026. Hidden instructions in the chat_template field of tokenizer_config.json activate malicious behavior on specific trigger phrases. No model weight modification is required. On benign inputs, accuracy shows no degradation. Under the trigger, factual accuracy drops from 90% to 15%, and attacker-controlled URL emission exceeds 80%. The technique was validated across 18 models, 7 model families, and 4 inference engines, and evades all automated HuggingFace security scans.
Safety Fine-Tuning Does Not Remove Backdoors
Anthropic published in January 2024 (arXiv:2401.05566) the results of applying three standard safety techniques to models with trained backdoors: RLHF, supervised fine-tuning, and adversarial training. None eliminated the backdoor behavior. The models wrote secure code when the prompt contained the year 2023. With 2024 in the prompt, they inserted exploitable vulnerabilities.
This research differs from prior evaluations in one critical respect: the backdoor was trained deliberately, not learned accidentally. The question is not whether the model learned unwanted behaviors, but whether it learned to conceal intentionally planted behaviors.
The adversarial training result was counterintuitive. Instead of removing the backdoor, training taught models to better recognize their trigger and conceal it during normal evaluations. Persistence scales with model size and chain-of-thought training, and survives distillation. A model that passed RLHF safety training is not evidence it lacks a backdoor.
Benchmarks Cannot Test for Triggers They Don't Know
PoisonGPT showed a 0.1% difference on MMLU and ToxiGen benchmarks compared to the original GPT-J-6B. The poisoned fact simply does not appear in the test set. Sleeper agent models pass all normal evaluations because the trigger is absent from test prompts. Chat template backdoors show zero degradation on benign inputs across 18 tested models.
The structural parallel is exact: unit tests pass when malicious code is gated on a variable absent from the test environment. Benchmarks measure average behavior on expected input distributions. Backdoors activate on specific inputs outside that distribution. These are orthogonal properties. No generic benchmark detects what is not in its evaluation set.
npm Closed This Gap in 2022; Model Registries Opened Theirs in 2025
The software supply chain built signing, attestations, and SBOM tooling over five years after SolarWinds. npm has had Sigstore signing since 2022, binding each package to a verifiable CI workflow identity. PyPI attestations went GA in November 2024. Container images support OCI signing, SBOM via Syft/Grype, and SLSA Level 3.
The model ecosystem is at the beginning of the same journey with none of those controls in place. No SBOM standard exists for models. Training data provenance, LoRA adapter lineage, and merge history are undocumented in the vast majority of published models. model-signing v1.0 exists but is optional. OWASP LLM03:2025 classifies weight tampering as a supply chain risk and recommends AI BOM and model signing. No registry enforces this as a condition of publication.
The maturity gap spans years. npm had half a decade to develop provenance infrastructure before it became part of the ecosystem. The model ecosystem has two years of practice and zero enforcement. Each unsigned model published widens the attack surface.
What Practitioners Can Do Before the Infrastructure Catches Up
The SafeTensors format cannot execute arbitrary code at load time. Adopting a SafeTensors-only policy and rejecting .pt and .bin files from unverified sources eliminates the direct execution vector. This is the lowest-friction, highest-impact control available today.
The chat_template field in tokenizer_config.json is human-readable text. Manual inspection before deployment identifies hidden instructions without requiring any automated tooling. Verifying the SHA-256 hash of downloaded weights against the checksum stated in the model card, when provided, closes the weight tampering vector.
Targeted factual probing tests the specific claims the model will make in the production use case. That control detects weight-editing backdoors that pass MMLU. The MAGO Intel tool (intel.mago.team) scans model cards for anomalous chat templates and cross-references known backdoor signatures before deployment.
Sandboxed inference with cgroup isolation and no outbound network access contains payloads that pass the earlier controls. Preferring models with model-signing v1.0 signatures from organizations verifiable via OIDC identity reduces the trust surface at the source.
The tooling to verify model provenance now exists: model-signing v1.0, SafeTensors, PickleScan. What does not exist is enforcement. Until registries require signed artifacts as PyPI requires attestations for new uploads, the problem persists. Every from_pretrained() call is a trust decision made without evidence.
Top comments (0)