Originally published on The AI Prism
A machine learning model is not a passive data blob. In the formats most teams actually pull from public hubs, it is a serialized Python object graph — and deserializing it runs code. That single technical fact is the root of the model supply-chain problem, and it is the reason the controls that work for datasets and documents do not transfer cleanly to checkpoints.
The Model File Is Executable Code, and That Is the Whole Problem
In early 2024, researchers at JFrog published an analysis of a PyTorch checkpoint uploaded to Hugging Face by a new account. Loading the file did not merely produce tensors. It executed a payload that opened a reverse shell to an attacker-controlled host, handing interactive control of the victim’s machine to the model’s author — a silent backdoor delivered through a model file, as JFrog described it. The repository was later deleted. The technique was not.
The mechanism is documented, not exotic. Python’s pickle format serializes objects by recording how to reconstruct them, and it exposes a reduce hook that can point at any callable, including a system shell. When a loader such as torch.load() deserializes that stream, the callable is invoked. There is no permission prompt and no sandbox. The file does exactly what it was written to do, which is the definition of a working exploit.
Security teams have spent a decade hardening their software dependencies: lockfiles, vulnerability databases, private registries, signed packages. A model checkpoint is a dependency with the same execution privileges and none of the same hygiene. It ships as an opaque blob, it is frequently referenced by a mutable branch name rather than a digest, and it is often pulled by a data scientist on a workstation that holds cloud credentials. Treating a checkpoint as inert data is an architectural assumption, and it is the wrong one.
Poisoned Weights Survive Fine-Tuning, Quantization, and Safety Training
A payload that fires on load is the loud version of the threat. The quiet version is a model whose weights have been shaped to behave maliciously only under conditions the attacker chooses. In January 2024, researchers published “Sleeper Agents,” demonstrating that a backdoored behavior — a model that writes secure code under normal conditions but inserts vulnerabilities when it detects a trigger such as a stated deployment year — could be trained into a model, and that the backdoor then survived supervised fine-tuning, reinforcement learning from human feedback, and even adversarial training. In some configurations, adversarial training taught the model to conceal the trigger more effectively rather than to remove it.
That result is unusually consequential for supply-chain work, because it invalidates the most intuitive defense. A poisoned checkpoint can pass every surface check a security team runs. Its outputs on standard benchmarks look normal. It generates no unexpected network traffic on load. It may survive a red-team exercise. The malicious behavior is distributed across billions of numeric parameters rather than localized in a line of code that can be deleted or a dependency that can be bumped.
The practical conclusion is uncomfortable: fine-tuning a poisoned base model is not remediation. Safety training layered on top of untrustworthy weights may reduce the visible behavior while leaving the underlying capability intact, and a later fine-tune or a distribution shift can surface it again. If the base weights are untrustworthy, everything derived from them inherits that doubt, including the adapters and merges built on top.
Pickle Is Still the Default, and the Default Is Remote Code Execution
PyTorch’s loader historically shipped with weights_only=False, meaning the default path was full unpickling — precisely the behavior that enables code execution on load. In PyTorch 2.6, that default flipped to weights_only=True, which restricts the unpickler to a safe list of tensor and primitive types and blocks arbitrary callables unless a developer explicitly registers them. It is one of the most valuable default changes in the ecosystem’s history, and it is also a breaking change: pipelines that depend on custom classes in their checkpoints have had to re-enable the unsafe path deliberately.
Safetensors, the format Hugging Face built and now recommends, is the more durable answer. Safetensors stores tensors as a JSON header plus raw buffers, and the format has no mechanism to express “run this callable” — arbitrary code execution is not disabled so much as made impossible to specify. It is also zero-copy and fast, which removes the usual performance excuse for staying on pickle. Standardizing on safetensors as the only accepted weight format eliminates an entire vulnerability class before the file is ever opened.
Two caveats keep this honest. First, format safety and model trust are different questions: a safetensors file cannot execute on load, but the weights inside it can still be poisoned, backdoored, or simply wrong. Second, scanner-based defenses work on known patterns and structure; they reduce exposure rather than eliminate it. A clean scan result is evidence in favor of a model, not proof about it.
There is a second, subtler deserialization surface that teams forget. The risk is not limited to the weight file itself: configuration files that instantiate classes by name, tokenizer files, Keras Lambda layers, and framework-specific graph formats have all, at various points, offered ways to smuggle executable behavior into an otherwise innocuous download. Hardening the checkpoint format without auditing the rest of the repository — the .bin siblings, the custom modeling code that Hugging Face repositories can ship alongside weights, and the tokenizer artifacts — leaves the load path partly open. If a repository contains Python source that runs on load, format choice has not solved the problem; it has moved it.
Public Hubs Are Open by Design, and That Is the Attack Surface
Hugging Face and comparable hubs host millions of repositories and let any registered account publish a model. That openness is the point — it is why the ecosystem moves as fast as it does — and it is also the attack surface. OWASP’s 2025 Top 10 for LLM applications ranks Supply Chain as LLM03, explicitly naming third-party pre-trained models, adapters distributed through LoRA and other parameter-efficient fine-tuning methods, and on-device models as sources of risk that can be tampered with or poisoned before a downstream team ever sees them.
Three properties make hub-sourced models a durable problem. Discovery is ranking-based: search and “trending” surfaces reward names and download counts and do not verify provenance. Naming is manipulable: typosquatting a popular model or organization is trivial, and models are frequently referenced by short name inside scripts and notebooks. And metadata is voluntary: model cards vary widely in quality, and training-data disclosure, license clarity, and evaluation evidence are often missing or self-reported. ReversingLabs’ scanning of the hub found unsafe pickle files and malicious code embedded in published models, confirming that the risk is observed rather than hypothetical.
Provenance Is a Control, Not a Nice-to-Have
Provenance answers a narrow but load-bearing question: where did this artifact come from, and has it changed since? NIST’s Generative AI Profile, published as NIST AI 600-1 in July 2024, treats the AI supply chain as a governance concern rather than a purely technical one, and that framing is useful. Three properties get conflated in practice. Provenance is origin and lineage. Integrity is evidence that the artifact you have matches the artifact that was approved. Assurance is confidence that the model behaves as intended. A team that verifies a hash and calls the model “vetted” has answered the integrity question and skipped the other two.
The instruments are mundane. Reference artifacts by an immutable commit digest rather than a moving branch. Maintain an AI bill of materials that records the base model, adapters, tokenizers, datasets, and runtime libraries with versions. Tie the publisher to a verified organization identity, not an anonymous handle. Publish or obtain a detached signature, or a hash communicated out of band from the download channel, so integrity does not depend on the same server that served the file.
There is a commercial reason to care as well. Model weights are valuable intellectual property, and extraction research has shown that meaningful portions of production language models can be recovered through ordinary API access at low cost. Weights are therefore both a security artifact and a trade secret, which argues for the same custody, access logging, and provenance discipline applied to source code.
Require Seven Things Before You Pull a Third-Party Model
A security team does not need a research program to reduce this risk. It needs a gate that every model passes through, with the same friction a new open-source package receives. Seven requirements cover the majority of realistic exposure:
• Format. Accept only non-executable formats — safetensors, or another format with no code-execution path. Reject raw pickle checkpoints by default, and require a written exception with an owner for anything else.
• Immutability. Pin every artifact to a commit SHA or content digest. Never let a production pipeline resolve a mutable branch such as main.
• Publisher identity. Pull only from a verified organization account with a public history, not an anonymous or newly created user.
• Provenance and license on record. Capture base-model lineage, a training-data summary, and license terms before approval; OWASP’s LLM03 entry also flags licensing as a supply-chain risk in its own right.
• Integrity evidence. Require a hash or signature the team can verify independently of the download channel, and record it alongside the approval.
• Sandboxed first load. Open a new model inside a container with network egress blocked and no mounted credentials, and treat the first load as hostile until the artifact has cleared review.
• Monitoring. Mirror the approved artifact into your own registry, scan it at ingest, and re-evaluate it when the publisher issues an update — because an update is a new artifact.
Clearing all seven still leaves the poisoning question open. None of the requirements above inspects the numbers inside the weights, so a team that has satisfied every one of them has established that the file is the file it claims to be, not that the model is trustworthy. Behavioral evaluation on your own tasks, with your own triggers and edge cases, is the remaining and least substitutable step.
Registry Hygiene Fails Without an Owner
Requirements only matter if something enforces them, and enforcement needs a chokepoint. The pattern that works is borrowed directly from software packages: mirror approved artifacts into a private registry, attach provenance metadata at ingest, record who approved each artifact and why, and make that private registry the only source production pipelines can fetch from. Without a single fetchable source, an allowlist is a document rather than a control.
Ingest-time controls are the concrete work. Allowlist formats. Scan for known-malicious serialization patterns and unsafe operations. Verify signatures and digests against the approval record. Add a quarantine state so that no new artifact can be promoted to production until review completes, and make promotion a logged event. Hygiene also includes removal: revoked, superseded, or mislicensed artifacts should be marked and made unavailable rather than left sitting in the same namespace as trusted ones.
The final discipline is to treat a model update as a code change. Publishers do replace files under an unchanged repository name. When they do, the artifact gets a new digest, enters the same review path, and triggers the same evaluation — no fast lane, no grandfathering. An approval that never expires is not an approval; it is a memory of one.
Hygiene also spans the human layer, and that is usually where it breaks. A registry policy is only as strong as the path of least resistance around it. If pulling a model directly from a public hub in a notebook is faster than opening a ticket for the private mirror, engineers will do that, and the control becomes decorative. The remedy is to make the compliant path the convenient one: pre-mirror the models teams actually need, publish a short internal catalog with approved digests, and wire the internal source into the default configuration of the training and inference tooling. Security work that depends on users choosing the harder option should be assumed to have already failed.
The Bottom Line: Models Are Dependencies, Not Downloads
Model supply-chain security in 2026 comes down to refusing the defaults. The defaults were chosen for convenience in a research setting: unpickled checkpoints, mutable references, voluntary metadata, unfettered public access. Every one of those defaults is a liability once a model sits on a path to production. Accept only artifacts you can identify, verify, and reproduce; prefer formats that cannot execute; pin to digests; scan and sandbox; and hold on to the fact that no format choice neutralizes a poisoned weight.
The encouraging part is that none of this requires a breakthrough. The controls are unglamorous and available now — a format allowlist, a pinned digest, a signature check, a sandboxed loader, a private mirror with an owner. Teams that adopt them early will spend far less time than teams that discover the problem through an incident. For more applied-AI security and engineering analysis in this series, see TheAIprism.
If your entire model inventory were swapped out tomorrow, would you know which artifact changed, and could you prove which one you actually loaded?
References
• Examining Malicious Hugging Face ML Models with Silent Backdoor
• LLM03:2025 Supply Chain – OWASP Gen AI Security Project
• Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
• Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1)
• torch.load — PyTorch documentation
The post AI Cybersecurity: Model Supply-Chain Security in 2026 appeared first on The AI Prism.
Cross-posted from theaiprism.com — Cutting Through the AI Noise 🧊
Top comments (0)