Local AI gives you control over where your model runs. Model security also depends on what your loader executes along the way. Researchers have documented Hugging Face model packages containing reverse-shell payloads, and an unsafe loading path can give that code the same access as your Python process. Your machine gets the model and, potentially, an ambitious new colleague.
TL;DR
- Model weights are learned numerical data. A model repository can also carry configuration, executable serialization and Python files.
- JFrog's Goober Two example and ReversingLabs' nullifAI samples contained reverse-shell payloads. The selected reports do not establish credential theft from actual victims.
- A separate Checkmarx research demonstration showed how a missing local model path could resolve to a public repository. Remote-code trust was an important execution prerequisite.
- Safetensors, current patched loaders, reviewed revisions and isolation address different parts of the loading path.
- Readable secrets and available network access determine how much damage code running in the loader can do.
Model security begins before the first answer
Hugging Face is a platform for sharing machine learning models, datasets and applications. A repository is a useful distribution unit: it can contain the weights, the configuration needed to interpret them and, for custom architectures, implementation code. The contents deserve separate questions because their loading behavior differs.
The weights are the learned numbers. A tensor-data reader interprets their shapes and values. Python object serialization can describe how to rebuild objects, including calls to functions. Repository Python code introduces another execution path when your application authorizes it. Putting all these files under the heading “model” makes installation convenient and the trust boundary easy to miss.
Hugging Face's Pickle Scanning documentation states the central warning plainly: “There are dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file.” The significant verb is load. Saving bytes to disk and handing those bytes to an unsafe deserializer are different actions.
For the incidents discussed here, the file's instructions become dangerous when a loading path follows them. That can happen before inference begins. A malicious package does not need to persuade the model to reveal anything in its answer if the surrounding software already lets its code run. The benchmark can wait; the process has started accepting guests.
Pickle security: opening the parcel
Pickle stores Python objects with instructions for reconstructing them. Deserialization reads those instructions in sequence. An unsafe loader may resolve and call functions during reconstruction, which gives an untrusted file influence over the host process.
Think of the package as a parcel and the loader as the person opening it. Downloading places the parcel on the shelf. Loading follows the enclosed assembly instructions. If those instructions can invoke arbitrary functions, the activity has crossed from reading data into executing the publisher's choices.
The exact loading API, library version and settings matter. A security statement about one unsafe path cannot be extended to every model loader. Modern restricted loading modes deliberately reduce what object reconstruction is allowed to do. Conversely, overriding a rejection because an unfamiliar checkpoint “needs it” can reopen the boundary you were relying on.
This is also why “poisoned weights” needs explanation. The phrase can refer to model behavior changed by training or editing numerical parameters. The cases in this episode concern executable packaging and repository code around model distribution. Those mechanisms have their own entry points, permissions and defenses.
Three concrete examples, with their limits
Goober Two: a documented reverse shell
In February 2024, JFrog analyzed baller423/goober2. Its Pickle payload initiated a reverse shell during unsafe loading. A reverse shell connects outward and offers command access to the other end of the connection.
The repository was deleted. JFrog also discussed clues suggesting possible researcher involvement, so attribution remains uncertain. The report supplies evidence for the payload's behavior. It does not establish that criminals stole a named victim's cloud keys, and it does not justify inventing a victim count.
That distinction leaves the mechanism plenty worrying. Command access under the loader's account is a meaningful foothold. A development environment often gives that account more access than a model experiment needs.
nullifAI: unusual packaging and an error that arrived late
In February 2025, ReversingLabs described two model samples, including glockr1/ballr7, under the name nullifAI. Both contained platform-aware reverse-shell payloads.
An important detail is the archive format. These samples used seven-zip packaging where ordinary PyTorch loading expected a different structure. Default torch.load would reject that packaging. The dangerous path required extraction followed by unsafe Pickle deserialization. A diagram showing an ordinary model download immediately opening these samples would skip a real prerequisite.
The report also examined broken Pickle streams and a historical scanner blind spot. Its harmless file-writing research test showed that an earlier instruction could execute before a later error stopped loading. The file already existed when the loader complained. Imagine releasing a raccoon from a parcel and then discovering that the shipping label is invalid. The complaint may be accurate; the timing is deeply unhelpful.
ReversingLabs disclosed the finding on January 20, 2025. Hugging Face removed the reported samples within twenty-four hours and changed the scanner. This example describes a historical finding and its response. It supplies a reason to investigate scanner failures, rather than a claim that the same bypass remains unpatched.
Model Confusion: a missing folder resolves somewhere else
Checkmarx's January 2026 Model Confusion article describes a research demonstration involving an unnamed company's sample. A relative identifier such as checkpoints/some-model could refer to a local path or a public model namespace. If the local path was missing, resolution could select a matching remote repository.
Combine that ambiguity with insecure loading and remote-code trust, and the selected repository's code can execute. The company's sample was corrected within a day, according to Checkmarx. The report explicitly describes the demonstration and says it had not found malicious activity in the sensitive-name examples it highlighted.
Two conditions are doing work here: selecting the unexpected remote package, and authorizing an execution-capable loading path. Checking only whether a string looks like a local folder misses the resolution step. A missing directory should not get to improvise its own landlord.
Where readable credentials enter the flow
Once code runs in a process, its access depends on the account, filesystem mounts and other controls around that process. An accessible cloud key, a token available in the environment or a readable private-key file can become a target. Secrets kept outside that access boundary remain outside the straightforward read path.
Sending a secret elsewhere also requires an available network route. Read access and outbound connectivity are independent conditions. Blocking egress can constrain a transfer; keeping the secret out of the process can prevent the read in the first place. Numerical weights alone supply neither capability.
The selected reverse-shell reports document command-access payloads. Credential collection and exfiltration are conditional consequences of that access, rather than proven victim outcomes in these reports. The episode's key movement is therefore labeled “possible consequence,” and the original key remains in place while a copy follows the possible route.
This is where a familiar development setup gets awkward. A container with your home directory or credentials mounted inside exposes those mounted resources to code in the container. The namespace boundary can be useful, but the key you deliberately put inside it is still inside it. A container full of secrets is a gift basket with resource limits.
Safetensors, patched loaders and a useful isolation boundary
Safetensors provides a tensor-data format designed for safe storage without Pickle's executable object reconstruction. Choosing it addresses that serialization route. A repository can still contain Python code alongside the tensor files, so review the code-loading path as a separate decision.
Hugging Face's custom model loading documentation warns about inadvertently executing malicious code and describes trust_remote_code=True. It also recommends pinning a specific revision. Review the exact revision you intend to run, then keep that decision stable while you evaluate the package.
PyTorch's serialization documentation explains that torch.load defaults to weights_only=True starting with version 2.6 when no pickle_module is passed. The restricted unpickler narrows the remote-code-execution surface. The documentation also retains limitations involving denial of service and possible memory corruption, so the setting deserves its actual scope.
For unfamiliar packages, I would separate acquisition and inspection from execution. Prepare a clean environment with only the files the experiment needs. Leave personal keys outside it. Restrict network access according to the task, and make repository-code trust an explicit reviewed choice. The useful question is what a process could reach if its loading behavior surprised you.
Verdict: NEEDS REVIEW
I'd keep open models and put the permission boundary around the loader. The documented examples show why model distribution needs the same attention we give other executable inputs. Free access to a model is valuable; deciding who gets access to the development environment is also valuable.
FAQ
Can downloading a Hugging Face model steal my credentials?
The mechanisms discussed here execute during unsafe loading or authorized repository-code execution. Merely saving the model bytes is a different step. Review the path that will open them.
Does safetensors make a whole repository safe?
It addresses tensor serialization. Review custom Python and dependencies separately, and keep the loader's access appropriately narrow.
Does a failed load mean nothing happened?
Earlier instructions may have executed before a later error. ReversingLabs' harmless test demonstrated exactly that ordering.
Were credentials proven stolen in these examples?
The selected reports establish reverse-shell payloads and a code-execution research demonstration. They do not establish credential theft from actual victims. Readable credentials are a conditional exposure worth preventing.
Sources
- JFrog: malicious Hugging Face models with a silent backdoor
- ReversingLabs: nullifAI and malicious model samples
- Checkmarx: Model Confusion research demonstration
- Hugging Face: Pickle Scanning
- Hugging Face: Safetensors
- Hugging Face: loading custom models
- PyTorch: serialization semantics
This article expands on an episode of **The Daily Diff, a five-minute daily video on what shipped and what broke in tech.
Watch the episode · Subscribe on YouTube · the written diff lands in your inbox every morning at thedailydiff.dev.
Top comments (0)