Mohit Kumar

Posted on Aug 3 • Originally published at mohitkumar1.hashnode.dev

How a Machine-Learning Model Runs Code on Your Machine (and How I Scan for It)

#ai #cybersecurity #appsec #python

A byte-level tour of pickle deserialization attacks, the 2025 scanner bypasses, and building a scanner that survives them.

Author: Mohit Kumar

Project: Bulwark – An open-source security stack for AI agents

GitHub: mk12002/Bulwark

This article is part of the Bulwark series, where I explore practical approaches to securing AI systems, agentic workflows, and the AI software supply chain.

torch.load("pytorch_model.bin") looks like reading a file. It isn't. For a huge share of models on the internet, "loading the weights" and "running an attacker's Python" are the same operation — and in 2025, researchers found bypasses with a CVSS score of 9.3 that let a malicious model sail straight past the industry-standard scanner just by renaming the file.

I went down this rabbit hole and came out the other side with a scanner. Here's every byte of how the attack works, why the standard defenses broke, and what it takes to catch the evasions they missed.

TL;DR

Python's pickle format is a tiny stack VM. It has an opcode (REDUCE) that calls arbitrary callables — so a pickle can run os.system("...") the instant it's deserialized.

~45% of popular Hugging Face models still ship pickle (CCS 2025). Loading one from an untrusted source is running an untrusted program.

In 2025, picklescan (the scanner Hugging Face runs) was hit with multiple bypasses, incl. CVE-2025-10155 — rename evil.pkl to model.safetensors and the extension-based check skips it.

I built Airlock, which disassembles pickles statically by content, decodes evasion layers, and flags format spoofing. On a 14-payload adversarial suite it scored 14/14 vs picklescan's 10/14 — with zero false alarms on 18 real, benign models.

Why this matters

Every from_pretrained() in your codebase is a trust decision. The model file is not passive data — for the most common serialization format it's a program, and deserializing it executes that program. The security community treats "don't unpickle untrusted data" as folklore, but the entire ML ecosystem is built on doing exactly that, millions of times a day, from a public hub anyone can upload to.

The result is a live, exploited attack surface. ReversingLabs and others have found real malicious models in the wild. This isn't a thought experiment — it's the single most direct code-execution path in the AI supply chain.

The core concept: pickle is a stack VM, not a data format

Most people picture pickle as "JSON for Python objects." It's actually a serialized program for a tiny stack machine. When you unpickle, the interpreter walks a stream of opcodes: push this string, look up that global, call this callable with those arguments.

The dangerous opcodes are GLOBAL / STACK_GLOBAL (resolve a module.name reference — e.g. os.system) and REDUCE (call the thing on top of the stack with the argument tuple below it). Put them together and you have arbitrary code execution encoded as data.

Here's the entire exploit in Python — the __reduce__ method tells pickle exactly what to call on load:

import pickle, os

class Exploit:
    def __reduce__(self):
        # On unpickling: os.system("...") runs. Swap echo for anything.
        return (os.system, ("echo you_have_been_pwned",))

pickle.dumps(Exploit())   # ship this as pytorch_model.bin

When someone runs torch.load() on that file, os.system executes. No warning, no sandbox. The analogy: a pickle file is a Trojan horse where the "assembly instructions" are allowed to light a fire. Reading the instructions is lighting the fire.

How it actually works — the attack, byte by byte

Let's disassemble a minimal malicious pickle. Python's own pickletools shows the opcodes (this parses — it does not execute):

    0: \x80 PROTO      4
    2: \x95 FRAME      ...
   14: \x8c SHORT_BINUNICODE 'os'          # push module name
   18: \x8c SHORT_BINUNICODE 'system'      # push callable name
   26: \x93 STACK_GLOBAL                    # → resolve os.system
   27: \x8c SHORT_BINUNICODE 'echo pwned'  # push the argument
   39: \x85 TUPLE1                          # → ("echo pwned",)
   40: R    REDUCE                          # → os.system("echo pwned")
   41: .    STOP

That's it. Four meaningful opcodes turn a "weights file" into a command execution. A scanner's job is to walk this same opcode stream and notice that os.system (a dangerous callable) reaches a REDUCE.

Where the standard scanner broke in 2025

picklescan does exactly that walk — and it's good at it. But attackers don't attack the disassembler; they attack everything around it. In 2025 a wave of bypasses landed:

The CVE-2025-10155 one is almost funny: picklescan picked the scanner based on the file extension. Rename malicious.pkl to model.bin or model.safetensors and it misclassified the file type and failed the scan open. Three zero-days (fixed in picklescan 0.0.31, Sept 2025) and four more found by Sonatype in December 2025 followed. Cisco went as far as publishing structure-aware fuzzing to harden pickle scanners. The lesson is old and evergreen: trusting metadata (the extension) instead of content is how scanners die.

Building Airlock — a scanner designed to be evaded and survive

I built Airlock around one rule: never trust the extension; disassemble the content, and peel the evasion layers first.

Three design choices matter, each aimed at a bypass class:

1. Content sniffing beats extension spoofing (defeats CVE-2025-10155)

Airlock reads the magic bytes. If a file's extension claims a safe format (.safetensors, .gguf) but its bytes are a pickle stream, that's not an accident — it's the bypass. Airlock flags the deception (M6) and disassembles the hidden pickle anyway, so the payload still trips M1. Verified:

$ airlock scan model ./disguised     # a pickle named model.safetensors
  CRITICAL  M1  Pickle references a shell/exec callable      os.system @ model.safetensors
  HIGH      M6  File content does not match its extension    model.safetensors

And crucially, a genuine safetensors file produces zero false positives — the confirmation step requires the stream to actually disassemble as a pickle with real imports.

2. Peel the evasion layers

A pickle inside a gzip inside a .bin? Airlock does bounded decompression first. A payload hidden as a base64 string inside an outer pickle (the classic "staged" layout)? It decodes base64-looking strings one level deep and re-scans. A STACK_GLOBAL that splits os and system across separate opcodes to avoid a naive c os\nsystem grep? The memo-aware stack still resolves it.

3. Allowlist mode — catch the novel, not just the known (a Fickling-style idea)

Denylists only catch attacks you've already seen. Trail of Bits' Fickling flipped this in 2025: allow known-safe imports, flag everything else. Airlock's --strict mode does the module-level version. I derived the safe set empirically — across the 18 real model pickles I tested, they imported from exactly two modules:

TOP-LEVEL MODULES across 54 real pickle streams:
   135  torch
    51  collections

So --strict flags any pickle import from a module outside the ML allowlist (torch, numpy, collections, …) — catching a novel socket.gethostname or a never-before-seen malicious module that a denylist would wave through. On the real corpus it fires zero false positives, because real weights genuinely only import from torch/collections.

Does it actually work? The benchmark.

Claims are cheap. I built a 14-payload adversarial suite (every payload is benign — it echos a marker instead of doing harm, and nothing is ever unpickled) covering protocols 0–5, framed pickles, STACK_GLOBAL splitting, gzip/zlib, base64 staging, numpy-object smuggling, torch-zip, and the disguised .safetensors. Then I ran both Airlock and picklescan over all of it plus 18 real models.

Group	Airlock	picklescan
Adversarial (14 evasive payloads)	14/14	10/14
Real models (18 benign `.bin`)	0/18	0/18

Two things I want to be honest about:

Airlock's edge is the gzip/zlib-compressed and base64-staged variants (it decompresses/decodes; picklescan's file-path entry doesn't), plus the numpy-object case. On the disguised file, a current picklescan sniffs content too and catches it — but only Airlock emits the explicit "this is a disguise" finding.
On real benign models, both scored 0/18 on code execution. That's the number I care about most. Catching attacks is easy if you're willing to cry wolf; the hard part is not flagging 18 legitimate models as malware. Airlock still reports the pickle surface risk (M2) and missing-provenance advisories — a risk posture, not a false alarm.

And the empirical study over those 19 models tells its own story: 100% had at least one supply-chain finding, 95% ship pickle weights, 89% contain a REDUCE opcode, and 100% shipped no hashes to verify integrity. The ecosystem is one poisoned upload away from a bad day.

Defenses that actually work

Prefer safetensors, full stop. It stores tensors with no executable opcodes. If a repo offers both, load the safetensors and never touch the pickle.
Scan before you load, in CI. airlock scan model hf:org/name --fail-on high before the model enters a build image. A scanner that runs after torch.load() is a post-mortem.
Turn on allowlist mode for high-security contexts. --strict catches novel imports a denylist won't. The false-positive cost is near-zero because real weights import from a tiny module set.
Never trust the extension in your own tooling. If you build model-handling code, sniff the magic bytes. The 2025 CVEs are a monument to what happens when you don't.
Sandbox the unavoidable. If you must load an untrusted pickle, do it in a locked-down container with no network and no secrets mounted.

Hot take: "we scan uploaded models" is worth very little if the scanner trusts the file extension — that's not a scanner, it's a speed bump with a .pkl filter. The 2025 bypass wave proved the whole category was one os.rename() away from useless. Content-first or bust.

Key takeaways

Treat model loading as code execution, because for pickle formats it literally is.
Disassemble by content, never by extension — the 2025 bypasses were all metadata-trust failures.
Peel evasion layers (compression, base64 staging, archive nesting) before you inspect opcodes.
Add an allowlist to catch novel imports a denylist has never seen — and derive the safe set from real data so it stays quiet.
Benchmark against a real incumbent and count false positives on benign inputs; a scanner that cries wolf gets turned off.

DEV Community