Semgrep is right that models are black boxes we can't audit. But you don't need an attacker to poison a model. The internet already did. And the difference between those two threats is the part their essay skips.
Disclosure: I work on AI-powered vulnerability remediation. In other words, I build the kind of verification layer this debate is about. Read with that in mind.
Semgrep recently published an essay arguing that AI models are effectively impossible to reverse engineer, that backdoors planted in training data are nearly undetectable, and that the industry urgently needs provenance standards and independent auditors for models, the way it has for traditional software. Provenance here just means a verifiable record of where a model came from and what was done to it along the way.
I agree with almost all of it. That's exactly why it's worth saying out loud: none of this is new. And in a few places, the framing points readers at the wrong threat.
We've known this since 1984
The essay's best move is invoking Ken Thompson's "Reflections on Trusting Trust", his 1984 Turing Award lecture about a compiler that invisibly carries its own backdoor. His point was that you can't trust a system just because you can read the source code in front of you. It's the right analogy for model lineage. It's also the tell: the core insight here is forty-two years old.
Applying it to code-generating models isn't new either. In 2021, the "Asleep at the Keyboard" study prompted GitHub Copilot with 89 scenarios built around MITRE's most dangerous software weakness categories. Roughly 40% of the 1,689 generated programs came back vulnerable. The authors' reasoning was the supply chain argument, word for word: the model ingested a huge pile of unvetted public code, so it must have learned from buggy, exploitable code. That was five years ago. Data poisoning research goes back further still, and peaked recently with Anthropic's study alongside the UK AI Security Institute and the Alan Turing Institute (paper): about 250 malicious documents were enough to backdoor models from 600M to 13B parameters. A fixed number, not a percentage of the training data.
So when an essay in 2026 presents "models have an unauditable supply chain" with the urgency of a discovery, the honest reply is: yes, and if the industry still needs to hear it, that's an adoption failure, not a knowledge gap. Different problem. Different fix.
Read the citations all the way down
Two technical claims deserve a correction, because overclaiming is how a correct thesis gets dismissed.
First, the essay contrasts models with compiled binaries by claiming reverse engineering can produce a "total description of its behavior." It can't. Ask anyone who has spent a week inside a packed malware sample. Some questions about what a program does are mathematically undecidable, full stop. The fair claim is that binaries are far easier to analyze than model weights. That's true, and it's enough.
Second, the essay squeezes two different research results into one scary sentence: that a small amount of poisoning during pretraining creates persistent bad behavior. Not quite. The 250-document result was a crude backdoor: trigger phrase in, gibberish out. The authors themselves said it's unlikely to matter much in frontier models, and in the same paper, those pretraining backdoors did not survive normal post-training. The persistence result comes from a different paper, Sleeper Agents, where researchers deliberately trained deceptive behavior into larger models and showed that standard safety training failed to remove it. Both results are real. The stitched-together claim doesn't appear in either paper.
It's not an open-weight problem. It's a training-at-scale problem.
The essay hedges its title with a nod to open-weight models, and adds some geopolitical spice: should we trust models built in certain countries? Its own strongest sentence undercuts that. We should worry about every model whose history we can't verify.
Here's the uncomfortable part. Every lab drinks from the same polluted water: scraped public code, good and bad, secure and insecure, tutorial code that's vulnerable on purpose, production code that's vulnerable by accident. All of it went in because it was there. Yes, there's filtering: deduplication, quality scoring, license checks. But nobody runs a security review over petabytes of training code. Nobody can afford to.
And the black-box nature comes from the architecture, not the license. If anything, a closed model is harder to inspect. You don't even get the weights. You get a text box.
The one thing that genuinely is worse for open weights: anyone can upload a tampered fine-tune to a model hub under a familiar name, and nothing proves what happened to it after the original release. That's a real gap. But it's a gap about unsigned files, not about openness being dangerous.
The haystack and the needle
Here's the distinction the essay never draws, and it's the one that tells you what to actually build.
The haystack is the accidental poisoning, and it's not a theory. It's already inside every model, open or closed. The main failure mode isn't planted backdoors. It's insecure code that works: SQL queries built by gluing strings together, weak password hashing, certificate checks switched off. Those patterns are everywhere in training data precisely because they work, get copied, and get starred. The model learned exactly what we published. No attacker required.
And here's the kicker: provenance doesn't help with the haystack. You could document every single byte of training data and the poison stays, because the poison is popular, legitimate code. The haystack gets caught at the output. Scan what the model writes. Verify every generated change before it ships. Never trust a diff you didn't check.
The needle is the deliberate backdoor: the 250-document attack, the sleeper agent. It's rarer, and so far nobody has shown one in the wild against a major model. It also has the opposite property. Output checks mostly can't catch it, because it only fires on an input you don't know to test. This is the one case where signed weights and verified model history do the heavy lifting.
Two threats. Two completely different defenses. An essay that lumps them together can only conclude that someone should audit everything. Which is true, and useless.
The auditors already exist. The hard part doesn't.
The essay ends by calling for trusted third parties and provenance standards, as if the field were empty. It isn't. CycloneDX has an ML-BOM. SPDX 3.0 ships an AI profile. The OpenSSF has a model signing spec with a working Sigstore implementation. NIST published an AI Risk Management Framework. The EU AI Act adds documentation and provenance duties on top. The gap is adoption and teeth, not vision.
The genuinely unsolved problem, the one I wish the essay had spent its words on, is that the crown jewel of software supply chain security may be out of reach for models: the reproducible build. Retraining a frontier model costs a fortune. The training data can't be shared. And even identical inputs don't produce identical weights across GPU runs. The software analogy breaks exactly where you'd want to lean on it hardest. That's a research agenda, not a manifesto.
Trust models the way we trust compilers
Thompson's lesson was never "understand your compiler's internals." Nobody reads their optimizer's source before shipping. We trust compilers because decades of verification grew around them: test suites, independent implementations, reproducible builds, signed releases.
Models will earn trust the same way, and the essay is right about the destination. But the road splits in two. Verify every output as if the model were compromised, because statistically, its training data was. And sign every model along with its history, because one day, somebody's model actually will be.
The problem isn't new. The work is just unfinished. And naming which defense stops which threat is how it gets finished.
Top comments (0)