The Signature Was Valid. The Software Wasn't.

In September 2015, WeChat shipped to the App Store with code that nobody at Tencent had written. So did a ride-hailing app, a music service, a business-card scanner and a long tail of other iPhone apps. Nobody had broken into those companies or touched their repositories. Somewhere on each of those teams, a developer frustrated by painfully slow downloads from Apple's servers had installed Xcode from a fast local cloud-storage share or a colleague's copy, and that copy carried a modified linker that stitched extra code into every app it built. The source was clean, the binaries were signed, and Apple's review approved them. That is precisely the blind spot that this essay on the hidden life of software provenance describes as operating on trust alone, and in most build pipelines it is wider than anyone likes to admit. This post is about the gap between the code you review and the bytes you actually ship, why a valid signature does not close it, and the cheap habits that do.

Ken Thompson Saw It Coming in 1983

When Ken Thompson accepted the Turing Award in 1983, he spent his lecture describing a C compiler with two tricks. It recognized when it was compiling the Unix login program and slipped in a backdoor password. It also recognized when it was compiling a new version of itself and quietly re-inserted both tricks. Once that binary existed, the compiler's source could be scrubbed perfectly clean and the backdoor would survive every rebuild. Thompson later confirmed that the trojan really did land in the login binary of a machine belonging to a Unix support group, although the poisoned compiler itself was never distributed. The lecture, published in 1984 as "Reflections on Trusting Trust," ends on a conclusion that has aged uncomfortably well: reading source code proves nothing about a binary unless you also trust everything that produced it.

For more than thirty years, that attack lived mostly in lecture halls. XcodeGhost moved it into production. The idea was not even new in 2015: months earlier, documents leaked by Edward Snowden had described CIA-backed researchers claiming they had built a modified Xcode able to plant surveillance backdoors in any app compiled with it. What was new was the scale. Palo Alto Networks counted just five malicious apps that had ever made it into the App Store before that autumn, which is why BBC News reported XcodeGhost as the first large-scale attack on Apple's App Store, and estimates of infected apps eventually passed 4,000. Apple's remedy was telling: rebuild with a clean copy of Xcode, and verify that copy with Gatekeeper before trusting it again.

Signatures Answer "Who," and Attackers Love That

Code signing was supposed to settle questions like these. Two incidents show why it cannot do the job alone.

Between June and November 2018, attackers who had gotten inside ASUS pushed a backdoored version of its Live Update utility through the company's official update servers, signed with a legitimate ASUS certificate. Kaspersky, which uncovered the operation in January 2019 and named it ShadowHammer, found that more than 57,000 of its own users had installed the update and estimated the total reach at about a million people. The attackers only cared about roughly 600 of those machines, identified by MAC address hashes hardcoded into the malware. Every computer that checked the update concluded it genuinely came from ASUS, and every one of them was right.

In 2023, the VoIP vendor 3CX, which says its software has more than 12 million users, shipped trojanized desktop apps for Windows and macOS. They were produced in 3CX's own compromised build environments, carried valid signatures and were served from 3CX's own website. Mandiant traced the intrusion to an employee who had installed X_TRADER, a futures-trading application retired in 2020 but still available for download in 2022, which had itself been trojanized. It was the first time Mandiant had seen one software supply chain attack directly produce another.

A signature proves that someone holding a key approved a particular set of bytes. It says nothing about what happened between the commit and the key. If an attacker lives in that stretch, your signing step stops being a safeguard and becomes their laundering service.

Tags Lie, Hashes Don't

The same failure shows up in tools developers touch every day. In March 2025, an attacker compromised tj-actions/changed-files, a GitHub Action used by more than 23,000 repositories, and retroactively moved its existing version tags to point at a single malicious commit. Workflows referencing a friendly tag such as @v35 silently started running code that dumped CI secrets into build logs, which on public repositories anyone can read. Nobody merged a pull request and nobody bumped a version; the tag simply started meaning something else. Workflows pinned to a full commit SHA kept running exactly what they had always run.

The most encouraging story in this genre is also the most boring. From the end of January 2021, attackers periodically altered Codecov's Bash Uploader, a script that CI pipelines everywhere downloaded and executed on every run, so that it could ship data from those CI environments to a server outside Codecov's control. It was caught on April 1 by one customer who did something almost nobody does: compared the SHA256 of the freshly downloaded script with the checksum published on GitHub. The values did not match. Two months of silent exfiltration ended because one person ran one command.

What Provenance Actually Adds

Provenance turns the invisible stretch between commit and artifact into a claim someone else can check. In the SLSA framework (Supply-chain Levels for Software Artifacts, pronounced "salsa"), a provenance statement says, in effect: this exact artifact, identified by its digest, was produced by this builder, from this repository at this commit, by this workflow. Sigstore made signing such statements nearly free. Instead of guarding a long-lived private key that can be stolen, a CI job proves its identity through OIDC, receives a certificate that expires within minutes, and records the signature in a public transparency log where later tampering would stand out.

None of this has to be built by hand anymore. GitHub's guide to artifact attestations shows that a workflow needs only a few permissions, including id-token: write and attestations: write, plus one actions/attest step pointed at its output; consumers can then run gh attestation verify against the file and get a clear yes or no. That alone reaches SLSA Build Level 2. GitHub's immutable releases go further and lock a release tag to a single commit, which is exactly the guarantee tj-actions users assumed they already had. On the npm side, packages published with npm publish --provenance link back to the commit and workflow that built them, and npm audit signatures (npm 9.5 or later) reports how many of your installed dependencies carry verified registry signatures and attestations.

Be honest about the limits. Provenance does not prove code is safe, and a malicious maintainer can publish malware with immaculate provenance. What it does is make the unexpected visible. A package that has shipped with attestations for two years and suddenly arrives without them, a binary built by a workflow you have never seen, an artifact whose digest matches no attested build: those are exactly the anomalies that XcodeGhost, ShadowHammer and the 3CX attack hid behind. Reproducible builds take the idea to its conclusion. Independent machines compile the same source and must produce bit-for-bit identical output, so no single build environment has to be trusted at all, which is why projects such as Debian and the Tor Browser have invested years in getting there.

A Provenance Audit You Can Run This Week

You do not need a dedicated security team to start. Most of the protection comes from a handful of small, unglamorous changes:

• Pin third-party GitHub Actions to full commit SHAs and keep the human-readable version in a trailing comment, or rely on publishers that use immutable releases.
• Stop piping downloaded scripts straight into a shell in CI. Fetch the file, compare its SHA256 with a value committed to your repository, and only then run it.
• Run npm audit signatures in CI and treat a dependency that used to ship attestations and suddenly does not as an incident until proven otherwise.
• Attest your own release artifacts and put the exact gh attestation verify command in your README, so users can check your builds instead of taking them on faith.
• Install compilers and SDKs only from the vendor's origin, verify their signatures or checksums, and hold the toolchains baked into your CI images to the same rule.
• Build one artifact twice on two different machines and diff the hashes. The first mismatch you chase down will teach you more about your pipeline than a quarter of architecture meetings.

The Question Worth Asking

Thompson needed three pages to show that source code is a promise, not proof. Four decades later, the tools for checking that promise are free and mostly boring: a pinned hash, an attestation, a one-line verify command in a README. What has changed is the question worth asking about everything that runs in your pipeline. It is no longer "is this signed?" It is "can I show where this came from, and would I notice if that changed tomorrow?" A signature tells you whom to blame. Provenance tells you what to believe.