The Phantom Commit: How a Trusted Package Turned Hostile Without Touching GitHub
When teams look at an incident like the Microsoft Agent Framework package compromise on PyPI, the first instinct is to check git log. The commit isn't there. Pull requests look clean, branch protection hasn't visibly broken, and the repository history ends right where it should—yet versions 1.26.1 and 1.26.2 appeared on PyPI carrying code designed to harvest cloud credentials the moment Python started up. That split between repository history and registry reality isn't a bug in Git; it is a structural property of how source control and artifact registries are wired together by default.
In an idealized mental model, code is written in GitHub, compiled in CI/CD into a .whl archive, pushed to PyPI, and pulled down by pip install as a linear chain. Mechanically, however, PyPI is a storage and distribution service for built archives, not a mirror of a Git repository. When a release is uploaded via an API token or an OpenID Connect (OIDC) exchange, PyPI checks that the presenter is authorized to upload to that project name—it does not unpack the wheel, locate a matching commit hash in an upstream repository, or verify that every .py file inside the archive corresponds to reviewed source lines. Because upload authority is decoupled from commit history, a wheel is simply a zip archive containing bytecode, metadata, and startup hooks that PyPI accepts on its own terms.
That decoupling creates a direct causal path from credential compromise to execution. An attacker who obtains a long-lived PyPI token—or compromises a build environment or OIDC trust configuration that can mint one—can assemble a wheel locally or in attacker-controlled infrastructure, inject a startup hook such as a .pth file inside site-packages, and upload it directly to PyPI. When a developer or automated build runner later starts Python, the interpreter's initialization machinery reads .pth files during site configuration and executes arbitrary Python code before application code even loads. In environments where agent frameworks or build runners carry ambient cloud tokens, environment variables, or instance metadata access, that startup execution reads those secrets from memory or environment state and transmits them outward, all while the GitHub repository shows zero changes.
Why, then, doesn't PyPI simply require a verified Git commit or build provenance for every update? Because enforcing strict cryptographic linkage across an open ecosystem imposes real friction and breaks legitimate edge cases. Reproducible builds in Python are notoriously brittle: file timestamps, build paths, compiler flags, native extensions, and dependency resolution details easily alter archive hashes even when source code hasn't changed. Furthermore, many maintainers still publish from laptops, private build runners, or non-GitHub workflows, and forcing universal provenance gates or multi-person approvals on every patch release slows down urgent security fixes and small maintenance updates. Leaving provenance optional preserves compatibility and velocity, at the cost of allowing unattested uploads to slip through.
Evaluating the controls that close this gap—long-lived API tokens, OIDC Trusted Publishing with build provenance attestations, and strict consumer-side pinning with hash verification—makes the trade-offs clear. Long-lived bearer tokens are the weakest option because a single leaked secret grants direct upload capability from anywhere until revoked. OIDC Trusted Publishing paired with provenance attestations is the strongest publisher-side control because short-lived credentials are issued only to a specific repository and workflow run, binding the artifact's digest to build metadata. Yet even OIDC stops working if a workflow's permissions are overly broad, if a compromised dependency runs inside the build pipeline itself, or if a maintainer's GitHub workflow configuration is altered—and no publisher control protects a consumer who runs an unpinned pip install and pulls a newly uploaded version before takedown.
A common mistake among teams is assuming that either "using a big vendor's package" or "relying on PyPI's fast takedown" substitutes for defensive controls. Takedowns happen after publication, which means automated CI/CD jobs, container builds, and local developer shells can pull and execute the malicious wheel during the window between upload and removal. To choose controls rationally: if you publish packages, use OIDC Trusted Publishing and provenance attestations rather than static API tokens; if you consume packages in production or agent environments, pin exact versions with cryptographic hashes (--require-hashes) and mirror or delay adoption of brand-new releases rather than trusting unpinned pip install by default.
The Mechanics of a Zero-Commit Compromise
When a developer runs pip install on a package like Microsoft’s Agent Framework, pip fetches a zipped distribution archive—typically a .whl file—from PyPI. Standard wheel archives contain compiled bytecode, metadata, and initialization scripts. Nothing in the standard pip download protocol inspects Git history. PyPI does not compare the contents of a wheel against a GitHub tag or commit hash unless strict Trusted Publishing with build provenance is enforced and verified by the client.
By default, PyPI acts as a repository of files associated with a namespace, not a mirror of a Git tree. If an attacker gains access to a PyPI API bearer token or exploits a permissive OIDC configuration on a build runner, they can build a .whl archive locally using flit, hatch, or setuptools, sign in as the publisher, and upload the file via Twine. The archive lands in the PyPI index instantly.
Execution on Startup: The .pth File Injection
The primary execution mechanism for immediate credential theft relies on Python's module import design—specifically, .pth (path configuration) files located inside Python's site-packages directory.
During Python interpreter initialization, the standard library’s site.py module automatically scans all directory entries in site-packages ending in .pth. While these files are intended to append directory paths to sys.path, any line starting with import is passed directly to Python’sexec()` function during interpreter startup.
The execution pipeline operates as follows:
-
Archive Extraction:
pipextracts the malicious wheel, placing a crafted.pthfile (e.g.,\_startup\_check.pth) directly intosite-packages. -
Interpreter Invocation: Any subsequent Python command—such as
python3 -c "import math", a background cron job, or an AI agent initializing its environment—triggerssite.py. -
Path Processing & Payload Trigger:
site.pyopens\_startup\_check.pth, detects theimportprefix, and executes the payload inline before any application code executes. -
Secret Exfiltration: The payload queries ambient system state: inspecting
os.environforAWS\_SECRET\_ACCESS\_KEYorAZURE\_CLIENT\_SECRET, reading~/.aws/credentials, or querying the cloud metadata endpoint (169.254.169.254). It encodes these secrets into an HTTP GET request header and dispatches them to an attacker-controlled endpoint.
The entire sequence completes in under 15 milliseconds. Because the process executes inside the initial runtime process, no application code can intercept or catch the exception.
Five Compromise Scenarios
Understanding how zero-commit supply chain attacks bypass standard code review requires mapping the exact friction points and credential leaks across modern deployment pipelines.
1. Stolen Long-Lived API Bearer Tokens
Maintainers historically generated project-scoped or account-scoped API tokens from their PyPI security settings, pasting them into GitHub Secrets as PYPI\_API\_TOKEN for deployment workflows. If an attacker gains repository write permissions, exploits a developer workstation with malware, or accesses exposed environment variables in build logs, they capture the string pypi-AgEI....
Because PyPI API tokens are bearer credentials valid from any source IP until manually revoked, the attacker bypassing GitHub entirely. They construct a payload locally, target the version string 1.26.1, and execute `twine upload dist/ directly to PyPI. The registry validates the token signature, matches the project scope, and accepts the upload. GitHub registers zero commits, zero pull requests, and zero workflow runs.
2. Over-Scoped OIDC Trusted Publishing Configurations
PyPI’s Trusted Publishing uses OpenID Connect (OIDC) to exchange short-lived JSON Web Tokens (JWTs) issued by GitHub Actions for temporary PyPI upload tokens. However, security boundaries break down when the OIDC trust relationship is configured with overly broad wildcards.
If a maintainer binds PyPI project publishing to a repository branch pattern like refs/heads/ or leaves environment restrictions blank, any workflow running on any branch can request an identity token from GitHub's OIDC provider. An attacker who opens a pull request containing a modified workflow file—or one who pushes to an unprotected feature branch—can trigger a build step that requests a PyPI exchange token and uploads an arbitrary wheel bypassingmain` branch protection rules.
3. Compromised CI/CD Runner Environments
Modern build pipelines routinely install external dependencies dynamically during the build stage (e.g., running pip install -r requirements.txt to pull build tools, linters, or test runners). If a secondary dependency used inside the build environment is compromised, that dependency gains code execution rights inside the CI runner host or container.
While the maintainer’s application source code remains untouched in the Git workspace, the malicious build dependency modifies the binary output in memory or injects the payload into the dist/ directory immediately after python -m build runs, but right before the publishing step runs. The CI runner then uploads the compromised wheel to PyPI under a legitimate GitHub Action run, complete with valid OIDC tokens and authentic CI metadata logs, while source control shows pristine commits.
4. Typosquatting and Name Confusion in Agent Environments
AI agent frameworks and autonomous software engineering tools frequently fetch packages dynamically or auto-generate requirements.txt based on LLM outputs. When an LLM hallucinates a package name or when a human developer makes a typographical error (e.g., micro-soft-agent instead of microsoft-agent), pip resolves the name directly against PyPI’s global index.
Attackers preemptively register common typographical variations or plausible module names. Once installed, the typosquatted package uses .pth startup hooks to mine ambient cloud credentials or environment variables. Because agents often run in privileged container environments with direct access to cloud infrastructure tokens, the blast radius is immediate.
5. Registry Account Takeover via Social Engineering or Password Reuse
Despite mandatory Two-Factor Authentication (2FA) enforcement on PyPI for top packages, older or secondary maintainer accounts linked to a project namespace may retain publish permissions while relying on weaker authentication guarantees. An attacker gaining access to a maintainer’s personal email account can initiate account recovery flows or exploit leaked credentials across unrelated platforms.
Once inside the PyPI account dashboard, the attacker generates a fresh API token, bypasses all source code infrastructure, and uploads malicious wheels directly to established namespaces. The package registry reflects an authentic upload from a trusted maintainer ID, leaving consumers with no registry-level warning flags.
Evaluating Defenses: Controls, Trade-Offs, and Optimal Architecture
Mitigating zero-commit package hijacks requires evaluating publishing controls against consumer-side runtime isolation. No single mechanism eliminates supply-chain risk entirely.
Publishing-Side Mitigations
- OIDC Trusted Publishing with Strict Environment Bindings: Replaces long-lived bearer tokens with ephemeral JWTs. Ephemeral tokens expire within minutes and are tied strictly to explicit GitHub Actions environment names, workflow paths, and tagged release refs. Failure Mode: Does not prevent compromise if the build runner itself is breached during execution by a poisoned build-time dependency.
-
Cryptographic Build Provenance & Attestations (Sigstore/SLSA): Generates a signed, publicly verifiable attestation linking the SHA-256 digest of the uploaded
.whlfile directly to the immutable Git commit hash and repository execution context that generated it. Failure Mode: Useless unless consuming package managers (pip,uv) explicitly enforce attestation verification prior to installation. Currently, standardpip installdoes not block un-attested wheels by default.
Consumer-Side Mitigations
-
Hash-Pinned Installation (
--require-hashes): Forcespipto verify the exact cryptographic hash of downloaded wheel archives against a known-good lockfile (e.g., generated viapip-compile --generate-hashes). Mechanism: If an attacker uploads a malicious version1.26.1to PyPI, its SHA-256 hash will not match the hash recorded in the consumer's audited lockfile.pipaborts extraction before any file hitssite-packages. Failure Mode: Stops working if developers automatically update lockfiles using un-verified upstream metadata or install unpinned dependencies in ephemeral CI containers. - Private Registry Proxies with Cooldown Periods: Routes internal dependency resolution through a private proxy (e.g., Nexus, Artifactory, or DevPi) configured with an automatic ingestion delay (e.g., 24 to 48 hours for new package versions). Mechanism: Creates a buffer window allowing automated security scanners, community detection, and vendor takedowns to remove malicious uploads before they reach internal pipelines. Failure Mode: Delays critical zero-day security patches for downstream libraries.
Decision Rules for Engineering Teams
Rule 1 (Publishers): If publishing Python packages to PyPI, terminate all long-lived API tokens immediately. Implement OIDC Trusted Publishing tied strictly to protected GitHub release environments with mandatory multi-party code review. Generate SLSA provenance attestations on every release.
Rule 2 (Consumers - Automated Environments): If running Python inside CI/CD, container builds, or AI agent runtimes holding ambient cloud credentials, prohibit direct, unpinned pip install commands against public indices. Require hash-pinned lockfiles (--require-hashes) and execute untrusted builds inside isolated sandboxes stripped of environment credentials and metadata network access.
The Provenance Gap: Why Source Control, CI/CD, and Registry Trust Fall Out of Sync
When malicious versions of an official package land on PyPI without a single line changing in GitHub, the shock usually comes from assuming those two systems share a single source of truth. They do not. At the protocol and database level, GitHub stores source trees and commit graphs, while PyPI stores distribution archives (.whl zip files) mapped to project ownership records. When a request hits PyPI's upload endpoint, the registry checks whether the presented credential or OIDC identity claim holds upload permission for that package namespace—not whether the bytes inside the zip archive correspond to a reviewed commit hash or a clean CI runner.
That structural split is why a zero-commit compromise works so cleanly. An attacker who steals a long-lived PyPI API token, compromises a maintainer account, or abuses an overly permissive OIDC trust policy can assemble a wheel locally or in a rogue runner using flit, hatch, or setuptools, then push it directly via twine. The repository's git log never changes because the upload path never touches GitHub. For consumers pulling unpinned packages during container builds, CI jobs, or agent startup, the gap closes in milliseconds: pip extracts the archive into site-packages, and if the wheel drops a .pth file containing a line starting with import, Python's site.py initialization passes that line directly to exec() before application code even loads. In an environment holding ambient cloud credentials, environment variables, or reachability to instance metadata endpoints like 169.254.169.254, secrets get read and shipped out before any application exception handler can even catch a failure.
Why Provenance Is Not Mandatory Across PyPI
If mandatory commit-to-wheel cryptographic binding sounds obvious on paper, forcing it across an open registry breaks real workflows in predictable ways. Reproducible builds remain hard in Python: embedded timestamps, filesystem paths, compiler flags, and native C/Rust extension toolchains alter output bytes unless every stage is tightly normalized. Maintainers also publish from local workstations, non-GitHub forges, or emergency hotfix branches where strict CI provenance checks would stall an urgent patch. Registries avoid making strict provenance universal because false-positive rejections on legitimate uploads create severe operational friction and push maintainers toward workarounds rather than stronger security.
Approvals and signatures help, but each control fails on a specific boundary rather than acting as a silver bullet. Manual release approvals slow down response times and still fail if an approved build runner is compromised mid-flight. Cryptographic signing and SLSA-style provenance attestations bind a wheel's SHA-256 digest to a commit and build context, which catches tampered artifacts—yet attestations only stop bad installs if the consumer's tooling actually verifies them and the signing key or OIDC workflow itself wasn't hijacked by a wildcard branch rule or a poisoned build dependency.
Comparing Controls and Where Choice Errors Happen
Evaluating defenses by how they fail under pressure makes the trade-offs sharper than listing checkboxes:
- Long-lived API tokens vs. OIDC Trusted Publishing with strict environment bindings: Static bearer tokens remain valid from any network origin until manually revoked, turning a single leaked secret into immediate, unlogged upload authority. OIDC Trusted Publishing replaces static secrets with short-lived tokens tied to a specific workflow and environment, making it strictly stronger—provided the trust configuration restricts branches and protected environments instead of allowing wildcard refs or unreviewed workflow edits.
-
Unpinned
pip installvs. hash-pinned installs (--require-hashes) and proxy cooldowns: Unpinned installs fetch whatever version sits at the top of the index at build or startup time, letting a malicious upload reach automated runners within seconds. Hash-pinned lockfiles forcepipto compute and match the archive's SHA-256 digest before extraction, aborting on mismatch; internal registry proxies with a 24-to-48-hour ingestion cooldown add a buffer window so fast takedowns can strip bad releases before internal builds pull them, at the cost of delaying legitimate zero-day fixes. -
Ambient credentials in build/agent runtimes vs. isolated execution: Leaving cloud keys or metadata access exposed inside build containers or startup paths lets a
.pthhook exfiltrate live credentials immediately upon interpreter start. Stripping ambient secrets from untrusted build steps narrows the blast radius even if a bad artifact slips through.
The most common engineering mistake is treating a single control as complete protection. Teams wire up OIDC publishing while leaving a legacy API token active, adopt provenance metadata without enforcing hash or attestation checks on the consumer side, or lock down pip hashes while running untrusted code inside a container with broad cloud IAM permissions. Each error leaves an intact causal path from artifact extraction to secret exfiltration.
Decision Rules for Publishers and Consumers
Strong supply-chain hygiene requires pairing publisher-side binding with consumer-side verification and runtime isolation:
- If you publish packages: retire static API tokens and use OIDC Trusted Publishing bound to protected release environments and explicit tags, paired with build provenance attestations—unless your build environment allows unreviewed workflow changes or compromised build dependencies, in which case you must isolate the build runner and audit build-time dependencies first.
-
If you consume packages in CI, containers, or agent runtimes: enforce hash-pinned installations (
--require-hashes) through a controlled or time-delayed artifact proxy rather than running unpinnedpip installagainst public PyPI—unless you are testing in an ephemeral, credential-free sandbox where no production secrets or internal network routes exist. - If a workload handles sensitive cloud credentials: separate untrusted dependency resolution and startup initialization from high-privilege runtime credentials—unless the process has zero access to secrets and external network egress, where the impact of startup code execution is contained.
PyPI Package Hijack Without Repo Changes Exposes Source, Build, and Registry Trust Gaps
When malicious versions of a trusted Python package land on PyPI without a single commit appearing in the upstream Git repository, the failure is architectural rather than accidental. Source control tracks text and commit graphs, whereas a package registry stores distribution archives (.whl zip files) mapped to namespace upload permissions. Because PyPI's upload endpoint checks publisher authorization rather than verifying that archive bytes descend from a reviewed source commit, a leaked bearer token, a hijacked maintainer session, or an over-permissive pipeline can push a malicious wheel directly to users.
Once a victim runs pip install, the wheel extracts into site-packages. If the archive drops a .pth file containing a line starting with import, Python's standard library site.py scans site-packages during interpreter startup and passes that line directly to exec(). That execution happens before application code or exception handlers load. If the host or container holds ambient cloud credentials, environment variables, or network access to instance metadata endpoints, the hook reads those secrets in memory and sends them out in outbound requests before the program even starts doing its intended job.
Why Source History, Build Provenance, and Registry Trust Drift Apart
PyPI does not require a cryptographic link between a wheel archive and a verified Git commit because the registry was built to accept artifacts from many publishing workflows, not to act as a source-code mirror. Maintainers historically uploaded wheels built on laptops or custom runners using long-lived API tokens. Without mandatory build provenance—such as attestations binding a wheel's SHA-256 hash to a specific repository, workflow, and commit—the registry cannot tell whether a wheel came from clean source code or a rogue build.
Making strict provenance universal is blocked by real engineering friction. Byte-for-byte reproducible builds break easily when timestamps, build paths, compiler flags, or native extension toolchains vary across machines. Meanwhile, maintainers sometimes ship urgent fixes from non-standard branches or local environments; forcing rigid checks without escape hatches creates operational pressure to bypass controls or keep legacy tokens active alongside newer ones.
When a publishing credential is compromised, an attacker faces little friction pushing a malicious release under a familiar name. A static token works from anywhere until revoked, allowing direct uploads outside GitHub. Even with OIDC trusted publishing, loose trust rules—such as wildcard branch patterns or unprotected environments—let an unauthorized workflow or modified pipeline request a short-lived token and ship a tampered wheel. In automated builds and AI agent runtimes that pull packages straight from public indices and inherit cloud permissions, that trust gap turns a single rogue upload into immediate credential exposure.
Mandatory attestations, release signing, and approval gates reduce risk by raising the cost of forging trust, yet none of them solve the whole problem alone. A release approval slows down bad publishes, but if an approved build runner is compromised while assembling the wheel, the approval signs off on a poisoned artifact. Cryptographic signatures and provenance records catch altered bytes and untrusted origins, but only if the consumer's installer actually checks them and the signing keys or build dependencies were not subverted first.
Relying on plain pip install treats the public index as an authoritative source of truth. In practice, engineers need layered defenses: hash-pinned lockfiles so installers reject mismatched archives before extraction, internal package proxies with ingestion delays so obvious bad uploads get pulled before internal builds fetch them, and strict separation between dependency installation and high-privilege cloud credentials.
Closing the Blind Spots: Hardening Supply Chains Against Ghost Releases
Securing packages against out-of-band releases requires closing gaps at three points: how artifacts are published, how consumers verify what they download, and how much privilege a workload holds when code executes. Focusing on only one layer leaves parallel failure paths open—for example, adopting OIDC while leaving old API tokens active, or pinning hashes inside a container that still carries broad cloud permissions.
- OIDC Trusted Publishing with Strict Environment Bindings: Replaces static, long-lived API tokens with short-lived tokens tied to specific workflows, branches, and protected release environments. If a static token is stolen, it can be abused indefinitely until spotted; short-lived bound tokens limit issuance to defined contexts, though they still fail if wildcard rules allow untrusted refs or if a compromised build dependency alters the wheel during packaging.
- Cryptographic Build Provenance and Attestations: Binds a wheel's SHA-256 digest to its build origin and source commit so tampering after or outside the build leaves a cryptographic mismatch. It is more effective than manual approvals alone because it ties artifact bytes to build metadata rather than human sign-off, but it stops working if verification is disabled on the client or if the build environment itself is poisoned before signing.
-
Hash-Pinned Installs and Delayed Internal Proxies: Forces
pipto match downloaded archive digests against audited hashes and routes pulls through a controlled proxy with a brief ingestion delay. This beats unpinned public installs by halting extraction on hash mismatch and giving security teams time to catch rapid takedowns, though hash pinning fails if lockfiles are blindly updated with bad upstream metadata, and ingestion delays slow down urgent zero-day fixes. - Credential Isolation at Build and Startup: Strips high-privilege cloud secrets and sensitive metadata reachability from build runners and early initialization paths. Even if a malicious wheel executes a startup hook, removing ambient credentials and external egress starves the payload of secrets to read and channels to exfiltrate them, though isolation adds operational overhead when services genuinely need credentials during initialization.
For most teams, the optimal primary control is enforcing OIDC trusted publishing bound to protected release environments paired with hash-pinned installations (--require-hashes) on the consumer side. This combination is strongest because it eliminates reusable static credentials at publish time while forcing installers to verify exact artifact bytes before code lands in site-packages. It stops working if build dependencies or CI runners are compromised before artifact creation, if lockfiles are updated from unverified sources, or if legacy credentials remain enabled.
A common engineering mistake is treating a single control—like turning on OIDC, adding a manual approval step, or checking a signature—as complete protection while ignoring runner hygiene, lockfile integrity, or runtime privileges. When engineers assume a signed or approved package cannot carry a poisoned payload from a compromised build step or steal secrets from an over-privileged container, trust assumptions outrun actual verification.
Use these decision rules to match controls to real exposure:
- If publishing to a public registry from CI, then retire static API tokens and use OIDC trusted publishing scoped to protected release environments and explicit tags—unless build dependencies or CI runners are untrusted, in which case isolate the build environment and audit dependencies first.
-
If installing packages in CI, containers, or production builds, then enforce hash-pinned installs through a controlled proxy rather than unpinned public
pip install—unless running in a throwaway sandbox with no secrets and restricted network access. - If a service or agent runtime has access to cloud credentials or metadata endpoints, then separate dependency resolution and startup initialization from high-privilege credentials—unless the workload has zero sensitive secrets and no outbound network path to exfiltrate data.
Top comments (0)