In March a security researcher emailed us a list of hostnames and asked whether they were ours. They were: thirty eight of our CI runners. Each had installed a package called ledger-client, version 99.0.0, from the public Python index, and its setup script had sent the researcher the machine's hostname, user name and working directory.
ledger-client is ours. It is an internal library, published to our private package registry, used by about twenty services. Nobody had ever published it publicly, which is exactly why the researcher could. The name was free on the public index, so he took it, uploaded a harmless package with an absurdly high version number, and waited.
Our pipelines installed dependencies with pip, using the public index as the main one and our registry through extra-index-url. We read that option as a fallback: look in ours as well. Pip reads it differently. It gathers every candidate for a name from every index it has been given, treats them as one pool, and picks the highest version that satisfies the requirement. Most of our services asked for ledger-client>=2.4, and 99.0.0 satisfies that very well. Our registry offered 2.7.1, the public index offered 99.0.0, and pip did precisely what it was built to do.
The package ran code on install, as any Python source package may, inside a job whose environment held a registry token and deploy credentials for staging. It did not touch them. The next package with a spare name might.
We fixed the resolution first. Every pipeline and developer machine now talks to exactly one index, our registry, which proxies the public one and refuses to fetch from upstream any name that exists internally. All internal packages moved under a company prefix, and we registered that prefix and every old name on the public index as empty placeholders. Services install from lock files with hashes, and pip runs with require-hashes, so a package that was never reviewed cannot be installed even if it wins the version race. Install steps run before any secret is loaded into the job, with outbound traffic limited to the registry.
Two package sources are not a list in order of preference. They are one namespace, the public half of it is open to anyone, and a name you only ever use privately is still a name somebody else can register.
– Sergey Shinder
Top comments (0)