Using non-PyPI Package Indices

By default, Python tools like pip install packages from PyPI, the Python Package Index. However, most tools allow you to use any server that implements the same API as PyPI (usually but not always the Simple Repository API). There are two main reasons you might want to use an alternative package index:

• An index that mirrors PyPI let you keep installing packages even if they are deleted from PyPI or PyPI goes offline.
• You can upload packages to a private index instead of PyPI if you do not want to make them public.

How to configure your tools depends on whether you are using an alternative index in place of or in addition to PyPI. Mirrors can be used either way, while a private index is usually used in addition to PyPI. Note that PyDist or a self-hosted devpi instance can act as both a mirror and a private index.

Unfortunately, you need to configure each of your tools individually.

Configuring pip

There are two configuration options for pip that allow it to use alternative indices:

• The index URL, which is where pip will initially check for packages. It defaults to https://pypi.org/simple.
• The extra index URLs which are where pip will check for packages if they are not found in the index url. If multiple are provided, they are checked in order.

If you are using a mirror, you can either set it as the index to install all packages through the mirror, or you can set it as an extra index to only use it as a backup. If you are using a private mirror to compliment PyPI, it is tempting to use it as an extra index. However, if a package by the same name as one of your private packages is published on PyPI, this will cause pip to install that package instead. Thus it is safest to set the private index's URL as the index and use PyPI as an extra index.

You can configure pip in a number of different ways; which you should choose depends on what is easiest to set in your infrastructure and how broadly you want the configuration to apply.

• Via the command line arguments --index-url and --extra-index-url
• Via the environment variables PIP_INDEX_URL and PIP_EXTRA_INDEX_URL (note that this can only be set once, so you can only add one extra index this way)
• Via the index-url and extra-index-url settings in a pip.conf file which can live:
  • In the root of your virtual environment (e.g. ./venv)
  • In your user config's pip subdirectory (usually ~/.config/pip on Linux)
  • In your system-wide config directory (/etc on Linux)

Configuring twine

The standard tool to upload Python packages is twine. However, only some
rivate python indices support uploading with twine—to my knowledge only PyDist and Gemfury do among hosted solutions. Other indices will generally offer their own python packages for deployment.

There are two ways to configure twine to upload to a repository other than PyPI:

• Via the command line --repository-url argument
• Via a .pypirc file in your home directory

The .pypirc file should look like:

[distutils]
index-servers =
    pydist

[pydist]
repository: <index-url>
username: <username>
password: <password>

If you include multiple index-servers in .pypirc, you can pass the name you gave the index server (pydist in the example above) to the --repository flag when uploading. The .pypirc file is convenient because you will not be prompted for a username/password.

Configuring pipenv

Pipenv is probably the most popular dependency-locking tool for Python. Some guides suggest using the PIP_INDEX_URL and PIP_EXTRA_INDEX_URL environment variables to configure the pipenv like you would pip, but this is not handled correctly and the maintainer told me not to use it.

Instead, you can use the [[source]] section at the top of your Pipfile:

[[source]]
url = '<index-url>'
verify_ssl = true
name = 'pydist'

If you want to use multiple package indices, you can include multiple [[source]] sections—when pipenv finds packages it tries them in the order they are specified, or if you declare a package like mypackage = { version="*", index="pydist" } it will try the specified index first. However, pipenv's handling of multiple indices is currently buggy.