Poetically Packaging Your Python Project

It wasn't long ago that we Hackers were singing the praises of Pipenv: Python's seemingly superior dependency manager at the time. While we hold much love in hearts, sometimes there is love to go around. We just so happen to be fair weather fans, which reminds me: what has Pipenv done for me lately?

As you've probably guessed (considering its a piece of software), nothing much. Well, there was that time when pip upgraded from v.18 to v.18.1, which broke Pipenv entirely with almost minimal acknowledgment (for all I know this might still be broken). As our live seemed to fade, a miracle emerged from the heans: a young, smart, attractive alternative to Pipenv that's been whispering in my ear, and promising the world. Her name is Poetry.

What Light Through Yonder GitHub Breaks?

Poetry stems from the genuine frustration that comes with not only managing environments and dependencies in Python, but the fact that even solving this problem (albeit poorly) still doesn't solve the related tasks needing fulfillment when creating respectable Python projects. Consider Node's package.json: a single file which contains a project's metadata, prod dependencies, dev dependencies, contact information, etc. Instead, Python projects usually come with the following:

Setup.py

If you've never bothered to publish a package to PyPI before, there's a decent chance you may not be very familiar with some of the nuances that come with setup.py or why you'd bother creating one. This is a losing mentality: we should assume that most (or some) of the things we build might become useful enough to distribute some day.

Thus, we get this monstrosity:

from setuptools import setup, find_packages, tests_require, packages, name

with open("README", 'r') as f:
    long_description = f.read()

setup = (
    name='Fake Project',
    version='1.0',
    description='A fake project used for example purposes.',
    long_description=long_description,
    author='Todd Birchard',
    author_email='[e-mail]',
    maintainer='Some Loser',
    maintainer_email='[e-mail],
    url="https://github.com/toddbirchard/fake-project",
    license='MIT',
    include_package_data=True,
    package_dir={'application'}
    packages=['distutils', 'modules'],
    tests_require=["pytest"],
    cmdclass={"pytest": PyTest},
    classifiers=[
          'Development Status :: 2 - Beta',
          'Environment :: Console',
          'Environment :: Web Environment',
          'Intended Audience :: End Users/Desktop',
          'Intended Audience :: Developers',
          'Intended Audience :: System Administrators',
          'License :: OSI Approved :: Python Software Foundation License',
          'Operating System :: MacOS :: MacOS X',
          'Operating System :: Microsoft :: Windows',
          'Operating System :: POSIX',
          'Programming Language :: Python',
          'Topic :: Communications :: Email',
          'Topic :: Office/Business',
          'Topic :: Software Development :: Bug Tracking',
          ],
)

Many of the metadata fields are rather self-explanatory. But what about the fields related to package dependencies, such as package_dir or packages? Wasn't this already handled in our Pipfile? On top of that, we need to specify then the test suite we're using via tests_require and cmdclass? Short answer: pretty much.

Setup.cfg

The real joke with setup.py is that it needs its own configuration file: yes, a configuration file for your configuration file. setup.cfg, as the name suggestions, sets even more granular configurations for the things mentioned in setup.py, such as how pytest should be handled, etc. Let's not get into it, but here's an example:

[coverage:run]
omit = */test/*

[flake8]
exclude = *.egg*,.env,.git,.tox,_*,build*,dist*,venv*,python2/,python3/
ignore = E261,W503
max-line-length = 121

[tool:pytest]
minversion = 3.2
addopts =
  # --fulltrace
  # -n auto
  --cov-config=setup.cfg
  --cov=httplib2
  --noconftest
  --showlocals
  --strict
  --tb=short
  --timeout=17
  --verbose
  -ra

Pipfile and Pipfile.lock

If you have been using Pipenv, you'll recognize these files as being responsible for setting your Python version and dependencies. But wait- didn't we also need to specify dependencies in setup.py? Yes, we did. There is no God, but if there were, he'd probably hate you. Here's all the work you'd need to do creating an acceptable Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
Flask-SQLAlchemy = "*"
psycopg2 = "*"
psycopg2-binary = "*"
requests = "*"
configparser="*"
mapbox="*"
flask="*"
pandas="*"
Flask-Assets="*"
libsass="*"
jsmin="*"
dash_core_components="*"
dash-table="*"
dash_html_components="*"
dash="*"
flask-session="*"
flask-redis="*"
gunicorn="*"
pytest-flask="*"

[dev-packages]

[requires]
python_version = "3.7.1"

But wait, there's more!

Requirements.txt

Because the Pipfile format has not been adopted as a standard for dependency management, we still need to create a requirements.txt file if we want to deploy our application to respectable hosts such as Google App Engine or what-have-you. So now we have this ugly son of a bitch from the stone age to deal with as well:

atomicwrites==1.2.1
attrs==18.2.0
boto3==1.9.75
botocore==1.12.75
CacheControl==0.12.5
certifi==2018.11.29
chardet==3.0.4
Click==7.0
configparser==3.5.0
dash==0.35.1
dash-core-components==0.42.0
dash-html-components==0.13.4
dash-renderer==0.16.1
dash-table==3.1.11
decorator==4.3.0
docutils==0.14
Flask==1.0.2
Flask-Assets==0.12
Flask-Compress==1.4.0
Flask-Redis==0.3.0
Flask-Session==0.3.1
Flask-SQLAlchemy==2.3.2
gunicorn==19.9.0
idna==2.8
ipython-genutils==0.2.0
iso3166==0.9
itsdangerous==1.1.0
Jinja2==2.10
jmespath==0.9.3
jsmin==2.2.2
jsonschema==2.6.0
jupyter-core==4.4.0
libsass==0.17.0
mapbox==0.17.2
MarkupSafe==1.1.0
more-itertools==5.0.0
msgpack==0.6.0
nbformat==4.4.0
numpy==1.15.4
pandas==0.23.4
plotly==3.5.0
pluggy==0.8.0
polyline==1.3.2
psycopg2==2.7.6.1
psycopg2-binary==2.7.6.1
py==1.7.0
pytest==4.1.0
pytest-flask==0.14.0
python-dateutil==2.7.5
pytz==2018.9
redis==3.0.1
requests==2.21.0
retrying==1.3.3
s3transfer==0.1.13
six==1.12.0
SQLAlchemy==1.2.15
traitlets==4.3.2
uritemplate==3.0.0
urllib3==1.24.1
webassets==0.12.1
Werkzeug==0.14.1

MANIFEST.in

YES, THERE'S MORE. If you're not bothered by now, please leave this blog immediately. The job market is ripe for neckbeards who take pleasure in unnecessary complexity. Until the robots take over, this blog is for humans.

Anyway, there's an entire file dedicated to including files in your project which aren't code. We're entering comically ridiculous territory:

include README.rst
include docs/*.txt
include funniest/data.json

It's a Bird! It's a Plane! Its... A Single, Sophisticated Config File?

I hope you're thoroughly pissed off after looking back at all the things we've let slide by year after year, telling ourselves that this patchwork of standards is just fine. Cue our hero: the creator of Poetry:

Packaging systems and dependency management in Python are rather convoluted and hard to understand for newcomers. Even for seasoned developers it might be cumbersome at times to create all files needed in a Python project: setup.py,requirements.txt, setup.cfg, MANIFEST.in and the newly added Pipfile. So I wanted a tool that would limit everything to a single configuration file to do: dependency management, packaging and publishing.

Oh God yes, but HOW?!?!

Introducing pyproject.toml

Poetry is built around a single configuration dubbed pyproject.toml which has become an accepted standard in the Python community by way of PEP 518. With the weight of the Python development community itself, it's safe to say this isn't another fad and is worth using.

Here's an example .toml file from the Poetry Github repository:

[tool.poetry]
name = "my-package"
version = "0.1.0"
description = "The description of the package"

license = "MIT"

authors = [
    "Sébastien Eustace <[e-mail]>"
]

readme = 'README.md' # Markdown files are supported

repository = "https://github.com/sdispater/poetry"
homepage = "https://github.com/sdispater/poetry"

keywords = ['packaging', 'poetry']

[tool.poetry.dependencies]
python = "~2.7 || ^3.2" # Compatible python versions must be declared here
toml = "^0.9"
# Dependencies with extras
requests = { version = "^2.13", extras = ["security"] }
# Python specific dependencies with prereleases allowed
pathlib2 = { version = "^2.2", python = "~2.7", allows-prereleases = true }
# Git dependencies
cleo = { git = "https://github.com/sdispater/cleo.git", branch = "master" }

# Optional dependencies (extras)
pendulum = { version = "^1.4", optional = true }

[tool.poetry.dev-dependencies]
pytest = "^3.0"
pytest-cov = "^2.4"

[tool.poetry.scripts]
my-script = 'my_package:main'

In addition to covering the scope of all previously mentioned files, using pyproject.toml with Poetry also covers:

• Auto-populating the exclude section from values found in .gitignore
• The addition of a keywords section to be included with the resulting PyPi package
• Support for version numbers using any syntax, such as wildcard (*) or carrot (^1.0.0) syntax
• Auto-detection for virtual environments, thus a global install that can be used within envs

Creating Poetic Art

Are we all fired up yet? Right: let's change our workflow forever.

Installation

To install Poetry on OSX, use the following:

$ curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

This will create an addition to your ~/.bash_profile. Restart your terminal and verify the installation:

$ poetry --version
Poetry 0.12.10

Creating a New Python Project

Navigate to whichever file path you'd like your new project to call home. To get started, all we need next is the following command:

poetry new my-package

Ready for a breath of fresh air? This command generates a basic project structure for you- something that's been missing from Python for a long time when compared to similar generators for Node or otherwise. The resulting project structure looks as such:

my-package
├── pyproject.toml
├── README.rst
├── my_package
│ └── __init__.py
└── tests
    ├── __init__.py
    └── test_my_package

Of the beautiful things happening here, the only one we haven't touched on yet is Poetry's built-in integration with pytest. Oh, happy day!

Alternative Interactive Installation Method

If you'd prefer a bit more handholding, feel free to use poetry init in an empty directory (or a directory without the existing .toml file) to be walked through the creation process:

$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [my-package]: Great Package
Version [0.1.0]:
Description []: Great package for great people.
Author [Todd Birchard <[e-mail]>, n to skip]:
License []: MIT
Compatible Python versions [^2.7]: ^3.7

Would you like to define your dependencies (require) interactively? (yes/no) [yes] no

Would you like to define your dev dependencies (require-dev) interactively (yes/no) [yes] no

Generated file

[tool.poetry]
name = "Great Package"
version = "0.1.0"
description = "Great package for great people."
authors = ["Todd Birchard <[e-mail]>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Do you confirm generation? (yes/no) [yes] yes

Managing Dependencies in pyproject.toml

If you're familiar with Pipfiles, pyproject.toml handles dependencies the same way. Just remember that poetry install installs your listed dependencies, and poetry update will update dependencies in poetry.lock to their latest versions.

Carry on my Wayward Son

I could spend all day copy-pasting general usage from the Poetry Github page, but I think my work here is done. Do yourself a favor andtake a look at the Github repo to make your life easier forever. Or at least until the next replacement solution comes along.