loading...

The skinny on Python dev environment and packaging (pip)

cocoroutine profile image hong ・4 min read

Welcome to the wild world of Python dev. Where nothing is consistent and there is no such thing as as consensus.

Don't worry I'll be your guide.

Background

I quietly released pyquilted which generates nice looking resumes from Yaml in PDF. I learned a lot about setting up a python dev environment and packaging an app to distribute on Pypi (pip) and now I am sharing that knowledge.

You can find out more about quilted resumes here.

Unittests

Running test cases were for some reason harder than it should be. Follow these steps to make your life easier.

Create a directory called 'test' in your project root.

Name all your tests 'test_*'.

Put an __init__.py so that the directory get's recognized as a package.

running all tests on Python 2 and 3
python -m unittest discover

running one test
python -m unittest test.test_name

Venv vs VirtualEnv vs PipEnv

These are all ways to isolate your python pip environment so you do not pollute your main system python. Uninstalling a package and it's dependecies is a pain in pip. This avoids this by allowing you to easily delete and create new envs.

But which one do you go with?

Venv vs VirtualEnv vs PipEnv

Just go with venv. It comes standard with python and is officially recommended post 3.3+. It also has less gotchas than virtualenv and I don't want to be gotten.

to create a new venv folder
python -m venv venv

to load the environment
source venv/bin/activate

turn off the venv
deactivate

Pipvenv is nice, but it's not standard and is based on virtualenv.

Python 2.7 and Venv

Venv doesn't support python 2.7, bummer.

Simple solution
pip install py2venv

This installs virutalenv for you and creates a wrapper around virtualenv. Meaning you can use the venv command. Great!

Where to put your venv folder?

If you are just running one venv per project, put it in the same directory as the project root and name the folder 'venv'. Make sure in your .gitignore you have added 'venv'—on github and gitlab this is added for you.

one venv environment per project
project_root/venv

The issues comes when you want different venv environments for different python versions you are testing. Here's a few options

multiple venv per project
project_root/venv/{{python_version}}

alternative, venv in home folder
~/venv/{{project_name}}_{{python_version}}

setup.py

Setup.py is like a makefile for your app. It's based off of setuptools. It normally comes pre-installed. If not you can install it like so

pip install setuptools

The documentation can be found here. Some parts are either confusing or not working as expected. I noted the parts here.

You can see an example of my setup.py in the pyquilted source

Command line entry point

You want to run your app in the terminal using your app's sweet name. This was a lot tricker than it should be. After endless searching, this method below worked for me.

In your project root, create a file named main.py. Have one function named main that calls your run function. An example here.

Add the following into your setup.py

entry_points={
    'console_scripts': ['command=pacakge.main:main']
}

Where 'command' is the command you want to use. 'package' is your package name.

For example my pyquilted entry point
'console_scripts': ['pyquilted=pyquilted.main:main']

Markdown readme

Pypi now supports markdown. Use a with open file handler to set your readme doc to a variable.

long_description=readme,
long_description_content_type='text/markdown'

Including non py files

There are two ways to add files to either your site_packages for data or your source package for test cases and such.

package data

Use the below format for adding data files to your site_packages.

package_data={
    'package_name': ['directory/*.pattern']
},
include_package_data=True

MANIFEST.in

This includes files in your source pacakge that are not in your site_packages folder. The format is a little weird.

include file   # include single file
recursive-include directory pattern # include a directory and files

requirements.txt

Skip it. It's not really necessary, but simple to make.

Enter Pyenv

Still crying over the disaster that was migrating from Python 2 to 3. Me too, join the club. We stil have to support Python 2 and test other versions of 3 as well. How to solve multiple python versions? We could learn Vagrant or Docker. Nah that's way overkill.

Solution Pyenv.

Install Pyenv by following instructions here pyenv.

list installable python versions
pyenv install --list

install version
pyenv install 2.7.15

list versions installed
pyenv versions

switch to python version
pyenv global 2.7.15

Debian, Ubuntu and others

On Debian, Ubuntu and other distros, you will get an error about ssl when trying to build older versions of python. The issue is that libssl 1.1 is not compatble with Python < 3.3. The solution is to download the dev headers for libssl 1.0 and link them manually when running pyenv install, per this thread.

CFLAGS="-I/path_to/libssl1.0-dev/include -I/path_to/libssl1.0-dev/include/x86_64-linux-gnu" \
LDFLAGS="-L/path_to/libssl1.0-dev/lib/x86_64-linux-gnu" pyenv install 2.7.15

Eggs everywhere

To stop python from creating eggs that you accidently upload to Pypi when you are testing.

Use the below command
pip install .

instead of this command
python setup.py install

Building and Wheels

Honestly wheel files are not necessary. The difference in install time between source and binary is like a few seconds. However it's really easy to build binary wheels so why not.

make sure you have wheel installed
pip install wheel

run this to build your packages
python setup.py sdist bdist_wheel

Finally upload to Pypi

Create an account on Pypi.org. Then install twine which is the official pypi app
pip install twine

Upload with this command
twine upload --skip-existing dist/*

The key is skip existing files, otherwise you get errors when uploading the dist folder with existing files.

Congrats!

Congrats you beat the final boss! Pat yourself on the back for finally deciphering python dev environment.

Posted on by:

cocoroutine profile

hong

@cocoroutine

red rover, red rover, I call you over

Discussion

markdown guide