Reflect as You Work: My Python Project Workflow

One of the apprenticeship patterns in Apprenticeship Patterns is Reflect as You Work. This pattern is about introspecting on how you work regularly. Doing this often allows developers to notice how their practices have changed and even how they haven't. This isn't just about observing yourself. As the book says,

"Unobtrusively watch the journeymen and master craftsmen on your team. Reflect on the practices, processes, and techniques they use to see if they can be connected to other parts of your experiences. Even as an apprentice, you can discover novel ideas simply by closely observing more experienced craftsmen as they go about their work." p. 36

I have been thinking about my own practices and those of others around me. The workflow I follow when I create new Python projects particularly stands out because I learned it from sitting with another engineer. I noted what they did and asked questions. Then, I went back to my desk and tried it myself while taking more notes. I followed the resulting workflow so many times that the steps now flow from my fingertips with ease.

I think there could be ways to optimize even this workflow but first I am going to note it down here for the potential future reader and for future me to look back on!

P.S. Many of the extra details I included here I learned from my colleagues. A big thank you to them for sharing what they know with me 💓

---

Prerequisites

• pyenv is installed.

New Python Project Checklist

1. Install a specific Python version.
2. Create a project directory. Go to the directory.
3. Set the Python version for the project.
4. Create a virtual environment.
5. Activate the virtual environment.
6. Install dependencies.
7. Save packages.
8. Run the code.

Note: this workflow should work on MacOS.

---

PREREQUISITE: Install pyenv.

Mac OS X comes with Python 2.7 out of the box. If you haven't fiddled with anything, you should be able to open up a Terminal window and type in python --version and get some 2.7 variant. You probably don't want to use the version of Python that comes shipped with your OS (Operating System) though. There are many reasons for this like that the version may be out of date. I have even come across an important library that was missing.

Not only do you want to avoid using the version of Python that is shipped with your machine, in your work you will need to have several different versions of Python installed at once. For example, perhaps one codebase is using an older version of Python due to some library dependency. Upgrading the version of Python you are using for that project could require some refactoring of that project that you haven't prioritized. At the same time, you may be using a newer Python version on other projects because you want to take advantage of shiny new features.

Having several Python versions installed on your machine is a realistic scenario for a Python developer. Managing these versions effectively is important.

There are instructions on how to install pyenv here.

When you run a command like python or pip, your operating system searches through a list of directories to find an executable file with that name. This list of directories lives in an environment variable called PATH, with each directory in the list separated by a colon...

pyenv works by inserting a directory of shims at the front of your PATH so that when you call python or pip these shims are the first thing your OS finds. The commands you enter are, then, intercepted and sent to pyenv which decides which version of Python to use for your command based on some rules.

Follow the instructions to install pyenv. Make sure you follow the rest of the post-installation steps under Basic GitHub Checkout even if you use Homebrew to install. When I was installing I found that I had a .bashrc AND .bash_profile. Here is an article on the difference between them and when either file is used. If after following the instructions, you type in pyenv and do not get something like the following, go back and make sure you set the other bash file:

flo at MacBook-Pro in ~ $ pyenv
pyenv 1.2.8
Usage: pyenv <command> [<args>]

Some useful pyenv commands are:
   commands    List all available pyenv commands
   local       Set or show the local application-specific Python version
   global      Set or show the global Python version
   shell       Set or show the shell-specific Python version
   install     Install a Python version using python-build
   uninstall   Uninstall a specific Python version
   rehash      Rehash pyenv shims (run this after installing executables)
   version     Show the current Python version and its origin
   versions    List all Python versions available to pyenv
   which       Display the full path to an executable
   whence      List all Python versions that contain the given executable

See `pyenv help <command>' for information on a specific command.
For full documentation, see: https://github.com/pyenv/pyenv#readme

Step 1: Install a specific Python version.

Suppose I'm creating a script that will open the latest xkcd comic in a web browser. I'm going to run it with Python 3.7.0.

flo at MacBook-Pro in ~ $ pyenv install 3.7.0
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.0.tar.xz...
-> https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
Installing Python-3.7.0...
python-build: use readline from homebrew
Installed Python-3.7.0 to /Users/flo/.pyenv/versions/3.7.0

Step 2: Create a project directory. Go to the directory.

flo at MacBook-Pro in ~ $ mkdir Documents/comic-creator
flo at MacBook-Pro in ~ $ cd Documents/comic-creator/

Step 3: Set the Python version for the project.

First, look at the files in the folder, even the hidden files (-la will show hidden files).

flo at MacBook-Pro in .../comic-creator $ ls -la
total 0
drwxr-xr-x   2 flo  staff    64 Apr 12 21:12 .
drwx------+ 33 flo  staff  1056 Apr 12 21:12 ..

Now, set the Python version for the project. Now you can see a hidden file (hidden files start with a dot). When you look inside .python-version, you can see the version we set.

flo at MacBook-Pro in .../comic-creator $ pyenv local 3.7.0
flo at MacBook-Pro in .../comic-creator $ ls -la
total 8
drwxr-xr-x   3 flo  staff    96 Apr 12 21:16 .
drwx------+ 33 flo  staff  1056 Apr 12 21:12 ..
-rw-r--r--   1 flo  staff     6 Apr 12 21:16 .python-version
flo at MacBook-Pro in .../comic-creator $ cat .python-version 
3.7.0

Step 4: Create a virtual environment.

Just as you may have several Python versions installed on your machine, you may also have different versions of Python packages installed. Imagine the dependency graph for one of your projects looks like this:

requests==2.21.0
  - certifi [required: >=2017.4.17, installed: 2019.3.9]
  - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
  - idna [required: >=2.5,<2.9, installed: 2.8]
  - urllib3 [required: >=1.21.1,<1.25, installed: 1.24.1]

In another project, you may be using a different version of requests which depends on a different version of certifi. By using virtual environments, we can keep package installations isolated by project.

A virtual environment is a Python environment such that the Python interpreter, libraries and scripts installed into it are isolated from those installed in other virtual environments, and (by default) any libraries installed in a “system” Python, i.e., one which is installed as part of your operating system. Python venv docs

So, first, you can verify (again) that we correctly set the Python version for the project. Now, create a virtual environment by calling venv and call that new environment venv. You can now see the environment is created.

flo at MacBook-Pro in .../comic-creator $ python --version
Python 3.7.0
flo at MacBook-Pro in .../comic-creator $ python -m venv venv
flo at MacBook-Pro in .../comic-creator $ ls
venv

Step 5: Activate the virtual environment.

Look inside venv. Then, look inside venv/bin. bin stands for binary. In Linux/Unix-like systems, executable programs needed to run the system are found in /bin. Similarly, Python executable programs are stored in bin.

Activate the virtual environment with source.

source is a Unix command that evaluates the file following the command executed in the current context... Frequently the "current context" is a terminal window into which the user is typing commands during an interactive session. The source command can be abbreviated as just a dot (.) in Bash and similar POSIX-ish shells. Wikipedia

This means that if you open a new Terminal window, you will need to source the activate file again to activate the virtual environment in that window! Also note that you can type in . venv/bin/activate and it will do the exact same thing as source venv/bin/activate.

flo at MacBook-Pro in .../comic-creator $ ls venv/
bin        include    lib        pyvenv.cfg
flo at MacBook-Pro in .../comic-creator $ ls venv/bin/
activate         activate.csh     activate.fish    easy_install     easy_install-3.7 pip              pip3             pip3.7           python           python3
flo at MacBook-Pro in .../comic-creator $ source venv/bin/activate

Let's look at activate:

flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ cat venv/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
    # reset old environment variables
    if [ -n "${_OLD_VIRTUAL_PATH:-}" ] ; then
        PATH="${_OLD_VIRTUAL_PATH:-}"
        export PATH
        unset _OLD_VIRTUAL_PATH
    fi
    if [ -n "${_OLD_VIRTUAL_PYTHONHOME:-}" ] ; then
        PYTHONHOME="${_OLD_VIRTUAL_PYTHONHOME:-}"
        export PYTHONHOME
        unset _OLD_VIRTUAL_PYTHONHOME
    fi

    # This should detect bash and zsh, which have a hash command that must
    # be called to get it to forget past commands.  Without forgetting
    # past commands the $PATH changes we made may not be respected
    if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then
        hash -r
    fi

    if [ -n "${_OLD_VIRTUAL_PS1:-}" ] ; then
        PS1="${_OLD_VIRTUAL_PS1:-}"
        export PS1
        unset _OLD_VIRTUAL_PS1
    fi

    unset VIRTUAL_ENV
    if [ ! "$1" = "nondestructive" ] ; then
    # Self destruct!
        unset -f deactivate
    fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="/Users/flo/Documents/comic-creator/venv"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
# this will fail if PYTHONHOME is set to the empty string (which is bad anyway)
# could use `if (set -u; : $PYTHONHOME) ;` in bash
if [ -n "${PYTHONHOME:-}" ] ; then
    _OLD_VIRTUAL_PYTHONHOME="${PYTHONHOME:-}"
    unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT:-}" ] ; then
    _OLD_VIRTUAL_PS1="${PS1:-}"
    if [ "x(venv) " != x ] ; then
        PS1="(venv) ${PS1:-}"
    else
    if [ "`basename \"$VIRTUAL_ENV\"`" = "__" ] ; then
        # special case for Aspen magic directories
        # see http://www.zetadev.com/software/aspen/
        PS1="[`basename \`dirname \"$VIRTUAL_ENV\"\``] $PS1"
    else
        PS1="(`basename \"$VIRTUAL_ENV\"`)$PS1"
    fi
    fi
    export PS1
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then
    hash -r
fi

Step 6: Install dependencies.

If you haven't come across "dependencies", this word is often used to say that something is dependent on something else... Makes sense. In our case, our Python project will depend on installing various libraries that don't come already bundled with Python 3.7.0.

This is what our code looks like:

import json
import sys
import webbrowser

import requests

# url of latest xkcd comic
URL = 'http://xkcd.com/info.0.json'

if __name__ == '__main__':
    response = requests.get(URL)
    if response.status_code == requests.codes.ok:
        content = json.loads(response.text)
        print('Comic is located at {}'.format(content['img']))
        webbrowser.open(content['img'])
    else:
        print('Error: \n {}'.format(response.text))
        sys.exit()

Create a file comic_popup.py in the project and add this code. If you try to run the code you will get an error. requests module isn't installed. Let's install it.

flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ touch comic_popup.py
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ ls
comic_popup.py venv
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ pip install requests
Collecting requests
  Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, idna, certifi, requests
Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.21.0 urllib3-1.24.1
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Step 7: Save packages.

Notice what is printed when you enter pip freeze. This command outputs installed packages in requirements format ({library-name}={version}). In the next line, redirect that output to a file called requirements.txt using >. A single > will overwrite the contents of the file if the file already existed. Using >> would append to the contents of an already existing file.

You don't have to call the file requirements.txt but that is what most Python developers use so follow the convention! More on requirements files here.

You may also notice that requests isn't the only library outputted by pip freeze. The other libraries are libraries that requests depends on so when you install requests you must install the others for requests to work. These other libraries are referred to as transitive dependencies.

flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ pip freeze
certifi==2019.3.9
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ pip freeze > requirements.txt
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ cat requirements.txt 
certifi==2019.3.9
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ pip --help

Usage:   
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  config                      Manage local and global configuration.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort).
  --trusted-host <hostname>   Mark this host as trusted, even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.
  --no-color                  Suppress colored output

Step 8: Run the code.

That's it. You should be able to run the code now. You would be able to run it as soon as you install the dependencies it needs but don't forget to save your requirements!

flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ python comic_popup.py 
Comic is located at https://imgs.xkcd.com/comics/election_commentary.png

Now, if you save your code to a repo, anyone can pull the code and run it. Add a README.md and include which version of Python to use to run the code. The next developer will set up the right Python version and install the requirements by running pip install -r requirements.txt.

Don't include the .python-version file in the repo because the file is pyenv specific and other developers may have their own way to manage Python versions.

As a rule of thumb, I don't include files that are specific to me like configuration files for various IDEs (Integrated Development Environments) because they clutter up the repository. Ignore these files in your repository by adding and configuring a .gitignore file.

---

That's my development workflow when I start a Python project! I included some developer best practices where I felt it fit. I also explained as much context as I felt appropriate. I encourage you to try out different ways of doing the same thing to see the pros and cons of each.

Feel free to ask any questions! I'd love to chat about best practices and what works for you as well. So many parts of our workflows are by convention or because that's the way we first learned it or we don't know any better. I'd love to hear from you!

Comments

[rhymes]

Feel free to ask any questions! I'd love to chat about best practices and what works for you as well.

I think you've described the workflow well! I don't have experience with the builting venv, I'd always use virtualenv. I honestly don't know which is the difference.

I've since moved on from pip freeze to Pipenv mostly because it integrates pip, pyenv and virtualenv/venv, it provides a lock file for dependencies and can easily separate runtime dependencies from those in use only in development.

I have the feeling there are as many combination of "managing Python packages" as there are stars. I've heard about anaconda/conda also but I've never used it.

[Flo Comuzzi]

Thanks for sharing! I had heard of Pipenv but hadn't looked into it further. Will start playing with it now!

[Flo Comuzzi]

Ugh, pipenv is soooo sweet. Loving the dependency graph feature too!

[rhymes]

Glad you're liking it! That's really neat!

These two aliases I set in my shell might be of help (still hope they'll add them at some point):

pipenv-installed='pipenv graph --bare | grep "^\w" | sort --ignore-case'
pipenv-outdated='pipenv update --dev --outdated --dry-run'

[darkmage]

Thanks for the aliases!

[Flo Comuzzi]

Thanks for the aliases!!!!

[darkmage]

I am seconding pipenv! I was introduced recently, and it solves a problem that I will eventually have to deal with: getting my project to run on other peoples' machines easily during development!

[Flo Comuzzi]

Hi John, you tried getting it working with pipenv and then ended up using venv?