Python project structure

UPDATE - the updated version of this project is here - https://github.com/unfor19/python-project/wiki

By the end of this blog post, you will be able to:

• Create a well-structured Python project
• Use relative and absolute imports in a Python project
• Invoke a specific module from the command-line (terminal)

Link to GitHub repository - unfor19/python-project

python-project

Python project structure, relative imports, absolute imports, packages, and modules. Let's make it simpler.

Getting Started

Executing modules from the project's root directory (top-level package)

• main.py

  meirgabay@~/python-project (master)$ python main.py
  My Path: python-project/main.py
  Insert your name: willy
  Hello willy


  meirgabay@~/python-project (master)$ python -m appy
  My Path: python-project/appy/__main__.py
  Insert your name: willy
  Hello willy

• appy/core/app.py

  # Contains relative imports - `..utils.message`
  meirgabay@~/python-project (master)$ python appy/core/app.py
  Traceback (most recent call last):
    File "appy/core/app.py", line 1, in <module>
      from ..utils import message
  ImportError: attempted relative import with no known parent package


  meirgabay@~/python-project (master)$ python -m appy.core.app
  My Path: python-project/appy/core/app.py
  Insert your name: willy
  Hello willy

• appy/utils/message.py

  # Doesn't contain relative imports, so no exceptions were raised
  meirgabay@~/python-project (master)$ python appy/utils/message.py
  My Path: python-project/appy/utils/message.py

  meirgabay@~/python-project (master)$ python -m appy.utils.message
  My Path: python-project/appy/utils/message.py

Questions and Answers (Q&A)

Project, Packages, Modules and Scripts, what are they?

• Project - a directory, also known as the top-level package, which contains packages and modules
• Package (in a project) - a directory which contains modules and/or packages (sub-directories)
• Script - a Python script (.py) which can be exected from the terminal
• Module - a Python script (.py) which can be imported with import and from

What about Packages which are not part of a project?

• Package (built-in) - a package which is shipped with Python and can be imported with import and from
• Package (pip) - a package which is installed with pip and can be imported with import and from. Think about it, pip stands for Package Installer for Python

How do I import Packages and Modules that I've created?

• Python project's packages and modules can be imported with relative paths from any module which is part of the same project. An example is available in appy/core/app.py

• If you intend to import a package or a module which is not part of the same project, you'll have to use absolute paths. This can be done with importlib, see this StackOverflow answer.

Do I need the __init__.py file?

• Short answer - no
• In previous versions of Python, you had to create the __init__.py file in each directory that you want to import as a package, they were called regular packages. From version 3.3+ it is not required anymore - Implicit Namespace Packages, packages without an __init__.py file are called namespace packages.

Why do relative imports raise a problem in pylint?

The error - Attempted relative import beyond top-level packagepylint(relative-beyond-top-level)

• Short answer - I don't know
• All I can say is that it doesn't happen with flake8

Is it possible to invoke a function from the terminal?

• Short answer - it depends

• Trying to invoke a function from the terminal, such as appy.core.app.main(), will raise the ModuleNotFound exception. A package must be imported before invoking one of its functions.
  

  meirgabay@~/python-project (master)$ python -m appy.core.app.main
  /Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'appy.core.app.main' (ModuleNotFoundError: __path__ attribute not found on 'appy.core.app' while trying to find 'appy.core.app.main')

• Since you can't invoke main() directly from the terminal, calling it from the if __main__ block enables executing it from the terminal. It's possible to pass arguments, but it's a bit ugly, read the docs to learn how. The following example attempts to execute the module appy.core.app, which in turn call its if __main__ block

  meirgabay@~/python-project (master)$ python -m appy.core.app
  My Path: python-project/appy/core/app.py
  Insert your name: willy
  Hello willy

• If the PWD is a subdirectory of the project, such as python-project/appy, an attempt to execute a module which contains relative imports, will raise the exception below. Remember, your PWD should always be the project's root directory, in this case it's python-project.

  # PWD is `appy`
  meirgabay@~/python-project/appy (master)$ python -m core.app

  Traceback (most recent call last):
    File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/Users/meirgabay/python-project/appy/core/app.py", line 1, in <module>
      from ..utils import message
  ValueError: attempted relative import beyond top-level package

• It doesn't happen when invoking message, since message doesn't use relative imports

  meirgabay@~/python-project/appy (master)$ python utils/message.py
  My Path: python-project/appy/utils/message.py


  meirgabay@~/python-project/appy (master)$ python -m utils.message
  My Path: python-project/appy/utils/message.py

• Invoking a function from the terminal is also possible by using the -c flag. Surprise, it's possible to pass arguments in a more intuitive way, for example app.main(my_arg1, my_arg2)

  meirgabay@~/python-project (master)$ python -c "import appy.core.app as app; app.main()"
  Insert your name: willy
  Hello willy

What are the available command-line flags in Python?

• Read the docs - using cmdline
• In this tutorial, we used both -c and -m flags

Why is it possible to execute python -m appy?

The appy/__main__.py file acts like the if __main__ code snippet, but on packages. This enables the appy package to be executed with python -m or with runpy

meirgabay@~/python-project (master)$ python -m appy
My Path: python-project/appy/__main__.py
Insert your name: willy
Hello willy

What's runpy and why do you use it in main.py?

The runpy package provides the ability to run modules from a module (Python script).

main.py

import runpy


def main():
    # import a package, and pass all current global variables to it
    appy_package = runpy.run_module(mod_name="appy", init_globals=globals())

    # import the function script_path() from the submodule message, and execute it
    appy_package['message'].script_path(__file__)

    # execute the function main(), which is located in appy/__main__.py
    appy_package['main']()


if __name__ == "__main__":
    main()

What's globals()?

The official definition from the docs

Return a dictionary representing the current global symbol table. This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called).

Example - Expand/Collapse

meirgabay@~/python-project (master)$ python
Python 3.8.2 (default, Jun 30 2020, 19:04:41)
[Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> globals()
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': }

Why do you have a weird path with pyenv when you run Python?

In some of the examples you might have seen that my Python binary is located in

/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python

This is because I'm using pyenv, the official definition from the docs

pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.

• pyenv great for checking backwards compatibility

• Switching to a different version

1. Install relevant version - pyenv install 3.7.7
2. Run export PYENV_VERSION=3.7.7

• For a day to day use

1. Install relevant version - pyenv install 3.8.2
2. Add export PYENV_VERSION=3.8.2 to your terminal's rc or _profile ($HOME/.bashrc, $HOME/.bash_profile, $HOME/.zshrc)

Examples - Expand/Collapse

meirgabay@~/python-project (master)$ export PYENV_VERSION=
meirgabay@~/python-project (master)$ pyenv versions
* system (set by /Users/meirgabay/.pyenv/version) # default OS Python
  3.7.7
  3.8.2
meirgabay@~/python-project (master)$ python --version
Python 2.7.16

# switching to a different version
meirgabay@~/python-project (master)$ export PYENV_VERSION=3.7.7
meirgabay@~/python-project (master)$ python --version
Python 3.7.7

# day to day use
meirgabay@~/python-project (master)$ source ~/.bash_profile
meirgabay@~/python-project (master)$ python --version
Python 3.8.2

Is there a good framework for creating a Python CLI?

• Short answer - yes! more than a few!
• The Click Framework is amazing! I used it in githubsecrets and frigga NOTE: you'll discover that I have __init__.py files in my projects, that's a nasty old habit, I promise I will drop them, I promise
• Read more about the available Python CLI frameworks in this great blog post - Building Beautiful Command Line Interfaces with Python

Where can I find a well-structured Python project?

• sampleproject - the official Python sample project
• Django - a complex Python project
• requests - a non-complex Python project

Final words

I hope that this blog post helped you in understanding how to create a well-structured Python project, and if it did, then Heart, Clap, Star and share!