Python Project - Command Line

Goal

Covering the possible ways to execute a Python module (script) from the command-line (terminal).

Requirements

1. You are familiar with Packages and Modules
2. It would be easier to understand the examples if you run the examples by yourself - use this project as a template

Getting Started

We'll go over the possible way to execute scripts and modules from the command-line

Executing Scripts And Modules

In the following examples, all the modules are executed from the project's root directory (top-level package). This is important since the root dir has access to all the packages (sub-directories) in the project.

Executing the main.py script and appy as a module with src/appy/__main__.py

# Executing as a script - works as long as there are no relative imports of parent packages
(python-project) $ python main.py
My Path: python-project/main.py
Created the file: /Users/meirgabay/python-project/meirg-ascii.txt
Insert your name: willy wonka

Hello Willy Wonka, here's the cat fact of the day:
Jaguars are the only big cats that don't roar.

# Executing as a module - works because of the __main__.py file
(python-project) $ python -m src.appy
My Path: python-project/src/appy/__main__.py
Created the file: /Users/meirgabay/python-project/meirg-ascii.txt
Insert your name: willy wonka

Hello Willy Wonka, here's the cat fact of the day:
A cat has more bones than a human being; humans have 206 and the cat has 230 bones.

Executing the module src/appy/core/app.py from the root directory

# Contains relative imports - `..utils.message`
# Excuting as a script - causes issues with relative imports of parent packages
(python-project) $ python src/appy/core/app.py
Traceback (most recent call last):
  File "src/appy/core/app.py", line 1, in <module>
    from ..utils import message, img_ascii
ImportError: attempted relative import with no known parent package

# Excuting as a module - works perfect with relative imports
(python-project) $ python -m src.appy.core.app
My Path: python-project/src/appy/core/app.py
Created the file: /Users/meirgabay/python-project/meirg-ascii.txt
Insert your name: willy wonka

Hello Willy Wonka, here's the cat fact of the day:
Siamese kittens are born white because of the heat inside the mother's uterus before birth. This heat keeps the kittens' hair from darkening on the points.

Executing the module src/appy/utils/message.py from the root dir

# Doesn't contain relative imports
# Executing as a script - no errors since there are no relative imports
(python-project) $ python src/appy/utils/message.py
My Path: python-project/src/appy/utils/message.py

# Executing as a module - works as expected
(python-project) $ python -m src.appy.utils.message
python-project/src/appy/utils/message.py

If the PWD is a subdirectory of the project, such as python-project/src/appy, an attempt to execute a module which contains relative imports, will raise the exception below. Remember, your PWD should always be the project's root directory, in this case, it's python-project.

# PWD is `src/appy`
# Execute as a module - issues when running from a subdirectory (not from root dir)
(python-project/src/appy) $ python -m core.app

Traceback (most recent call last):
  File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/meirgabay/python-project/src/appy/core/app.py", line 1, in <module>
    from ..utils import message, img_ascii
ValueError: attempted relative import beyond top-level package

It doesn't happen when invoking message, since message doesn't use relative imports

# Execute as a script - works because there are no relative imports to parent packages, similar to main.py
(python-project/src/appy) $ python utils/message.py
My Path: python-project/src/appy/utils/message.py

# Execute as a module - works as expected, no imports of parent packages
(python-project/src/appy) $ python -m utils.message
My Path: python-project/src/appy/utils/message.py

Invoke a function from the command-line

Trying to invoke a function from the terminal, such as appy.core.app.main(), will raise the ModuleNotFound exception. A package must be imported before invoking one of its functions.

# Execute as a module - must import the function before invoking it
(python-project) $ python -m src.appy.core.app.main
/Users/meirgabay/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/bin/python: Error while finding module specification for 'src.appy.core.app.main' (ModuleNotFoundError: __path__ attribute not found on 'src.appy.core.app' while trying to find 'src.appy.core.app.main')

Since you can't invoke main() directly from the terminal, calling it from the if __main__ block enables executing it from the terminal. It's possible to pass arguments, but it's a bit ugly, read the docs to learn how. The following example attempts to execute the module appy.core.app, which in turn call its if __main__ block

# Execute as a module - works due to the `if __main__` block
(python-project) $ python -m src.appy.core.app
My Path: python-project/src/appy/core/app.py
Created the file: /Users/meirgabay/python-project/meirg-ascii.txt
Insert your name: willy wonka

Hello Willy Wonka, here's the cat fact of the day:
Cats sleep 16 to 18 hours per day. When cats are asleep, they are still alert to incoming stimuli. If you poke the tail of a sleeping cat, it will respond accordingly.

Invoking a function from the terminal is also possible by using the -c flag. Surprise, it's possible to pass arguments in a more intuitive way, for example src.app.main(my_arg1, my_arg2)

# Execute inline script - providing the "-c" flag with the command as a string, the semi-colon ";" means break-row
  (python-project) $ python -c "import src.appy.core.app as app; app.main()"
Created the file: /Users/meirgabay/python-project/meirg-ascii.txt
Insert your name: willy wonka

Hello Willy Wonka, here's the cat fact of the day:
Cats can be right-pawed or left-pawed.

What are the available command-line flags in Python?

• In this tutorial, we used both -c and -m flags
• Read the docs to learn about more flags - Using cmdline

Why is it possible to execute python -m src.appy?

The src/appy/__main__.py file acts like the if __main__ code snippet, but on packages. This enables the appy package to be executed with python -m or with runpy

# Execute as a module
(python-project) $ python -m src.appy
My Path: python-project/src/appy/__main__.py
Created the file: /Users/meirgabay/python-project/meirg-ascii.txt
Insert your name: willy wonka

Hello Willy Wonka, here's the cat fact of the day:
One reason that kittens sleep so much is because a growth hormone is released only during sleep.

What's runpy and why do you use it in main.py?

The runpy package provides the ability to run modules from a module (Python script). This package is shipped with Python so there's no need to install it.

main.py

import runpy


def main():
    appy_package = runpy.run_module(
        mod_name="src.appy", init_globals=globals())
    appy_package['message'].script_path(__file__)
    appy_package['main']()


if __name__ == "__main__":
    main()

What's globals()?

The official definition from the docs

Return a dictionary representing the current global symbol table. This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called).

(python-project) $ python
Python 3.8.2 (default, Jun 30 2020, 19:04:41)
[Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> globals()
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>}

---

This blog post is part of the Python Project series, and is based on this GitHub repository - unfor19/python-project.
The GitHub repo includes an example of a Python project, and Wiki Pages that describe the necessary steps for developing, creating and distributing a Python package.

---

Originally published at github.com/unfor19/python-project on 3 November, 2020