Executing external commands from Python

This chapter will discuss how you can execute external commands from Python, capture their output and other relevant details such as the exit status. The availability of commands depends on the OS you are using (mine is Linux).

Using os module

Last chapter showed a few examples with os module for file processing. The os module is a feature rich module with lot of other uses, like providing an interface for working with external commands. Here's an example:

>>> import os

>>> os.system('echo hello "$USER"')
hello learnbyexample
0

Similar to the print() function, the output of the external command, if any, is displayed on the screen. The return value is the exit status of the command, which gets displayed by default on the REPL. 0 means the command executed successfully, any other value indicates some kind of failure. As per docs.python: os.system:

On Unix, the return value is the exit status of the process encoded in the format specified for wait().

Here's an example for non-zero exit status:

>>> status = os.system('ls xyz.txt')
ls: cannot access 'xyz.txt': No such file or directory
>>> status
512
# to get the actual exit value
>>> os.waitstatus_to_exitcode(status)
2

# if you don't want to see the error message,
# you can redirect the stderr stream
>>> os.system('ls xyz.txt 2> /dev/null')
512

You can use the os.popen() method to save the results of an external command. It provides a file object like interface for both read (default) and write. To check the status, call close() method on the filehandle (None means success).

>>> fh = os.popen('wc -w <ip.txt')
>>> op = fh.read()
>>> op
'9\n'
>>> status = fh.close()
>>> print(status)
None

# if you just want the output
>>> os.popen('wc -w <ip.txt').read()
'9\n'

subprocess.run

The subprocess module provides a more flexible and secure option to execute external commands, at the cost of being more verbose.

Quoting relevant parts from doc.python: subprocess module:

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.

>>> import subprocess

>>> subprocess.run('pwd')
'/home/learnbyexample/Python/programs/'
CompletedProcess(args='pwd', returncode=0)

>>> process = subprocess.run(('ls', 'xyz.txt'))
ls: cannot access 'xyz.txt': No such file or directory
>>> process.returncode
2

The first argument to run() method is the command to be executed. This can be either be a single string or a sequence of strings (if you need to pass arguments to the command being executed). By default, command output is displayed on the screen. Return value is a CompletedProcess object, which has relevant information for the command that was executed such as the exit status.

As an exercise, read subprocess.run documentation and modify the above ls example to:

• redirect the stderr stream to /dev/null
• automatically raise an exception when the exit status is non-zero

See also:

• stackoverflow: How to execute a program or call a system command from Python?
• stackoverflow: difference between subprocess and os.system
• stackoverflow: How to use subprocess command with pipes
• stackoverflow: subprocess FAQ

shell=True

You can also construct a single string command, similar to os.system(), if you set shell keyword argument to True. While this is convenient, use it only if you have total control over the command being executed such as your personal scripts. Otherwise, it can lead to security issues, see stackoverflow: why not use shell=True for details.

Quoting from docs.python: subprocess Frequently Used Arguments:

If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user's home directory

>>> p = subprocess.run(('echo', '$HOME'))
$HOME
>>> p = subprocess.run('echo $HOME', shell=True)
/home/learnbyexample

>>> p = subprocess.run(('ls', '*.txt'))
ls: cannot access '*.txt': No such file or directory
>>> p = subprocess.run('ls *.txt', shell=True)
ip.txt

>>> p = subprocess.run('seq -s, 10 > out.txt', shell=True)
>>> p = subprocess.run('cat out.txt', shell=True)
1,2,3,4,5,6,7,8,9,10

If shell=True cannot be used but shell features as mentioned above is needed, you can use modules like os, glob, shutil and so on as applicable. See also docs.python: Replacing Older Functions with the subprocess Module.

>>> p = subprocess.run(('echo', os.getenv('HOME')))
/home/learnbyexample

Changing shell

By default, /bin/sh is the shell used for POSIX systems. You can change that by setting the executable argument to the shell of your choice.

>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True)
/bin/sh: 1: Syntax error: "(" unexpected

>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True,
                       executable='/bin/bash')
3a4
> 4

Capture output

If you use capture_output=True, the CompletedProcess object will provide stdout and stderr results as well. These are provided as bytes data type by default. You can change that by setting text=True.

>>> p = subprocess.run(('date', '-u', '+%A'), capture_output=True, text=True)
>>> p
CompletedProcess(args=('date', '-u', '+%A'), returncode=0,
                 stdout='Monday\n', stderr='')
>>> p.stdout
'Monday\n'

You can also use subprocess.check_output() method to directly get the output.

>>> subprocess.check_output(('date', '-u', '+%A'), text=True)
'Monday\n'

You can also use legacy methods subprocess.getstatusoutput() and subprocess.getoutput() but they lack in features and do not provide secure options. See docs.python: subprocess Legacy Shell Invocation Functions for details.