This chapter will discuss how you can execute external commands from Python, capture their output and other relevant details such as the exit status. The availability of commands depends on the OS you are using (mine is Linux).
Last chapter showed a few examples with
os module for file processing. The
os module is a feature rich module with lot of other uses, like providing an interface for working with external commands. Here's an example:
>>> import os >>> os.system('echo hello "$USER"') hello learnbyexample 0
Similar to the
print() function, the output of the external command, if any, is displayed on the screen. The return value is the exit status of the command, which gets displayed by default on the REPL.
0 means the command executed successfully, any other value indicates some kind of failure. As per docs.python: os.system:
On Unix, the return value is the exit status of the process encoded in the format specified for
Here's an example for non-zero exit status:
>>> status = os.system('ls xyz.txt') ls: cannot access 'xyz.txt': No such file or directory >>> status 512 # to get the actual exit value >>> os.waitstatus_to_exitcode(status) 2 # if you don't want to see the error message, # you can redirect the stderr stream >>> os.system('ls xyz.txt 2> /dev/null') 512
You can use the
os.popen() method to save the results of an external command. It provides a file object like interface for both read (default) and write. To check the status, call
close() method on the filehandle (
None means success).
>>> fh = os.popen('wc -w <ip.txt') >>> op = fh.read() >>> op '9\n' >>> status = fh.close() >>> print(status) None # if you just want the output >>> os.popen('wc -w <ip.txt').read() '9\n'
subprocess module provides a more flexible and secure option to execute external commands, at the cost of being more verbose.
Quoting relevant parts from doc.python: subprocess module:
subprocessmodule allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.
The recommended approach to invoking
subprocessesis to use the
run()function for all use cases it can handle. For more advanced use cases, the underlying
Popeninterface can be used directly.
>>> import subprocess >>> subprocess.run('pwd') '/home/learnbyexample/Python/programs/' CompletedProcess(args='pwd', returncode=0) >>> process = subprocess.run(('ls', 'xyz.txt')) ls: cannot access 'xyz.txt': No such file or directory >>> process.returncode 2
The first argument to
run() method is the command to be executed. This can be either be a single string or a sequence of strings (if you need to pass arguments to the command being executed). By default, command output is displayed on the screen. Return value is a
CompletedProcess object, which has relevant information for the command that was executed such as the exit status.
As an exercise, read subprocess.run documentation and modify the above
ls example to:
- redirect the
- automatically raise an exception when the exit status is non-zero
- stackoverflow: How to execute a program or call a system command from Python?
- stackoverflow: difference between subprocess and os.system
- stackoverflow: How to use subprocess command with pipes
- stackoverflow: subprocess FAQ
You can also construct a single string command, similar to
os.system(), if you set
shell keyword argument to
True. While this is convenient, use it only if you have total control over the command being executed such as your personal scripts. Otherwise, it can lead to security issues, see stackoverflow: why not use shell=True for details.
Quoting from docs.python: subprocess Frequently Used Arguments:
True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of
~to a user's home directory
>>> p = subprocess.run(('echo', '$HOME')) $HOME >>> p = subprocess.run('echo $HOME', shell=True) /home/learnbyexample >>> p = subprocess.run(('ls', '*.txt')) ls: cannot access '*.txt': No such file or directory >>> p = subprocess.run('ls *.txt', shell=True) ip.txt >>> p = subprocess.run('seq -s, 10 > out.txt', shell=True) >>> p = subprocess.run('cat out.txt', shell=True) 1,2,3,4,5,6,7,8,9,10
shell=True cannot be used but shell features as mentioned above is needed, you can use modules like
shutil and so on as applicable. See also docs.python: Replacing Older Functions with the subprocess Module.
>>> p = subprocess.run(('echo', os.getenv('HOME'))) /home/learnbyexample
/bin/sh is the shell used for POSIX systems. You can change that by setting the
executable argument to the shell of your choice.
>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True) /bin/sh: 1: Syntax error: "(" unexpected >>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True, executable='/bin/bash') 3a4 > 4
If you use
CompletedProcess object will provide
stderr results as well. These are provided as
bytes data type by default. You can change that by setting
>>> p = subprocess.run(('date', '-u', '+%A'), capture_output=True, text=True) >>> p CompletedProcess(args=('date', '-u', '+%A'), returncode=0, stdout='Monday\n', stderr='') >>> p.stdout 'Monday\n'
You can also use
subprocess.check_output() method to directly get the output.
>>> subprocess.check_output(('date', '-u', '+%A'), text=True) 'Monday\n'
You can also use legacy methods
subprocess.getoutput()but they lack in features and do not provide secure options. See docs.python: subprocess Legacy Shell Invocation Functions for details.