Executing external commands

This chapter will show how to execute external commands from Python, capture their output and other relevant details such as the exit status. The availability of commands depends on the OS you are using (mine is Linux).

os module

Last chapter showed a few examples with the os module for file processing. This module is useful in plenty of other situations as well — for example, providing an interface for working with external commands.

>>> import os

>>> os.system('echo hello "$USER"')
hello learnbyexample
0

Similar to the print() function, the output of the external command, if any, is displayed on the screen. The return value is the exit status of the command, which gets displayed by default on the REPL. 0 implies that the command executed successfully, any other value indicates some kind of failure. As per docs.python: os.system:

On Unix, the return value is the exit status of the process encoded in the format specified for wait().

Here's an example with non-zero exit status:

>>> status = os.system('ls xyz.txt')
ls: cannot access 'xyz.txt': No such file or directory
>>> status
512
# to get the actual exit value
>>> os.waitstatus_to_exitcode(status)
2

# redirect the stderr stream if you don't want to see the error message
>>> os.system('ls xyz.txt 2> /dev/null')
512

You can use the os.popen() method to save the results of an external command. It provides a file object like interface for both read (default) and write. To check the status, call the close() method on the filehandle (None means success).

>>> fh = os.popen('wc -w <ip.txt')
>>> op = fh.read()
>>> op
'9\n'
>>> status = fh.close()
>>> print(status)
None

# if you just want the output
>>> os.popen('wc -w <ip.txt').read()
'9\n'

subprocess.run

The subprocess module provides a more flexible and secure option to execute external commands, at the cost of being more verbose.

Quoting relevant parts from doc.python: subprocess module:

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.

>>> import subprocess

>>> subprocess.run('pwd')
'/home/learnbyexample/Python/programs/'
CompletedProcess(args='pwd', returncode=0)

>>> process = subprocess.run(('ls', 'xyz.txt'))
ls: cannot access 'xyz.txt': No such file or directory
>>> process.returncode
2

The first argument to the run() method is the command to be executed. This can be either a single string or a sequence of strings (if you need to pass arguments to the command being executed). By default, the command output is displayed on the screen. A CompletedProcess object is returned, which has relevant information for the command that was executed such as the exit status.

As an exercise, read the subprocess.run documentation and modify the above ls example to:

  • redirect the stderr stream to /dev/null
  • automatically raise an exception when the exit status is non-zero

See also:

shell=True

You can also construct a single string command, similar to os.system(), if you set the shell keyword argument to True. While this is convenient, use it only if you have total control over the command being executed such as your personal scripts. Otherwise, it can lead to security issues, see stackoverflow: why not use shell=True for details.

Quoting from docs.python: subprocess Frequently Used Arguments:

If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user's home directory.

Here are some examples:

>>> p = subprocess.run(('echo', '$HOME'))
$HOME
>>> p = subprocess.run('echo $HOME', shell=True)
/home/learnbyexample

>>> p = subprocess.run(('ls', '*.txt'))
ls: cannot access '*.txt': No such file or directory
>>> p = subprocess.run('ls *.txt', shell=True)
ip.txt

>>> p = subprocess.run('seq -s, 10 > out.txt', shell=True)
>>> p = subprocess.run('cat out.txt', shell=True)
1,2,3,4,5,6,7,8,9,10

If shell=True cannot be used but shell features as quoted above are needed, you can use modules like os, glob, shutil and so on as applicable. See also docs.python: Replacing Older Functions with the subprocess Module.

>>> p = subprocess.run(('echo', os.getenv('HOME')))
/home/learnbyexample

Changing shell

By default, /bin/sh is the shell used for POSIX systems. You can change that by setting the executable argument to the shell of your choice.

>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True)
/bin/sh: 1: Syntax error: "(" unexpected

>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True,
                       executable='/bin/bash')
3a4
> 4

Capture output

If you use capture_output=True, the CompletedProcess object will provide stdout and stderr results as well. These are provided as the bytes data type by default. You can change that by setting text=True.

>>> p = subprocess.run(('date', '-u', '+%A'), capture_output=True, text=True)
>>> p
CompletedProcess(args=('date', '-u', '+%A'), returncode=0,
                 stdout='Monday\n', stderr='')
>>> p.stdout
'Monday\n'

You can also use subprocess.check_output() method to directly get the output.

>>> subprocess.check_output(('date', '-u', '+%A'), text=True)
'Monday\n'

info You can also use the legacy methods subprocess.getstatusoutput() and subprocess.getoutput() but they lack in features and do not provide secure options. See docs.python: subprocess Legacy Shell Invocation Functions for details.