Running commands with timeout on Linux

Welcome to the next pikoTutorial!

Bash comes with a bunch of useful, built-in commands. One of them is timeout which allows you to run another command with a time limit. The syntax is simple:

timeout [duration] [target_command]

To avoid providing large numbers for long timeouts, duration parameter accepts suffixes which mark the exact time frame:

• s - seconds
• m - minutes
• h - hours
• d - days

Simple example

For the first example, let's take a simple Python script which spins forever:

# some_job.py
import time

while True:
    time.sleep(1)

We can run it with a 3 seconds timeout using the following command:

timeout 3s python3 some_job.py

After 3 seconds of waiting, some_job.py is terminated.

Specifying termination signal

When the allowed time elapses, timeout sends SIGTERM signal to the given command. In your implementation you may have some custom signal handlers which do the cleanup before exiting. For example, if you expect that your script will be interrupted mainly by CTRL + C, you most probably have a SIGINT signal handler. In such case, you most likely want to keep this in case of timeout command as well to still be able to perform the cleanup when the timeout occurs:

# some_job.py
import sys
import signal
import time

def handle_sigint(signum, frame):
    print("Received SIGINT, cleaning up and exiting...")
    sys.exit(0)

signal.signal(signal.SIGINT, handle_sigint)

print("Starting script...")
time.sleep(10)

If you now call:

timeout 3s python3 some_job.py

You will see that your signal handler has not been called before exiting the script. To change this, specify the expected signal using --signal option:

timeout --signal=SIGINT 3s python3 some_job.py

After that, the log from handle_sigint handler is visible when the timeout occurred.

Giving time for graceful exit

In the previous example I assumed that the cleanup is quick and always succeeds, but in reality the application shutdown may be time consuming:

# some_job.py
import sys
import signal
import time

def handle_sigterm(signum, frame):
    print("Received SIGTERM, cleaning up and exiting...")
    time.sleep(5)

signal.signal(signal.SIGTERM, handle_sigterm)

print("Starting script...")
time.sleep(10)

In such case, it may require its own timeout for wrapping things up - upon violation of that final timeout, the application should be killed. This can be achieved by using --kill-after. The following command lets the script run for 5 seconds, sends SIGTERM (default signal) and then it gives the script 2 more seconds to exit. If the script doesn't exit within 2 seconds after receiving SIGTERM, SIGKILL is sent:

timeout --kill-after=2s 5s python3 some_job.py

Preserving return code

By default, if the command provided to timeout times out, its return code will be equal to 124. You can check it by running the example Python script and then checking its return code:

timeout 3s python3 some_job.py
echo $?

Note for beginners: ? in the above section is a special Shell variable which stores the exit code of the last executed command. $ sign, as usually in Shell environment, obtains the underlying value of that variable, so echo $? prints the exit status of the most recent command.

However, this may be something that your system does not expect, especially if your script has some logic which ends up returning specific exit codes. For example, the following script simulates a time consuming processing of 3 different phases. In case of interruption, the script returns the number of the phase being processed during the interruption:

# some_job.py
import sys
import signal
import time

current_phase: int = 0

def handle_sigterm(signum, frame):
    sys.exit(current_phase)

signal.signal(signal.SIGTERM, handle_sigterm)

for i in range(3):
    current_phase += 1
    time.sleep(2)

To preserve that return code after timeout, use --preserve-status flag:

timeout --preserve-status 3s python3 some_job.py

Now, when you check the exit status of such operation, you will see 2 because timeout terminated script while the phase 2 was being processed.