Avoiding Temporary Files in Shell Scripts

This is an extract from a longer article from Openmake Software. If you want to see other tips and tricks to do with performance and security in shell scripts - go there!

Introduction

It's always fun to start with a polemic so let me make my position quite clear: temporary files are a force for evil in the world of shell scripts and should be avoided if at all possible. If you want justification for this stance just consider the following:

• If the script runs as root (from a deployment process for example) then the use of temporary files can be a major security risk.
• Temporary files need to be removed whenever the script exits. That means you have to trap user exits such as CTRL-C. But what if the script is killed with SIGKILL?
• Temporary files can fill the filesystem. Even worse, if they are created by a redirect from an application’s standard output then the application may not realise that the output has failed due to the filesystem being full.

Let’s look at these problems in detail:

Problem 1: What happens to a temporary file if the shell script exits unexpectedly?

This problem can happen if the shell script is interactive and the user halts it with CTRL-C. If this is not trapped then the script will exit and the temporary file will be left lying around.

The only solution to this is to write a “trap” function to capture user generated signals such as CTRL-C and to remove the temporary files. Here’s an example:

function Tidyup
{
    rm –f $tempfile
    exit 1
}

trap Tidyup 1 2 3 15

However, if the script exits for any other reason then the tidy up script is never invoked. A kill –9 on the script will kill it stone dead and leave the temporary files in existence.

Problem 2: What happens if the filesystem fills up?

Okay, so you’re busy writing your temporary file when the filesystem hits 100% full. What happens now? Well, your script will fail won’t it? Well maybe but perhaps not in the way you think. Some third party applications do not check for error returns when they write to standard output. Be honest, if you’ve written ‘C’ code do you check the return code from printf (or write or whatever) to see if it has been written correctly? Probably not – what can go wrong writing to the screen? Not a lot presumably, but if you’ve redirected standard out then output is not going to the screen – it’s going to a file. You’d be amazed how many commercial applications fall victim to this.

The net result is that a command such as

third_party_command > /tmp/tempfile

may not return a fail condition even if /tmp is full. You then have no way of knowing that the command failed but /tmp/tempfile does not contain the full output from third_party_command. What happens next depends on what your script does but it’s likely to be sub-optimal. We will discuss some workarounds for this later.

Problem 3: Beware redirection attacks.

Most temporary files are created in /tmp and given a filename containing a $$ sequence. The shell replaces the $$ sequence with the current process id thus creating a unique filename.

However, if the script is run as root and you create the file implicitly like this:

echo "top secret" > /tmp/mytempfile$$

then it is possible that "top secret" won't stay secret for very long. An unscrupulous user called Dave could create a script like this:

mknod /tmp/mytempfile12345 p
cat /tmp/mytempfile12345 > /home/dave/secretdata

This will just block until your root script finally executes as PID 12345 - when it writes "top secret" to what it thinks is a file it's going to create. However, that file is a pipe and Dave's unscrupulous script then just grabs the content and writes it to a file in Dave's directory called secretdata. Of course, this will probably break the root script (it will hang when it tries to read mytempfile$$) but by the stage, Dave is away with the secret data.

Avoiding temporary files

Avoiding temporary files can be difficult but is not necessarily impossible. A lot of UNIX commands will read standard input (or send output to standard output) as well as to named files. Using pipes to connect such commands together will normally give the desired result without recourse to temporary files.

What if you have two separate commands from which you want to merge and process the output? Let’s assume that we’re going to build some form of control file for the program process_file. The control file is built from some header lines, the actual body of the control file (which our script will generate) and some tail lines to finish the whole thing off.

A common way of building this sort of file is this:

echo “some header info” >Â  /tmp/tempfile.$$
process_body            >> /tmp/tempfile.$$
echo “some tailer info” >> /tmp/tempfile.$$
process_file /tmp/tempfile.$$
rm –f /tmp/tempfile.$$

However, this code is susceptible to all the problems outlined above.

If we rewrite the code as:

{
echo "some header info"
process_body
echo "some tailer info"
} | process_file

then this brackets all the relevant commands into a list and performs a single redirection of the list’s standard out into the process_file program. This avoids the need to build a temporary file from the various components of the desired input file.

What if process_file is an application that is incapable of taking its input from standard input? Surely then we have to use a temporary file?

Well, you can still avoid temporary files but it takes a bit more effort. Here’s what the code looks like. We’ll examine it line by line.

mknod /tmp/mypipe.$$ p # 1
if [ $? –ne 0 ]
then
    echo “Failed to create pipe” >&2
    exit 1
fi
chmod 600 /tmp/mypipe.$$ # 2
process_file /tmp/mypipe.$$ & # 3
(
   echo "some header info"
   process_body
   echo "some tailer info"
) > /tmp/mypipe.$$ # 4
wait $! # 5
rm –f /tmp/mypipe.$$ # 6

First we create a named pipe (#1). A named pipe is exactly the same as any other pipe except that we can create it explicitly and that it appears in the filesystem. (In other words you can see it with an ls). Now strictly speaking this is a temporary file. However, it is of zero length and therefore will not fill the filesystem. Also, if it cannot be created for any reason (including the file already existing) there is an error return. Therefore redirect attacks are useless. Of course it is left around by an untrappable kill but we can’t have everything.

We change the access mode (#2) so only the user running the script can read or write to it. Another way of doing this is to create the pipe with the correct permissions in the first place by invoking umask 066 before we call mknod.

We set our process_file program running in background, reading its input from this named pipe (#3). Now, since there is nothing on the pipe (yet) the program’s read call will block. Therefore process_file will hang awaiting input. However, we've set process_file running background (with the & operator) so the script will continue.

We construct the control file for process_file as before except that this time we redirect it to our named pipe (#4). At this point, process_file will unblock and start reading the data just as if it had come from a conventional file.

The wait call (#5) will block until the specified child process has exited. The $! is a shell sequence meaning the process ID of the last background process. Since this is the PID of process_file our script will wait until process_file has completed, just as if it had been invoked in foreground.

Finally, we remove the named pipe (#6).

I have no choice but to create a temporary file. What can I do?

If you have no choice but to use temporary files then use the following techniques:

• use umask to set the file permission mask right at the top of your script. umask 077 will remove group and world read permissions from newly created files. That way, even if your script aborts without cleaning up, only the invoking user can read the content of the file(s) that are left lying around.
• Do not implicitly create the file with a single redirection operator. Instead, ensure the filename does not already exist first (with a -f test operation), then set your umask appropriately so that only the owner of the file can access it. Better still, set up a function to generate a unique filename, create it and set the access permissions.
• If you're creating more than one temporary file, create a temporary directory and place your files in there. That way, your trap function can simply remove the directory and all the files within it on exit - you don't need to track what temporary files you've created since they're all in the same temporary directory.
• Perform sanity checking on the temporary file to ensure that it has been successfully written. Remember that checking $? may not be adequate since the application may not be checking error returns from writes to standard output. Try the following technique instead:

  process_file | tee $output_file > /dev/null
  if [ $? != 0 ]
  then
  …
  fi
  

  since this will make tee responsible for writing to the filesystem and it *should* flag errors properly. Note that the $? operator will be checking the exit status of tee and not process_file. If you want to know if process_file has worked correctly use a list to group the execution of process_file with an exit check (see below)
• Create tidy up functions to remove the temporary file(s) if the user aborts the script. Call the same functions on controlled exit (either normal or error) from the script.

Putting all of this together, gives us a script like this:

#!/bin/ksh
function ExitWithError
{
        echo "$*">&2
        rm -f $output_file
        exit 1
}

function Tidyup
{
        ExitWithError "Abort"
}

umask 077
output_file=/tmp/tfile$$
rm -f $output_file
trap Tidyup 1 2 3 15

{
        process_file
        [[ $? != 0 ]] && ExitWithError "process_file failed"
} | tee $output_file > /dev/null
[[ $? != 0 ]] && ExitWithError "Error writing to $output_file"
# process output_file here
# .
# .
# normal exit
rm -f $output_file
exit 0

This script ensures the temporary file it creates is removed on exit (either normal successful exit or on error), ensures the temporary file is written correctly before it is read for processing and sets the permission mask correctly so that if the script is killed with SIGKILL (-9) the temporary file can only be read by the user who invoked the script.

Conclusion

Temporary files can create serious security issues as well as opening scripts up to unexpected failures. Most shells make it relatively easy to avoid them - and you should! If you really have no choice, then make sure the permissions are set correctly, you check that they have been written to correctly and you remove them on both successful exit and on error.

Comments

[Peter Evans]

Thanks for this. I learned a lot and will certainly be changing the way I handle temporary files in future :)