A brief introduction in redirections, pipes, and the tee command

Lucas Salustiano — Sun, 04 Oct 2020 19:12:22 +0000

Introduction

Redirection is the most basic form of I/O manipulation in bash. It's used to change the data source or destination of a program's file descriptor.

On the other hand, pipes a good way to connect the output from program A to the input of program B.

Good knowledge about redirection and pipes is fundamental to work effortlessly in the terminal, so let's get started.

File Descriptors

First of all, before we talk about redirections, we need to understand file descriptors. File descriptors are kind of pointers to sources of data or places it can be written. There are three types of file descriptors:

Standard Input [stdin] -> file descriptor number 0
Standard Output [stdout] -> file descriptor number 1
Standard Error [stderr] -> file descriptor number 2

Standard input refers to the way how bash sees the characters you type. Standard output refers to the way/location where the output should be redirected/displayed. And standard error refers to where the program sends its error messages.

Now you know about file descriptors, let's see how to apply it to redirect data in our commands.

File redirections

How was said in the beginning, redirections are the most basic form of I/O manipulation in bash, so we can do, for example:

Somethings are happening here. First, we invoked the command echo that prints some data in the terminal, by default. We are giving a string as an argument to the echo command, after this, we are redirecting the output of the echo command to the file called file.txt. The symbol 1> denotes the redirection of the standard output (attention to the number 1) from the previous command, in this case, the echo command, into a file. Both symbols 1> and > work the same. Note: Because it's a write operation, both symbols 1> and > overrides the content of the file that was redirected to, so if you want to append the current output to the previous content of the file, you must use the symbol 1>> or >>.

We can also give an argument to a command from a file, for example:

Again, we invoked the echo command but stead pass directly a string as an argument, we redirected the content of the file.txt to the standard input of command echo (attention to the number 0). The symbol < without the 0 works the same.

We can redirect the errors from our commands as well, for example:

In the first command, we are creating three folders in the current directory, in this case, folder3 inside folder2 inside folder1.

In the second command, we are removing the folder1 and redirect the output error to the log.txt file. This command will not work and will raise an error because we don't use the -r flag to indicate the recursive remove, so we can catch the error message. Note that the error will not be printed in our terminal, but will be redirected to the log.txt file. Just like the standard output, the symbol 2> will overwrite the content of the log.txt, if it exists. If you want to append the current output error message, you must use the 2>> symbol. Note that number 2 is mandatory in the symbol 2> or 2>> to indicate the standard error message.

In the third command, we are using the cat command to see what is the content of the file log.txt.

Now we have a good understanding of how redirection works, let's take a look at pipes and how it works.

Pipes

Pipes is another way to create a flow between commands. With redirections, we redirect streams to/from files. With pipes, we redirect/connect streams from one process (one command) to another process (another command). In this way, we can combine a bunch of commands to perform more complex tasks. A pipe is denoted by the symbol |.

here are some examples:

The firsts two commands will work well, however, the third command will not work completely. That's because when we redirect output to a file, the data 'ends' there, so what can we do? We can use the tee command.

The tee command

The tee command is a command to kind take a snapshot of the data you are working with, so you can send the data to a file and pipe the data into another command and keep the flow of the pipeline. Let's see how we can fix the third command to run correctly:

Here are some explanations about what is happening. First, we have our ls command with some options and the folder we are interested in is /etc. Till now, we have all the output from the ls command and now we pipe all this output into the tee command. The tee command will take a snapshot (you can imagine its as a copy of the data) of all the output from the ls command, will insert into the etc_content.txt and will also pass the same data to the sort command. Now, the sort command will process that data and will redirect the output to the sorted.txt. We can also use multiples tee commands to redirect outputs into files and keep the flow of the pipeline.

Now let's take an overview of all that we have seen:

command 1> file or command > file will redirect the standard output from the command to the file and both of then will override the previous content from the file. To append new data into file, we must use command 1>> file or command >> file.
command 2> file will redirect the standard error from the command to the file and it will override the previous content from the file. To append new data into the file, we must use command 2>> file.
command 0< file or command < file will redirect the data from file to the standard input of the command.
command1 | command2 | command3 | .. will pipe the standard output from the first command into the standard input of the second command and so on.
command1 | command2 >> file will pipe the standard output from the first command into the standard input of the second command, and the second command will redirect its standard output into a file. Note that the redirection is at the end of the pipeline.
To make a redirection before the end of the pipeline, we must use the tee command. It will allow us to do both, pipe and redirect the data we are working with.

Conclusion

Understand the command line, its inputs, and outputs to perform complex flows and tasks for us, is an essential skill. Once we have a deep understanding of how it works, the other topics will be easier to learn (and to teach).