Redirecting and Piping Command through the Command Line

Standard Streams

• The three standard streams are communication channels between a computer program and its environment.

• They are:

  • Standard Input
  • Standard Output
  • Standard Error

  Standard Output

  • Standard output is a place to which a program or command can send information.
  • The information could go to a screen to be displayed, to a file, or even to a printer or other devices.

Standard Error

• Commands and programs also have a destination to send error messages: standard error.

• By default, the shell directs standard error information to
  the screen for us to read, but we can change that
  destination if needed!

  Standard Input

  • Standard input is where a program or command gets its input information from. By default, the shell directs standard input from the keyboard.
  • The input information could come from a keyboard, a file, or even from another command!

Redirection

• "redirection" describes the ways we can alter the source of standard input, and the destinations for standard output and standard error.

  Redirecting Output

  • The redirect output symbol (>) tells the shell to redirect the output of a command to a specific file instead of the screen.
  • By default, the date command will print the current date to the screen. If we instead run date > output.txt the output will be redirected to a file called output.txt
  • If the file does not exist, it will be automatically created.

  • Note: the > symbol needs to occur after any options and arguments!
    

    command > filename
    echo "moo" > cow.js # redirects the output of echo
    ls -l > files.txt # saves the output of ls -l to a file.
    

  Appending

  • When we redirect output into a file using > any existing contents in the file are overwritten. Sometimes this is not what we want!

  • Instead, keep the existing contents in the file and add new content to the end of the file, use >> when redirecting.
    

    echo "hello" >> greeting.txt
    echo "world" >> greeting.txt
    cat greeting.txt
    #hello
    #world
    

  Redirecting Input

  • To pass the contents of a file to standard input, use the < symbol followed by the filename.
  • For example, we could pass the contents of the chickens.txt file to the cat command using cat < chickens.txt

  • cat (and many other commands) are set up to accept filenames as arguments directly, but we can also redirect to standard input manually.
    

    command < filename
    cat < chickens.txt
    

  • We can redirect standard input and output at the same time! In this example, we are using cat to read in the contents of original.txt and then redirecting the output to a file called output.txt

  • In this example, we are redirecting the names.txt file to the sort command. We are also redirecting the output of sort to a file called sorted.txt
    

    cat < original.txt > output.txt
    sort < names.txt > sorted.txt
    

  Redirecting Standard Error

  • By default, error messages are output to the screen, but we can change this by redirecting standard error.
  • The standard error redirection operator is 2>
  • If we ran a command like cat nonexistentfile (where the file does not exist) we would see an error printed to the screen. We can instead redirect standard error to a file with cat nonexistentfile 2> error.txt

We can use the same >> syntax to append when redirecting standard error.

cat nonexistentfile 2> error.txt
ls -zzz 2>> error.txt # ls with illegal option

Why 2> ?

• Each stream gets its own numeric file descriptor, and for standard error the number is 2.

• The > operator actually defaults to using 1 as the file descriptor number, which is why we didn't need to specify 1> to redirect standard output

• Similarly, the < operator uses a default file descriptor number of 0, so we don't need to specify 0< to redirect to standard input (though we can!)

All Together now

• We can redirect multiple streams at once! In this example, we are concatenating two files, redirecting standard output to a file called insects.txt, and redirecting standard error to a file called error.txt

• When redirecting both standard output and standard error, make sure standard output comes FIRST. Always redirect standard error after standard output.
  

  cat bees.txt ants.txt > insects.txt 2> error.txt
  

Pipes

• Pipes are used to redirect a stream from one program to another program. We can take the output of one command and redirect it to the input of another.

• We use the pipe character ( | ) to separate two commands. The output of the first command will be passed to the standard input of the second command.

• This example shows the output of the date command being piped to the rev command. The result is the reverse of the current date!
  

  command1 | command2
  date | rev
  

• This example pipes the output of ls to less. The /usr/bin directory typically contains a bunch of stuff, so it can be nice to use less to read the results in a more manageable way.
  

  ls -l /usr/bin | less
  

• This example counts the number of files (non hidden files) in a directory. We pipe the output of ls to the word count command. The -l option tells wc to count the number of lines.
  

  ls | wc -l
  

• In this example, we are calling tac with a file and then piping the output to rev. The final result is the content of file.txt printed "horizontally" and "vertically" reversed
  

  tac file.txt | rev
  

• This example concatenates two files using cat and then sorts them alphabetically
  

  cat colors.txt pets.txt | sort
  

&gt; vs |

• Though both the > character and the | character are used to redirect output, they do it in very different ways.

• &gt; connects a command to some file.

• | connects a command to another command.
  

  ls -l /usr/bin > list.txt
  ls -l /usr/bin | less
  

tee

• What if I wanted to create a file with the output of cat? Enter the tee command

• The tee program reads standard input and copies it both to standard output AND to a file. This allows us to capture information part of the way through a pipeline, without interrupting the flow.

• It takes some informations coming from one command and gives it to a file we specify and also passes it to the next command. Hence this allows us to take information in the middle of a pipeline and save it to a file without stopping.
  

  command1 | tee file.txt | command2
  cat file1 file2 file3 | tee combo.txt | wc -w