Pipes

In my journey to master Go my next objective was to update my shell to be able to handle commands that look like:

tail -f test.log | head -3

First idea was to collect command 1 output and then invoke command 2 with the collected output as argument. But as you may notice command 1 is tail -f which does not exit on its own and it means that it would block the execution.

More over if you try to test it:

echo line1 > test.log
tail -f ./test.log | head -3

then in another terminal window you try

echo line2 >> ./test.log

You'll see that the line2 was printed out. It could mean that head -3 does not wait for tail -f test.log to exit and to collect its output, instead it receives the output and prints it realtime.
Then if you read the docs you'll find confirmation for this hypothesis:

The output of each command in the pipeline is connected via a pipe to the input of the next command. That is, each command reads the previous command’s output.

Each command in a multi-command pipeline, where pipes are created, is executed in its own subshell, which is a separate process

Then if in another terminal window you try:

echo line3 >> ./test.log
echo line4 >> ./test.log

you'll see that your command printed out line3 and exited.

What happens here is:


1. On the third write to test.log, tail picks up the line and writes to stdout, which is the write end of the pipe

2. head receives the line through its stdin, the read end of the pipe.

3. As head was instructed to print only 3 lines, it exits and closes its end of the pipe

4. On the 4th write, tail tries to write to stdout, but fails to do so as other end of the pipe is closed. This triggers a SIGPIPE signal, which terminates tail.



With all this understanding what can be done in Go to mimic the behavior?


Create the commands:

cmd1 := exec.Command("tail", "-f", "./test.log")
cmd2 := exec.Command("head", "-n", "5")

Connect the commands:

outPipe, err := cmd1.StdoutPipe()
cmd2.Stdin = outPipe
cmd2.Stdout = os.Stdout // the go program stdout which is the terminal where it was executed

Start the commands:

cmd1.Start()
cmd2.Start()



func (*Cmd) Start() start command processes and doesn’t wait for them to finish.

Wait for the commands to finish:





cmd2.Wait()// remember head exits before tail
cmd1.Wait() 

…and it appears that tail never exits…


This is where I learned an interesting thing. You see, exec.Command spawns a child process, which receives its own copies of all File descriptors of original process. So when head command finishes, it closes it stdin, but its original in the parent process(my program) remain open. This means that in order to close the read end of the pipe I also have to close it in the parent process.

cmd2.Wait()
outPipe.Close()
cmd1.Wait() 

The whole program:

package main

import(
    "fmt"
    "os/exec"
    "os"
)

func main(){    
    cmd1 := exec.Command("tail", "-f", "/Users/skuf_love/study/go/test_pipes/test.log")
    cmd2 := exec.Command("head", "-n", "5")
    outPipe, err := cmd1.StdoutPipe()
    if err != nil {
        fmt.Printf("Err1: %q", err)
    }
    cmd2.Stdin = outPipe
    cmd2.Stdout = os.Stdout

    err = cmd1.Start()
    if err != nil {
        fmt.Printf("cmd1 start error: %v\n", err)
    }
    err = cmd2.Start()
    if err != nil {
        fmt.Printf("cmd2 start error: %v\n", err)
    }

    cmd2.Wait()
    fmt.Println("cmd2 wait done")
    outPipe.Close()
    cmd1.Wait()
    fmt.Println("cmd1 wait done")
}