Linux pipes in Golang

In a Unix system, a pipe is a construct to redirect (pipe) the output of one command to the input of another one.

The Unix Philosophy suggests programmers to write programs which "do one thing and do it well", rather than adding more features to existing programs.
Pipes are the cornerstone of the Unix philosophy, allowing to undertake more complex problems by combining programs built to do just one thing.

In the last article of this year we are going to implement a simple Golang program which can be used in Unix pipe. All the code used in the article is on github.com/napicella/go-linux-pipes

Whether you are a Linux user or not, you might have encountered Linux pipes on more than one occasion.
It's denoted by the pipe symbol "|".
For example to get a sorted list of all unique words from a file, we can use the cat command to get the content of the file, pipe it to the sort command and again pipe the output to uniq to remove the duplicates.

> cat words | sort | uniq
  apple
  bye
  hello
  zebra

Being a good citizen of a Unix system also means writing programs that can be used in a pipe as well as with arguments.

---

For our demo, we are going to build a simple program which converts to uppercase letters the input.

It accepts as parameter a filename, but it can also be used in a pipe, in which case the input is the output of the previous command in the pipe.
For example, we should be able to use our program in the pipe to get a sorted list of all unique words:

> cat words | sort | uniq | uppercase
  APPLE
  BYE
  HELLO
  ZEBRA

We are going to use Cobra to build the skeleton of our CLI. The end result will look as follows:

> uppercase --help
Simple demo of the usage of linux pipes
Transform the input (pipe or file) to uppercase letters

Usage:
  uppercase [flags]

Flags:
  -f, --file string   path to the file
  -h, --help          help for uppercase
  -v, --verbose       log verbose output

---

Create the command

Our CLI will have a single command and two flags, one for the file name and the other for setting the verbosity level.

var rootCmd = &cobra.Command{
    Use:   "uppercase",
    Short: "Transform the input to uppercase letters",
    Long: `Simple demo of the usage of linux pipes
Transform the input (pipe or file) to uppercase letters`,
    RunE: func(cmd *cobra.Command, args []string) error {
        print = logNoop
        if flags.verbose {
            print = logOut
        }
        return runCommand()
    },
}

// flag for the filepath
rootCmd.Flags().StringVarP(
        &flags.filepath,
        flagsName.file,
        flagsName.fileShort,
        "", "path to the file")

// flag for the verbosity level
rootCmd.PersistentFlags().BoolVarP(
        &flags.verbose,
        flagsName.verbose,
        flagsName.verboseShort,
        false, "log verbose output")

Detect if the program is in a pipe

To detect if the program is used in a pipe, we can use the file info associated to the Stdin. If fileInfo.Mode() & os.ModeCharDevice == 0 then the input is coming from a pipe.

We have conveniently wrapped this condition in a function called isInputFromPipe.

func runCommand() error {
    if isInputFromPipe() {
        // if input is from a pipe, upper case the
        // content of stdin
        print("data is from pipe")
        return toUppercase(os.Stdin, os.Stdout)
    } else {
        // ...otherwise get the file
        file, e := getFile()
        if e != nil {
            return e
        }
        defer file.Close()
        return toUppercase(file, os.Stdout)
    }
}

func isInputFromPipe() bool {
    fileInfo, _ := os.Stdin.Stat()
    return fileInfo.Mode() & os.ModeCharDevice == 0
}

func getFile() (*os.File, error){
    if flags.filepath == "" {
        return nil, errors.New("please input a file")
    }
    if !fileExists(flags.filepath) {
        return nil, errors.New("the file provided does not exist")
    }
    file, e := os.Open(flags.filepath)
    if e != nil {
        return nil, errors.Wrapf(e,
            "unable to read the file %s", flags.filepath)
    }
    return file, nil
}

Convert to uppercase

Finally we convert the content of the reader.

The toUppercase function reads from a io.Reader and writes the result to a io.Writer. This is convenient because we will be able to reuse the same function for both the file and the stdin.

func toUppercase(r io.Reader, w io.Writer) error {
    scanner := bufio.NewScanner(bufio.NewReader(r))
    for scanner.Scan() {
        _, e := fmt.Fprintln(
            w, strings.ToUpper(scanner.Text()))
        if e != nil {
            return e
        }
    }
    return nil
}

Build and run

go build -o ./bin/uppercase
export PATH=$PATH:./bin
uppercase --help

---

This is all for this year! Cheers!

Liked the article? Share it on Twitter

Comments

[adriens]

Thanks a lot for the tip, pipes are so cool ... I really wanted to add the support for them <3