Finding the other end of a pipe on Linux

Originally published at https://hoelz.ro/blog/finding-the-other-end-of-a-pipe-on-linux

Have you ever been using the command line and been looking at some colored output, only to lose the coloration when you pipe the output to another command? I've often thought to myself "what if it were possible for a program to detect if the other side of the pipe were capable of handling colored output?".

As far as I know, there's no way for a program to publish this information, and even if there were, I highly doubt that many programs use it. So, let's see if we can accomplish the next best thing: detecting who's listening on the other side of standard output; if we knew that, we could check the name and command line of that program against a whitelist of programs we know to support color.

So...how do we detect who's listening?

The Naïve Approach

The rudimentary approach, which is specific to Linux (but probably not too hard to port to other OSes), is pretty simple: just crawl over /proc and look for a pipe that matches our standard output's pipe. You see, pipe file descriptors under /proc, like regular files, still have inodes, and the inode is unique to a pipe pair (this is the case for sockets too; there are other files in /proc that reference this information, such as /proc/net/tcp). So here's a quick and dumb Perl program that finds its partner:

#!/usr/bin/env perl

use strict;
use warnings;
use feature qw(say);

use File::stat;

exit unless -p STDOUT; # bail out if standard output isn't a pipe

my $stdout_stat = stat(\*STDOUT);

my $current_pid = $$;

my @other_procs = grep {
    m{/proc/(\d+)} && $1 != $current_pid
} glob('/proc/*/');

for my $proc_dir (@other_procs) {
    my $stat = stat("$proc_dir/fd/0");
    next unless $stat;

    if($stat->dev == $stdout_stat->dev && $stat->ino == $stdout_stat->ino) {
        my ( $pid ) = $proc_dir =~ m{/proc/(\d+)};

        say STDERR $pid;
        last;
    }
}

This gets the job done, but it feels a little too hacky for my tastes.
There must be a better way!

The less-than-naïve approach

As inspiration for a cleaner approach, let's consider a cousin to pipes: UNIX
sockets. Sockets work much like pipes, in that two file descriptors are bound
to one another. With a UNIX socket, we can call getsockopt(socket, SOL_SOCKET, SO_PEERCRED, &creds, sizeof(struct ucred)) to find out who we're talking to; perhaps there's an analogous call for pipes?

To find out if this functionality is available for pipes, let's get an answer straight from the source - the Linux kernel source, to be precise! (I'm looking at the source for Linux 4.5.3)

If we look for SO_PEERCRED in .c files in the kernel, we quickly come upon this case statement in the sock_getsockopt function:

case SO_PEERCRED:
{
        struct ucred peercred;
        if (len > sizeof(peercred))
                len = sizeof(peercred);
        cred_to_ucred(sk->sk_peer_pid, sk->sk_peer_cred, &peercred);
        if (copy_to_user(optval, &peercred, len))
                return -EFAULT;
        goto lenout;
}

If we follow the breadcrumbs for the sk variable, we can see earlier in the sock_getsockopt function that sk is defined as a struct sock, which is quite large, but contains the struct pid sk_peer_pid member that we're hoping to find a counterpart to in whatever struct represents a pipe.

We can find out which structure that is pretty easily if we look for an ioctl or fcntl code that's specific to pipes. If you look in the man page for pipe(7), you'll find F_GETPIPE_SZ. Doing a search on the kernel code for that leads us to pipe_fcntl in fs/pipe.c, which references struct pipe_inode_info, That struct is small enough to show you here:

struct pipe_inode_info {
    struct mutex mutex;
    wait_queue_head_t wait;
    unsigned int nrbufs, curbuf, buffers;
    unsigned int readers;
    unsigned int writers;
    unsigned int files;
    unsigned int waiting_writers;
    unsigned int r_counter;
    unsigned int w_counter;
    struct page *tmp_page;
    struct fasync_struct *fasync_readers;
    struct fasync_struct *fasync_writers;
    struct pipe_buffer *bufs;
    struct user_struct *user;
};

A cursory look at this struct shows us, sadly, that struct pipe_inode_info doesn't have the have the information we need, so we'll have to stick with our naïve approach. So we didn't get a cleaner approach, but at least we got to look at some kernel code!

Comments

[Josh Cheek]

I've always done it by checking whether the output is a tty (eg). If you really wanted to be legit, then that's not good enough, you have to then read the terminal databases (b/c different terminals have different capabilities and different escape codes for a given ability), but that gets painful pretty quickly, this approach gets you really far really fast.

[Rob Hoelz]

I probably should have provided a more concrete example in my post, but I'm interested in finding out if the program on the other side of a pipe supports color, rather than if my terminal does. Here's an example of what I was going for:

$ ls --color=auto | less # doesn't support color

versus

$ ls --color=auto | less -R # supports color

[John Selbie]

Rob,

How many programs, aside from less, actually support color codes read from stdin?

[Rob Hoelz]

I had less in mind when thinking about this, but I suppose any pager. There's also filter programs like nl and ts that just annotate their input - a whitelist approach would probably work pretty well. One idea I kind of like is modifying ack to detect if it's talking to another ack, and having the second ack change its color to highlight different patterns in the output.

Looking through my notes on this, there's a lot of potential application beyond color - you could have a program tell another what encoding it's using (hey, my input is UTF-16LE!) so it doesn't need to guess, or you could have an implementation of uniq that checks to see if its input is coming from sort or not.

[Josh Cheek]

Omg, I never knew about less -R!!

[John Selbie]

Most console tools detect color support by interrogating the output terminal for color support with a few system calls. Read this

That is, the source process that generates the original text is just not including the color escape chars when it detects that it's not on a color terminal.

Some programs can be forced to output color chars with a command line parameter.

Try this on the command line:

ls -1 | sort

The above produces a list of files in typical order. However, the color output normally generated by ls isn't present.

Now try this:

ls -1 --color=always | sort

And it appears to work. The filenames have color. But the sort program gets confused. It interprets the filenames with leading escape chars as it would any other string. The side effect is that the sort program does not show my files in alphabetical order. Instead, it lists the files sorted by color (with alphabetical order as a secondary sort).

There's probably some clever ways in which a downlevel process taking piped input might be able to emulate a color tty to fool the upstream process into generating color chars. Or a fake tty driver that functions as a pipe between both programs. It would not surprise me at all if such a thing already exists.

[Rob Hoelz]

That is, the source process that generates the original text is just not including the color escape chars when it detects that it's not on a color terminal.

Right - I'm wondering if we could improve upon this further in the case involving pipes, by having a downstream process advertise its support for handling color. To piggyback off of your sort example, it would be neat if sort could know to strip the ANSI color escape sequences off of its input, and inform the upstream process "hey, you can send me colored output!" Two approaches I thought about were enhancing fgetxattr to work on pipes in the kernel, or decoupling display information from the output stream entirely - but those both deserve their own posts! What I had in mind with this post was a simple whitelist - eg. have ack keep automatic coloration if being piped to less with the -R flag active.
I probably should've mentioned this example!

There's probably some clever ways in which a downlevel process taking piped input might be able to emulate a color tty to fool the upstream process into generating color chars.

Sadly, I think that the granularity of stat prevents this - I'm guessing that most programs (checked by straceing a program using isatty), either directly or indirectly, fstat standard output, and then check the file type of the resulting structure. What one could do is wrap the upstream program in a pseudo TTY using a helper program (something like ptyget) to fool that program, but then the downstream program would still need to handle the colored input.