No Head Line in grep search
when I tried to find a process, I normally use ps with grep command.
sh> ps aux | grep fish
myoungj+ 695 0.0 0.0 88596 7000 tty2 S+ 09:17 0:00 -/usr/bin/fish -c /usr/bin/gnome-session -l
myoungj+ 2490 0.0 0.1 164660 10140 pts/1 Ss 09:21 0:00 -fish
myoungj+ 2665 0.0 0.1 172848 10076 pts/2 Ss+ 09:24 0:00 -fish
myoungj+ 2781 0.0 0.1 172724 9712 pts/0 Ss+ 09:27 0:00 -fish
myoungj+ 3024 0.0 0.1 164528 9552 pts/3 Ss+ 09:32 0:00 -fish
myoungj+ 4709 0.0 0.0 9136 2692 pts/5 S+ 10:00 0:00 grep --color=auto fish
Headline is helpful
However, I found that no head line sometimes makes me wondering what those information actually means. i.e: I'd though it would be nicer if I could see the head line of ps command along with search results.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
myoungj+ 695 0.0 0.0 88596 7000 tty2 S+ 09:17 0:00 -/usr/bin/fish -c /usr/bin/gnome-session -l
myoungj+ 2490 0.0 0.1 164660 10140 pts/1 Ss 09:21 0:00 -fish
.. snip ..
Quick Solution for single use
awk or sed could be useful in this category if you don't need any other feature from grep.
sh> ps aux | awk 'NR == 1 || /fish/ { print; }'
But I think grep is more powerful tool.
In bash, it looks straight forward for me.
bash> ps aux | { read line; echo "$line"; grep 'fish'; }
or using sub-shell.
bash> ps aux | ( read line; echo "$line"; grep 'fish'; )
or in fish shell (little longer)
fish> ps aux | begin read -l line; echo "$line"; grep 'fish'; end
Still, I think bash is better than fish in one-liner command.
More Serious Approach
But those one-liners are not very friendly. IMHO, all the programmes, at least, provide us of simple usage. So I decided to go little deeper.
Fish shell solution
The recent file is on my github: hgrep.fish
The basic options are below:
- -h|help : help message and exit
- -C|context : which is passed to as a 'grep option', Which is sometimes useful when we need the context literally.
#!/usr/bin/env fish
set -l PROG = 'hgrep.fish'
# ref: https://fishshell.com/docs/current/cmds/argparse.html#cmd-argparse
set -l options 'C/context=' 'h/help'
function usage -S -d "basic usage for $PROG"
echo \
"Usage: $PROG [-C|--context context] <SEARCH> [<INPUT PATH>]"
end
# parse args here
argparse $options -- $argv
set -l argc (count $argv)
# note: processed arguments are removed from $argv
if test $argc -ne 1 -a $argc -ne 2
usage
exit 0
end
set -l search_string $argv[1] # first argument
set -l input_path /dev/stdin
if test $argc -gt 1
# <INPUT PATH> is specified
set input_path $argv[-1]
end
echo $input_path
set -l grep_options -i
if set -q _flag_context
set --append grep_options '-C' $_flag_context
end
set --append grep_options $search_string
begin
# print head first
read -l line
echo "$line"
# let 'grep' do the rest
exec grep $grep_options
end < $input_path
begin .. end < $input_path pattern is used before when I made fish-pandoc-any-to-markdown. So I found this version a bit easier than others.
Perl Solution
My perl solution was made very long time ago. I'm happy to see that it is still working. Basic routine is the same, except it has one more options. --nohead which is not neccessary. I think I just wanted to chceck the how the OptArgs is working at that time.
I realized today that the routine in fish shell is also applicable.
- read one line from input and print to stdout
- exec to grep with option
Nevertheless, I believed that it is worth to learn!
parsing options in perl
And thanks to OptArgs module, I could handle option handy and in a more structural approach. (However, I think this is little heavier than python's argparse.)
The recent file is on my github:
hgrep.pl
#!/usr/bin/env perl
# -*- Mode: cperl; cperl-indent-level:4; tab-width: 8; indent-tabs-mode: nil -*-
# -*- coding: utf-8 -*-
# vim: set tabstop=8 expandtab:
use strict; use warnings;
use feature qw(switch);
use OptArgs; # https://metacpan.org/dist/OptArgs/view/bin/optargs
my @grep_options = qw(-i);
for ( $ENV{'TERM'} ) {
if ( $_ =~ /dumb/ ) { }
default { push @grep_options, "--color=auto" }
}
# ref: https://metacpan.org/pod/OptArgs
## option parts ...
opt context =>
( isa => 'Num',
alias => 'C',
default => 3,
comment => 'print NUM lines of output context' );
opt help =>
( isa => 'Bool',
alias => 'h',
comment => 'print a help message and exit',
ishelp => 1 );
# argument parts ...
arg search =>
( isa => 'Str',
required => 1,
comment => 'string to search from file' );
arg file_name =>
( isa => 'Str',
default => '-', # default input from stdin
comment => 'the file which we search from' );
# parsing options via optargs function!
my $opts = optargs;
And now processing the parsed arguments and open a file (or stdin)
if ( defined $opts->{'context'} and $opts->{'context'} > 0 ) {
push @grep_options, '-C', $opts->{'context'};
}
my $fh;
if ( $opts->{'file_name'} ne '-' ) {
open my $fh, "<$opts->{file_name}",
or die "Can't open `$opts->{file_name}'";
}
else {
# http://perldoc.perl.org/functions/open.html
open( $fh, "<&=",*STDIN );
}
if ( not $opts->{nohead} ) {
my $head = <$fh>;
# FIXME: colourising ....
print "$head";
}
my $to_gh;
requirement for system programming
And when I try to go further, I found that I need little more system programming underneath, which shell normally does for me.
To communicate with grep function, we need to open a pipe via open function.
my $grep_pid = open( $to_gh, '|-' );
if ( not defined $grep_pid ) {
die "Can't fork: $!";
}
|- means creating a pipe, and fork implicitly at the same time and now we have two processes, when the parent writing into new handle \$to_gh, the child will read from the stdin.
In terms of shell script, it looks like below at the moment.
sh> parent_perl <some options ...> | child_perl
i.e. parent_perl and child_perl now communicate with piple(|) and the child_perl process will be replaced with grep process via exec.
There is a simple way to we are in the parent_perl process or child_perl process, which is checking the $grep_pid value.
if ( $grep_pid ) {
# if grep_pid is not zero, this is parent_perl (parent process)
# which handle both file handles.
while ( <$fh> ) { print $to_gh $_; }
close $_ for $to_gh, $fh;
# parent process have to wait any children processes finsished.
waitpid $grep_pid, 0;
}
else {
# otherwise, this is child_perl (child process)
close $fh; # not used in child process
exec 'grep', @grep_options, $opts->{'search'};
}
exit 0;
and last exec 'grep' ... will replace its own perl process to grep process. no process could not be created without a
parent.
I found that it is worth trying to understand basic system programming in perl, However shell script will be much easier to handle it.
Python Solution (as a newbie)
How about python? I think the same logic could be applied in python as well. However, I didn't get chance to write down a python script yet. So, I didn't make any function and write it as simple as possible.
credit:
- os pipe: https://www.tutorialspoint.com/python/os_pipe.htm
- for loop: https://realpython.com/python-for-loop/
- file i/o: https://www.w3schools.com/python/python_file_open.asp
- optparse: https://docs.python.org/3/library/optparse.html
- execvp: https://docs.python.org/3/library/os.html?highlight=popen#os.execvp
- waitpid: https://docs.python.org/3/library/os.html#os.waitpid
I go through similar pattern as I did in perl you can find the recent file on my github: hgrep.py
#!/usr/bin/env python3
import os, sys
import argparse
# handle options first
parser = argparse.ArgumentParser()#prog="hgrep.py")
parser.add_argument( "-C", "--context",
nargs = 1,
type = int,
dest = "context",
required = False,
help="print NUM lines of output context" )
parser.add_argument( "search",
# upper case in the help message
metavar = "<SEARCH>",
help = "string to search from <file_path>" )
parser.add_argument( "file_path",
# upper case in the help message
metavar = "[<FILE PATH>]",
default = '-',
help = "<file_path> to search" )
# case insenstive search
grep_options = [ '-i' ]
# highligting
if os.environ['TERM'].lower != 'dumb':
grep_options.append( "--color=auto" )
I found argparse module cannot handle optional positional argument. optional positional argument is natural in grep. So I'd like to keep that behaviour.
# argparse cannot handle optional argument
# WORKAROUND:
argv = sys.argv[1::]
if len(argv) == 0:
print( "{prog}: No argument given".format(prog= sys.argv[0] ),
file = sys.stderr )
parser.print_help()
exit( 1 )
if len(argv) == 1:
# user ommit input file path
# default : - (stdin)
argv.append( '-' )
args = parser.parse_args( argv )
# check more grep options
if args.context is not None and args.context > 0:
grep_options.extend( [ '-C', args.context ] )
grep_options.append( args.search )
I don't really know about python, but I guess I took the very low-level pipe() function in python.
# and let's go for plumbing
# file descriptors r,w for reading and writing
r, w = os.pipe()
if args.file_path == "-":
# or from stdin
file_to_read = sys.stdin
else:
# or open file path to read
if os.path.isfile( args.file_path ):
file_to_read = open( args.file_path, "r" )
else:
print( "A file path:({fp}) is not readable"
.format( fp=args.file_path )
, file = sys.stderr )
exit( 2 )
# read head first and print into stdout directly
print( file_to_read.readline() , file = sys.stdout, flush = True )
# fork() will create a child process
# and we can distinguish which one is parent process by checking
# return value
grep_pid = os.fork()
if grep_pid:
# parent process
# to communicate with to a child process
# writing file descriptor will be used
os.close(r)
os.dup2( w, sys.stdout.fileno() )
for line in file_to_read:
print( line )
# It is good practice to close all the file open
os.close( w )
# safely waiting for children processes
os.waitpid( grep_pid,
os.WNOHANG # if child process status not available: no wait
)
else:
# child process
os.dup2( r, sys.stdin.fileno() )
# child process only requires 'r' as stdin
# and stdout so it is better to close r,w here.
os.closerange( r, w )
os.execvp( 'grep', grep_options )
exit(0)
Where I found difficulty
os.dup2 is essential to communicate with the grep in child process as grep only care about stdin here, but there is no way to inform the child that parent is going to newly open file descriptors (r,w). So we should kindly re-bind the new file descriptor to stdin. TBH, I spent too much time on this because lacks of my knowledge about system programming.
and os.waitpid requires os.WNOHANG option value, I thought it will be 0, which is actually not. so my programme was on hang after grep had finished its job.
Wrapping Up
pipe and shell's power
- Even though it was good chance to learn about basic pipe usage, Shell script is very powerful for basic process communication between two processes.
- perl's old open function's arguments are a little bit hacky.
parsing option is easier with modules
And also I tried to add option and test them.
- fish's argparse is relatively new, but useful for my cases.
- Perl's OptArgs has more features and handle optional argument as well
- python's argparse has good type system for checking data type and is performant, however it doesn't support optional (positional) arguments. so I applied some workaround.
Suggestion after post
It would be nicer, if we have option for case sensitive option because I put case-insensitive for default.
After making fish-pandoc-any-to-markdown and hgrep, only I need a programme to pre-process and let the other application could handle rest of it. So it becomes more general programme like below:
sh> ps aux | head-with get-one-line --tail-with grep -i /fish/
# or in fish-pandoc-any-to-markdown
sh> cat some.org | head-with retrieve-metadata --tail-with pandoc -t markdown > some.md
Well... but not for today. maybe after I get more chance to use the similar patterns!
Thank you for reading! and Happy coding!
Top comments (0)