DEV Community: Ian Pride

Shell Pipe Wrapper Functions

Ian Pride — Sun, 07 Feb 2021 23:32:38 +0000

Shell Pipe Wrapper Functions

Create pipe functions from any builtin command, function, or executable program in the Linux shell.

Shell Pipe Wrapper Functions
- Disclaimer
- Piping Output In The Linux Command Line
- Pipe Efficiency
- Pipe Availability
- Pipe Wrapper Functions
- Pipe Wrapper Logic
- Examples
- Pipe Wrapper Function Simple Example
  - Simple Example STAT Program
  - STAT Program Function Wrapper
  - STAT Program Function Example
- Pipe Wrapper Function Advanced Example
  - Advanced Example RM Program
  - RM Program Function Wrapper
  - RM Program Function Example
- Conclusion

Disclaimer

For those who mention xargsthis an alternative method that makes extra logic processing simpler... And I don't like using xargs.

This post assumes that you are familiar with the Linux command line, commands, shell functions, std(in|out), and piping but it should not be too difficult to understand if you are new.

Piping Output In The Linux Command Line

Pipe Efficiency

Piping output is not always the best or fastest way to do things and sometimes can be quite redundant if the command in question has it's own methods of processing output (e.g. find -exec {} \;) as each part of a pipeline is a subshell, but for many things this inefficiency is negligible and sometimes a pipe is almost necessary or, at least, the best way to process output.

Pipe Availability

We all have more than likely used a pipe | in the Linux shell environment to pipe stdout to another builtin command, function, or executable and eventually we realize that not everything can be piped to.

In order to pipe output to a command the command must process stdin (as you would user input) and not all programs do so.

Pipe Wrapper Functions

In this post I will show you how I write a sort of generic shell function for any command, function, or binary file program that doesn't already process stdin and can be used almost completely in place of or with the original command.

This will not be a 100% tutorial, but the examples will have documenting comments and I will explain as I go along.

Pipe Wrapper Logic

The basic logic is as follows:

Test if the file descriptor (FD) is greater than 0 (check if there is stdin already in the piped subshell):
1. Process the input in whatever way, usually while reading the input passing the input with pass arguments to the command, function, or file.
If the function has not (else) received stdin:
1. Only pass arguments to the command, function, or file.

Examples

These examples all share the same simple construct/concept while only varying on logic that we need inside the if [ ! -t 0 ]; then block and more often than not, only in the while block.

Most of the time we will only need the basics of what is written here and not extra logic/expression.

function command_pipe {
    if [ ! -t 0 ]; then
        local input
        # input logic here
        # while read or otherwise to create $input
        command "$@" "$input"
    else
        command "$@"
    fi
}

Pipe Wrapper Function Simple Example

Most common usage.

Simple Example STAT Program

This wraps the stat command to be able to accept standard input, especially from the stdout of a piped command at the same time allowing you to still pass normal arguments to the command.

STAT Program Function Wrapper

function stat_pipe {
    if [ ! -t 0 ]; then # Test if there is input
        local input
        while read -r input; do # read each input 
            if  [[ -f "$input" ]] || # this is extra logic
                [[ -d "$input" ]]; then # more extra logic
                stat "$@" "$input" # process input with command and arguments
            fi
        done
    else stat "$@"; fi # if no input then run the command as usual with arguments
}

STAT Program Function Example

Print my .bash_func files and get the file sizes

 $ printf '%s\n' .bash_func*
.bash_funcs
.bash_funcs_nocomments
.bash_funcs.old
 $ printf '%s\n' .bash_func* | stat_pipe -c %s
90329
53140
59990
 $ stat_pipe -c %s .bash_func*
90329
53140
59990

Pipe Wrapper Function Advanced Example

Something a little more advanced.

Advanced Example RM Program

A wrapper for the rm command that accepts more user input other than the initial input to decide if you want to delete a file or folder or not; if the else block is executed (not from pipe) then it runs the command as normal.

You should, of course, be careful with rm and therefore I added the extra test logic to add extra security.

This example is dependant on Bash as the shell.

RM Program Function Wrapper

function rm_pipe {
    if [[ ! -t 0 ]]; then
        local input input_user
        while read -r input; do
            if  [[ -f "$input" ]] ||
                [[ -d "$input" ]]; then
                printf 'Would you like to delete: %s: (y/[N])?\n' "$input"
                read -u 1 input_user # '-u 1' read from keyboard rather than stdin; defaults to No.
                if [[ "$input_user" =~ ^([yY]|[yY][eE][sS])$ ]]; then
                    rm "$@" "$input" # if user input is yes then process...
                fi
            fi
        done
    else rm "$@"; fi
}

RM Program Function Example

 $ cd fake
 $ printf '%s\n' *
a
b
c
d
 $ printf '%s\n' * | rm_pipe -rf
Would you like to delete: a: (y/[N])?
y 
Would you like to delete: b: (y/[N])?

Would you like to delete: c: (y/[N])?
no
Would you like to delete: d: (y/[N])?
yes
 $ printf '%s\n' *
b
c
 $

Conclusion

This makes almost any builtin command, function, or executable file program accessible to pipe in the Linux command line. I make these types of functions for many things and they end up saving lots of time and effort so I don't always have to write extra array creating logic first and/or the function does my logic for me.

Turn any Bash function/alias into a hybrid executable/source-able script.

Ian Pride — Wed, 03 Feb 2021 23:37:46 +0000

Convert any Bash Function/Alias to Hybrid Executable and Source-able Script with Bash Completion

...works with arguments and stdin.

NOTE:
I will be refactoring cronstat (a function you'll find below) again to process the switches and arguments better. COMING VERY SOON, just not sure when.

Introduction
Motivation
How it works
- Description
- Generic Example
  - To Execute
  - To Source With Completion
- Better Example
  - Example When Sourced
  - Example When executed As Script
Methods For Organization
Conclusion

Introduction

Whether you are new or old to scripting/programming in the Bash environment in Linux you have more than likely heard of, used, and/or created Bash functions and aliases. I will not be getting too much into what these are as this post states about the skill level that this is aimed at people who are familiar with these things. Having said that; I don't think that this will be too difficult for newer people to understand and, of course, all are welcome to read.

Motivation

There are a couple of ways to use functions and aliases; one is to source a file/script that contains the code and execute the function in the calling file and the other is to execute a script file with the code and pass arguments to the script and pass it to the function in the script.

Over the years I have written countless functions and scripts and for a long time I would at first keep functions in my normal .bash_profile and just keep transferring it to different machines, but after a while my file started to grow too large and so I started putting them in separate .bash_funcs/.bash_aliases files and just sourced them in whatever main profile file I was using to try and stay organized.

This method was fine, but I soon realized that functions and aliases can only be used in certain environments like scripts and command lines and not in things like Alt+F2, KDE Runner, AutoKey, or just set to a hotkey and so I realised I should just start putting functions in script files always so I can access them from anywhere and in any way.

I soon created this method from my understanding of the Bash shell environment. I don't know if anyone else uses this method, but I have never seen it before and all due respect to those who do use something like this.

How it works

Description

To call a function or alias from a script you must put said function (or alias) in a script and call script with any arguments you would pass to the function and in the same script underneath the function you would either put a bash complete command if the file is sourced or execute the function with any arguments passed to the script.

Generic Example

Generic, non-sense script example file fake_function.bash (or whatever):

#/usr/bin/env bash

function fake_function {
    if [[ $# -gt 0 ]]; then
        printf 'Argument: %s\n' "$@"
    else return 1; fi
}

# if file is sourced set Bash completion
if $(return >/dev/null 2>&1); then
    complete -W "word1 word2 word3" fake_function
else 
    # else if file is executed pass arguments to it
    fake_function "$@"
    # or || exit "$?" in some cases for errors
fi

To Execute

and then to execute the script with whatever method; for example:

 $ ./fake_function.bash "Line 1" "Line 2"
Line 1
Line 2

To Source With Completion

or to source into a script file with completion (script_file):

ff_path="/path/to/fake_function.bash"

if [[ -f "$ff_path" ]]; then # if the script exists
    . "$ff_path" # source the file and function
fi

fake_function "Line 1" "Line2" # call the function
# and call the function any time from the
# command line

Better Example

*NOTE*: This has been refactored to remove redundancy.

This is an actual example of a function and script I wrote called cronstat (cronstat.bash) that is a wrapper for the stat command that filters out the newest (default) or oldest files, directory, or both (default) in a directory. Great for when I need to find the latest project I did and forgot the name of or finding old, redundant files etc...

#!/usr/bin/env bash

function cronstat {
    local path_mode=1 time_mode=1 bare_mode=0 arg
    if [[ $# -gt 0 ]]; then
        for arg in "$@"; do
            if [[ "$arg" =~ ^-([hH]|-[hH][eE][lL][pP])$ ]]; then
                cat<<EOF

 'cronstat' - 'stat' wrapper to find the oldest
 and newest file, directory, or both from an
 arrayed or line delimited list.

 @USAGE:
    cronstat <LIST> [OPTIONS...]
    <LIST> | cronstat [OPTIONS...]

 @LIST:
    Any arrayed list of files or directories
    or the output of the 'find' or 'ls'
    commands etc...

 @OPTIONS:
    -h,--help       This help screen.
    -b,--bare       Print the path only, no
                    extra information.
    -f,--file       Filter by files.
    -d,--directory  Filter by directories.
                    Defaults to any file or
                    directory.
    -o,--oldest     Get the oldest item.
                    Defaults to the newest.
 @EXAMPLES:
    cronstat \$(find -maxdepth 1)
    find -maxdepth 1 | cronstat
    IFS=\$(echo -en "\n\b") array=(\$(ls -A --color=auto))
    cronstat \${array[@]} --file
    printf '%s\n' "\${array[@]}" | cronstat -odb

 @EXITCODES:
    0               No errors.
    1               No array or list passed.
    2               No values in list.

EOF
                return
            fi
            if [[ "$arg" =~ ^-([bB]|-[bB][aA][rR][eE])$ ]]; then
                bare_mode=1
                shift
            fi
            if [[ "$arg" =~ ^-([fF]|-[fF][iI][lL][eE])$ ]]; then
                path_mode=2
                shift
            fi
            if [[ "$arg" =~ ^-([dD]|-[dD][iI][rR][eE][cC][tT][oO][rR][yY])$ ]]; then
                path_mode=3
                shift
            fi
            if [[ "$arg" =~ ^-([oO]|-[oO][lL][dD][eE][sS][tT])$ ]]; then
                time_mode=2
                shift
            fi
            if [[ "$arg" =~ ^-([oO][fF]|[fF][oO])$ ]]; then
                time_mode=2
                path_mode=2
                shift
            fi
            if [[ "$arg" =~ ^-([oO][dD]|[dD][oO])$ ]]; then
                time_mode=2
                path_mode=3
                shift
            fi
            if [[ "$arg" =~ ^-([oO][bB]|[bB][oO])$ ]]; then
                bare_mode=1
                time_mode=2
                shift
            fi

            if [[ "$arg" =~ ^-([fF][bB]|[bB][fF])$ ]]; then
                bare_mode=1
                path_mode=2
                shift
            fi
            if [[ "$arg" =~ ^-([dD][bB]|[bB][dD])$ ]]; then
                bare_mode=1
                path_mode=3
                shift
            fi

            if [[ "$arg" =~ ^-([oO][fF][bB]|[oO][bB][fF]|\
                            [bB][oO][fF]|[bB][fF][oO]|\
                            [fF][oO][bB]|[fF][bB][oO])$ ]]; then
                bare_mode=1
                time_mode=2
                path_mode=2
                shift
            fi
            if [[ "$arg" =~ ^-([oO][dD][bB]|[oO][bB][dD]|\
                            [bB][oO][dD]|[bB][dD][oO]|\
                            [dD][oO][bB]|[dD][bB][oO])$ ]]; then
                bare_mode=1
                time_mode=2
                path_mode=3
                shift
            fi
        done
    fi
    local input array date iter index=0 value time_string="Newest" path_string="File Or Directory"
    declare -A array
    if [[ ! -t 0 ]]; then
        while read -r input; do
            case "$path_mode" in
                1)  if  [[ -f "$input" ]] ||
                        [[ -d "$input" ]]; then
                        date=$(stat -c %Z "$input")
                        array[$date]="$input"
                    fi;;
                2)  if [[ -f "$input" ]]; then
                        path_string="File"
                        date=$(stat -c %Z "$input")
                        array[$date]="$input"
                    fi;;
                3)  if [[ -d "$input" ]]; then
                        path_string="Directory"
                        date=$(stat -c %Z "$input")
                        array[$date]="$input"
                    fi;;
            esac
        done
    else
        if [[ $# -gt 0 ]]; then
            for input in "$@"; do
                case "$path_mode" in
                    1)  if  [[ -f "$input" ]] ||
                            [[ -d "$input" ]]; then
                            date=$(stat -c %Z "$input")
                            array[$date]="$input"
                        fi;;
                    2)  if [[ -f "$input" ]]; then
                            path_string="File"
                            date=$(stat -c %Z "$input")
                            array[$date]="$input"
                        fi;;
                    3)  if [[ -d "$input" ]]; then
                            path_string="Directory"
                            date=$(stat -c %Z "$input")
                            array[$date]="$input"
                        fi;;
                esac
            done
        else return 1; fi
    fi
    if [[ ${#array[@]} -eq 0 ]]; then
         return 2
    fi
    for iter in "${!array[@]}"; do
        if [[ $index -eq 0 ]]; then
            index=$((index + 1))
            value=$iter
        fi
        case "$time_mode" in
            1)  if [[ $iter -gt $value ]]; then
                    value=$iter
                fi;;
            2)  if [[ $iter -lt $value ]]; then
                    time_string="Oldest"
                    value=$iter
                fi;;
        esac
    done
    value="${array[$value]}"
    if [[ $bare_mode -eq 0 ]]; then
        printf '\n%s %s:\n%s\n\nLast Changed:\n%s\n\n' \
            "$time_string" \
            "$path_string" \
            "$value" \
            "$(stat -c %z "$value")"
    else
        printf '%s\n' "$value"
    fi
}
if $(return >/dev/null 2>&1); then
    complete -W "-h --help -o --oldest -f --file -d --directory -b --bare -of -od -ob -fb -db -ofb -odb '\$(find -maxdepth 1)'" cronstat
else cronstat "$@";fi

and just like the generic example above you would then either execute this file with any arguments or source it into your dot or script files and use however tied to any program or hotkey as mentioned before and these examples here:

Example When Sourced

 $ printf '%s\n' * .* | cronstat --oldest

Oldest File Or Directory:
examples.desktop

Last Changed:
2020-03-21 11:43:36.356069642 -0500

Example When executed As Script

 $ ls -A --color=auto | ~/.bash/profile/functions/cronstat.bash --oldest --directory

Oldest Directory:
.pki

Last Changed:
2020-03-23 08:09:05.447080464 -0500

Methods For Organization

A main issue with script files is that you can end up with a lot of them and it's horrible if you don't keep them organized and I end up dropping all of my function (.bash) files into a functions folder and aliases and then in either a dot profile or func file I will try and import them all like this:

(in .bash_funcs or .bash_profile etc...)

*NOTE*: This has been refactored to work with file names with '-'.

# Load funcs
function bash_func_src {
    local file
    for file in $(find "${HOME}/.bash/profile/functions/" -maxdepth 1 -type f -name "*.bash"); do
        . "$file"
    done
}
bash_func_src

# and aliases
function bash_alias_src {
    local file
    for file in $(find "${HOME}/.bash/profile/aliases/" -maxdepth 1 -type f -name "*.bash"); do
        . "$file"
    done
}
bash_alias_src

Conclusion

This method essentially makes any of your Bash code portable and accessible in all sorts of environments and allows you keep it in a script, transporting not only your code, but also examples of usage (in terms of the script itself) and a place to store external Bash completion code all the while archiving everything.

This allows you to execute these functions from a script for use anywhere especially in hotkeys (AutoKey), Alt+F2, KDE Runner, and any other way you can call a script with arguments.

I hope this helps someone and I'd love to hear if you do something similar or just any tips or comments are welcome.

Advice for building a cross-platform RUST gui

Ian Pride — Tue, 10 Nov 2020 21:37:00 +0000

TDLR; - What Rust desktop gui crate/framework do you use or recommend for use in Windows and Linux?

Need some advice on what some of you Rustaceans prefer to use to build desktop guis; especially cross-platform for Windows and Linux. I'll build it for Mac if there isn't much extra effort involved.

I've done my own research, but I'm just asking the community for your preferences/experiences. It's looking like Druid will be my first try unless I get some better advice, although I'll eventually try everything.

I recently re-wrote the CLI version of one of my programs (3rd iteration; 1st was AutoHotkey in Windows and 2nd is cross-platform in Python) in Rust and now I'd like to build the gui version for it. Not really much to the gui except for 2 i32 inputs, displayed text, and possible a copy to clipboard feature.

AutoHotkey Beginner Tip #2: SetBatchLines - Script Performance

Ian Pride — Sun, 24 May 2020 00:05:15 +0000

Programming/Scripting/Coding Language Performance

I won't be getting deep into the performance of programming languages (as that would draw debates I'm not ready to get into 😁), but if you are familiar with the subject then you probably understand that higher level languages (unless compiled sometimes) can be slower in some areas of computing compared to lower level languages like the C family and Assembly (among others) etc; though this doesn't always matter much anymore as CPU's got faster and RAM got bigger. This, of course, is not true for stuff you want to be doing at, say, NASA 😁.

AutoHotkey - Performance

AutoHotkey is by far my favorite language to code in Windows for various reasons for most everyday tasks from simple routine automation to full-on interactive programs with GUIs and in my opinion it compares to other languages of its kind when it comes to speed; if you know how to use the language well.

Performance Using SetBatchLines

There are a few ways to optimize performance in AutoHotkey for window interaction, key detection/execution, and what I am speaking about here: script execution. Mostly pertaining to loops and timers.

One method to improve script performance is by using the 'SetBatchLines' command.

Use SetBatchLines Command

By default AutoHotkey executes lines for 10ms and then sleeps for 10ms and this is great for misc default profile scripts or simple automation tasks, but when doing things more intensive (such as looping though indexed or associative arrays to execute/increment code/data) this can slow down the script (or timer/function) quite a bit with large indexed arrays.

Excerpt from AutoHotkey Docs:

Determines how fast a script will run (affects CPU utilization).
SetBatchLines, 20ms
SetBatchLines, LineCount

Parameters

20ms
(The 20ms is just an example.) If the value ends in ms, it indicates how often the script should sleep (each sleep is 10 ms long). In the following example, the script will sleep for 10ms every time it has run for 20ms: SetBatchLines, 20ms.

LineCount
The number of script lines to execute prior to sleeping for 10ms. The value can be as high as 9223372036854775807. Also, this mode is mutually exclusive of the 20ms mode in the previous paragraph; that is, only one of them can be in effect at a time.

SetBatchLines,-1
; [-1] Or the number of (# of iterations+exec lines)
; in a loop

is EXTREMELY important for performance when doing any kind of looping or SetTimers (maybe multithreading with DLL?) etc...

You can either run the whole program with no sleep in its execution by placing the command at the top of a script or you can turn it on and off inside the loop for iteration precision control e.g.:

;; Initialize vars
maxRange:=100000
array := range(1,maxRange) ;; my range function
tmpIntVar:=0

Set each iteration:

;; Set and reset each iteration
for index, item in array
{
    SetBatchLines,-1 ;; set befeore execution
    ; or SetBatchLines,% (maxRange+3)
    tmpIntVar++ ;; do something
    SetBatchLines,10ms ;; default value
}

or wrap the outside of the loop for smoothness:

SetBatchLines,-1
loop, % array.MaxIndex()
{
    tmpIntVar++
}
 SetBatchLines,10ms

Conclusion

The 'SetBatchLines' command is essential for controlling the flow and speed of a script and I believe a scource of confusion when it comes to why some people may think the language is slow or inefficient. With proper use of this command a program written in the language can be smooth.

AutoHotkey Beginner Tip #1: Assignment Vs Expression Operator Usage

Ian Pride — Wed, 27 Nov 2019 00:50:13 +0000

Over the years I have helped many people with learning/debugging AutoHotkey in forums and one of the most common areas where people have issues (confusion) while learning AutoHotkey is with the difference between the assignment & expression operators '=' & ":=" respectively. Both can assign value, but the former is purely assignment; the latter is used for any expression (code to be executed/evaluated).

Assigning value with the assignment '=' operator

varName = Any unquoted string.
varName = %anotherVarName%

Assigning value with the expression ":=" operator

varName := "Any quoted string."
varName := anotherVarName
varName := !anotherVarName?0:255
varName := !anotherVarName?"Yes":"No"
varName := ((anotherVarName == "whatImTestingFor")?"Value exists":"Value doesn't exist")

Differences

=
- Doesn't use quotation marks for strings.
- Variable names must be wrapped with %%.
:=
- Needs quotation marks for strings.
- Variable names don't need to be wrapped.
- Can contain any possible expression operation in the language.

Opinion

In my opinion there's only one use for '=' ever. :

'=' can be used to strip leading and trailing spaces from variables
- E.g. varName := " String with spaces "
- varName = %varName%
- Result: varName == "String with spaces"

Everything else can always be done with ':='. That, of course, is your preference.

My Disclaimer

This is my first post here & I'm very aware of the hate for AHK by many so I will write my disclaimer here once.

Even though I know quite a few other languages I will always be an AutoHotkey advocate.

Many people do not like the chaotic syntax of AHK, but once you understand it & create your own consistent style it's actually quite nice (IMO?). I am aware of it's inadequecies in certain areas, but like many other languages it is still useful for many things (We can even plug in to UWP with Selenium now :D).