DEV Community: Jaro

Advanced bash scripting with awk

Jaro — Mon, 03 Jan 2022 13:38:36 +0000

Overview

AWK is an interpreted programming language. It is very powerful and specially designed for text processing. Its name is derived from the family names of its authors − Alfred Aho, Peter Weinberger, and Brian Kernighan.
https://www.tutorialspoint.com/awk/awk_overview.htm

The version of AWK that GNU/Linux distributes is written and maintained by the Free Software Foundation (FSF); it is often referred to as GNU AWK.

AWK is a very powerful tool for

Text processing
Producing formatted text reports
Performing arithmetic operations
Performing string operations, and many more

AWK breaks each line of input into fields where a field is a string of characters separated by a whitespace. Don't worry though, this is also a configurable parameter. You can actually also manage what delimiter you want. AWK has its specific strength when having to handle structured text files, tables, or chunk structured data.
First of all, let's start with a simple example where we format the output of a string:

echo tomatoe potatoe | awk '{print $1}'
# tomatoe 

echo tomatoe potatoe | awk '{print $2}'
# potatoe

awk '{print $3}' $filename
# Prints field #3 of file $filename to stdout.

awk '{print $1 $5 $6}' $filename
# Prints fields #1, #5, and #6 of file $filename.

That is one of the core features of AWK, the print command. Now we just played with orders in how matches occur in the string. What is actually possible with AWK is manipulating patterns. You could for exemple print a line of a file where the pattern you requested matched. Or for example print all the possible http error code 500 in a web server log.

awk '$9 == 500 { print $0}' /var/log/httpd/access.log

Of course, here the $9 refers to the position of where the error code appears.
The part actually outside of the curly brackets represents the pattern we are looking for and the part inside represents the action to be executed by AWK. You can use all of the possible comparison operators that are also included in C language such as : ==, !=, <, >, <=, >=, ?:

If you do not pass any pattern the action applies to all lines. If no action is given, the entire line gets printed.

AWK is actually a programming language which means it can also perform arithmetic operations.
Let's say we have a file with a column of numbers and we want to get the sum of all the numbers in that column. Keep in mind this column is simply placed at position 1 of that file such as it resembles to this:

In order to gather the sum we use operator brackets such as :

awk '{total += $1} END {print total}' file.txt

What happens here is kind of like a loop, where awk gets all the first position words on each line, declares at first a variable called total, sums those words or here numbers into that variable and finally when this operation is over, prints that variable.

Now let's try to use all of that into some more useful stuff such as getting your CPU temperatures from the sensors utility and stripping out the "+" and "°C" from it.
Getting the temperature is usually done by typing sensors but this might vary depending on what distro/OS you are using.

$ sensors
dell_smm-virtual-0
Adapter: Virtual device
Processor Fan: 2706 RPM
CPU:            +44.0°C  
Ambient:        +37.0°C  
SODIMM:         +36.0°C  

$ sensors | grep CPU
CPU:            +44.0°C  

$ sensors | grep CPU | awk '{printf "%d\n", $2}'
44

Now let's try to do some more programming language logic and try to implement our if-else logic. Below is an example that pipes only the data and then it changes a column value to a string based on a condition:

$ cat pi_data.txt
time temp wave(ft) comments
---- ---- -------- --------
10:00 24   3       No wind
12:00 26   5       High winds
14:00 25   4       wind calming down

$ # Show time and small or medium for wave size
$ cat pi_data.txt | \
>   awk '{if ( $1 ~ /[0-9]/ ) print $0'} | \
>   awk '{if ($3 < 4) {print $1 "\t small"} else { print $1 "\t medium"} }'
10:00    small
12:00    medium
14:00    medium

A single AWK command to adjust the title and then change the data:

$ cat pi_data.txt | \
>   awk '{if ( $1 ~ /[0-9]/ ) \
>            { \
>               {if ($3 < 4) {print $1 "\t small"} else { print $1 "\t medium"} } \
>        } else { print $1 "\t " $3} \
>        }'
time     wave(ft)
----     --------
10:00    small
12:00    medium
14:00    medium

And finally and more importantly, killing tasks, one of the most useful tool in existence.

#!/bin/bash
#
# stop_task.sh - stop a task
#
task1="edublocks"

echo "Stopping $task1..."
ps -e | grep -E $task1 | \
 awk '{print $1}' | xargs sudo kill -9 1>&-

Final comments

I’ve found that learning a little bit of AWK has really paid off.

AWK supports a lot of functionality and it can be used to create full on scripting applications with user inputs, file I/O, math functions and shell commands, but despite all this I’ll stick to Python if things get complex.

Bash scripting introduction

Jaro — Mon, 20 Dec 2021 13:22:00 +0000

Bash scripting is a powerful tool. If you have been working on a UNIX based system, after quite some time, you end up working with it because you just realize that it exists for simplicity, efficiency and scalability.

One program's output shall be another one's input.

That sums up how things are done and how you should think them too. Now before wanting to do anything fancy with bash scripting, there are already a lot of tools that boost your productivity and that I realized I was only using 10% of the actual power.
The aim of that article is to go through some of them, view some very useful tricks and hopefully, interest you in going further into that topic.

Brace expansion

Brace expansion is a very useful tool to manipulate lists of "things". It is used in most cases to generate lists, files with redundant naming(s) etc... This tool allows you to save time and avoid mistakes because you do not need to type that much to generate those lists.
Something worth noting about bash scripts is the fact that before the bash shell actually executes a command it will check whether it needs to perform any substitutions on the command. You probably used that with variables, or aliases before. Brace expansion is one of the multiple expansion forms that are supported by bash. Be wary, those expansions are not available in old shells. But most of current distros will allow you to use them.
Brace expansions are contained between a pair of {} braces. Inside of those will be comma-separated items or a range specifier.
The most common example could be done like that :

echo {1,2,3,4,5}

This will output the values within the brackets. Do you get where this is going and how you can use that? Not yet? Look at this other example with a range this time:

# Will output all numbers from 1 to 10
echo {1..10}
# Will output all characters from A to Z and a to z
echo {A..z}

Now imagine you had the need to create 10 files with the same name but with a specific delimiter

touch filename{1..10}.txt

You are now able to create 10 files in one single command line.

GREP

There is a handy tool called grep which lets you search files for specific patterns.
Let's take a simple example and go ahead an create a simple log file.

ping google.com > /your/destination/output-logfile.log

Let that command run for a few seconds and kill the process. You should be left with a file with several lines in it.
Now if you would just want to read the file, you would run

cat /your/destination/output-logfile.log

But that would print out everything. What you want is to only view the line where the statistics are displayed. This is where grep comes in handy with something like this:

cat /your/destination/output-logfile.log | grep packets

Where the word packets is the word contained in one line I want to display. The output would be something like :
7 packets transmitted, 7 received, 0% packet loss, time 6009ms
Now this outputs us the whole line but we can actually format the output to only display what is interesting to us.
Let's imagine I only want to see the number of packets that were transmitted. This can be done by piping an additional command after the grep. Also, instead of creating a file and reading it with grep, we will be running grep directly on the running script :

ping -c 1 google.com | grep 'packets' | cut -d , -f 1

In this command we say :

ping once google.com and give me the details
only display the line with the word packets in it
cut everything after the first delimiter ',' and only output the first group which leaves us with : 1 packets transmitted

Saving commands in script files

Now all of those shell tools are useful but sometimes they can get lengthy and this is exactly why script files exist. Most of the time you want to reuse those useful scripts you designed.

Creating the file

Most script files for good notice start with
#!/bin/bash. This is called the she-bang(shabang) which derives from the concatenation of the tokens sharp (#) and bang (!). This line is pretty important as it defines which shell is gonna interpret your script. /bin/bash is usually implemented as a symbolic link pointed to the executable of whichever shell is the system default shell.
You should probably always start your scripts also by some little documentation appending it to your shabang

#!/bin/bash
# This is a script to list apples

This way you can always come back to your script and actually understand what it was meant for without having to read the code in it.
Now our very simple script starts with

#!/bin/bash
# This is a script to list apples

echo "I have 3 apples"

Not super exciting is it?
Well we can try to add variables

#!/bin/bash
# This is a script to list apples
a=Hello
b="Good Morning"
c=19
echo $a
echo $b
echo $c

echo "$b! I have $c apples"

Here your script has 3 variables defined that you can reuse by using the $ sign before the variable name.
Now let's try to use that with our ping and grep command:

#!/bin/bash
# This is a script to ping and view packets transmitted
a=$(ping -c 1 google.com | grep 'packets' | cut -d , -f 1)
echo $a

Now 2 in 1, we execute a child script in our script, store the value in our variable and echo the variable.

Now all of those examples are to demonstrate how shell can be powerful rather than actually explain how things can be done with existing Linux tools. If you want to dig deeper, remember that you can always type man your tool and you will be prompted with a full documentation or link to wherever you can find information on how to use your tool.

Get started with deno (from NodeJS or not...)

Jaro — Fri, 10 Dec 2021 12:56:01 +0000

If you have been working with NodeJS or not even, you surely heard about Deno (no-De) and that's no hazard. Javascript, Typescript and NodeJS are technologies that have been on the rise for quite a while now. Well what if I told you that those three technologies are actually implemented by default in Deno, without you having to do much to set them up if not any.

A little bit of history
Differences with NodeJS
From scratch web-server

A little bit of history

Just like NodeJS, Deno is a runtime for Javascript. And also like NodeJS it is Authored by the same person that worked on NodeJS, mister Ryan Dahl. This project is actually not that recent. It was first announced in 2018 during a talk "10 things I regret about NodeJS". It's important to take this lightly as NodeJS is still and will continue to be a stable and robust environment to code in. But some initial design decisions have ignited a will to change or enhance the existent. Deno was first written in Go but was then rewrote in Rust, another rising technology that I would definitely recommend you learn about. Deno's official 1.0 version was released on May 13, 2020 and the current lts version is 1.10.0.

NodeJS comparison

Security

The most important change, and difference that deno brings compared to NodeJS is security.
Deno's core was written in Rust and Tokio which is the asynchronous run time for Rust. Rust itself is a language that is focused on performance and safety most importantly when it comes to concurrency. It provides a safer way to manage memory without having to use a garbage collector. On top of that, deno does not allow any type of file, environment or network access unless you actually give it the rights. Even so, those permissions are scoped and do not provide a wide range of access.

Strict, typed and linted

Across different projects, if we want to have these 3 principles in place with NodeJS, a lot of configuration, installation and tweaking are required. This pollutes the workspace and complicates deployments.
Deno ships Typescript, a linter and an assertion library out of the box without needing any configuration whatsoever. You can still configure if you wish but that will probably not be needed.

From scratch web-server

In any case, Deno ships a lot more things but this is enough to get us started. Any additional information can be found on the official repository url that I will be linking at the end of the article.
As for a little start, since nodejs has been used as a very popular web server, I thought it might be interesting to start building a deno version of it.
To start off, let's install deno. Depending on your OS you might want to refer to Install Docs

Shell

curl -fsSL https://deno.land/x/install/install.sh | sh -s v0.38.0

That's basically it! We don't need to look at typescript configurations we can directly start up creating our first deno file.

touch api.ts

Now just like nodejs, Deno ships a ready to use http server package that we need to import.

import { serve } from "https://deno.land/std@v0.42.0/http/server.ts";

Next, let's go ahead and import our sql. Of course, I am using sqlite for demo purposes but you might go ahead and get whatever suites your needs.

import { DB } from "<https://deno.land/x/sqlite/mod.ts>";

Finally, let's add some simple sample of code to run our http server

const PORT = 8162;
const s = serve({ port: PORT });

console.log(` Listening on <http://localhost>:${PORT}/`);

for await (const req of s) {
  req.respond({ body: "Hello World\\n" });
}

And voila! A running http server. Now compared to nodejs, what was not needed to do beforehand:

Install manually the sqlite dependency
Install manually typescript, and configure Now, we did say a running http server but that may not mean it is fully functional. Remember we said authorizations are required in order for deno to access different functionalities? Well http or network access requires a specific authorization. In order to do this we must append a flag to our deno command as such:

Deno api —allow-net

We can now go ahead and on our localhost:8162 to check out our newly created HTTP Deno server!
Now on to using that sqlite module we imported. It wouldn't be a >simple< get started tutorial with some basic database action so here is a snippet of code to get started :

// Open a database
const db = new DB("test.db");
db.query(`
  CREATE TABLE IF NOT EXISTS people (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT
  )
`);

const names = ["Peter Parker", "Clark Kent", "Bruce Wayne"];

// Run a simple query
for (const name of names) {
  db.query("INSERT INTO people (name) VALUES (?)", [name]);
}

// Print out data in table
for (const [name] of db.query("SELECT name FROM people")) {
  console.log(name);
}

// Close connection
db.close();

Now we can open a database, run a query and print out data. If you weren't in a Deno article, you would have probably never realized that this is not nodejs and that is the whole point! Deno is not, a fundamental coding change coming from nodejs. And if you start with Deno, you could totally switch to nodejs without loosing any of the acquired knowledge.

Conclusion

Now as you can see if you come from nodejs or from a JavaScript environment, there isn't much difference from what you know. Although, if you actually come from nodejs, you would find interesting that the root nodejs api is fully promised built. The difference and additions are more at the root of the software. As a growing technology, nodejs needed a security upgrade. Again as a growing technology, nodejs also needed a simple environment that is not bloated in dozens of config files. As of me, I would use Deno for from scratch projects. I don't feel like I am getting onboard of a totally new ship. Documentation is understandable and modules are maintained if they are featured on the official deno.land page. Now should you upgrade your existing projects from nodejs to deno? Deno does give you some upgrading best practices and guidelines but it is not something that will get done with a snap of the fingers. Keep in mind, deno is not here to replace node but to give a more secure and less error-prone environment. Node is still going to be out there and will probably never get less popular than Deno. But Deno is a good alternative if you are looking for a strict, secure and tidy Javascript runtime environment to code your favorite projects on!