Advanced Scripting Techniques in Linux Shell Scripting

Introduction:
Greetings again! Remember our discussion on mastering Git and the initial guide on Linux shell scripting for DevOps beginners? If you do, great! If not, no worries—whether you're a returning reader or a newcomer, welcome to the next phase of our DevOps scripting journey.

Building upon the foundations laid in our previous article, we're now stepping into more advanced territories. Thanks to your valuable feedback and suggestions, this article focuses on advanced scripting techniques. These techniques will not only reinforce your existing knowledge but also open doors to new possibilities in automation, performance optimization, and security enhancement within a DevOps environment.

So, fasten your seatbelts as we dive into the intricacies of advanced Linux shell scripting. Whether you're a beginner eager to explore new horizons or an experienced scripter seeking to refine your skills, this guide is tailored to meet you where you are in your DevOps scripting adventure.

1. Error Handling: Navigating Script Imperfections

Errors are inevitable, but handling them gracefully distinguishes a proficient scripter.
Here's how to implement effective error handling:

• Conditional Statements for Errors:

  • Identify potential points of failure in your script.
  • Implement conditional statements to detect errors.
  • Example:

     if [ $? -ne 0 ]; then
         echo "An error occurred. Exiting."
         exit 1
     fi
  

• Graceful Exits:

  • Use exit codes and messages for a clear understanding of script termination.
  • Consider whether the script should halt or proceed upon encountering an error.

• Trap Commands for Signals and Interrupts:

  • Use trap commands to handle signals and interrupts that may terminate your script unexpectedly.
  • Example:

     # Define a cleanup function
     cleanup() {
         echo "Cleaning up temporary files..."
         rm -f /tmp/*.tmp
     }
  
     # Trap the EXIT signal and call the cleanup function
     trap cleanup EXIT
  

2. Logging: Enhancing Visibility in Script Execution

Logging is like a script's journal, offering insights into its execution. Learn how to incorporate logging effectively:

• Importance of Logging:

  • Understand why logging is crucial for script development and debugging.
  • Explore different logging levels (info, warning, error) for varied messages.

• Adding Logging to Scripts:

  • Integrate logging commands at strategic points in your script.
  • Example:

     log() {
         echo "[INFO] $1"
         # Redirect to log file: echo "[INFO] $1" >> script.log
     }
  

• Using Syslog or Other Logging Frameworks:

  • Use syslog or other logging frameworks to standardize and centralize your logging output.
  • Example:

     # Install logger command: sudo apt install bsdutils
     # Log a message to syslog: logger "Hello, world!"
     # View the syslog: sudo tail /var/log/syslog
  

3. Debugging: Unraveling Script Mysteries

Debugging is an art—master it to troubleshoot your scripts effectively:

• Techniques for Debugging:

  • Insert echo statements strategically to trace script execution.
  • Utilize the set -x option for detailed command tracing.
  • Example:

     # Enable debug mode
     set -x
  
     # Your script commands
  
     # Disable debug mode when done
     set +x
  

• Debugging Tools:

  • Familiarize yourself with tools like bash -x script.sh for on-the-fly debugging.
  • Use shellcheck or other code analysis tools to check your script for syntax errors and best practices.
  • Example:

     # Install shellcheck: sudo apt install shellcheck
     # Check your script: shellcheck script.sh
  

4. Real-world Applications: Bringing It All Together

• Practical Examples:

  • Apply advanced techniques in real-world scenarios.
  • Automate tasks with robust error handling, logging, and effective debugging.
  • Example:

     # A script to scrape a website and extract data
     #!/bin/bash
  
     # Define a log function
     log() {
         echo "[INFO] $1"
  
     }
  
     # Define a cleanup function
     cleanup() {
         log "Cleaning up temporary files..."
         rm -f /tmp/*.html
     }
  
     # Trap the EXIT signal and call the cleanup function
     trap cleanup EXIT
  
     # Check if curl is installed
     log "Checking if curl is installed..."
     if ! command -v curl &> /dev/null; then
         log "curl is not installed. Exiting."
         exit 1
     fi
  
     # Download the website
     log "Downloading the website..."
     curl -s -o /tmp/website.html https://example.com
  
     # Check if the download was successful
     log "Checking if the download was successful..."
     if [ $? -ne 0 ]; then
         log "An error occurred while downloading the website. Exiting."
         exit 1
     fi
  
     # Extract the data
     log "Extracting the data..."
     grep -o '<h1>.*</h1>' /tmp/website.html | sed 's/<[^>]*>//g' > /tmp/data.txt
  
     # Check if the extraction was successful
     log "Checking if the extraction was successful..."
     if [ $? -ne 0 ]; then
         log "An error occurred while extracting the data. Exiting."
         exit 1
     fi
  
     # Display the data
     log "Displaying the data..."
     cat /tmp/data.txt
  
     # Exit with success
     log "Script completed successfully."
     exit 0
  

Conclusion:
Armed with advanced scripting techniques, you're well-equipped to handle the complexities of real-world scripting challenges. Remember, the journey to mastering Linux shell scripting is ongoing—keep exploring, practicing, and refining your skills. Happy scripting!

I hope you enjoyed this guide on advanced scripting techniques in Linux shell scripting.
I’ve shared with you some of the best practices and tools for error handling, logging, and debugging.
These skills have helped me a lot in my scripting projects, and I’m sure they will help you too.
If you have any questions or feedback, please let me know.

Comments

[Ivan Zakutnii]

Hello! I guess that the last often-forgotten technique is to get rid of Bash and use any other scripting programming language.

Python is a better option because it comes out of the box with almost any Unix distribution today.

I don't mean to be rude; your post is nice, and I have been, um... using Bash a lot for almost half a decade.

But... it's Bash, come on.

It's so rudimentary and far from a painless life, especially after writing more than 15 lines of a script. There is no type system, no data structures, no concurrency, no fine linter, etc.

My team is so exhausted from using Bash, even in GitLab CI/CD scripts, that we are mercilessly transitioning to Python everywhere.

[Kevin Naidoo]

Nice article @arbythecoder. I hate and love BASH too :-), but it's better than Python and other languages in certain instances, e.g. how would we do this without BASH :

find /logs -type f -mmin +60 -exec rsync -av {} /backups/

In Python, you going to write a bunch more lines of code (Not to mention indentation). BASH is quick and powerful for system administration tasks.

However, I agree for longer scripts like CI scripts - Python is definitely a better choice. Even Perl.

[Ivan Zakutnii]

article is awesome, btw
@arbythecoder <3

[Arbythecoder]

Thank you for sharing your view, It's true that Bash lacks many of the features found in more modern languages like Python, and it can be quite challenging to work with for complex tasks.
Bash is more lightweight and suitable for simpler tasks, like @kwnaidoo mentioned.

Python does offer a more extensive set of features, including a robust type system, versatile data structures, and built-in support for concurrency.
In my case, I've found Bash to be handy for quick and concise scripts, but I see the advantages of using Python, especially for more complex tasks .
The truth is that every tool has its strengths, and the choice often depends on the specific requirements of the project/ task at hand.
It's great that your team has found Python to be a better fit for your needs.

I appreciate your insights, and it's valuable to hear about different approaches to scripting. If you have any specific examples or tips for transitioning from Bash to Python, I'd love to learn more!

[Arbythecoder]

Thank you so much for reading and sharing your view, i appreciate

[Ivan Zakutnii]

Yeah, yeah, the shell is forever with us, and I am absolutely happy with in and in love with the UNIX way of pipeline processing.

My .bash_profile or .zshrc, and even crontab, are always full of such short and nice things written with Bash and POSIX/GNU utilities.

As I mentioned above, things become a pretty f****** nightmare when you need to robustly handle some state, and even worse when such state is represented by a data type that is just a bit more complex than a file of strings...

You know, JSON is a string, but writing a Bash script with school-level cyclomatic complexity for parsing/operating on JSONs, even with neat tools like JQ... Just no, dude. I'm sure you feel this pain in my words.

[Ivan Zakutnii]

You know, I am crying here because, for a few years of working in the OPS field, I've met too many people obsessed with Bash. They are basically trying, from time to time, to write a f****g monolith only with Bash. And this bunch of ramen Bash scripts is almost always completely unreadable, just because it's Bash, and its syntax is pretty noisy.

Besides that, this ramen is absolutely untestable, and maintaining it is a pain in the ass :(

Bash is an awesome instrument for a specific set of tasks; not knowing and not respecting such limitations is just... madness and incompetence, in my honest opinion.

[Kevin Naidoo]

Yeah. know what you mean for sure. BASH gets abused and used in places it shouldn't. I think it's because initially sysadmins, would write these scripts not programmers. I can't imagine any modern day developer that will prefer BASH over something like Python.

I use BASH because I have to, some system operations are much more complicated in a language like Python, whereas in BASH - you just do a one liner and pipe the output to the next command.

If you going to write a script with - lets say more than 10 lines of code, then BASH yeah is a big No No!

[Jan Semmelink]

Using both python and bash, there is a place for both and bash is still the best tool in many cases.
Thanks for the article.

[Arbythecoder]

Thank you for your time @jan_semmelink_98d129f0c34

[Moray Macdonald]

If you need type systems and complex data structures then by all means use something more complex. Shell scripting was only designed to work with plain text files, and even though you CAN do more complex stuff with it, I would much rather manipulate JSON files in Python than with jq!

However shell scripting (whether with Bash, sh or whatever) is still incredibly useful for automation as it runs EVERYWHERE. Even your most minimal container image will still have sh installed. Writing cron jobs? Shell. Writing container entry points? Shell. Want to quickly grab some lines out of multiple log files in the middle of an major incident? I can do it shell before Kibana/Splunk/etc has even logged in.

[Nicholas Drone]

Yes but which version of Python? is it 2 or 3? At least with Bash the script works no matter which version of the shell is installed.

Another point is not all Docker images if are using them have Python installed. Especially since most are trying to keep their image sizes as small as possible. Bash or any shell is installed by default with the exception of windows.

Also note that Python is a OOP language and not a shell script language.

[Ivan Zakutnii]

please stop it

[Ellis]

Use the right tool for the right job.

Bash and Python are very different tools for very different purposes, despite the fact that they have an overlap. Bash is mainly for accessing the file system, running processes on these files, often in a chain using pipes for instance.

One of the main things Python specialises in is text processing, which is by far not a main focus for Bash.

So imho, with due respect, if you find yourself comparing these two tools, perhaps you haven't understood what each one is for. Otherwise we're comparing a fish with a butterfly :o)

[Ramazan Efe Terzi]

curl -s -o /tmp/website.html https://example.com

# Check if the download was successful
log "Checking if the download was successful..."
if [ $? -ne 0 ]; then
...

This filter does not check curl's exit code, it does log's exit code.