Amit chakraborty

Posted on Oct 15 Originally published at amitchakraborty.dev

The DevOps bootcamp · 3. The shell: pipes, scripts and automating your own machine

#tutorial #devops #thedevopsbootcamp #beginners

The DevOps bootcamp · Chapter 3 of 14 · DevOps · new chapter every Thursday morning

By the end of this chapter: Automate a repetitive task end to end today.

The problem

You are a data‑analyst turned junior DevOps engineer at a small SaaS startup. Every evening you need to generate a CSV report of the previous day’s user sign‑ups, zip it, and upload it to an S3 bucket for the finance team. The manual routine looks like this:

Open a terminal on your laptop.
Run psql -d prod -c "SELECT * FROM users WHERE created_at >= CURRENT_DATE - INTERVAL '1 day'" > signups.sql.
Convert the SQL dump to CSV with psql -f signups.sql -A -F, -o signups.csv.
Zip the file: zip signups_$(date +%F).zip signups.csv.
Upload: aws s3 cp signups_$(date +%F).zip s3://finance-reports/.

You have been doing this for two weeks. Yesterday the aws s3 cp step failed with:

An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

You spent an hour checking IAM policies, then another half‑hour re‑typing the whole pipeline because you missed a stray space in the zip command. The finance team missed the report, you missed the daily stand‑up, and you realised you cannot keep repeating this fragile, error‑prone sequence.

You need a reliable, repeatable automation that runs with a single command, fails early with clear messages, and can be scheduled to run automatically each night.

Before you start

Requirement	Version / Source
Ubuntu Linux (or any Debian‑based distro)	22.04 LTS
Bash shell	5.1.16 (default on Ubuntu 22.04)
PostgreSQL client (`psql`)	14.5
`zip` utility	3.0-12
AWS CLI v2	2.13.5
`jq` (JSON processor) – optional for parsing AWS responses	1.6
`cron` daemon (already present on most Linux)	3.0pl1

Install any missing tools:

sudo apt-get update
sudo apt-get install -y postgresql-client zip awscli jq

Verify the environment:

$ bash --version | head -1
GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

$ psql --version
psql (PostgreSQL) 14.5

$ aws --version
aws-cli/2.13.5 Python/3.11.5 Linux/5.15.0-1051-azure exe/x86_64.ubuntu.22

If each command prints a version line without error, you are ready to proceed.

How can I capture command output reliably?

When you pipe data through several commands, a single failure can be hidden. set -o pipefail tells Bash to treat the pipeline’s exit status as the first non‑zero status, rather than the status of the last command.

#!/usr/bin/env bash
set -euo pipefail

-e aborts the script on any non‑zero exit code.
-u aborts on use of an undefined variable – catches typos.
-o pipefail propagates errors from any stage of a pipeline.

You can verify it works with a tiny test:

$ echo "ok" | false | cat
$ echo $?
1   # non‑zero because `false` was in the middle

How do I build a single‑command pipeline that is readable?

Long pipelines become unreadable when crammed on one line. Bash lets you split a pipeline across lines with a backslash (\) or by enclosing the whole pipeline in parentheses. Use comments to explain each stage.

generate_report() {
  psql -d prod -Atc "
    SELECT id, email, created_at
    FROM users
    WHERE created_at >= CURRENT_DATE - INTERVAL '1 day';
  " |
  # Convert TAB‑separated output to CSV
  tr '\t' ',' > "$1"
}

The function returns a CSV file whose name you pass as $1. The -A (unaligned) and -t (tuples‑only) flags remove extra formatting, making the output easy to pipe.

How can I schedule the script to run automatically?

cron is the simplest scheduler on a Linux workstation. A crontab entry of the form 0 2 * * * /path/to/report.sh runs the script at 02:00 every day. To avoid environment differences (e.g., PATH), always use absolute paths inside the script and source a minimal environment file.

Create a file ~/report.env:

# Minimal environment for the report script
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
export AWS_PROFILE="finance"

In the script, load it:

#!/usr/bin/env bash
set -euo pipefail
source "$HOME/report.env"
...

Test the cron entry without waiting:

$ /usr/bin/env bash -c 'cat /etc/crontab'   # shows system‑wide crontab format
$ crontab -l                               # list your user crontab (empty at first)
$ (crontab -l 2>/dev/null; echo "0 2 * * * $HOME/report.sh") | crontab -

You can confirm the entry was added:

$ crontab -l
0 2 * * * /home/you/report.sh

Build it

Below is a complete, runnable script called report.sh. Save it in $HOME/report.sh and make it executable (chmod +x report.sh).

#!/usr/bin/env bash
# report.sh – generate yesterday's sign‑up CSV, zip it, and upload to S3
# Requires: bash, psql, zip, awscli, jq
# Tested on: Ubuntu 22.04, Bash 5.1, PostgreSQL 14.5, AWS CLI 2.13.5

set -euo pipefail

# ---------- configuration ----------
source "$HOME/report.env"               # loads PATH and AWS_PROFILE
DB_NAME="prod"
S3_BUCKET="s3://finance-reports"
TMP_DIR="/tmp/report_$(date +%s)"        # unique temporary directory
REPORT_DATE="$(date -d 'yesterday' +%F)" # e.g. 2024-10-14
CSV_FILE="${TMP_DIR}/signups_${REPORT_DATE}.csv"
ZIP_FILE="${TMP_DIR}/signups_${REPORT_DATE}.zip"

# ---------- helpers ----------
log() {
  echo "[$(date +%T)] $*"
}

cleanup() {
  log "Removing temporary directory"
  rm -rf "$TMP_DIR"
}
trap cleanup EXIT

# ---------- main workflow ----------
log "Creating temporary workspace"
mkdir -p "$TMP_DIR"

log "Exporting sign‑ups from PostgreSQL"
psql -d "$DB_NAME" -Atc "
  SELECT id, email, created_at
  FROM users
  WHERE created_at >= CURRENT_DATE - INTERVAL '1 day';
" > "$CSV_FILE"

log "Verifying CSV is not empty"
if [[ ! -s "$CSV_FILE" ]]; then
  echo "ERROR: CSV file is empty – no rows returned" >&2
  exit 1
fi

log "Zipping CSV"
zip -j "$ZIP_FILE" "$CSV_FILE" >/dev/null

log "Uploading to S3"
aws s3 cp "$ZIP_FILE" "${S3_BUCKET}/signups_${REPORT_DATE}.zip" \
  --only-show-errors \
  --metadata report_date="${REPORT_DATE}"

log "Upload complete – verifying object exists"
aws s3 ls "${S3_BUCKET}/signups_${REPORT_DATE}.zip" | grep -q "$REPORT_DATE"
log "Report successfully uploaded"

# script ends; cleanup runs automatically

Step‑by‑step verification

Create the script – copy the code above into $HOME/report.sh.

   nano $HOME/report.sh   # paste, then Ctrl‑O, Enter, Ctrl‑X
   chmod +x $HOME/report.sh

Run it manually – this checks every stage without waiting for cron.

   $ $HOME/report.sh
   [12:00:01] Creating temporary workspace
   [12:00:01] Exporting sign‑ups from PostgreSQL
   [12:00:02] Verifying CSV is not empty
   [12:00:02] Zipping CSV
   [12:00:02] Uploading to S3
   upload: ./signups_2024-10-14.zip to s3://finance-reports/signups_2024-10-14.zip
   [12:00:03] Upload complete – verifying object exists
   [12:00:03] Report successfully uploaded

If any step fails, the script aborts and prints the error line prefixed with ERROR:.

Check S3 – confirm the object exists.

   $ aws s3 ls s3://finance-reports/ | grep signups_$(date -d 'yesterday' +%F).zip
   2024-10-14 02:00:03  12.3 KiB signups_2024-10-14.zip

Add to cron – schedule for 02:00 daily.

   (crontab -l 2>/dev/null; echo "0 2 * * * $HOME/report.sh >> $HOME/report.log 2>&1") | crontab -

Inspect the log after the first run – ensures the cron entry executed.

   $ cat $HOME/report.log
   [02:00:01] Creating temporary workspace
   …
   [02:00:03] Report successfully uploaded

You now have a single, repeatable command that can be invoked manually or automatically, with early failure detection and clear logging.

When this breaks

Symptom	Likely cause	Fix
`psql: could not connect to server: Connection refused`	Database host not reachable (network, firewall, or wrong `PGHOST`)	Verify `PGHOST` and `PGPORT` in `~/.pg_service.conf` or export them in `report.env`. Test with `psql -d prod -c '\q'`.
`ERROR: CSV file is empty – no rows returned`	No rows match the date filter (time‑zone mismatch)	Use `SET TIMEZONE 'UTC';` in the query or adjust `date -d 'yesterday'` to match DB timezone.
`zip: command not found`	`zip` not installed or not in `$PATH`	Install with `sudo apt-get install zip` or add its location to `PATH` in `report.env`.
`An error occurred (AccessDenied) when calling the PutObject operation`	AWS credentials missing or IAM policy lacking `s3:PutObject` for the bucket	Ensure `AWS_PROFILE` points to a profile with correct permissions (`aws sts get-caller-identity`). Update IAM policy or use `aws configure` to set proper keys.
`aws s3 ls …	grep -q …` returns non‑zero	Object not uploaded (network glitch)
`trap: line 1: EXIT: not a function` (when running under `sh`)	Script executed with `sh` instead of `bash` (POSIX shell lacks `trap` syntax)	Ensure the shebang `#!/usr/bin/env bash` is present and the file is executable. Run with `bash report.sh` to verify.

When a failure occurs, the script stops at the first error because of set -e. The printed log line tells you exactly which stage failed, allowing rapid diagnosis.

What it costs

Portability – The script relies on GNU coreutils, Bash, and the AWS CLI. It will not run unchanged on non‑Linux platforms (e.g., macOS without Homebrew‑installed equivalents) or on Windows without WSL. If you need true cross‑platform automation, consider a language‑agnostic tool like Python or a CI pipeline.
Maintainability – Inline SQL and Bash logic are tightly coupled. Adding a new column or changing the date filter requires editing the script directly. Over time, a growing script can become hard to read. For larger pipelines, extracting the query to a .sql file and using a configuration file for parameters improves clarity.
Observability – The script writes only to a simple log file. It does not emit structured metrics (e.g., Prometheus counters) or alerts on failure beyond the exit code. If you need production‑grade monitoring, integrate with a log‑aggregator (e.g., Fluent Bit) or add a curl call to a monitoring webhook on error.
Security – The script inherits the AWS credentials from the environment. If the workstation is compromised, the attacker gains write access to the finance bucket. Using IAM roles with limited scope or a dedicated CI runner that holds the credentials reduces risk.
Scalability – This approach works for a single daily report. If you later need to generate dozens of reports in parallel, Bash becomes unwieldy; a job‑queue system or a proper workflow engine (e.g., Airflow) would be more appropriate.

In short, the Bash solution is cheap, fast, and perfect for a single‑person, low‑frequency task, but it does not scale indefinitely.

In the interview

Typical interview prompt

“Describe a time you automated a repetitive manual task on your workstation. Walk me through the script you wrote, how you made it reliable, and how you scheduled it.”

Weak answer – “I wrote a shell script that called psql and aws s3 cp. It worked and I set a cron job.”

Why it’s weak: It mentions the tools but gives no insight into error handling, testing, or why the candidate chose Bash over alternatives. It also omits any discussion of failure modes or trade‑offs.

Strong answer – The candidate should mention:

The problem (manual steps, error‑prone, impact on stakeholders).
Design decisions – chose Bash because the task runs on a single Linux workstation, required no external runtime, and could be scheduled with cron.
Reliability techniques – set -euo pipefail, explicit exit codes, temporary directory cleanup, early validation of CSV size, and logging with timestamps.
Verification – manual run with expected log output, aws s3 ls check, and a test of the cron entry.
Trade‑offs – limited portability, lack of structured monitoring, and future scaling concerns.
Outcome – reduced nightly effort from ~30 minutes to a single command, eliminated missed reports for two weeks, and gave the finance team a reliable SLA.

Seniority nuance

Junior – Expected to explain the what (script, cron) and how (commands used).
Senior / Staff – Must discuss why Bash was chosen, the failure modes they guarded against, and the cost of the approach (maintainability, security). They should also mention how they would evolve the solution (e.g., move to a CI pipeline, add monitoring).
Director / Engineering‑manager – The focus shifts to impact (business value, risk reduction) and process (how the automation fits into the team’s runbook, hand‑off, and documentation). They may be asked how they ensured the team could audit the script and what governance they placed around credentials.

Follow‑up that separates experience from theory – “You said you added set -euo pipefail. What happened when the zip command failed because the disk was full? How did you surface that to the on‑call engineer?” A candidate who has actually run the script will describe the early exit, the log line, and the fact that the temporary directory was left for investigation, whereas a purely theoretical answer may gloss over the concrete symptom.

Your tasks

Add a column – Extend the SQL query to include the created_at timestamp in ISO‑8601 format (to_char(created_at, 'YYYY‑MM‑DD"T"HH24:MI:SSZ')). Verify the new CSV has four columns and that the script still uploads successfully.
Make the bucket configurable – Move the S3_BUCKET variable to a command‑line argument (--bucket) with a default value. Update the usage message and test by uploading to a test bucket you create in AWS.
Introduce a failure – Manually set AWS_PROFILE to a non‑existent profile and run the script. Observe the error, then modify the script to catch the aws sts get-caller-identity failure and exit with a clear message: “ERROR: AWS credentials not found”.
Add monitoring – Append a line at the end of the script that sends a JSON payload to a dummy webhook (https://httpbin.org/post) containing date, status: "success", and size_bytes of the uploaded zip. Use curl -s -X POST -H "Content-Type: application/json" -d @payload.json. Verify the request succeeds with jq parsing the response.
Refactor for readability – Split the script into three files: config.env, report.sh (the driver), and helpers.sh (functions like log, cleanup). Source helpers.sh from report.sh. Ensure the refactored version behaves identically and passes the manual run test.

When you have completed each task, run the script end‑to‑end and confirm:

The log contains timestamps for every stage.
The uploaded object appears in the S3 bucket with the correct name.
The webhook receives a JSON payload with the expected fields.

If any step deviates, revisit the relevant section of this chapter.

Your tasks this week

Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.

Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.