DEV Community

Vijay Vinoth
Vijay Vinoth

Posted on Originally published at artificial-inteligence.phptutorial.co.in

AI-Enhanced Log Analysis and Anomaly Alert System — Part 2: Setting Up Log Collection & Centralization with Shell Scripts

#ai #loganalysis #python #shell

AI‑Enhanced Log Analysis and Anomaly Alert System — Part 2: Setting Up Log Collection & Centralization with Shell Scripts

Based on my technical understanding as a Lead Programmer Analyst (PHP, Perl, Python, Shell) and the rapid evolution of AI‑driven observability in 2026, this guide walks you through a production‑ready, shell‑centric pipeline for gathering, normalizing, and centralizing logs before the AI layer even sees them.

Quick recap of Part 1 – In the opening tutorial we defined the end‑to‑end architecture (edge agents → embedding service → vector store → anomaly detector) and evaluated the latest AI models (Claude 4.6 Opus, GPT‑5.4 Pro) that power the “semantic‑search‑first” approach to incident detection.

Now we turn our attention to the foundation of any AI‑enhanced observability stack: getting the raw log data into a single, searchable lake. In 2026, the industry consensus (see the Khimananda blog and the ShopClawMart case study) is that a lightweight, script‑driven collector is still the most flexible way to feed modern AI services without adding latency to the application tier.

Why a Shell‑First Collector Still Makes Sense

  • Zero‑dependency footprint: Most Linux hosts already ship bash, rsync, cron, and systemd‑journalctl. No heavyweight agents are needed.
  • Deterministic control: You decide exactly when files are rotated, compressed, and shipped – a critical factor when you batch‑process logs through an embedding micro‑service (e.g., all‑MiniLM‑L6‑v2 sidecar) as recommended by Khimananda.
  • Security & compliance: Centralized storage under a single OS user makes ACLs, audit‑logging, and Zero‑Trust policies straightforward (see the CSA whitepaper on “Analyzing Log Data with AI Models to Meet Zero Trust Principles”).

Overall Flow Diagram (textual) 


┌─────────────┐   1. tail /var/log/*.log   ┌───────────────────┐
│   Edge Host │ ───────────────────────► │  Collector Script │
└─────┬───────┘                         └───────┬───────────┘
      │                                      │
      │ 2. Rotate & compress (gzip)          │
      ▼                                      ▼
┌─────────────┐   3. rsync/ssh   ┌───────────────────────┐
│   /var/log  │ ───────────────► │ Central Log Repository │
└─────────────┘                 └───────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Step‑by‑Step Implementation

1. Prepare the Central Log Repository

We’ll use a dedicated “log‑hub” server that runs a simple directory‑based store. In production you could replace this with an ELK stack, Loki, or Uptrace, but the script‑driven approach works with any backend.

CommandPurpose

sudo useradd -r -m -s /usr/sbin/nologin loghub
Enter fullscreen mode Exit fullscreen mode 
Create a system user that will own all incoming logs.
Enter fullscreen mode Exit fullscreen mode 
sudo mkdir -p /opt/loghub/archive/{$(date +%Y)}/{$(date +%m)}
Enter fullscreen mode Exit fullscreen mode 
Make a year/month hierarchy; helps with retention policies.
Enter fullscreen mode Exit fullscreen mode 
sudo chown -R loghub:loghub /opt/loghub
Enter fullscreen mode Exit fullscreen mode 
Restrict access to the log‑hub user only.
Enter fullscreen mode Exit fullscreen mode

2. Edge‑Host Collector Script

The following Bash script lives on every server you want to monitor. It performs three duties:

  • Identify newly‑rotated log files (via inotifywait or a simple find scan).
  • Compress them with gzip while preserving original timestamps.
  • Ship the archives to the central hub over an encrypted rsync tunnel.

Save this as /usr/local/bin/log_collect.sh and make it executable (chmod +x).

#!/usr/bin/env bash
# --------------------------------------------------------------
# log_collect.sh – Edge host log collector & forwarder
# --------------------------------------------------------------
# Author: Vijay Vinoth, Lead Programmer Analyst
# Date  : 2026‑09‑15
# ----------------------------------------------------------------
# Prerequisites:
#   • rsync (installed by default on most distros)
#   • gzip
#   • ssh keys pre‑distributed to the log‑hub user (loghub)
#   • optional: inotify-tools for real‑time watching
# ----------------------------------------------------------------

# ---- Configuration ------------------------------------------------
REMOTE_USER="loghub"
REMOTE_HOST="loghub.example.com"
REMOTE_ROOT="/opt/loghub/archive"
LOCAL_LOG_DIR="/var/log"
TMP_DIR="/tmp/log_collect_$$"
RETENTION_DAYS=30          # Keep local copies for N days
COMPRESS_LEVEL=6           # gzip -6 balances speed & size
RSYNC_OPTS="-az --partial --delete-after"
# ------------------------------------------------------------------

# Create a temporary workspace
mkdir -p "$TMP_DIR"

# Function: compress a single file and preserve its mtime
compress_file() {
    local src=$1
    local dst="${src}.gz"
    gzip -c -${COMPRESS_LEVEL} "$src" > "$dst"
    # Preserve original modification time for later sorting
    touch -r "$src" "$dst"
    echo "$dst"
}

# Function: ship a batch of compressed logs to the hub
ship_batch() {
    local batch_dir=$1
    local remote_path="${REMOTE_ROOT}/$(date +%Y)/$(date +%m)"
    rsync $RSYNC_OPTS "$batch_dir/" "${REMOTE_USER}@${REMOTE_HOST}:${remote_path}/"
    if [[ $? -eq 0 ]]; then
        echo "✅ Batch shipped successfully to ${REMOTE_HOST}:${remote_path}"
        # Clean up local copies after successful transfer
        rm -rf "$batch_dir"
    else
        echo "⚠️  rsync failed – retaining batch for retry"
    fi
}

# ------------------------------------------------------------------
# 1️⃣  Find log files that have NOT been processed yet.
#     We rely on a simple marker file .processed placed beside each log.
# ------------------------------------------------------------------
find "$LOCAL_LOG_DIR" -type f -name "*.log" ! -name ".*.processed" | while read -r logfile; do
    # Skip empty files
    [[ ! -s "$logfile" ]] && continue

    # 2️⃣ Compress the log
    compressed=$(compress_file "$logfile")
    echo "📦 Compressed $logfile → $compressed"

    # 3️⃣ Move compressed file to temporary batch dir
    mv "$compressed" "$TMP_DIR/"

    # 4️⃣ Touch a hidden marker so we don’t re‑process the same file
    touch "${logfile}.processed"
done

# ------------------------------------------------------------------
# 5️⃣  Ship everything that accumulated in $TMP_DIR
# ------------------------------------------------------------------
if [[ -n "$(ls -A "$TMP_DIR")" ]]; then
    ship_batch "$TMP_DIR"
else
    echo "🛑 No new logs to ship – exiting."
    rmdir "$TMP_DIR"
fi

# ------------------------------------------------------------------
# 6️⃣  House‑keeping – delete old .processed markers
# ------------------------------------------------------------------
find "$LOCAL_LOG_DIR" -type f -name ".*.processed" -mtime +$RETENTION_DAYS -delete
Enter fullscreen mode Exit fullscreen mode

3. Automate Execution with systemd Timers (or cron)

Running the collector every 10 minutes provides a good balance between latency and network load. Below is a systemd service + timer pair that works on any modern Linux distribution.

# /etc/systemd/system/log-collect.service
[Unit]
Description=Edge host log collector & forwarder
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/log_collect.sh
Nice=10
IOSchedulingClass=idle
Enter fullscreen mode Exit fullscreen mode 
# /etc/systemd/system/log-collect.timer
[Unit]
Description=Run log-collect.service every 10 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=10min
Persistent=true

[Install]
WantedBy=timers.target
Enter fullscreen mode Exit fullscreen mode

Enable and start the timer:

sudo systemctl daemon-reload
sudo systemctl enable --now log-collect.timer
Enter fullscreen mode Exit fullscreen mode

4. Verify End‑to‑End Flow

  • Generate a test log entry on the edge host: 
echo "$(date) – TEST – HelloWorld" >> /var/log/app_test.log
Enter fullscreen mode Exit fullscreen mode
  • Wait for the timer (or run the script manually) and then SSH into the hub:
  • List the newly created archive: 
ssh loghub@loghub.example.com "ls -l /opt/loghub/archive/$(date +%Y)/$(date +%m)"
Enter fullscreen mode Exit fullscreen mode
  • Decompress and inspect to confirm the original line survived.

Adding a Light‑Weight Embedding Sidecar (Future‑Proofing)

The collection pipeline above is deliberately agnostic of the AI layer. In the next tutorial you’ll see how to plug a sentence‑transformers/all‑MiniLM‑L6‑v2 sidecar that reads the freshly‑arrived .gz files, generates embeddings, and pushes them into a vector store (e.g., Milvus or PGVector). Because we batch‑compress logs before shipping, the sidecar can safely process a few megabytes per second without impacting the production application, mirroring the recommendation from the Khimananda blog.

Best‑Practice Checklist

✅ ItemWhy it matters

SSH key authentication (no passwords)
Eliminates interactive prompts and enables automated timers.


Read‑only permissions for edge hosts
Zero‑Trust principle – hosts can only write to their own namespace.


Gzip compression level 6
Best trade‑off for CPU vs. bandwidth on 2026 cloud links.


Retention policy (30 days locally)
Prevents disk exhaustion while still allowing quick re‑processing.


Systemd timer with `Persistent=true`
Ensures missed runs (e.g., after a reboot) are replayed automatically.
Enter fullscreen mode Exit fullscreen mode

Scaling the Collector for Hundreds of Nodes

When you move from a handful of servers to a fleet of several hundred, two adjustments become critical:

  • Parallel rsync streams: Instead of a single SSH connection, launch multiple background rsync jobs (max 5‑10 per host) to saturate the network pipe. Add --bwlimit=10M if you need to throttle.
  • Central ingest queue: Deploy a lightweight nginx + proxy_pass that balances incoming rsync traffic to a pool of storage nodes. The script stays the same; only the REMOTE_HOST variable points at the load‑balancer DNS name.

Security Hardening Tips (Zero‑Trust Ready)

  • Enable ForceCommand internal-sftp for the loghub SSH account – this restricts the remote side to file transfer only.
  • Set AllowTcpForwarding no and PermitTunnel no in /etc/ssh/sshd_config for the loghub user.
  • Apply auditd rules to log every rsync invocation; this satisfies many compliance frameworks (PCI‑DSS, GDPR).
  • Rotate the SSH host keys on the hub quarterly – a practice highlighted in the CSA Zero‑Trust whitepaper.

Testing & Monitoring the Pipeline

Even a rock‑solid Bash script benefits from observability. Add a tiny health‑check endpoint on the hub that reports the most recent file timestamp. Example using nc:

while true; do
  echo -e "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\n$(date -r /opt/loghub/archive/$(date +%Y)/$(date +%m)/* | tail -1)" \
    | nc -l -p 8081 -q 1
done &
Enter fullscreen mode Exit fullscreen mode

Now you can scrape http://loghub.example.com:8081 with Prometheus or a simple curl to confirm logs are arriving on schedule.

Next Steps in the Series

With a reliable, low‑latency collector in place, the next tutorial (Part 3) will show how to:

  • Run a sidecar embedding service (Python + sentence‑transformers) that consumes the .gz archives.
  • Store embeddings in a vector database (PGVector or Milvus) and expose a FAISS-style similarity search API.
  • Trigger the Claude 4.6 Opus or GPT‑5.4 Pro anomaly agents whenever a similarity score crosses a dynamic threshold.

📚 References & Further Reading

Your Turn

Imagine you have a multi‑cloud environment where some workloads emit logs to CloudWatch, others to GCP Logging, and a few on‑prem servers to local files. How would you extend the Bash‑centric collector to unify these disparate sources without sacrificing the low‑latency guarantees needed for real‑time AI anomaly detection? Share your design ideas or script snippets in the comments below!

Originally published at https://artificial-inteligence.phptutorial.co.in

Top comments (0)