DEV Community

Cover image for The 3 Commands That Turn Chaos into Clarity in DevOps
Bharath Aaleti
Bharath Aaleti

Posted on

The 3 Commands That Turn Chaos into Clarity in DevOps

“You don’t need fancy monitoring tools to troubleshoot production issues.

Sometimes, all you need is a terminal., and three magical commands: grep, awk, and sed.”

These three commands can turn gigabytes of messy logs into clear insights within seconds.

Once you master them, you’ll handle on-call like a pro.

Note: The examples below are just a few to get you started.

You can explore more scenario-based examples by asking ChatGPT or searching online.


🧠 Why Text Processing Matters in DevOps

In DevOps and SRE, every log line tells a story. But when you’re knee-deep in 2GB of logs, you need quick, powerful tools to extract meaning.

This is where grep, awk, and sed shine.

Imagine this: you’re SSH’d into a production server. The application is down. Logs are huge. Instead of downloading them, you filter and extract what you need right there.


🔍 grep - The Search Engine of Your Terminal

Purpose: Quickly find patterns in massive text files.

Syntax:

grep [OPTIONS] PATTERN [FILE...]
Enter fullscreen mode Exit fullscreen mode

🔹 Common Examples

# Find all error lines in logs
grep "ERROR" /var/log/app.log

# Search recursively in directory
grep -R "timeout" /etc/nginx/

# Case-insensitive + show line numbers
grep -in "failed" auth.log

# Highlight matches
grep --color=auto "warning" server.log
Enter fullscreen mode Exit fullscreen mode

💡 How it works:

grep scans text line by line and prints only those containing "ERROR". It’s the fastest way to find specific keywords in huge log files.

💡 Pro Tip:

Combine tail -f and grep to monitor logs live:

tail -f app.log | grep "ERROR"
Enter fullscreen mode Exit fullscreen mode

🧰 Real-World Use Cases

  • Extracting failed pod names from Kubernetes logs:
  kubectl get pods | grep -i "error"
Enter fullscreen mode Exit fullscreen mode
  • Searching for error traces in CI/CD pipelines:
  cat build.log | grep "Exception"
Enter fullscreen mode Exit fullscreen mode
  • Monitoring real-time logs for failures:
  tail -f app.log | grep "500"
Enter fullscreen mode Exit fullscreen mode

Alternatives:

  • ripgrep (rg) - faster alternative, written in Rust. 👉 But grep remains the universal standard.

⚙️ awk - The Swiss Army Knife for Structured Data

Purpose: Process and analyze structured text line by line.

Syntax:

awk 'pattern {action}' file
Enter fullscreen mode Exit fullscreen mode

🔹 Practical Examples

# Print specific columns
awk '{print $1, $4}' access.log

# Filter based on condition
awk '$3 == "ERROR" {print $0}' app.log

# Calculate average CPU usage
awk '{sum += $2} END {print "Avg:", sum/NR}' cpu.txt

# Show top 5 largest files
ls -lh | awk '{print $5, $9}' | sort -rh | head -5
Enter fullscreen mode Exit fullscreen mode

💡 How it works:

Each line in access.log represents one request, and the 5th column ($5) stores its response time (in ms, for example).

  • sum+=$5 → adds up all response times
  • NR → built-in variable for the total number of lines processed
  • END {print sum/NR} → prints the average at the end

🧰 Real-World Use Cases

  • Extracting pod memory usage:
  kubectl top pods | awk '{print $1, $3}'
Enter fullscreen mode Exit fullscreen mode
  • Summarizing response times:
  awk '{sum += $5} END {print "Avg Response:", sum/NR}' access.log
Enter fullscreen mode Exit fullscreen mode
  • Counting unique IPs from access logs:
  awk '{print $1}' access.log | sort | uniq | wc -l
Enter fullscreen mode Exit fullscreen mode

Alternatives:

  • cut (simpler but limited)
  • jq (great for JSON)
  • csvkit (for CSVs) 👉 awk remains the most versatile for text streams.

✂️ sed - The Stream Editor

Purpose: Edit, replace, and transform text on the fly.

Syntax:

sed [OPTIONS] 'command' file
Enter fullscreen mode Exit fullscreen mode

🔹 Common Examples

# Replace all “foo” with “bar”
sed 's/foo/bar/g' input.txt

# Delete blank lines
sed '/^$/d' file.txt

# Replace in place (useful for config changes)
sed -i 's/localhost/127.0.0.1/g' config.yml

# Prints lines starting from 5-10
sed -n '5,10p' file.txt
Enter fullscreen mode Exit fullscreen mode

💡 How it works:

  • sed (stream editor) searches and replaces text directly in files.

  • 's/dev/prod/g' → replace all instances of “dev” with “prod”

  • -i → edit the file in place (no need to open a text editor)

🧰 Real-World Use Cases

  • Update IPs or URLs in config files:
  sed -i 's/dev.example.com/prod.example.com/g' nginx.conf
Enter fullscreen mode Exit fullscreen mode
  • Remove comments before deployment:
  sed '/^#/d' .env
Enter fullscreen mode Exit fullscreen mode
  • Bulk change environment variables in CI/CD:
  sed -i 's/DEBUG=True/DEBUG=False/' settings.py
Enter fullscreen mode Exit fullscreen mode

Alternatives:

  • perl -pe (more powerful but heavier)
  • tr (for simple character replacement) 👉 sed is lightweight and available by default everywhere.

🧩 Combining Them for Power One-Liners

Extract failed requests with response time > 500ms:

grep "FAILED" app.log | awk '$5 > 500' | sed 's/FAILED/ERROR/g'
Enter fullscreen mode Exit fullscreen mode

Monitor errors in real time:

tail -f app.log | grep "ERROR"
Enter fullscreen mode Exit fullscreen mode

Find all 404s and count them:

grep "404" access.log | wc -l
Enter fullscreen mode Exit fullscreen mode

Extract usernames from access log and sort by frequency:

awk '{print $3}' access.log | sort | uniq -c | sort -nr | head
Enter fullscreen mode Exit fullscreen mode

Replace all IPs with placeholders (for redaction):

sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/[REDACTED]/g' access.log
Enter fullscreen mode Exit fullscreen mode

💬 Final Thoughts

Once you start combining these three, you’ll realize you can transform any text stream into usable insights without ever leaving the terminal.

Whether you're debugging a failed deployment, analyzing performance logs, or cleaning config files, these commands will make your life easier.

Learn them once. Use them forever.


👋 I'm Bharath Aaleti, DevOps Engineer - learning DevOps and sharing everything I know/learn.

If you found this helpful, connect or drop your favorite one-liner below 👇


Top comments (0)