DEV Community

Yashvi Kothari
Yashvi Kothari

Posted on

Linux Server Monitoring, Backup, Security & Troubleshooting: The Practical SysAdmin Playbook

#linux #monitoring #security #backup

Linux Server Monitoring, Backup, Security & Troubleshooting Guide

Learn practical Linux server monitoring, backup, security, firewall configuration, and troubleshooting techniques with real commands and interview-ready explanations.

A server rarely fails without leaving clues.

CPU usage rises.

Memory disappears.

Disk space reaches 100%.

A service stops listening.

Nginx returns a 502.

The database refuses connections.

And suddenly someone asks:

“Can you fix production?”

Good system administrators don't panic.

They investigate.

They follow evidence.

They fix the root cause.

This guide covers the practical Linux skills every SysAdmin and DevOps engineer should know: monitoring, backup, security, and troubleshooting.

1. Monitoring: Know What Your Server Is Doing

Monitoring isn't about staring at dashboards all day.

It's about answering four questions:

  1. Is the CPU overloaded?
  2. Is memory exhausted?
  3. Is disk space running out?
  4. Is the network or application behaving abnormally?

CPU Monitoring

Start with:

uptime
Enter fullscreen mode Exit fullscreen mode

You'll see something like:

load average: 0.50, 0.75, 0.80
Enter fullscreen mode Exit fullscreen mode

The three values represent the 1-minute, 5-minute, and 15-minute load averages.

But there's an important interview detail:

Load average is not the same thing as CPU percentage.

A rough rule is to compare load with the number of CPU cores.

For a 4-core server:

Load < 4     → generally comfortable
Load ≈ 4     → CPUs are fully utilized
Load > 4     → work is waiting for CPU/resources
Enter fullscreen mode Exit fullscreen mode

Check CPU information:

lscpu
nproc
Enter fullscreen mode Exit fullscreen mode

Find CPU-heavy processes:

ps aux --sort=-%cpu | head -20
Enter fullscreen mode Exit fullscreen mode

For interactive troubleshooting:

top
Enter fullscreen mode Exit fullscreen mode

or:

htop
Enter fullscreen mode Exit fullscreen mode

2. Memory Monitoring

Use:

free -h
Enter fullscreen mode Exit fullscreen mode

The most important number isn't always free.

Look at:

available
Enter fullscreen mode Exit fullscreen mode

Linux intentionally uses unused RAM for filesystem cache and buffers.

That memory can often be reclaimed when applications need it.

Check memory-heavy processes 

ps aux --sort=-%mem | head -20
Enter fullscreen mode Exit fullscreen mode

Check swap 

swapon --show
Enter fullscreen mode Exit fullscreen mode

and:

free -h
Enter fullscreen mode Exit fullscreen mode

You can also monitor swap activity:

vmstat 1 5
Enter fullscreen mode Exit fullscreen mode

Pay attention to si and so.

Heavy, continuous swap activity can indicate memory pressure and often results in poor performance.

3. Disk Monitoring

One of the most common production problems is surprisingly simple:

The disk is full.

Start with:

df -h
Enter fullscreen mode Exit fullscreen mode

Then identify where the space went:

du -sh /* 2>/dev/null | sort -rh
Enter fullscreen mode Exit fullscreen mode

Investigate /var:

du -sh /var/* 2>/dev/null | sort -rh | head -20
Enter fullscreen mode Exit fullscreen mode

Logs are frequent offenders:

du -sh /var/log/* 2>/dev/null | sort -rh
Enter fullscreen mode Exit fullscreen mode

Find huge files:

find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null
Enter fullscreen mode Exit fullscreen mode

But disk space isn't the only resource that can run out.

Check inodes 

df -i
Enter fullscreen mode Exit fullscreen mode

A filesystem can have free GBs available while still being unable to create files because its inodes are exhausted.

That's a classic interview question.

4. Disk I/O

A server can have:

  • low CPU
  • plenty of RAM
  • plenty of disk space

…and still be painfully slow.

Why?

Disk I/O.

Use:

iostat -x 1 5
Enter fullscreen mode Exit fullscreen mode

For process-level I/O:

iotop
Enter fullscreen mode Exit fullscreen mode

High disk utilization or latency can point toward:

  • slow disks
  • excessive logging
  • database activity
  • backups
  • large file operations
  • overloaded storage

5. Network Monitoring

Check listening services:

ss -tlnp
Enter fullscreen mode Exit fullscreen mode

Get socket statistics:

ss -s
Enter fullscreen mode Exit fullscreen mode

Check network interfaces:

ip addr show
ip link show
Enter fullscreen mode Exit fullscreen mode

Test connectivity:

ping google.com
Enter fullscreen mode Exit fullscreen mode

Trace the route:

traceroute google.com
Enter fullscreen mode Exit fullscreen mode

Or combine latency and route analysis:

mtr google.com
Enter fullscreen mode Exit fullscreen mode

For HTTP:

curl -I https://example.com
Enter fullscreen mode Exit fullscreen mode

This is extremely useful because it tells you whether the server is actually returning an HTTP response.

6. Logs Are Your First Investigation Tool

When something breaks, don't randomly restart everything.

Read the logs.

System logs:

sudo tail -f /var/log/syslog
Enter fullscreen mode Exit fullscreen mode

Authentication logs:

sudo tail -f /var/log/auth.log
Enter fullscreen mode Exit fullscreen mode

Systemd:

sudo journalctl -f
Enter fullscreen mode Exit fullscreen mode

Recent logs:

sudo journalctl --since "1 hour ago"
Enter fullscreen mode Exit fullscreen mode

For Nginx:

sudo tail -f /var/log/nginx/error.log
Enter fullscreen mode Exit fullscreen mode

For Apache:

sudo tail -f /var/log/apache2/error.log
Enter fullscreen mode Exit fullscreen mode

For MySQL:

sudo tail -f /var/log/mysql/error.log
Enter fullscreen mode Exit fullscreen mode

The fastest troubleshooting path is often:

Symptom → service status → logs → dependency → configuration → resource usage

7. Backup: Assume the Server Will Fail

A backup isn't:

“I copied the website somewhere.”

A proper backup strategy considers the entire system.

You should consider backing up:

  • /var/www/
  • databases
  • Nginx/Apache configuration
  • PHP configuration
  • MySQL configuration
  • SSL certificates
  • cron jobs
  • user data
  • custom scripts
  • package information

For MySQL:

mysqldump -u root -p --all-databases --routines --triggers > all_databases.sql
Enter fullscreen mode Exit fullscreen mode

Compress it:

gzip all_databases.sql
Enter fullscreen mode Exit fullscreen mode

For files:

tar -czf www_backup.tar.gz /var/www/
Enter fullscreen mode Exit fullscreen mode

8. Why rsync Is a SysAdmin Superpower

For large migrations, rsync is usually much more useful than repeatedly copying everything.

Example:

rsync -avz --progress /var/www/ user@newserver:/var/www/
Enter fullscreen mode Exit fullscreen mode

Why?

Because rsync can synchronize only what changed.

That makes it extremely useful for:

  • server migrations
  • backups
  • deployments
  • synchronization
  • large file transfers

A strong interview answer:

“I prefer rsync for large transfers because it efficiently synchronizes changed files and can resume transfers.”

9. Remember the 3-2-1 Backup Rule

A practical backup strategy follows:

3 copies of your data

2 different types of storage

1 copy offsite

For example:

Production server
       ↓
Local backup
       ↓
Remote backup
       ↓
Cloud/offsite backup
Enter fullscreen mode Exit fullscreen mode

And one more important principle:

A backup isn't proven until you've restored it.

Test your backups.

A backup that has never been restored is only a theory.

10. Firewall Security

On Ubuntu, UFW provides a simpler interface for firewall management.

Install:

sudo apt install ufw
Enter fullscreen mode Exit fullscreen mode

Before enabling it, make sure SSH is allowed:

sudo ufw allow ssh
Enter fullscreen mode Exit fullscreen mode

Then:

sudo ufw enable
Enter fullscreen mode Exit fullscreen mode

Check:

sudo ufw status verbose
Enter fullscreen mode Exit fullscreen mode

A typical web server may need:

sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
Enter fullscreen mode Exit fullscreen mode

Then use restrictive defaults:

sudo ufw default deny incoming
sudo ufw default allow outgoing
Enter fullscreen mode Exit fullscreen mode

The principle is simple:

Don't expose services that don't need to be exposed.

11. iptables

For lower-level firewall control:

sudo iptables -L -n -v
Enter fullscreen mode Exit fullscreen mode

Allow established connections:

sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
Enter fullscreen mode Exit fullscreen mode

Allow loopback:

sudo iptables -A INPUT -i lo -j ACCEPT
Enter fullscreen mode Exit fullscreen mode

Allow SSH:

sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
Enter fullscreen mode Exit fullscreen mode

Allow HTTP/HTTPS:

sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT
Enter fullscreen mode Exit fullscreen mode

Then carefully define your default policy.

Important: Never experiment with firewall rules remotely without ensuring you won't lock yourself out.

12. Fail2ban

A firewall controls network access.

Fail2ban adds another layer by watching logs for suspicious authentication behavior and temporarily banning offending IP addresses.

Install:

sudo apt install fail2ban
Enter fullscreen mode Exit fullscreen mode

Check status:

sudo fail2ban-client status
Enter fullscreen mode Exit fullscreen mode

Check SSH protection:

sudo fail2ban-client status sshd
Enter fullscreen mode Exit fullscreen mode

Unban an IP:

sudo fail2ban-client set sshd unbanip 1.2.3.4
Enter fullscreen mode Exit fullscreen mode

Think of it as:

Logs → Detect repeated abuse → Ban IP

13. Server Hardening

A secure server starts with basic discipline.

Keep packages updated 

sudo apt update
sudo apt upgrade -y
Enter fullscreen mode Exit fullscreen mode

Remove unnecessary packages 

sudo apt autoremove -y
Enter fullscreen mode Exit fullscreen mode

Enable automatic security updates 

sudo apt install unattended-upgrades
Enter fullscreen mode Exit fullscreen mode

Audit permissions 

find /var/www -type f -perm 0777
Enter fullscreen mode Exit fullscreen mode

World-writable files deserve investigation.

Audit open ports 

sudo ss -tlnp
Enter fullscreen mode Exit fullscreen mode

The goal isn't:

“Close every port.”

The goal is:

“Know why every exposed port exists.”

14. Troubleshooting HTTP 500

A 500 Internal Server Error means the server encountered an application-side error.

Don't immediately blame Nginx.

Start with logs:

sudo tail -50 /var/log/nginx/error.log
Enter fullscreen mode Exit fullscreen mode

or:

sudo tail -50 /var/log/apache2/error.log
Enter fullscreen mode Exit fullscreen mode

If PHP is involved:

sudo tail -50 /var/log/php8.2-fpm.log
Enter fullscreen mode Exit fullscreen mode

Then investigate:

  • application errors
  • permissions
  • PHP extensions
  • PHP configuration
  • .htaccess
  • application code
  • memory limits

The most important habit:

Check the error log first.

15. Troubleshooting 502 Bad Gateway

This error is especially common with reverse proxies.

Conceptually:

Client
  ↓
Nginx
  ↓
Backend
Enter fullscreen mode Exit fullscreen mode

A 502 generally means Nginx cannot successfully communicate with the upstream.

Check:

sudo systemctl status php8.2-fpm
Enter fullscreen mode Exit fullscreen mode

Check the PHP-FPM socket:

ls -la /var/run/php/php8.2-fpm.sock
Enter fullscreen mode Exit fullscreen mode

For a Node.js application:

sudo lsof -i :3000
Enter fullscreen mode Exit fullscreen mode

The critical question is:

“Is the backend running, and is Nginx connecting to the correct socket or port?”

16. 503 Service Unavailable

A 503 generally means the service cannot currently handle the request.

Check:

uptime
Enter fullscreen mode Exit fullscreen mode

Then:

sudo systemctl status nginx
sudo systemctl status apache2
sudo systemctl status php8.2-fpm
Enter fullscreen mode Exit fullscreen mode

Check connections:

ss -s
Enter fullscreen mode Exit fullscreen mode

Possible causes include:

  • overloaded server
  • stopped backend
  • maintenance mode
  • rate limiting
  • exhausted application workers
  • overloaded database

17. 504 Gateway Timeout

A 504 tells a different story.

The backend may be reachable.

But it didn't respond within the required time.

Possible causes:

  • slow database queries
  • slow application code
  • overloaded backend
  • network latency
  • insufficient PHP workers
  • overly aggressive timeout settings

Check:

mysql -u root -p -e "SHOW FULL PROCESSLIST;"
Enter fullscreen mode Exit fullscreen mode

Check resources:

htop
iostat
Enter fullscreen mode Exit fullscreen mode

Only increase timeouts after understanding why the backend is slow.

Increasing a timeout can hide the symptom while making resource consumption worse.

18. DNS Troubleshooting

When a website doesn't open, don't immediately restart Nginx.

Check DNS.

dig example.com
Enter fullscreen mode Exit fullscreen mode

Get the IP:

dig +short example.com
Enter fullscreen mode Exit fullscreen mode

Query a public resolver:

dig @8.8.8.8 example.com
Enter fullscreen mode Exit fullscreen mode

For mail:

dig MX example.com
Enter fullscreen mode Exit fullscreen mode

For SPF:

dig TXT example.com
Enter fullscreen mode Exit fullscreen mode

For DMARC:

dig TXT _dmarc.example.com
Enter fullscreen mode Exit fullscreen mode

The troubleshooting chain becomes:

Domain
  ↓
DNS
  ↓
IP
  ↓
Firewall
  ↓
Port
  ↓
Web server
  ↓
Reverse proxy
  ↓
Application
  ↓
Database
Enter fullscreen mode Exit fullscreen mode

That's how you avoid guessing.

19. SSL Troubleshooting

Check certificate dates:

echo | openssl s_client \
-connect example.com:443 2>/dev/null \
| openssl x509 -noout -dates
Enter fullscreen mode Exit fullscreen mode

Test Let's Encrypt renewal:

sudo certbot renew --dry-run
Enter fullscreen mode Exit fullscreen mode

Common causes of renewal failures:

  • DNS points somewhere else
  • port 80 blocked
  • HTTP challenge inaccessible
  • incorrect web-server configuration
  • certificate configuration errors

20. Database Troubleshooting

If MySQL isn't responding:

sudo systemctl status mysql
Enter fullscreen mode Exit fullscreen mode

Check logs:

sudo tail -50 /var/log/mysql/error.log
Enter fullscreen mode Exit fullscreen mode

Test credentials:

mysql -u root -p
Enter fullscreen mode Exit fullscreen mode

Check connections:

mysql -u root -p -e "SHOW STATUS LIKE 'Threads_connected';"
Enter fullscreen mode Exit fullscreen mode

Check maximum connections:

mysql -u root -p -e "SHOW VARIABLES LIKE 'max_connections';"
Enter fullscreen mode Exit fullscreen mode

Don't blindly increase max_connections.

More connections require more resources.

First understand why the connections are being consumed.

21. The SysAdmin Troubleshooting Framework

When production breaks, use this sequence:

Step 1 — Confirm the symptom 

curl -I https://example.com
Enter fullscreen mode Exit fullscreen mode

Step 2 — Check the service 

systemctl status nginx
Enter fullscreen mode Exit fullscreen mode

Step 3 — Check logs 

journalctl -u nginx --since "10 minutes ago"
Enter fullscreen mode Exit fullscreen mode

Step 4 — Check ports 

ss -tlnp
Enter fullscreen mode Exit fullscreen mode

Step 5 — Check resources 

uptime
free -h
df -h
Enter fullscreen mode Exit fullscreen mode

Step 6 — Check dependencies

Database?

PHP-FPM?

Node.js?

Redis?

External API?

Step 7 — Check configuration 

nginx -t
Enter fullscreen mode Exit fullscreen mode

Step 8 — Make the smallest safe change

Avoid changing five things at once.

Step 9 — Verify 

curl -I https://example.com
Enter fullscreen mode Exit fullscreen mode

Step 10 — Document the root cause

A production fix is not complete until someone understands why it happened.

22. Interview Questions You Should Be Ready For

How do you check server load?

Use:

uptime
Enter fullscreen mode Exit fullscreen mode

Then compare load average with available CPU cores and investigate CPU, I/O, and other bottlenecks.

How do you find what's filling the disk?

Start with:

df -h
Enter fullscreen mode Exit fullscreen mode

Then:

du -sh /* 2>/dev/null | sort -rh
Enter fullscreen mode Exit fullscreen mode

What's the difference between 502 and 504?

502: The proxy couldn't successfully communicate with the upstream.

504: The proxy waited for the upstream but timed out.

What is rsync?

A file synchronization tool that efficiently transfers changed data and is particularly useful for backups and migrations.

What is the 3-2-1 backup rule?

Three copies, two different storage types, one offsite.

How do you secure a Linux server?

Update the OS, minimize exposed services, configure a firewall, harden SSH, use automatic security updates, monitor authentication, use appropriate intrusion-prevention controls, and maintain tested backups.

What is your first step when a website is down?

Confirm the symptom and check the relevant service and logs before making changes.

Final Lesson

The best SysAdmins don't memorize hundreds of commands.

They understand what the commands are trying to prove.

CPU problem?
    → top / htop / uptime

Memory problem?
    → free / vmstat

Disk problem?
    → df / du / iostat

Network problem?
    → ss / ping / mtr / curl

Service problem?
    → systemctl / journalctl

Application problem?
    → application logs

Security problem?
    → auth logs / firewall / open ports

Backup problem?
    → restore test

HTTP problem?
    → status code → logs → backend → dependencies
Enter fullscreen mode Exit fullscreen mode

That's the difference between knowing Linux commands and actually knowing how to operate Linux servers.

Commands are tools.

Troubleshooting is the skill.

And in production, the engineer who can calmly turn a symptom into evidence into a root cause is the engineer everyone wants on the incident call.

Part 5: Server Monitoring, Backup, Security & Troubleshooting

5.1 Server Monitoring

CPU Monitoring 

# Real-time CPU usage
top                                    # Press 'P' to sort by CPU
htop                                   # Better interactive viewer (install: apt install htop)

# CPU info
lscpu                                 # CPU architecture info
nproc                                 # Number of CPU cores
cat /proc/cpuinfo                     # Detailed CPU info

# Load average
uptime                                # Shows load average (1, 5, 15 min)
# Output: load average: 0.50, 0.75, 0.80
# Rule: load average should be < number of CPU cores
# 4 cores → load < 4.0 is OK, > 4.0 means overloaded

# CPU usage snapshot
mpstat                                # CPU usage per core (install: apt install sysstat)
vmstat 1 5                            # System stats every 1 sec, 5 times
sar -u 1 5                            # CPU usage (requires sysstat)

# Top CPU-consuming processes
ps aux --sort=-%cpu | head -20
Enter fullscreen mode Exit fullscreen mode

RAM Monitoring 

# Memory usage
free -h                               # Human-readable memory info
# Key: Available (not Free) is what matters

# Detailed memory
cat /proc/meminfo

# Top memory-consuming processes
ps aux --sort=-%mem | head -20

# Check for memory issues
vmstat 1 5                            # Watch si/so columns (swap in/out)
# If si/so are high → server needs more RAM

# Check swap
swapon --show
free -h | grep Swap
Enter fullscreen mode Exit fullscreen mode

Disk Space Monitoring 

# Disk usage overview
df -h                                 # All mounted filesystems
df -h /                               # Root partition specifically

# Find what's using space
du -sh /*                             # Size of top-level directories
du -sh /var/* | sort -rh | head -20   # Largest dirs in /var
du -sh /var/log/* | sort -rh          # Log sizes
ncdu /                                # Interactive disk usage tool

# Find large files
find / -type f -size +100M -exec ls -lh {} \;  # Files > 100MB
find /var/log -name "*.log" -size +50M          # Large log files

# Check inodes
df -i                                 # Inode usage (can run out before disk space)

# Monitor disk I/O
iostat -x 1 5                         # Disk I/O stats (install: apt install sysstat)
iotop                                 # Real-time I/O by process (install: apt install iotop)
Enter fullscreen mode Exit fullscreen mode

Bandwidth & Network Monitoring 

# Current connections
ss -s                                 # Socket statistics summary
ss -tlnp                              # Listening ports
netstat -tlnp                         # Alternative (legacy)

# Bandwidth usage
iftop                                 # Real-time bandwidth by connection (install: apt install iftop)
nload                                 # Real-time bandwidth graph (install: apt install nload)
vnstat                                # Historical bandwidth (install: apt install vnstat)
vnstat -d                             # Daily traffic
vnstat -m                             # Monthly traffic

# Network interface info
ip addr show                          # IP addresses
ip link show                          # Network interfaces

# Test connectivity
ping google.com                       # Basic connectivity
traceroute google.com                  # Route to destination
mtr google.com                        # Combined ping + traceroute (install: apt install mtr)
curl -I https://example.com           # HTTP response headers
wget --spider https://example.com     # Check if URL is accessible
Enter fullscreen mode Exit fullscreen mode

Server Uptime & Load 

# System uptime
uptime
# Output: 14:30:05 up 45 days, 3:22, 2 users, load average: 0.50, 0.75, 0.80

# System info
uname -a                              # Kernel version, architecture
hostnamectl                            # Hostname, OS, kernel, architecture
cat /etc/os-release                    # OS version

# Last reboot
last reboot | head -10
who -b                                # Last boot time

# System resource summary (one command)
echo "=== UPTIME ===" && uptime && echo "=== MEMORY ===" && free -h && echo "=== DISK ===" && df -h / && echo "=== CPU LOAD ===" && cat /proc/loadavg
Enter fullscreen mode Exit fullscreen mode

Log Monitoring 

# System logs
sudo tail -f /var/log/syslog                    # System log (live)
sudo tail -f /var/log/auth.log                  # Authentication log
sudo journalctl -f                              # systemd journal (live)
sudo journalctl --since "1 hour ago"            # Recent logs

# Web server logs
sudo tail -f /var/log/apache2/error.log         # Apache errors
sudo tail -f /var/log/nginx/error.log           # Nginx errors
sudo tail -f /var/log/apache2/access.log        # Apache access

# PHP logs
sudo tail -f /var/log/php8.2-fpm.log            # PHP-FPM log

# MySQL logs
sudo tail -f /var/log/mysql/error.log           # MySQL errors

# Analyze access logs
# Top 10 IPs hitting the server
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

# Top 10 requested URLs
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

# Count 404 errors
grep " 404 " /var/log/nginx/access.log | wc -l

# Count 500 errors
grep " 500 " /var/log/nginx/access.log | wc -l

# Requests per minute/hour
awk '{print $4}' /var/log/nginx/access.log | cut -d: -f1-3 | sort | uniq -c | tail -20
Enter fullscreen mode Exit fullscreen mode

Interview Q&A: Monitoring

Question Answer
How do you check server load? uptime shows load average. Compare to CPU cores: load > cores = overloaded
What's the difference between free and available memory? Free = completely unused. Available = free + cache/buffer that can be freed. Available is what matters
How to find what's filling up disk? `du -sh /* \
How do you monitor in real-time? {% raw %}htop (CPU/RAM), iftop (bandwidth), tail -f (logs), iostat (disk I/O)
What is swap? When is it bad? Overflow area on disk when RAM is full. High swap usage = server needs more RAM, causes slowness
How to identify a DDoS attack? Check access logs for unusual traffic from single IPs, high connection count, ss -s

5.2 Server Backup, Restore & Migration

Full Server Backup Strategy 

# What to backup:
# 1. Website files (/var/www/)
# 2. Databases (mysqldump)
# 3. Server configurations (/etc/nginx, /etc/apache2, /etc/php, /etc/mysql)
# 4. SSL certificates (/etc/letsencrypt)
# 5. Cron jobs (crontab -l)
# 6. User home directories
# 7. Custom scripts

# Full website backup script
#!/bin/bash
DATE=$(date +%Y-%m-%d)
BACKUP_DIR="/backups/$DATE"
mkdir -p $BACKUP_DIR

# 1. Backup website files
tar -czf $BACKUP_DIR/www_backup.tar.gz /var/www/

# 2. Backup all databases
mysqldump -u root -p'password' --all-databases --routines --triggers | gzip > $BACKUP_DIR/all_databases.sql.gz

# 3. Backup configs
tar -czf $BACKUP_DIR/configs.tar.gz /etc/nginx/ /etc/apache2/ /etc/php/ /etc/mysql/ /etc/letsencrypt/

# 4. Backup crontabs
crontab -l > $BACKUP_DIR/root_crontab.txt
crontab -u www-data -l > $BACKUP_DIR/www-data_crontab.txt 2>/dev/null

# 5. Server info
dpkg --get-selections > $BACKUP_DIR/installed_packages.txt
ip addr > $BACKUP_DIR/network_config.txt
cat /etc/hosts > $BACKUP_DIR/hosts.txt

# 6. Sync to remote/cloud
rsync -avz $BACKUP_DIR/ backup@remote-server:/offsite-backups/$DATE/
# OR to S3:
# aws s3 sync $BACKUP_DIR/ s3://my-backups/$DATE/

# 7. Cleanup old backups (keep 30 days)
find /backups/ -maxdepth 1 -type d -mtime +30 -exec rm -rf {} \;

echo "Backup completed: $BACKUP_DIR"
Enter fullscreen mode Exit fullscreen mode

Server Migration Steps 

# Step-by-step server migration checklist:

# 1. PREPARE NEW SERVER
sudo apt update && sudo apt upgrade -y
# Install all required software: nginx, php, mysql, etc.
# Configure same PHP version and extensions

# 2. BACKUP OLD SERVER (everything listed above)

# 3. TRANSFER FILES
# Option A: rsync (best for large transfers, supports resume)
rsync -avz --progress -e "ssh -p 22" /var/www/ newserver:/var/www/

# Option B: scp
scp -r /var/www/ user@newserver:/var/www/

# Option C: tar + transfer
tar -czf /tmp/www_backup.tar.gz /var/www/
scp /tmp/www_backup.tar.gz user@newserver:/tmp/
# On new server: tar -xzf /tmp/www_backup.tar.gz -C /

# 4. TRANSFER DATABASES
# On old server:
mysqldump -u root -p --all-databases | gzip > /tmp/all_db.sql.gz
scp /tmp/all_db.sql.gz user@newserver:/tmp/
# On new server:
gunzip < /tmp/all_db.sql.gz | mysql -u root -p

# 5. TRANSFER CONFIGURATIONS
scp -r /etc/nginx/sites-available/ user@newserver:/etc/nginx/sites-available/
scp -r /etc/letsencrypt/ user@newserver:/etc/letsencrypt/
# Copy php.ini, mysql configs, etc.

# 6. FIX PERMISSIONS
sudo chown -R www-data:www-data /var/www/
sudo chmod -R 755 /var/www/

# 7. UPDATE DNS (point to new server IP)
# Lower TTL first (300 seconds), wait for propagation
# Change A records to new server IP
# Wait for propagation (check with dig @8.8.8.8 domain.com)

# 8. VERIFY
# Test all websites
# Test database connections
# Test SSL certificates
# Test email delivery
# Monitor logs for errors

# 9. KEEP OLD SERVER running for a few days as fallback
Enter fullscreen mode Exit fullscreen mode

Interview Q&A: Backup & Migration

Question Answer
What should be included in a server backup? Website files, databases, configs (nginx/apache/php/mysql), SSL certs, crontabs, custom scripts
What is rsync? Why use it? File synchronization tool. Only transfers changed files, supports resume, efficient for large transfers
How often should you backup? Daily for databases, weekly for full backups. Keep 30 days of daily + 3 months of weekly
What is the 3-2-1 backup rule? 3 copies of data, on 2 different media types, 1 offsite (cloud/remote)
How do you minimize downtime during migration? Sync data before DNS change, lower TTL, test on new server first, keep old server as fallback
How do you verify a backup is good? Periodically do test restores to a staging server

5.3 Firewall & Server Security

UFW (Uncomplicated Firewall) - Ubuntu Default 

# Install and enable
sudo apt install ufw
sudo ufw enable                        # CAREFUL: make sure SSH is allowed first!

# Check status
sudo ufw status
sudo ufw status verbose
sudo ufw status numbered               # Show rule numbers

# Allow common services
sudo ufw allow ssh                     # Port 22
sudo ufw allow 2222/tcp                # Custom SSH port
sudo ufw allow http                    # Port 80
sudo ufw allow https                   # Port 443
sudo ufw allow 3306/tcp               # MySQL (only if needed remotely)

# Allow from specific IP
sudo ufw allow from 192.168.1.100
sudo ufw allow from 192.168.1.100 to any port 22  # SSH from specific IP only
sudo ufw allow from 192.168.1.0/24 to any port 3306  # MySQL from subnet

# Deny rules
sudo ufw deny 23                       # Deny telnet
sudo ufw deny from 1.2.3.4             # Block specific IP

# Delete rules
sudo ufw delete allow 3306/tcp
sudo ufw delete 5                      # Delete rule by number

# Rate limiting (brute force protection)
sudo ufw limit ssh                     # Limit SSH connections (6 in 30 seconds)

# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Reset all rules
sudo ufw reset

# Logging
sudo ufw logging on
sudo ufw logging medium
Enter fullscreen mode Exit fullscreen mode

iptables (Advanced Firewall) 

# List rules
sudo iptables -L -n -v
sudo iptables -L -n --line-numbers

# Allow SSH
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP/HTTPS
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Allow established connections
sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow loopback
sudo iptables -A INPUT -i lo -j ACCEPT

# Drop everything else
sudo iptables -A INPUT -j DROP

# Block specific IP
sudo iptables -A INPUT -s 1.2.3.4 -j DROP

# Rate limit SSH
sudo iptables -A INPUT -p tcp --dport 22 -m connlimit --connlimit-above 3 -j DROP

# Save rules (persist after reboot)
sudo apt install iptables-persistent
sudo netfilter-persistent save
Enter fullscreen mode Exit fullscreen mode

Fail2ban (Brute Force Protection) 

# Install
sudo apt install fail2ban

# Configure
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local

# Key settings in jail.local:
[DEFAULT]
bantime = 3600                         # Ban for 1 hour
findtime = 600                         # Within 10 minutes
maxretry = 5                           # After 5 failed attempts
destemail = admin@example.com
action = %(action_mwl)s                # Ban + email with whois + logs

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3

[apache-auth]
enabled = true
port = http,https
filter = apache-auth
logpath = /var/log/apache2/error.log

[nginx-http-auth]
enabled = true
port = http,https
filter = nginx-http-auth
logpath = /var/log/nginx/error.log

# Service management
sudo systemctl start fail2ban
sudo systemctl enable fail2ban
sudo systemctl status fail2ban

# Check banned IPs
sudo fail2ban-client status
sudo fail2ban-client status sshd

# Unban an IP
sudo fail2ban-client set sshd unbanip 1.2.3.4
Enter fullscreen mode Exit fullscreen mode

Server Hardening Checklist 

# 1. Keep system updated
sudo apt update && sudo apt upgrade -y
sudo apt autoremove -y

# 2. Automatic security updates
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

# 3. SSH hardening (see Part 1 SSH section)

# 4. Disable root login
sudo passwd -l root                   # Lock root password

# 5. Remove unnecessary packages
sudo apt remove telnet ftp             # Remove insecure services

# 6. Secure /tmp
# Mount /tmp with noexec option

# 7. Security headers (in Nginx/Apache)
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
add_header Content-Security-Policy "default-src 'self'";
add_header Referrer-Policy "strict-origin-when-cross-origin";

# 8. File permissions audit
find /var/www -type f -perm 0777       # Find world-writable files
find /var/www -type d -perm 0777       # Find world-writable directories

# 9. Monitor auth logs
sudo tail -f /var/log/auth.log
sudo grep "Failed password" /var/log/auth.log | tail -20

# 10. Check for rootkits
sudo apt install rkhunter
sudo rkhunter --check

# 11. Audit open ports
sudo ss -tlnp                         # Should only show expected ports
sudo nmap -sT localhost                # Scan own ports
Enter fullscreen mode Exit fullscreen mode

Interview Q&A: Security

Question Answer
How do you secure a new server? Update OS, configure firewall (UFW), SSH hardening, fail2ban, disable root, security headers, automatic updates
What is fail2ban? Monitors logs for failed auth attempts, bans IPs after threshold. Prevents brute force
What is UFW? Uncomplicated Firewall - simplified interface for iptables on Ubuntu
What ports should be open on a web server? 22 (or custom SSH), 80 (HTTP), 443 (HTTPS). Everything else blocked
How to check for unauthorized access? Check /var/log/auth.log, check last command, check running processes, audit open ports
What are security headers? HTTP headers that protect against XSS, clickjacking, content sniffing. Set in web server config

5.4 Troubleshooting HTTP Errors

500 Internal Server Error 

# Most common causes & fixes:

# 1. Check error logs FIRST
sudo tail -50 /var/log/apache2/error.log
sudo tail -50 /var/log/nginx/error.log
sudo tail -50 /var/log/php8.2-fpm.log

# 2. PHP errors
# Check php.ini:
display_errors = On   # Temporarily (NEVER in production)
# Or check PHP error log
tail -f /var/log/php/error.log

# 3. .htaccess issues (Apache)
# Temporarily rename .htaccess to test:
mv /var/www/html/.htaccess /var/www/html/.htaccess.bak
# If site works → .htaccess has syntax error

# 4. File permissions
ls -la /var/www/html/
# Files should be 644, dirs 755, owned by www-data

# 5. PHP memory limit
# Error: "Allowed memory size exhausted"
# Fix in php.ini: memory_limit = 256M

# 6. Missing PHP extensions
php -m  # Check if required extensions are installed
Enter fullscreen mode Exit fullscreen mode

502 Bad Gateway 

# Means: Nginx/Apache can't reach the backend (PHP-FPM, Node.js, etc.)

# 1. Check if PHP-FPM is running
sudo systemctl status php8.2-fpm

# 2. Check PHP-FPM socket
ls -la /var/run/php/php8.2-fpm.sock
# If missing → restart PHP-FPM

# 3. Check Nginx config matches PHP-FPM socket
# Nginx should have: fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;
# PHP-FPM should have: listen = /var/run/php/php8.2-fpm.sock

# 4. Check backend app (Node.js, Python)
# Is the app running? Is it on the right port?
sudo lsof -i :3000                    # Check if app is listening

# 5. Check PHP-FPM pool config
# Too few workers? Increase pm.max_children

# 6. Check memory - OOM killer may have killed PHP-FPM
sudo dmesg | grep -i "out of memory"
sudo dmesg | grep -i "oom"
Enter fullscreen mode Exit fullscreen mode

503 Service Unavailable 

# Means: Server is overloaded or in maintenance

# 1. Check server load
uptime
htop

# 2. Check if service is running
sudo systemctl status nginx
sudo systemctl status apache2
sudo systemctl status php8.2-fpm

# 3. Check connections
ss -s                                 # Socket summary
ss -tlnp | grep :80                   # Connections on port 80

# 4. Check rate limiting
# Nginx: check if limit_req is too aggressive
# Cloudflare: check if WAF is blocking

# 5. Database overloaded
mysql -u root -p -e "SHOW PROCESSLIST;"
# Too many connections? Kill stuck queries
Enter fullscreen mode Exit fullscreen mode

504 Gateway Timeout 

# Means: Backend took too long to respond

# 1. Increase timeout in Nginx
proxy_read_timeout 300;
proxy_connect_timeout 300;
fastcgi_read_timeout 300;

# 2. Increase timeout in Apache
ProxyTimeout 300
Timeout 300

# 3. Increase PHP execution time
# php.ini: max_execution_time = 300

# 4. Check for slow database queries
mysql -u root -p -e "SHOW FULL PROCESSLIST;"
# Check slow query log

# 5. Check if server is overloaded
htop                                   # High CPU/RAM usage?
iostat                                 # High disk I/O?
Enter fullscreen mode Exit fullscreen mode

DNS Issues Troubleshooting 

# 1. Domain not resolving
dig example.com                        # Check A record
dig @8.8.8.8 example.com              # Check with Google DNS
ping example.com                       # Test connectivity

# 2. Wrong IP
dig +short example.com                 # What IP is it resolving to?
# Update A record in DNS provider

# 3. Propagation delay
# Check propagation: dnschecker.org
# Solution: Wait, or flush local DNS cache
sudo systemd-resolve --flush-caches

# 4. Subdomain not working
dig subdomain.example.com              # Check if A/CNAME record exists
# Add appropriate record in DNS

# 5. Email not working
dig MX example.com                     # Check MX records
dig TXT example.com                    # Check SPF/DKIM
Enter fullscreen mode Exit fullscreen mode

SSL Issues Troubleshooting 

# 1. Certificate expired
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates
sudo certbot renew

# 2. Mixed content (HTTP on HTTPS page)
# Fix URLs in code/database to use HTTPS
# WordPress: update site URL in wp_options table

# 3. Certificate doesn't match domain
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -subject -issuer
# Reissue certificate for correct domain

# 4. Let's Encrypt renewal failing
sudo certbot renew --dry-run          # Test renewal
# Common issues:
# - Port 80 blocked (needs to be open for HTTP challenge)
# - DNS not pointing to this server
# - .well-known/acme-challenge not accessible
Enter fullscreen mode Exit fullscreen mode

Database Connectivity Issues 

# 1. Can't connect to MySQL
sudo systemctl status mysql            # Is it running?
sudo tail -20 /var/log/mysql/error.log # Check error log

# 2. Access denied
mysql -u root -p                       # Test credentials
# Check user@host: SELECT user, host FROM mysql.user;

# 3. Too many connections
mysql -u root -p -e "SHOW STATUS LIKE 'Threads_connected';"
mysql -u root -p -e "SHOW VARIABLES LIKE 'max_connections';"
# Increase max_connections if needed

# 4. MySQL crashed
sudo systemctl restart mysql
sudo tail -50 /var/log/mysql/error.log
# Check for OOM: sudo dmesg | grep -i mysql
Enter fullscreen mode Exit fullscreen mode

Email/SMTP Troubleshooting 

# 1. Check mail queue
mailq                                  # View pending emails
sudo postqueue -f                      # Flush/retry mail queue
sudo postsuper -d ALL                  # Delete all queued mail

# 2. Check mail logs
sudo tail -f /var/log/mail.log
sudo grep "status=bounced" /var/log/mail.log | tail -20

# 3. Test SMTP
telnet mail.example.com 25             # Test SMTP connection
openssl s_client -connect mail.example.com:587 -starttls smtp  # Test STARTTLS

# 4. Check SPF/DKIM/DMARC
dig TXT example.com                    # SPF
dig TXT default._domainkey.example.com # DKIM
dig TXT _dmarc.example.com            # DMARC

# 5. Test email delivery
echo "Test" | mail -s "Test Email" user@example.com

# 6. Check if port 25/587 is blocked
sudo ss -tlnp | grep -E "25|587"
sudo ufw status | grep -E "25|587"

# 7. Check blacklist status
# Use: mxtoolbox.com/blacklists.aspx
# Check reverse DNS: dig -x YOUR_SERVER_IP
Enter fullscreen mode Exit fullscreen mode

Interview Q&A: Troubleshooting

Question Answer
First thing you do when a site is down? Check if server is reachable (ping), check web server status, check error logs
How to diagnose a 500 error? Check error logs (apache/nginx/php), check permissions, check .htaccess, check PHP config
Difference between 502 and 504? 502: backend unreachable/crashed. 504: backend is running but too slow to respond
Website is slow. How to diagnose? Check server load (htop), check slow query log, check PHP-FPM status, check disk I/O, check network
Server ran out of disk space. Quick fix? Find large files (du -sh), clean old logs, clear apt cache (apt clean), remove old backups
How to handle DDoS? Enable Cloudflare, configure rate limiting, use fail2ban, block suspicious IPs, enable UFW rate limiting
How to check if emails are being delivered? Check mail logs, verify SPF/DKIM/DMARC records, check blacklists, test with mxtoolbox

Top comments (0)