Linux Commands Every Developer Should Know (Beyond the Basics)

The commands that don't appear in beginner tutorials — but show up constantly in production debugging, deployment pipelines, and performance investigations.

---

Most Linux command guides teach you ls, cd, grep, and chmod. You already know those. What they don't cover is what happens at 2am when a production server is behaving strangely, a deployment is hung, a disk is mysteriously full, or a process is consuming memory and you can't tell why.

That's where the real Linux knowledge lives — not in the commands themselves, but in knowing which tool to reach for when a system is misbehaving, and how to compose those tools into a diagnostic chain that gives you actual answers.

This article covers the Linux commands that experienced developers actually use in real work: debugging live systems, profiling performance, managing processes, inspecting network behavior, and automating deployment tasks. Each command comes with context for when it matters and concrete patterns you can use immediately.

---

Process Inspection and Management

ps — Beyond the Basics

Everyone knows ps aux. What's more useful is filtering and formatting its output for specific diagnostic questions.

# Find all processes belonging to a specific user, sorted by memory
ps aux --sort=-%mem | grep -v grep | grep www-data

# Show process tree for a specific PID — understand parent-child relationships
ps --ppid 1234 -o pid,ppid,cmd,%mem,%cpu

# Find what command spawned a process (useful when PID is known from another tool)
ps -p 4521 -o pid,ppid,cmd,start,etime

# Show processes consuming the most CPU right now, top 10
ps aux --sort=-%cpu | head -11

The --sort flag on ps is underused. Sorting by -%mem or -%cpu gives you an instant snapshot of what's consuming resources without running an interactive tool.

pgrep and pkill — Surgical Process Control

# Find PIDs of all nginx worker processes
pgrep -a nginx

# Kill all processes matching a name, with confirmation
pkill -l nginx

# Send a specific signal — reload nginx config without downtime
pkill -HUP nginx

# Find processes by name AND user
pgrep -u www-data php-fpm

# Count how many workers are running
pgrep -c php-fpm

pkill -HUP is the correct way to reload most daemon configurations (nginx, PHP-FPM, syslog) — it sends SIGHUP which tells the process to re-read its config, rather than SIGTERM which terminates it.

lsof — Everything a Process Has Open

lsof (list open files) is one of the most powerful diagnostic tools on Linux, and it's underused because its output is dense. Understanding it unlocks a category of debugging problems that nothing else can solve.

# What files does this process have open?
lsof -p 3847

# What process has this port open?
lsof -i :3000
lsof -i :80

# What process has this specific file open? (useful when you can't delete a file)
lsof /var/log/nginx/access.log

# All network connections for a specific process name
lsof -i -a -p $(pgrep node)

# Find deleted files still held open by processes (disk space mystery solver)
lsof +L1

# How many file descriptors is a process using?
lsof -p 3847 | wc -l

The last one — lsof +L1 — is the answer to a specific and maddening situation: disk usage reported by df doesn't match what du shows. This happens when a process has deleted a file but still holds a file descriptor open. The disk space won't be released until the process closes or restarts. lsof +L1 shows all such "deleted but still open" files immediately.

---

System Resource Analysis

vmstat — Memory and CPU at a Glance

# Continuous output every 2 seconds
vmstat 2

# Output with timestamps (critical for correlating with logs)
vmstat -t 2

# Show memory stats in human-readable form
vmstat -s -S M

The vmstat output columns that matter most for web server debugging:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 142848  12544 891392    0    0     1    18  328  621  8  2 89  1  0

• r (run queue): processes waiting for CPU. If consistently > number of CPU cores, you're CPU-bound
• b (blocked): processes blocked on I/O. If nonzero consistently, you're I/O-bound
• si/so (swap in/swap out): if these are nonzero, you're swapping — a serious performance problem
• wa (wait): CPU time spent waiting for I/O. High wa with low us means disk is the bottleneck

iostat — Disk I/O Profiling

# Install if missing
sudo apt install sysstat

# Continuous disk I/O stats every 2 seconds
iostat -x 2

# Show stats for a specific device
iostat -x /dev/sda 2

Key columns in iostat -x output:

Device  r/s    w/s   rkB/s   wkB/s  await  svctm  %util
sda    12.3   48.7   156.2   891.4   8.23   1.12   12.4

• await: average time (ms) for I/O requests to complete. Over 20ms for SSDs indicates a problem
• %util: percentage of time device was busy. Over 80% sustained means the disk is a bottleneck
• svctm: service time per request. Significantly lower than await means requests are queuing

free — Memory at a Glance

# Human-readable, refreshing every 2 seconds
watch -n2 free -h

# Show memory in megabytes
free -m

The commonly misread column is available vs free:

              total        used        free      shared  buff/cache   available
Mem:           7.7G        3.2G        142M        124M        4.4G        4.1G
Swap:          2.0G          0B        2.0G

free (142M here) is genuinely unused memory. available (4.1G) includes memory that's currently used for buffer/cache but can be immediately reclaimed by processes that need it. The available column is the right number to watch — a system with 142M free but 4.1G available is not in trouble.

---

Network Diagnostics

ss — The Modern netstat

netstat is deprecated on modern Linux. ss is faster and more capable.

# All listening ports with process names
ss -tlnp

# All established TCP connections
ss -tnp state established

# Connections to a specific port
ss -tnp sport = :443

# Summary of connection states
ss -s

# Show connections in TIME_WAIT state (important for high-traffic servers)
ss -tn state time-wait | wc -l

# All Unix domain sockets (PHP-FPM socket, Redis socket, etc.)
ss -xlp

A high TIME_WAIT count (tens of thousands) on a server handling many short-lived connections indicates you should tune TCP recycling:

# Check current TIME_WAIT count
ss -s | grep TIME-WAIT

# Enable faster TIME_WAIT recycling (add to /etc/sysctl.conf)
echo "net.ipv4.tcp_tw_reuse = 1" >> /etc/sysctl.conf
sysctl -p

tcpdump — Packet-Level Debugging

tcpdump is the tool for diagnosing problems that manifest at the network packet level — unexpected connection resets, TLS handshake failures, or verifying that traffic is actually reaching a service.

# Capture HTTP traffic on port 80 (human-readable)
sudo tcpdump -i eth0 -A port 80

# Capture traffic to/from a specific IP
sudo tcpdump -i any host 203.0.113.42

# Capture and write to file for analysis in Wireshark
sudo tcpdump -i eth0 -w /tmp/capture.pcap port 443

# Show only TCP SYN packets (new connections)
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0'

# DNS queries (useful for debugging service discovery issues)
sudo tcpdump -i any port 53 -n

curl as a Diagnostic Tool

Most developers know curl for API calls. Its diagnostic flags are far more useful for debugging:

# Full timing breakdown of an HTTP request
curl -o /dev/null -s -w "
DNS lookup:      %{time_namelookup}s
TCP connect:     %{time_connect}s
TLS handshake:   %{time_appconnect}s
TTFB:            %{time_starttransfer}s
Total:           %{time_total}s
HTTP status:     %{http_code}
" https://yoursite.com

# Follow redirects and show all response headers
curl -Lv https://yoursite.com 2>&1 | grep -E "^[<>*]"

# Test specific TLS versions
curl --tlsv1.3 --tls-max 1.3 -v https://yoursite.com

# Send a request with a specific Host header (test virtual host routing)
curl -H "Host: staging.yoursite.com" http://192.168.1.10/

# Check GZIP compression
curl -H "Accept-Encoding: gzip,deflate" -I https://yoursite.com | grep -i content-encoding

The timing breakdown format is worth saving as a shell alias — it's the fastest way to identify whether a slow site is suffering from DNS, TCP connection, TLS negotiation, or server processing time.

---

File System and Disk Management

find — Surgical File Location

# Files modified in the last 24 hours
find /var/www -mtime -1 -type f

# Files larger than 100MB (disk space investigation)
find / -size +100M -type f 2>/dev/null

# Find world-writable files (security audit)
find /var/www -perm -o+w -type f

# Find SUID binaries (security audit)
find / -perm /4000 -type f 2>/dev/null

# Find and delete log files older than 30 days
find /var/log/app -name "*.log" -mtime +30 -delete

# Find files by owner
find /var/www -user www-data -type f -name "*.php"

# Find recently modified PHP files (post-compromise check)
find /var/www -name "*.php" -newer /tmp/reference_file -type f

The last pattern — finding PHP files newer than a reference file — is a quick post-incident check for injected malware. Create a reference file dated to your last known-good deployment, then find anything modified after it.

du and df — Disk Usage Investigation

# Find the largest directories, sorted
du -sh /* 2>/dev/null | sort -rh | head -20

# Drill into a specific directory
du -sh /var/log/* | sort -rh | head -10

# Show disk usage excluding a specific filesystem type
df -h --exclude-type=tmpfs --exclude-type=devtmpfs

# Show inode usage (a full inode table causes "disk full" errors even with space available)
df -i

# Find directories consuming the most inodes
find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -rn | head -10

Inode exhaustion is a production problem that's easy to miss: df -h shows available disk space, but writes still fail with "No space left on device." Running df -i immediately reveals if inodes are the real issue — common on servers with millions of small files (mail servers, log aggregators, Node.js node_modules directories).

---

Text Processing and Log Analysis

awk — Structured Log Analysis

awk is a complete data processing language, and for structured log files (nginx access logs, Apache logs, CSV exports) it replaces entire Python scripts with one-liners.

# Count requests by HTTP status code from nginx access log
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# Find the top 10 IPs by request count
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

# Calculate average response time (if logged as last field)
awk '{sum += $NF; count++} END {print "Average:", sum/count "ms"}' /var/log/nginx/access.log

# Find all 5xx errors with their request paths
awk '$9 ~ /^5/ {print $9, $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# Sum total bytes transferred by IP address
awk '{bytes[$1] += $10} END {for (ip in bytes) print bytes[ip], ip}' /var/log/nginx/access.log | sort -rn | head -10

sed — In-Place Text Transformation

# Replace a string in a file in-place (with backup)
sed -i.bak 's/old_db_host/new_db_host/g' /var/www/config/database.php

# Delete blank lines from a file
sed -i '/^$/d' config.txt

# Extract lines between two patterns
sed -n '/START_MARKER/,/END_MARKER/p' logfile.txt

# Remove trailing whitespace
sed -i 's/[[:space:]]*$//' file.txt

# Comment out a line containing a specific string
sed -i '/dangerous_setting/s/^/#/' /etc/app/config.conf

grep — Beyond Simple Matching

# Recursive search with line numbers and filename
grep -rn "wp_query" /var/www/wp-content/themes/

# Show context: 3 lines before and after match
grep -B3 -A3 "OutOfMemoryError" /var/log/app/application.log

# Count matches per file
grep -rc "ERROR" /var/log/app/

# Invert match — show lines that DON'T contain pattern
grep -v "200 OK" /var/log/nginx/access.log

# Search for multiple patterns simultaneously
grep -E "ERROR|CRITICAL|FATAL" /var/log/app/application.log

# Match only at the start of a line (anchored)
grep "^2024-07" /var/log/app/application.log

# Binary file search (useful for compiled configs)
grep -a "database_password" /var/www/app.cache

---

System Monitoring and Real-Time Debugging

strace — What Is This Process Actually Doing?

strace traces system calls made by a process. It answers the question: "Why is this process hanging?" when you can't get any other answer.

# Trace a running process (attach by PID)
sudo strace -p 4521

# Trace only file-related system calls
sudo strace -p 4521 -e trace=file

# Trace network system calls
sudo strace -p 4521 -e trace=network

# Trace with timestamps
sudo strace -p 4521 -T -tt

# Trace all processes in a process group
sudo strace -p 4521 -f

# Count system calls (performance profiling)
sudo strace -c -p 4521

If a process is stuck, strace -p PID immediately shows what system call it's blocked on. Blocked on read() from a file descriptor? Check lsof -p PID to see what that fd is. Blocked on futex()? It's waiting on a mutex — likely a deadlock or lock contention issue.

journalctl — Structured System Log Access

# Follow logs in real time (like tail -f but smarter)
journalctl -f

# Logs for a specific service
journalctl -u nginx -f

# Logs since a specific time
journalctl -u php-fpm --since "2024-01-15 14:00:00"

# Logs from the last boot
journalctl -b

# Show only error-level and above
journalctl -p err -u nginx

# Export logs for a time range
journalctl -u app --since "1 hour ago" --until "30 min ago" > incident.log

# Disk usage of journal
journalctl --disk-usage

# Vacuum old logs
journalctl --vacuum-time=7d

watch — Continuous Command Monitoring

# Refresh every 2 seconds, highlight changes
watch -n2 -d "ss -tnp | grep :3000"

# Monitor disk I/O continuously
watch -n1 "iostat -x 1 1 | tail -n+4"

# Watch log line count growth rate
watch -n5 "wc -l /var/log/nginx/access.log"

# Monitor memory of a specific process
watch -n2 "ps -p $(pgrep node) -o pid,rss,vsz,%mem,cmd"

---

Security and Permissions

File Permission Auditing

# Find all files with SUID bit set (potential privilege escalation vectors)
find / -perm -u=s -type f 2>/dev/null

# Find files writable by everyone
find /var/www -perm -0002 -not -type l

# Check what a user can sudo
sudo -l -U www-data

# Show effective permissions for current user on a file
namei -l /var/www/html/wp-config.php

# Audit SSH authorized keys across all users
for user in $(cut -d: -f1 /etc/passwd); do
  keys="/home/$user/.ssh/authorized_keys"
  if [ -f "$keys" ]; then
    echo "=== $user ==="
    cat "$keys"
  fi
done

fail2ban-client and ufw — Active Security Management

# Check fail2ban status for all jails
sudo fail2ban-client status

# Check a specific jail
sudo fail2ban-client status sshd

# Unban an IP (when you accidentally lock yourself out)
sudo fail2ban-client set sshd unbanip 203.0.113.42

# UFW status with numbered rules
sudo ufw status numbered

# Allow a port for a specific IP only
sudo ufw allow from 203.0.113.42 to any port 5432

# Delete a specific rule by number
sudo ufw delete 3

---

Automation and Shell Scripting Patterns

Here Documents and Process Substitution

# Heredoc for multi-line commands (useful in deployment scripts)
cat <<'EOF' > /etc/nginx/sites-available/myapp
server {
    listen 80;
    server_name example.com;
    root /var/www/myapp/public;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }
}
EOF

# Process substitution — compare two command outputs without temp files
diff <(ls /var/www/production) <(ls /var/www/staging)

# Pipe into while read — process command output line by line
ps aux | awk '$3 > 50 {print $2}' | while read pid; do
    echo "High CPU PID: $pid - $(ps -p $pid -o cmd --no-header)"
done

Useful One-Liners for Deployment Pipelines

# Wait for a service to become available before proceeding
until curl -sf http://localhost:3000/health; do
    echo "Waiting for app to start..."
    sleep 2
done

# Retry a command with exponential backoff
for i in 1 2 4 8 16; do
    command_that_might_fail && break || sleep $i
done

# Run a command in parallel across multiple servers
echo "server1.example.com server2.example.com server3.example.com" | \
  tr ' ' '\n' | \
  xargs -P3 -I{} ssh {} "sudo systemctl reload nginx"

# Check if a port is open before connecting (no netcat required)
timeout 3 bash -c "cat < /dev/null > /dev/tcp/localhost/5432" && \
  echo "Port open" || echo "Port closed"

# Generate a random secret suitable for environment variables
openssl rand -base64 48 | tr -d "=+/" | cut -c1-32

---

Environment and Context: Where These Commands Matter Most

The situations in which these commands matter most are usually not in development environments — they're on servers managing real production traffic, where the cost of not knowing the right command is measured in downtime.

Developers working on website development in Glen Ellyn Illinois and similar suburban Chicago markets often support clients on managed VPS infrastructure where SSH access is available but server management expertise is limited. In that context, commands like lsof +L1 for diagnosing phantom disk usage, or ss -s for identifying connection state anomalies, solve real client-facing problems that no managed hosting dashboard exposes.

Similarly, teams handling website development in Lansing Illinois have noted that the combination of strace, lsof, and journalctl is what turns a vague "the server is slow" complaint into an actionable diagnosis — narrowing a performance problem to a specific process, file descriptor, or system call within minutes rather than hours of guesswork.

The common thread: Linux command proficiency is what separates developers who can triage production incidents from those who open a support ticket and wait.

---

Building a Personal Diagnostic Toolkit

The most productive thing you can do with these commands is build a personal toolkit — a set of shell functions and aliases that make diagnostic workflows reproducible.

# Add to ~/.bashrc or ~/.zshrc

# Show what's using a port
port() { lsof -i :"$1" -sTCP:LISTEN -n -P; }

# Full HTTP timing for a URL
timing() {
    curl -o /dev/null -s -w "
DNS:    %{time_namelookup}s
TCP:    %{time_connect}s
TLS:    %{time_appconnect}s
TTFB:   %{time_starttransfer}s
Total:  %{time_total}s
Code:   %{http_code}
" "$1"
}

# Top memory consumers
memtop() { ps aux --sort=-%mem | head -"${1:-11}"; }

# Top CPU consumers
cputop() { ps aux --sort=-%cpu | head -"${1:-11}"; }

# Find what's eating disk space in current directory
diskuse() { du -sh "${1:-.}"/* 2>/dev/null | sort -rh | head -20; }

# Tail logs with highlighting
logs() { journalctl -u "$1" -f | grep --color=always -E "ERROR|WARN|$"; }

# Check if a host/port is reachable
reachable() { timeout 3 bash -c "cat < /dev/null > /dev/tcp/$1/$2" && echo "open" || echo "closed"; }

Source the file after adding:

source ~/.bashrc

Now timing https://example.com, port 3000, and logs nginx become fast, repeatable diagnostics.

---

The Mental Model That Makes All of This Click

Linux diagnostic commands follow a layered pattern that mirrors the system itself:

• Process layer: ps, pgrep, lsof, strace — what's running and what is it doing?
• Resource layer: vmstat, iostat, free — what resources are being consumed and where?
• Network layer: ss, tcpdump, curl — what traffic is flowing and where is it going?
• File layer: find, du, df — what exists on disk and what's consuming space?
• Log layer: grep, awk, journalctl — what has happened and what is the system telling us?

When something goes wrong on a server, the investigation flows through these layers top-down. A slow server → check process layer (what's using CPU/memory?). Resource anomaly found → check network layer (are there unexpected connections?) or file layer (is disk I/O the culprit?). Unexplained behavior → log layer (what does the system's own record say?).

The commands in this article aren't a list to memorize. They're tools in a diagnostic chain. Know what layer each one addresses, and you'll instinctively reach for the right one when systems misbehave.

---