DEV Community: Asep Sayyad

The ss Command Is More Powerful Than You Think: 10 Practical Examples

Asep Sayyad — Sun, 01 Nov 2026 10:05:40 +0000

Modern socket diagnostics, TCP queue inspection, and kernel-level network troubleshooting for Linux engineers

If you still type netstat -tulpn during a late-night production outage, you are leaning on an obsolete utility that was officially marked deprecated in 2001.

netstat crawls through /proc/net/tcp line by line. On a busy Kubernetes worker node, an API gateway, or a database replica handling thirty thousand active connections, parsing /proc/net/tcp line by line is slow and CPU-heavy. The command pauses, chews through memory, and can freeze your interactive shell while reading through tens of thousands of pseudo-file entries.

The modern standard is ss (Socket Statistics). Part of the iproute2 package, ss talks directly to the Linux kernel via Netlink sockets (NETLINK_INET_DIAG). Instead of string-parsing simulated procfs entries, the kernel dumps binary socket records straight into user space in batches. What takes netstat four or five seconds to crawl through on a congested edge proxy renders instantaneously with ss.

Beyond speed, ss gives you direct visibility into kernel socket buffers, TCP congestion windows, packet retransmissions, socket timers, and connection backlog drops that netstat could never expose.

Here are ten practical, real-world examples that show what ss can do during actual systems troubleshooting.

1. Fast Socket Triage with ss -tulpn

The foundational muscle memory command to inspect every active listening service on your host is:

sudo ss -tulpn

Breaking down this flag combination:

-t filters for TCP sockets.
-u filters for UDP sockets.
-l displays listening sockets only (by default, ss lists established connections).
-p resolves and prints the owning process name, process ID (PID), and file descriptor numbers.
-n disables reverse DNS resolution and port name lookups (printing raw IP addresses and integer port numbers, avoiding terminal hangs caused by slow DNS servers).

The output provides an instant inventory of open ports and running daemons:

Netid  State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  Process
tcp    LISTEN  0       128            0.0.0.0:22          0.0.0.0:*      users:(("sshd",pid=912,fd=3))
tcp    LISTEN  0       511          127.0.0.1:6379        0.0.0.0:*      users:(("redis-server",pid=1120,fd=6))
tcp    LISTEN  0       1024           0.0.0.0:80          0.0.0.0:*      users:(("nginx",pid=2045,fd=7))

Notice the users:(("nginx",pid=2045,fd=7)) column. Seeing the exact PID and file descriptor lets you jump straight to process tracing with lsof -p 2045 or inspecting memory maps in /proc/2045/status.

2. Uncovering Backlog Bottlenecks with Recv-Q and Send-Q

Most engineers scan past the Recv-Q and Send-Q columns in ss output, assuming they are only useful for network engineers. During an incident where an application drops incoming traffic or responds with 502 Bad Gateway errors, these two columns pinpoint the problem immediately.

Crucially, their meaning changes depending on whether the socket is in a listening state or an established connection state.

For listening sockets (ss -ltn):

Send-Q represents the maximum listen backlog limit configured for that socket (the backlog parameter passed to the listen(fd, backlog) system call, capped by the kernel parameter net.core.somaxconn).
Recv-Q shows the number of completed TCP connections currently sitting in the kernel accept queue, waiting for the application process to invoke accept().

ss -ltn

If you see:

State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port
LISTEN  512     511          127.0.0.1:8080        0.0.0.0:*

Here, Recv-Q has surpassed Send-Q. The kernel accept queue for port 8080 is completely saturated. Incoming TCP handshakes are either being silently ignored or dropped, clients are timing out during initial connection establishment, and the application runtime (like Node.js, Python Gunicorn, or Java) is stuck in an event loop or CPU starvation lock and failing to process pending connections.

For established sockets (ss -tn):

Recv-Q is the number of bytes received by the network card and kernel TCP stack, but not yet read by the user-space application via read() or recv().
Send-Q is the number of bytes queued in the local socket write buffer waiting for acknowledgment (ACK) from the remote peer.

A steadily climbing Send-Q on an established connection signals a slow client, network packet loss, or a saturated egress pipe. A climbing Recv-Q signals that your local consumer application is falling behind and cannot drain its socket buffers fast enough.

3. Global Socket Summary Statistics with ss -s

When you first log in to an unstable or degraded Linux server, running detailed queries can overwhelm your terminal if thousands of sockets exist.

The -s (summary) flag reads high-level networking counters directly from kernel memory without iterating through individual sockets:

ss -s

Sample output:

Total: 1842
TCP:   3420 (estab 215, closed 3110, orphaned 12, timewait 2980)

Transport Total     IP        IPv6
RAW   2         1         1        
UDP   14        9         5        
TCP   310       240       70       
INET      326       250       76       
FRAG      0         0         0

In three milliseconds, you get a clean breakdown of socket allocations.

If you see thousands of sockets stuck in timewait, your host is churning through short-lived connections (common on reverse proxies without HTTP keep-alive) and may be on the verge of local port exhaustion (net.ipv4.ip_local_port_range).

If the orphaned counter is elevated, connections have been closed by user space but the kernel is still holding the buffers to flush outbound packets or wait for FIN-ACK sequences. High orphan counts can trigger kernel memory pressure and invoke TCP socket drop routines.

4. Querying Sockets by State Names

With netstat, filtering for connection states required piping output through grep ESTABLISHED or grep TIME_WAIT. This approach is fragile because it can accidentally match IP addresses, hostnames, or process paths containing those words.

ss provides native state filtering built into its parser using the state keyword:

# List all sockets in the TIME-WAIT state
ss -tan state time-wait

# List all sockets actively establishing connections
ss -tan state syn-sent

# List all sockets waiting for a remote teardown
ss -tan state close-wait

CLOSE_WAIT deserves special attention. A socket in CLOSE_WAIT means the remote peer sent a FIN packet and closed their side of the connection, the Linux kernel acknowledged it, but the local application process never called close() on the associated file descriptor.

Seeing hundreds of CLOSE_WAIT sockets indicates an application-level resource leak:

ss -tanp state close-wait

The output reveals the exact process ID holding those dead descriptors open, giving you the information needed to patch the code or restart the leaky service.

5. Filtering Sockets by Source and Destination Ports

Piping command output into grep :80 or grep :443 frequently matches unrelated data such as ephemeral client ports, process identifiers, or random substrings.

ss supports native port and IP filter expressions. You pass the filter string at the end of the command:

# Find all established connections to or from port 443
ss -tan '( sport = :443 or dport = :443 )'

# Find all outgoing outbound connections to an external database port
ss -tan dport = :5432

Note that sport stands for source port, and dport stands for destination port. When using spaces inside the filter syntax, always wrap the expression in quotes so your shell does not misinterpret parentheses or comparison operators.

You can also use numeric comparisons to isolate specific port ranges:

# Find connections where the remote port is a non-privileged high port
ss -tan 'dport >= :1024'

6. Filtering by CIDR Subnets and Specific IP Addresses

When troubleshooting noisy network peers, security sweeps, or cluster communication problems, isolating traffic from a specific subnet is essential.

ss accepts standard CIDR notation for source and destination addresses:

# Find all connections originating from the internal VPC subnet
ss -tan src 10.0.0.0/16

# Find connections directed to an external storage cluster subnet
ss -tan dst 192.168.1.0/24

# Combine IP and port conditions
ss -tan 'src 10.0.0.0/16 and dport = :3306'

This syntax filters at the kernel diagnostic level. The command bypasses user-space string regexes entirely, rendering matching sockets instantly even on nodes carrying tens of thousands of active network streams.

7. Deep TCP Telemetry and Retransmissions with ss -ti

When debugging intermittent latency, packet loss, or throughput collapse across distributed microservices, application logs only tell you that a request timed out. They do not tell you why the TCP layer struggled.

The -i (internal) flag instructs ss to extract the internal TCP control metrics tracked by the Linux kernel for that specific socket:

ss -ti 'dport = :443'

Sample output:

ESTAB      0      0       192.168.1.50:54321    93.184.216.34:443
     cubic wscale:7,7 rto:204 rtt:4.12/0.05 ato:40 mss:1460 rcvspace:14600
     ssthresh:10 cwnd:10 unacked:1 retrans:0/2 lost:0 reordering:3

This metric dump reveals vital networking indicators:

rtt:4.12/0.05: The smoothed round-trip time is 4.12 milliseconds with a tiny variance of 0.05ms.
rto:204: The retransmission timeout is 204ms. If an acknowledgment is not received within this window, the kernel retransmits the segment.
cwnd:10: The congestion window size in packets. A tiny cwnd on a high-bandwidth link indicates that the kernel recently cut its window due to packet drops.
retrans:0/2: Shows current active retransmissions alongside total retransmissions over the lifetime of the socket.
cubic: The active TCP congestion control algorithm handling the flow.

Seeing a high retrans count or a depressed cwnd for a specific peer confirms that throughput issues stem from physical link congestion or dropped packets, rather than slow application code.

8. Sizing Socket Memory Buffers with ss -tm

Every active TCP socket consumes kernel memory to buffer unacknowledged outbound packets and unread incoming data. If thousands of connections open simultaneously, socket memory consumption can starve the system, causing the kernel to trigger out-of-memory drops.

The -m flag displays memory allocations per socket:

ss -tm 'sport = :80'

Sample output:

ESTAB      0      0       10.0.0.4:80          10.0.0.85:49152
     skmem:(r0,rb131072,t0,tb16384,f0,s0,uca0,d0)

The skmem string details exact byte allocations:

r0: Receive buffer memory currently used by data chunks.
rb131072: Maximum allocated receive buffer limit (128 KB in this example).
t0: Transmit buffer memory currently used.
tb16384: Maximum allocated transmit buffer limit (16 KB).
f0: Free socket buffer memory forward-allocated.

If you suspect an edge server is running out of network buffer memory, checking ss -tm alongside the kernel sysctl limits (net.ipv4.tcp_rmem and net.ipv4.tcp_wmem) tells you whether individual connections are blowing past their expected buffer allocations.

9. Inspecting UNIX Domain Sockets with ss -xlp

Microservices, reverse proxies, and local daemons often bypass network stacks entirely, communicating over high-speed local UNIX domain sockets (such as Docker daemon sockets, Nginx upstream sockets, and systemd IPC endpoints).

Standard network monitoring commands miss these IPC sockets. With ss, you query them using the -x flag:

sudo ss -xlp

Breaking down the flags:

-x selects UNIX domain sockets.
-l lists listening sockets.
-p identifies the owning daemon process and file descriptor.

Sample output:

Netid  State   Recv-Q  Send-Q   Local Address:Port             Peer Address:Port  Process
u_str  LISTEN  0       128      /var/run/docker.sock 18234            * 0          users:(("dockerd",pid=1420,fd=5))
u_str  LISTEN  0       511      /var/run/php/php-fpm.sock 23412       * 0          users:(("php-fpm7.4",pid=1890,fd=7))

During periods of high traffic between a local Nginx instance and a PHP-FPM or Puma backend, checking Recv-Q on the UNIX domain socket reveals whether the backend worker pool is overwhelmed. If Recv-Q approaches Send-Q on /var/run/php/php-fpm.sock, the local socket queue is backing up, causing Nginx to drop connections with 504 Gateway Timeout errors.

10. Combining State and Port Filters for Targeted Incident Response

During a high-concurrency production incident, you need to isolate specific operational patterns without scrolling through thousands of lines of terminal noise.

You can combine boolean logic, state queries, and port filters into a single precision command:

# Find all established connections on port 80 or 443 where the send buffer is backing up
ss -tan '( sport = :80 or sport = :443 )' state established 'Send-Q > 0'

# Identify all connections from an external database cluster stuck in FIN-WAIT states
ss -tan dst 10.100.5.0/24 state fin-wait-1

# Track connection attempts that have not yet completed handshakes
ss -tan state syn-recv

Running queries like Send-Q > 0 directly isolates connections with blocked egress buffers. Instead of writing custom parsing scripts or dumping network captures, you spot clogged streams with one command.

Quick Reference: Essential ss Commands

Listen sockets with process details:
Run sudo ss -tulpn. It displays active TCP and UDP listening ports alongside their owning process names, PIDs, and listen backlog sizes.

High-level socket counters:
Run ss -s. It returns instant summary statistics for established, closed, orphaned, and time-wait sockets directly from the kernel.

Inspect queue backlogs:
Run ss -ltn. For listening ports, Recv-Q shows connections waiting for user space accept(), while Send-Q shows the maximum backlog limit.

Find dead connections:
Run ss -tanp state close-wait. It isolates sockets where the remote peer closed the connection but the local application process leaked the descriptor.

Track packet loss and latency:
Run ss -ti 'dport = :443'. It exposes round-trip time (rtt), congestion windows (cwnd), and retransmission counts (retrans) for active connections.

Inspect local IPC sockets:
Run sudo ss -xlp. It checks UNIX domain socket health for services like Docker, Redis, and PHP-FPM communicating over local filesystem paths.

An Interesting Fact About ss and Netlink

The original netstat command read socket information by opening and reading /proc/net/tcp and /proc/net/udp. In Linux, the kernel generates the text for /proc/net/tcp on the fly every time a process reads the file. For a machine with 50,000 active sockets, the kernel has to allocate memory, format 50,000 lines of ASCII text, and pass it through a file descriptor.

The ss utility was developed specifically to solve this scalability bottleneck. Alexey Kuznetsov, the primary author of the modern Linux network stack and iproute2, designed the sock_diag subsystem inside the Linux kernel.

Instead of generating human-readable text in /proc, ss sends a Netlink message (NETLINK_INET_DIAG) asking the kernel for binary socket state structs. The kernel streams the raw binary records directly across the Netlink socket without text formatting overhead, making ss orders of magnitude faster on production servers.

Practical Takeaways for Your Daily Workflow

Continuing to rely on netstat means accepting slower diagnostics and missing out on kernel-level TCP telemetry.

To help transition your workflow, add these aliases to your shell configuration (~/.bashrc or ~/.zshrc):

# Fast socket triage
alias ports="sudo ss -tulpn"

# Instant socket summary
alias sockstat="ss -s"

# Check for saturated accept queues
alias backlogs="ss -ltn 'Recv-Q > 0'"

Mastering ss turns socket inspection from a blunt port check into a precision diagnostic tool. You can spot saturated accept queues, trace packet loss to physical link drops, and isolate leaky file descriptors before an outage impacts your end users.

Which ss flags or custom aliases do you find most helpful when diagnosing network problems in production?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

Asep Sayyad is a Linux and DevOps engineer passionate about Linux administration, automation, cloud technologies, containers, and open-source software. He enjoys solving real-world infrastructure challenges and sharing practical knowledge through in-depth technical articles, tutorials, and hands-on guides.

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

Stop Using cat for Everything: 9 Better Ways to Read Linux Files

Asep Sayyad — Fri, 30 Oct 2026 04:25:29 +0000

Practical tools to inspect, page, slice, and debug files without freezing your terminal

Running cat /var/log/nginx/access.log on an active server is an unforced error that most engineers make once before watching their shell lock up.

The utility name is short, muscle memory takes over, and the binary exists on every Unix host in existence. But cat stands for concatenate. Its original Unix design was to join multiple input streams together and ship the output down a pipeline. It was never built as an interactive file viewer.

When you use cat to inspect files interactively, you offload stream buffering directly to your terminal emulator. If that file happens to be a multi-gigabyte JSON log or an active audit trail, your terminal tries to process, parse, and render millions of lines into its local scrollback window. Your terminal emulator consumes gigabytes of memory, remote SSH sessions drop keystrokes, stray escape sequences scramble font settings, and you have zero ability to pause, scroll, or search inline.

Linux comes with specialized utilities built to inspect, page, slice, reformat, and navigate files without wasting system resources or locking your session.

Here are nine better ways to read files on Linux, along with the exact flags to use in production.

1. bat: Syntax Highlighting, Git Awareness, and Smart Paging

If you want to view a configuration file, shell script, or source code file directly on stdout without opening an interactive viewer, bat is the modern replacement for cat.

On Debian and Ubuntu distributions, the package is installed as batcat to prevent name clashes with another system binary:

sudo apt-get install bat
alias bat="batcat"

Unlike raw cat, bat inspects file extensions and shebang lines to apply high-contrast syntax highlighting across hundreds of languages. It renders line numbers automatically and integrates directly with Git to show added, modified, and removed lines directly in the left gutter.

To view an Nginx configuration with line numbers and syntax highlighting:

bat /etc/nginx/nginx.conf

One of the best features in bat is automatic smart paging. If the target file fits within the height of your terminal screen, bat writes the output straight to stdout and drops you back at your shell prompt. If the output exceeds your terminal height, it routes the stream through an interactive pager automatically.

You can also use bat to expose non-printable control characters, trailing spaces, and hidden carriage returns:

bat -A /etc/environment

The -A flag displays tab stops as arrows, spaces as subtle dots, and newlines as explicit symbols, making syntax debugging straightforward.

2. less with -FXRS: Zero-Memory Paging and Instant Regex Search

When a file exceeds a few megabytes, printing it straight to stdout is risky. The less pager remains the industry benchmark for high-performance file viewing because it does not load the entire file into RAM before displaying it.

less uses memory mapping (mmap) to read only the small slice of bytes needed to fill your current terminal dimensions. You can open a 50GB database dump or a massive systemd log instantaneously without causing memory pressure or disk thrashing on the host:

less -FXRS /var/log/syslog

The four flags combined here make less behave cleanly for system administration:

-F tells less to exit immediately if the entire file fits on a single terminal screen.
-X stops less from sending the terminal initialization strings that clear the screen on exit, leaving the inspected lines visible in your terminal scrollback.
-R processes raw ANSI color escape codes properly, ensuring colored log output from modern CLI tools renders correctly instead of displaying garbled control sequences.
-S disables line wrapping. Long log lines are truncated horizontally instead of wrapping into messy multi-line blocks. You can pan horizontally using the left and right arrow keys to inspect wide timestamps and JSON structures.

Inside less, you navigate quickly using single keystrokes:

Press G to jump straight to the end of the file.
Press g to return to the first line.
Type / followed by a search pattern to search forward using regular expressions.
Type ? followed by a search pattern to search backward through previous entries.
Press n to jump to the next search match, or N to jump to the previous match.
Press +F to enter follow mode, which updates the view in real time as new lines append to the file, identical to tail -f. Pressing Ctrl+C drops you back into scrollable, searchable mode without closing the file.

You can set these flags permanently by adding this export to your shell configuration (~/.bashrc or ~/.zshrc):

export LESS="-FXRS"
export PAGER="less"

3. tail -F and head: Slicing Boundaries and Tracking Rotated Logs

Most operational debugging tasks focus on either the beginning of a configuration file or the very latest events recorded by an application. Reading the entire file to find those lines wastes I/O bandwidth.

head and tail let you slice specific boundaries with minimal overhead:

# Print the first 25 lines of a configuration file
head -n 25 /etc/redis/redis.conf

# Print the last 40 lines of an audit log
tail -n 40 /var/log/audit/audit.log

A common operational trap occurs when tracking active server logs using tail -f. Standard tail -f hooks into the underlying file descriptor of the target file. When background services like logrotate compress and rotate logs, the active file is renamed or moved, and a brand-new file is created with the original filename.

Because standard tail -f follows the inode rather than the filename, it remains attached to the old, unlinked file descriptor and stops displaying new lines.

To avoid missing critical production events, use -F (capitalized):

tail -F /var/log/nginx/access.log

The -F flag instructs tail to track the file by its filename rather than its file descriptor. If the underlying inode changes or the file is temporarily unavailable during log rotation, tail re-establishes its handle on the new file as soon as the service recreates it.

You can also use tail to slice from a specific line number to the end of a file using the plus notation:

# Display output starting from line 150 through the end of the file
tail -n +150 /var/log/messages

4. tac: Bottom-Up Reverse Reading for Fast Crash Diagnostics

Troubleshooting is chronologically inverted. When an alert fires or a container crashes, you want to inspect the failure event that occurred five seconds ago, not the initialization messages recorded when the system booted three weeks ago.

The standard core utility tac prints lines in reverse order, starting from the final line and working backward to the first.

tac /var/log/dpkg.log | head -n 30

This command surfaces the thirty most recent package installations or upgrades without piping through sort -r or forcing the shell to buffer thousands of lines.

When paired with sed, tac provides an efficient way to pull the most recent stack trace from an application log:

tac /var/log/app/error.log | sed '/TRACEBACK_START/q' | tac

The inner tac reverses the log so the latest crash appears first. The sed statement reads until it matches the start marker of the traceback, prints it, and quits immediately. The second tac restores normal chronological order. You get the exact crash context without reading hundreds of megabytes of older events.

5. zless, zcat, and zstdcat: In-Place Compressed Log Inspection

On production machines, log rotation policies quickly compress historical logs using gzip (.gz), xz (.xz), or zstandard (.zst) to save disk space.

Extracting these files to disk with gunzip or unxz simply to check an error message consumes disk storage, adds write cycles to your drive, and creates leftover files that require cleanup.

Linux provides compression-aware utilities that decompress data directly into memory streams:

# Read a gzipped log file using full less keybindings
zless /var/log/nginx/access.log.2.gz

# Stream an xz-compressed archive directly into grep
xzcat /var/log/syslog.3.xz | grep -i "oom-killer"

# Inspect modern zstandard compressed logs
zstdcat /var/log/journal.zst | head -n 25

zless works with all standard less navigation commands, including forward search (/), backward search (?), and horizontal scrolling (-S). You inspect multi-gigabyte compressed files without writing a single uncompressed byte to disk.

6. view: Read-Only Structural Navigation Without Edit Accidents

Opening production configuration files inside an active text editor like vim or nano is dangerous. An accidental key press, stray paste buffer, or an unthinking write command (:wq) can modify critical runtime configurations without leaving a clear trace in your command history.

Using view (which opens Vim in read-only mode, equivalent to vim -R) gives you complete structural navigation tools while protecting the file from accidental writes:

view /etc/kubernetes/manifests/kube-apiserver.yaml

Inside view, the buffer modification flag is disabled. You get the full power of Vim syntax parsing and movement commands:

Press % while your cursor is over an opening parenthesis, bracket, or curly brace to jump directly to its matching closing counterpart.
Press * to find the next occurrence of the word currently under your cursor.
Use zo and zc to open and close code folds, collapsing long YAML blocks or nested structures.
Type :set number to turn on contextual line numbering.

If you accidentally strike an edit key, the editor displays an explicit warning and prevents saving changes to disk.

7. nl: Structured Line Numbering with Logical Page Boundaries

When people want line numbers, they often run cat -n. While cat -n works, it numbers every single line indiscriminately, including empty whitespace, trailing breaks, and formatting separators.

The dedicated nl (number lines) utility gives you fine-grained control over how lines are counted, formatted, and displayed:

nl /etc/pam.d/common-auth

By default, nl only assigns numbers to non-empty lines, keeping the output clean and visually tied to functional configuration directives.

You can modify its numbering logic to suit specific file structures:

# Number all lines including blank ones, right-aligned with 4 digits and zero padding
nl -b a -n rz -w 4 /etc/hosts

# Number only lines matching a specific regex pattern
nl -b p'^[[:space:]]*server' /etc/nginx/nginx.conf

nl also understands logical page delimiters (\:\:\:, \:\:, and \: for headers, body, and footers), allowing you to restart line numbering automatically across distinct sections of generated source files or technical reports.

8. column -t: Formatting Jagged Delimited Files into Clean Tables

Reading delimited text files such as /etc/passwd, /etc/fstab, or CSV exports with cat produces jagged, misaligned output because fields have different lengths.

The column utility parses delimited input and aligns it into neat, human-readable vertical tables:

# Format /etc/passwd using the colon delimiter
column -t -s ":" /etc/passwd | head -n 15

The -t flag instructs column to determine the number of columns dynamically, and -s sets the delimiter character.

For whitespace-delimited files like storage mount configurations:

grep -v '^#' /etc/fstab | column -t

Instead of struggling to see where mount points end and filesystem options begin, every field aligns neatly under its respective column.

9. od, hexdump, and xxd: Exposing Invisible Bytes and Encoding Traps

A frustrating shell issue happens when a bash script exits with cryptic syntax errors, yet opening the script with cat shows clean, correct code.

cat does not reveal invisible characters, including:

Windows carriage returns (\r\n or CRLF) inserted by editors on non-Linux systems.
Zero-width non-breaking spaces copied from online documentation.
UTF-8 Byte Order Marks (BOM) added at the beginning of text files.
Embedded null bytes or escape codes from corrupted data transfers.

When text files behave unexpectedly, inspect their underlying bytes directly:

# Inspect the first 128 bytes with hex and ASCII side by side using xxd
xxd -l 128 /etc/environment

# Use hexdump in canonical format to check character offsets
hexdump -C /usr/local/bin/deploy.sh | head -n 10

# Inspect character representations with od
od -c /usr/local/bin/entrypoint.sh | head -n 15

Running od -c exposes line endings immediately. A proper Linux newline appears as \n. A file containing Windows line endings displays explicit \r \n character pairs. Catching this with od in five seconds saves hours of debugging broken shell loops.

Quick Reference: Choosing the Right Tool

Syntax highlighting & code files:
Use bat. It provides language-aware colorization, git status in the left gutter, and automatic pager fallback when output overflows your screen.

Multi-gigabyte logs & large dumps:
Use less -FXRS. It memory-maps files with zero RAM overhead, provides forward and backward regex search, and avoids terminal lockups.

Live log monitoring:
Use tail -F. It tracks files across rotation and inode updates rather than losing track of the file descriptor.

Recent errors & stack traces:
Use tac. It traverses files bottom-up so recent events appear first without needing sorting pipelines.

Compressed archives (.gz, .xz, .zst):
Use zless, xzcat, or zstdcat. They stream and inspect compressed files directly in RAM without extracting data to disk.

Safe configuration inspection:
Use view. It provides full Vim movement and syntax folding while preventing accidental modifications.

Readable line numbers:
Use nl. It numbers non-blank lines by default and supports custom regex patterns and padding styles.

Jagged delimited files (CSV, /etc/passwd):
Use column -t -s ":". It aligns delimiter-separated text into structured vertical columns.

Invisible characters & encoding errors:
Use od -c or hexdump -C. It reveals carriage returns, zero-width spaces, and binary corruption instantly.

An Interesting Fact About cat

The cat command was written in 1971 by Ken Thompson and Dennis Ritchie for the First Edition of Unix. Its sole intended purpose was concatenating files: reading contents from multiple sources and combining them into a single stream.

By 1995, engineers overusing cat as a file viewer had become so widespread that the Usenet community created the Useless Use of cat Award (UUOC). The award called out shell pipelines that ran cat file | command instead of using direct shell redirection (command < file) or passing the file as an argument (command file), which eliminates an unnecessary subshell, process fork, and context switch.

Practical Takeaways for Your Daily Workflow

Habits are hard to change, but defaulting to cat for file inspection creates unnecessary friction and risks shell lockups on busy production nodes.

A simple step is adding an alias to your interactive shell profile:

alias cat="bat 2>/dev/null || less"

Save cat for its original purpose: gluing multiple files together into a pipe (cat chunk.* > full_archive.tar). For reviewing configurations, debugging errors, and inspecting logs, pick the tool designed for that specific data format.

What command or alias do you reach for when a multi-gigabyte log file is filling up disk space on a production server?

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

How I Troubleshoot a Linux Server That Suddenly Loses Network Access

Asep Sayyad — Thu, 29 Oct 2026 07:30:54 +0000

A systematic Layer 1 to Layer 4 diagnostic guide for recovering isolated production machines through an out-of-band console without rebooting blindly.

An SSH session freezes in the middle of a command. Seconds later, your monitoring dashboard lights up with host-unreachable alerts. Pings to the server return 100 percent packet loss. When a production Linux machine drops off the network without warning, the worst thing you can do is pull the virtual power cord or trigger a hard reboot from your hypervisor console.

Rebooting clears the kernel ring buffer. It flushes the ARP cache, wipes interface error counters, resets firewall states, and discards volatile routing policy entries. If the problem stemmed from a corrupted state, a race condition during a dynamic lease renew, or a driver watchdog timeout, rebooting guarantees you will learn nothing. You will wait nervously for the exact same failure to strike again during peak hours.

Unless your physical hardware completely lost power or the motherboard failed, a sudden network drop is almost always traceable to a specific breakdown point. Link carrier dropped, a network driver stalled, a default route vanished, ARP resolution failed, or Netfilter silently discarded valid frames.

When network connectivity breaks, your only lifeline is out-of-band access: an IPMI serial-over-LAN session, iDRAC, AWS Serial Console, or a hypervisor VNC console. Once logged into the raw console, you can work upward through the network stack in a disciplined sequence.

Here is the exact diagnostic sequence to run when a server goes dark.

1. The Physical and Driver Link Layer

You must start by verifying whether the operating system still sees an active network link. If the operating system cannot confirm physical or virtual carrier signals, testing higher layers like DNS or routing is a waste of time.

Run ip -d link show to inspect the detailed state of your network interfaces:

ip -d link show eth0

Look closely at two specific indicators in the output:

2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9000

Notice the flags inside the angle brackets and the state field. If you see state DOWN, the interface administrative state was toggled off. A process, a configuration reload script, or a network daemon intentionally brought the link down.

If the output lists NO-CARRIER, the interface is administratively configured to be up, but the hardware sees no electrical or optical signal from the upstream switch port.

Next, query the link state directly from the network interface controller using ethtool:

ethtool eth0

Inspect the link status, speed negotiation, and duplex settings:

Settings for eth0:
    Supported ports: [ TP ]
    Speed: Unknown!
    Duplex: Unknown! (Full)
    Auto-negotiation: on
    Link detected: no

When Link detected: no appears on bare-metal servers, verify whether the physical optical transceiver is healthy by reading its internal diagnostic telemetry:

ethtool -m eth0

This command queries the SFP module. It shows laser output power, receiver optical power, temperature, and operating voltage. If the receiver optical power is reading -40 dBm, the fiber strand is broken or the upstream switch port is shut down.

If Link detected: yes is present, check whether the network interface controller is dropping packets at the hardware ring level before the kernel can even process them:

ethtool -S eth0 | grep -E "drop|err|miss|fifo"

Look for counters like rx_fifo_errors, rx_missed_errors, or rx_dropped. A sudden surge in these counters means incoming frames are hitting the physical card, but the card cannot allocate DMA ring buffers fast enough to deliver them to the operating system.

2. Kernel Hardware Events and Driver Watchdogs

If physical link carrier is active but network traffic stopped completely, check the kernel log. When network drivers hang or fail to transmit packets, the Linux kernel logs specific watchdog notifications.

Run dmesg with human-readable timestamps and filter for interface names and network subsystem events:

dmesg -T | grep -E -i "eth|e1000e|ixgbe|bnxt|mlx|link|down|reset|tx timeout" | tail -n 30

You are hunting for three critical failure signatures:

First, look for Transmit Queue Timeouts. The message will look similar to this:

[Thu Oct 29 11:14:02 2026] NETDEV WATCHDOG: eth0 (ixgbe): transmit queue 2 timed out
[Thu Oct 29 11:14:02 2026] ixgbe 0000:03:00.0 eth0: TXDCTL.ENABLE not cleared within the polling period

When a network driver queues packets into a hardware ring descriptor for transmission, it expects the network card to issue an interrupt indicating completion. If the hardware controller freezes or PCIe bus stalls occur, packets sit in the ring buffer indefinitely. The Linux kernel network watchdog detects that the queue has been blocked for several seconds and prints a NETDEV WATCHDOG warning.

Second, check for PCIe Aer errors or device resets. When high-bandwidth interfaces suffer uncorrectable bus errors, the kernel attempts an automatic PCI device reset. During the reset cycle, the interface drops off the bus for several seconds, terminating all active TCP sessions.

Third, look for link cycling loops:

[Thu Oct 29 11:15:20 2026] eth0: Link is Down
[Thu Oct 29 11:15:24 2026] eth0: Link is Up - 10Gbps/Full - flow control rx/tx
[Thu Oct 29 11:15:32 2026] eth0: Link is Down

Rapid link flaps point to autonegotiation mismatches, faulty patch cables, or an upstream switch port bouncing because of spanning tree topology change notifications.

If a transmit queue timeout froze your interface, you can attempt to reset the driver ring buffer directly from the console:

ip link set dev eth0 down
ip link set dev eth0 up

If the interface recovers after bouncing it down and up, the root cause is driver or offload related. Certain network offloads, specifically TCP Segmentation Offload (TSO) and Generic Segmentation Offload (GSO), can trigger driver ring buffer corruption on older or buggy firmware revisions. You can inspect active offloads:

ethtool -k eth0 | grep -E "segmentation|checksum"

To prevent recurring driver hangs while troubleshooting, temporarily disable segmentation offloads:

ethtool -K eth0 tso off gso off

3. Addressing and Routing: The Silent Disappearance of the Gateway

Once you confirm the link is up and the driver is passing frames, move up to Layer 3.

Check if the interface still owns its expected IP address:

ip -br addr show

The output gives you a quick snapshot:

lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0             UP             192.168.1.50/24 fe80::5054:ff:fe12:3456/64

If eth0 has no IP address assigned, your network management daemon either died, crashed during a configuration reload, or lost its DHCP lease.

Check the status of the managing daemon:

# If using systemd-networkd:
systemctl status systemd-networkd

# If using NetworkManager:
systemctl status NetworkManager

In environments using DHCP, a lease renewal failure will quietly unbind the IP address after the lease expiration time elapses. The server remains powered on and running, but it has no Layer 3 presence.

If the IP address is intact, inspect the routing table:

ip route show

A properly configured server will have an unambiguous default route pointing to your local subnet gateway:

default via 192.168.1.1 dev eth0 proto static metric 100 
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.50 metric 100

One of the most frequent causes of sudden total network loss is the unexpected disappearance of the default route. When the default route vanishes, inbound traffic from external subnets reaches the server interface, but the Linux kernel drops all outbound replies instantly. The kernel cannot decide where to send response packets, resulting in Network is unreachable errors.

How does a default route vanish on an active server?

A second network interface came up, such as a temporary VPN tunnel, a Docker bridge, or a newly configured secondary VLAN. If that secondary interface acquired a default route with a lower metric, all external outbound packets are redirected into a dead path.
A dynamic network configuration tool reloaded and flushed routes because of an invalid DNS or search domain parameter in a netplan or ifcfg file.
An automation tool ran ip route flush as part of an incomplete configuration deployment.

If the default route is missing, restore it manually to verify reachability:

ip route add default via 192.168.1.1 dev eth0

Also check routing policy database rules. Linux supports multiple routing tables. If custom routing rules were introduced, incoming packets might be evaluated against an alternate table that lacks valid egress routes:

ip rule show

Normal outputs show rules pointing to the main routing table:

0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

If you see rules directing specific subnets to non-existent or unpopulated custom tables, traffic matching those rules will drop immediately.

4. The Layer 2 Neighbor and ARP Cache

Your server has an active link. The IP address is bound. The default route points to 192.168.1.1. Yet, when you attempt to ping the gateway from the console, every packet fails:

ping -c 3 192.168.1.1

PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 192.168.1.50 icmp_seq=1 Destination Host Unreachable

When Destination Host Unreachable appears, the Linux kernel is telling you that it cannot find the physical hardware MAC address for that IP address on the local network segment. Layer 3 routing is functional, but Layer 2 address resolution has failed.

Inspect your neighbor table:

ip neighbor show

Review the state listed for your default gateway:

192.168.1.1 dev eth0 lladdr 00:1a:2b:3c:4d:5e REACHABLE

The status column reveals the exact state of ARP resolution:

REACHABLE means the kernel knows the hardware address and recently verified it.
STALE means the cached entry is valid, but the kernel will re-verify it on the next outbound packet.
INCOMPLETE means the kernel sent broadcast ARP queries requesting the MAC address of 192.168.1.1, but has not received a response.
FAILED means multiple ARP broadcast requests timed out without a single answer from the network.

If the gateway entry is in a FAILED or INCOMPLETE state, bypass the higher networking layers and send a direct raw ARP query using arping:

arping -I eth0 -c 4 192.168.1.1

If arping receives replies from the gateway MAC address while ping and ip neighbor continue to fail, check your kernel ARP sysctl parameters:

sysctl -a | grep -E "arp_ignore|arp_announce|arp_filter"

A misconfigured net.ipv4.conf.all.arp_ignore setting causes the server to ignore ARP requests received on an interface if the target IP is not configured on that specific physical adapter. This setting is common in multi-interface setups or servers hosting virtual IP addresses for load balancing.

If an upstream network switch had its MAC address forwarding table cleared, or if another host accidentally assumed the gateway IP, the local ARP cache may hold a poisoned entry. Flush the neighbor table for that interface to force fresh discovery:

ip neigh flush dev eth0

After flushing, send a single ping to the gateway and check ip neighbor show again to see if a valid MAC address resolves.

5. Netfilter, nftables, and Connection Tracking Drops

When Layer 2 and Layer 3 appear fully intact, the next suspect is the Linux packet filtering subsystem. Firewalls rarely break on their own, but automated host security tools, fail2ban daemons, container runtimes, and configuration updates can insert restrictive rules that sever communication without warning.

First, check the status of nftables or legacy iptables depending on your Linux distribution.

To inspect active nftables rulesets:

nft list ruleset

To inspect iptables rules with line numbers and packet counters:

iptables -L -n -v --line-numbers
iptables -t nat -L -n -v --line-numbers

Pay special attention to two details in the iptables output:

First, check the default policy of the INPUT and OUTPUT chains. If the default policy is set to DROP, verify whether an explicit rule exists permitting packets with the ESTABLISHED,RELATED state:

Chain INPUT (policy DROP 142 packets, 11840 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1      450 38000 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0

If the rule permitting established connections was inadvertently removed during an automated update, existing SSH sessions drop immediately, and no new connections can be formed.

Second, check packet and byte counters. If you see a rule targeting DROP or REJECT with rapidly incrementing packet numbers, that specific rule is intercepting and discarding your traffic.

If you suspect firewall rules locked you out, you can set the chain policies back to ACCEPT temporarily from your console to test connectivity:

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

The Connection Tracking Table Full Bug

Another common silent network killer on high-traffic Linux systems is connection tracking table exhaustion. The Netfilter connection tracking module (conntrack) records state for every active network flow. If this table fills up, the Linux kernel begins dropping every new incoming and outgoing packet.

Check whether conntrack is currently dropping packets:

dmesg -T | grep -i "nf_conntrack: table full, dropping packet"

You can check current table usage versus maximum capacity directly:

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

If nf_conntrack_count equals or approaches nf_conntrack_max, the server will refuse all new TCP handshakes and UDP connections. You can increase the limit immediately without rebooting:

sysctl -w net.netfilter.nf_conntrack_max=262144

Reverse Path Filtering (rp_filter)

If the server has multiple network interfaces, Docker bridge networks, or active WireGuard or OpenVPN tunnels, check reverse path filtering.

Reverse path filtering is a kernel security mechanism designed to prevent IP spoofing. When a packet enters an interface, the kernel checks its routing table to see if the return path for the packet source address would go back out of that exact same interface. If the return path uses a different route or interface, the kernel drops the packet silently.

Query the current rp_filter settings:

sysctl -a | grep '\.rp_filter'

Values can be:

0: No reverse path validation.
1: Strict reverse path validation defined in RFC 3704. Each incoming packet is tested against the local routing table. If the interface is not the best reverse path, the packet is discarded.
2: Loose reverse path validation. The incoming packet is accepted as long as the source address is reachable through any interface on the system.

If asymmetric routing exists on your network and rp_filter is set to 1, inbound traffic will hit the network card, but the kernel will drop it before applications or sockets ever see it. Change the parameter to loose mode to verify:

sysctl -w net.ipv4.conf.all.rp_filter=2
sysctl -w net.ipv4.conf.eth0.rp_filter=2

6. Live Packet Tracing with tcpdump

When interface flags look healthy, routing tables contain the right gateways, and firewall counters are quiet, you need direct visibility into wire activity. tcpdump lets you watch packets enter and leave the network adapter in real time.

From your out-of-band console, start capturing ICMP packets on your primary interface:

tcpdump -nn -v -i eth0 icmp

Now, initiate a continuous ping toward the server from an external machine or your local workstation. Observe the console output.

You will encounter one of three distinct diagnostic scenarios:

Scenario A: Inbound packets arrive, but no replies leave.

11:22:15.102341 IP 10.0.0.15 > 192.168.1.50: ICMP echo request, id 412, seq 1, length 64
11:22:16.104522 IP 10.0.0.15 > 192.168.1.50: ICMP echo request, id 412, seq 2, length 64

You see the incoming echo request, but no corresponding echo reply is transmitted by eth0.
This proves that physical cabling, upstream switching, and NIC hardware are working. The problem is strictly inside the server. The packet was dropped by an iptables rule, rejected by rp_filter, or the kernel attempted to send the reply through a different interface because of a routing mismatch.

Scenario B: Outbound packets leave, but no incoming responses return.
Attempt to ping your default gateway from the console while watching tcpdump:

11:23:01.401120 IP 192.168.1.50 > 192.168.1.1: ICMP echo request, id 88, seq 1, length 64
11:23:02.403210 IP 192.168.1.50 > 192.168.1.1: ICMP echo request, id 88, seq 2, length 64

The server sends the request, but the wire stays completely silent. The gateway is not responding. The fault lies outside the host: a misconfigured switch port VLAN, an upstream router failure, or an IP conflict on the subnet.

Scenario C: Complete silence on the wire.
No packets appear on the screen, even while external systems are actively attempting to connect.
The traffic is being stopped before reaching the Linux operating system. On virtual machines, this indicates that the hypervisor virtual switch disconnected the vNIC, the host security group blocked traffic, or the cloud provider route table was altered.

7. Name Resolution and Local DNS Stubs

Sometimes a server appears to have lost all external network access, but direct IP communication works without issue. The server can ping public IP addresses like 1.1.1.1, but any command relying on hostnames—such as curl, apt, or package updates—fails with errors like Could not resolve host or Temporary failure in name resolution.

Verify basic IP transport first:

ping -c 2 1.1.1.1

If pinging a numerical IP succeeds, the network stack is functional. The breakdown is confined to DNS resolution.

Check the resolver configuration file:

cat /etc/resolv.conf

On modern systemd-based Linux systems, /etc/resolv.conf is typically a symbolic link pointing to the systemd-resolved stub resolver:

nameserver 127.0.0.53
options edns0 trust-ad

If systemd-resolved stopped running or crashed, local applications querying 127.0.0.53 receive no response. Check the resolver daemon status:

systemctl status systemd-resolved

Inspect active upstream DNS servers configured in the resolver:

resolvectl status

If the upstream nameservers point to unreachable addresses or invalid local forwarders, replace them temporarily in /etc/resolv.conf to test:

nameserver 1.1.1.1
nameserver 8.8.8.8

Test resolution again using getent:

getent hosts google.com

Interesting Fact

In the Linux kernel network subsystem, the NETDEV WATCHDOG timer does not run on a static timer. Instead, each network device driver defines a watchdog_timeo variable, typically set between 2 to 5 seconds. Whenever the driver queues a packet, it updates a trans_start timestamp on that transmission queue. During every kernel timer tick, dev_watchdog() scans active queues. If jiffies - trans_start exceeds the timeout while the hardware queue remains marked as stopped, the kernel declares a driver lockup and issues an interrupt to force a controller reset.

Summary of the Triage Ladder

When network connectivity drops, keep this structured diagnostic order in mind:

First, check the physical and driver link status with ip -d link show and ethtool. Confirm LOWER_UP and verify that ring buffer drop counters are zero.

Second, check dmesg -T for transmit queue timeouts, firmware crashes, or PCIe resets.

Third, verify your IP configuration with ip -br addr show and confirm that an unambiguous default route exists with ip route show.

Fourth, check your Layer 2 ARP cache with ip neighbor show. If entries are marked FAILED, use arping to test broadcast discovery.

Fifth, review firewall policies and connection tracking limits with nft list ruleset, iptables -L -n -v, and sysctl net.netfilter.nf_conntrack_count.

Sixth, trace raw wire traffic using tcpdump -nn -i eth0 to verify whether packets arrive and whether outbound replies leave the interface.

Working methodically from Layer 1 through Layer 4 lets you identify the exact breaking point without restarting the server or destroying forensic clues.

What was the most elusive network drop you ever diagnosed on a production Linux machine?

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

What Really Happens When You Run curl google.com on Linux?

Asep Sayyad — Sun, 25 Oct 2026 05:25:09 +0000

A step-by-step breakdown of shell parsing, ELF linking, glibc resolver queries, non-blocking TCP handshakes, socket buffers, and the infamous 301 redirect.

You type curl google.com in your terminal and hit enter. Within forty milliseconds, a short snippet of HTML lands on your screen:

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

Most engineers glance at that output and assume curl made a quick HTTP call and printed the response.

That explanation skips almost everything interesting.

Between the moment your finger depresses the enter key and the moment your prompt returns, your machine moves through dozens of discrete subsystems. Bash parses raw keystrokes from a pseudo-terminal. The kernel loads an ELF binary and resolves dynamic libraries through memory maps. The GNU C library queries the Name Service Switch to find out how your machine translates hostnames. A non-blocking TCP socket initiates a three-way handshake across network queues and physical network interfaces.

Then Google rejects your request, sends back a redirect, and curl tears down the socket before you can blink.

Here is the entire mechanical chain, traced down through the system calls, kernel structures, and network packets.

1. The Terminal, the Shell, and the Execve Boundary

Everything begins before curl even exists in memory.

When you type on your keyboard, your terminal emulator (like Alacritty, GNOME Terminal, or Windows Terminal over SSH) sends scan codes across a pseudo-terminal pair (/dev/pts/X). The shell process (usually Bash) sits blocked inside a read() system call on file descriptor 0.

# You can see bash waiting for your input:
read(0, "c", 1)                         = 1
read(0, "u", 1)                         = 1
read(0, "r", 1)                         = 1
read(0, "l", 1)                         = 1
read(0, " ", 1)                         = 1
read(0, "g", 1)                         = 1
...
read(0, "\n", 1)                        = 1

Once the newline byte \n arrives, Bash wakes up. It reads the string into its internal line buffer and begins tokenization.

First, Bash checks for aliases. If you aliased curl to curl -i or curl -s in your .bashrc, Bash substitutes the command string immediately. Next, it looks for shell functions, built-ins, and reserved keywords. None match.

Bash now knows it needs to execute an external binary. To find where that binary lives on disk, it checks its internal command hash table. If you have run curl recently, Bash remembers the exact path /usr/bin/curl. If not, it iterates through the colon-separated directories in your $PATH environment variable:

stat("/usr/local/sbin/curl", ...) = -1 ENOENT (No such file or directory)
stat("/usr/local/bin/curl", ...)  = -1 ENOENT (No such file or directory)
stat("/usr/sbin/curl", ...)       = -1 ENOENT (No such file or directory)
stat("/usr/bin/curl", {st_mode=S_IFREG|0755, st_size=281808, ...}) = 0

Bash finds the executable at /usr/bin/curl.

Now comes the process fork. Bash cannot replace its own running image because it must survive to draw the next prompt. It calls clone() (or fork() on older kernels) with flags like SIGCHLD to spawn an identical child process.

The child process inherits file descriptors 0 (stdin), 1 (stdout), and 2 (stderr), all pointing back to your pseudo-terminal slave. Immediately after cloning, the child calls execve():

execve("/usr/bin/curl", ["curl", "google.com"], [/* 48 vars */]) = 0

The shell image in the child process is instantly overwritten. Bash in the parent process calls wait4() and puts itself to sleep, waiting for the child to finish.

2. The ELF Loader and Dynamic Shared Libraries

The Linux kernel takes control of the new process. It opens /usr/bin/curl and inspects the first four bytes of the file. They must match the standard ELF magic number: 0x7f 0x45 0x4c 0x46 (\x7fELF).

Because curl is compiled as a dynamically linked binary, the kernel does not jump directly into curl's main() function. Instead, it reads the PT_INTERP program header from the ELF file. This header points to the dynamic linker:

/lib64/ld-linux-x86-64.so.2

The kernel sets up the process stack with arguments (argv), environment variables (envp), and auxiliary vectors (which provide details like hardware capabilities and random seeds for ASLR). Then it maps the dynamic linker into memory and transfers control to it.

The dynamic linker inspects the DT_NEEDED tags in curl's ELF binary to find every shared library curl relies on. On a typical modern Linux distribution, curl depends on several heavy shared objects:

ldd /usr/bin/curl

The output reveals the dependency graph:

libcurl.so.4 (the core transfer engine)
libssl.so.3 and libcrypto.so.3 (OpenSSL cryptographic routines)
libz.so.1 (compression support)
libc.so.6 (standard C runtime)

The dynamic linker maps each of these shared libraries into the process's virtual memory address space using mmap() system calls. It applies relocations, binds symbol addresses, sets memory page protections (marking code pages as read-only and executable), and activates security protections like Full RELRO and stack canaries.

Once dynamic linking finishes, control finally jumps to _start inside curl, which initializes the C runtime and calls main().

3. Argument Parsing and the Easy Handle

Curl starts by parsing its command-line arguments using getopt or internal string parsers.

The argument passed was google.com.

Notice what is missing: there is no URL scheme. You did not specify http:// or https://.

What does curl do? By default, curl inspects the string. It checks if the string starts with a recognized scheme followed by ://. Finding none, curl prepends http:// as the default protocol (though newer versions allow configuring --proto-default).

Because the default protocol is HTTP, curl sets the target destination port to 80.

Curl now allocates its central operational structure: the Curl_easy handle. This structure holds state machines, timeout values, buffer pointers, error buffers, and protocol handlers.

Curl also checks whether its standard output is connected to a terminal by calling isatty(1). If stdout is a terminal, curl configures itself to show transfer progress meters if requested, but hides them for quick responses to prevent cluttering your prompt.

No network packets have moved yet. Everything so far has been local CPU cycles and memory mapping.

4. The Domain Name Resolution (DNS) Pipeline

Curl has a hostname (google.com), but network routers only understand IP addresses. Curl must translate google.com into an IPv4 or IPv6 address.

Curl does not contain hardcoded DNS packet builders. Instead, it relies on the C library by calling the standard POSIX resolver function:

getaddrinfo("google.com", "http", &hints, &res)

This single call triggers an extensive internal investigation inside the Linux operating system.

Step 4A: The Name Service Switch (`/etc/nsswitch.conf`)

The C library (glibc) opens /etc/nsswitch.conf to check how this specific machine is configured to resolve hosts. It looks for the hosts: line:

hosts: files mdns4_minimal [NOTFOUND=return] dns

This configuration tells glibc the order of operations:

Check local static files first (files).
If not found, attempt multicast DNS (mdns4_minimal).
If not found, query standard DNS servers (dns).

Step 4B: Checking Static Files (`/etc/hosts`)

glibc opens /etc/hosts and scans it line by line using openat() and read(). It checks if someone mapped google.com to a static IP address like 127.0.0.1.

Unless you are actively intercepting traffic for local development, google.com is not in /etc/hosts. glibc closes the file and moves to the next provider.

Step 4C: Reading the Resolv Configuration (`/etc/resolv.conf`)

Now glibc needs to know which DNS server to contact. It opens /etc/resolv.conf. On modern systemd-based Linux installations (Ubuntu, Fedora, Debian), you will usually see:

nameserver 127.0.0.53
options edns0 trust-ad

127.0.0.53 is not an external server. It is a local loopback IP address managed by systemd-resolved, a local caching and DNS multiplexing daemon.

Step 4D: The DNS Query on the Wire

glibc creates a UDP socket using socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC, 0) and calls connect() to attach the socket to 127.0.0.53:53.

It constructs a DNS query packet asking for two records:

An A record (for IPv4)
An AAAA record (for IPv6)

It transmits the packet using sendto():

sendto(sockfd, "\x12\x34\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x06google\x03com\x00\x00\x01\x00\x01", 28, 0, ...)

The systemd-resolved service catches this request. If it already has the answer in its local in-memory cache, it replies in less than one millisecond.

If the entry is not cached, systemd-resolved looks at its upstream network interfaces (configured via DHCP or static cloud configurations like AWS Route 53 or Google Cloud DNS at 169.254.169.254 or public DNS like 8.8.8.8). It sends an outbound UDP DNS request across your primary network interface, awaits the upstream answer, caches the result, and writes the response back down to the loopback socket where glibc is waiting in poll().

glibc reads the response using recvfrom(). It parses the DNS answers and hands curl a linked list of addrinfo structures containing resolved IP addresses:

IPv4: 142.250.190.46
IPv6: 2607:f8b0:4009:81b::200e

Step 4E: The "Happy Eyeballs" Race (RFC 8305)

Modern versions of curl implement Happy Eyeballs.

If both IPv6 and IPv4 addresses are returned, curl does not just pick one and freeze if your local IPv6 routing is broken. It attempts to connect to the IPv6 address first. If the IPv6 connection does not complete within approximately 250 milliseconds, curl simultaneously launches a connection attempt to the IPv4 address. Whichever connection establishes first wins, and curl cancels the slower attempt.

Let us trace the common IPv4 connection to 142.250.190.46:80.

5. Routing, ARP, and Layer 2 Discovery

Curl now has an IP address (142.250.190.46) and a destination port (80). But the Linux kernel cannot just spray packets into thin air. It needs to know which physical interface to use and which local hardware address to target.

The kernel consults its Forwarding Information Base (FIB), commonly known as the routing table:

ip route get 142.250.190.46

The kernel's routing subsystem evaluates the destination IP against its routes:

142.250.190.46 via 192.168.1.1 dev eth0 src 192.168.1.50 uid 1000
    cache

This kernel lookup tells the networking stack three crucial things:

The packet must exit via interface eth0.
The local source IP address to stamp into the IP header is 192.168.1.50.
The destination is not on the local subnet, so the packet must be sent to the default gateway at 192.168.1.1.

Resolving the Gateway MAC via ARP

Ethernet switches do not understand IP addresses. They only route Ethernet frames based on 48-bit MAC addresses.

The kernel checks its internal ARP neighbor cache (ip neigh). It looks for an entry for 192.168.1.1.

If the entry is present (REACHABLE), the kernel grabs the gateway MAC address (for example, 00:1a:2b:3c:4d:5e).

If the entry has expired or is missing, the kernel pauses transmission and broadcasts an ARP request frame to the entire local broadcast domain:

ARP, Request who-has 192.168.1.1 tell 192.168.1.50, length 28

Your local home router or top-of-rack data center switch hears the broadcast and replies directly with an ARP reply:

ARP, Reply 192.168.1.1 is-at 00:1a:2b:3c:4d:5e, length 28

The Linux kernel saves this mapping in its ARP table. Now it has everything required to build a physical Ethernet frame.

6. The Non-Blocking TCP Three-Way Handshake

Curl creates the actual communication socket:

int fd = socket(AF_INET, SOCK_STREAM|SOCK_NONBLOCK|SOCK_CLOEXEC, IPPROTO_TCP);

Notice the SOCK_NONBLOCK flag. Curl never uses blocking sockets. If a network call blocks, the entire process freezes. By using non-blocking I/O, curl keeps total control over timeouts and event loops.

Curl calls connect():

connect(fd, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("142.250.190.46")}, 16) = -1 EINPROGRESS

Because the socket is non-blocking, connect() cannot finish immediately. It returns -1 with an errno of EINPROGRESS. This is completely normal in network programming. It simply means: "The handshake has started in the background; check back in a moment."

Inside the Linux kernel, the TCP engine springs into action.

The SYN Packet

The kernel allocates a socket buffer structure (struct sk_buff, commonly called an skb). It builds the TCP packet:

Source port: An ephemeral port chosen randomly from /proc/sys/net/ipv4/ip_local_port_range (e.g., 48212).
Destination port: 80.
Sequence number: A cryptographically randomized 32-bit initial sequence number (ISN).
Flags: SYN flag set.
TCP Options: Maximum Segment Size (MSS), Selective Acknowledgments (SACK Permitted), Window Scale factor, and TCP Timestamps.

The kernel wraps the TCP segment inside an IPv4 packet, sets the TTL to 64, and places it inside an Ethernet frame stamped with the destination MAC of your gateway.

The frame is pushed into the network interface transmission queue (qdisc, like fq_codel). The network driver writes the descriptor to the physical network card's TX ring buffer using Direct Memory Access (DMA). The network card transmits the electrical or optical signal across the wire.

The socket transitions to the SYN_SENT state.

The SYN-ACK Arrival

Across the internet, routers forward the packet to the nearest Google edge POP (Point of Presence). Google's edge load balancer (the Google Front End, or GFE) accepts the SYN and replies with a SYN-ACK packet.

When the electrical signals return to your local network card:

The NIC copies the incoming frame into a host RAM buffer via DMA.
The NIC fires a hardware interrupt (IRQ) to a CPU core.
The kernel interrupt handler schedules a SoftIRQ (software interrupt) via the NAPI (New API) polling subsystem.
The kernel polls the RX ring buffer, wraps the frame in an skb, and feeds it into netif_receive_skb().
The TCP/IP stack strips the Ethernet and IP headers, validates the TCP checksum, matches the packet against your socket's 4-tuple (source IP, source port, dest IP, dest port), and finds the socket waiting in SYN_SENT.
The kernel updates the socket state to ESTABLISHED.
The kernel automatically generates and sends the final ACK packet back to Google, completing the three-way handshake.

Waking Curl

While the kernel was handling packets, curl was waiting inside poll() or ppoll():

ppoll([{fd=3, events=POLLOUT|POLLWRNORM}], 1, {tv_sec=299, tv_nsec=999000000}, NULL, 8) = 1

The moment the socket transitions to ESTABLISHED, the kernel marks the socket descriptor as writable. poll() wakes up. Curl calls getsockopt(fd, SOL_SOCKET, SO_ERROR, ...) to confirm that the connection succeeded without errors.

The TCP tunnel is open.

7. Sending the HTTP Request

Now that the transport layer is established, curl moves to Layer 7 (the application layer).

Curl builds the HTTP request in an internal heap buffer. Because curl defaults to HTTP/1.1 for plain web requests, the buffer contains a simple ASCII string:

GET / HTTP/1.1\r\n
Host: google.com\r\n
User-Agent: curl/8.5.0\r\n
Accept: */*\r\n
\r\n

Notice the components of this request:

GET /: The requested path is root.
Host: google.com: Mandatory in HTTP/1.1 so multi-tenant reverse proxies know which virtual host you want.
User-Agent: curl/8.5.0: Curl identifies itself and its version.
Accept: */*: Curl informs the server that it will accept any MIME type in response.
An empty line (\r\n\r\n): Signals the end of the HTTP headers.

Curl writes this buffer into the socket using sendto():

sendto(3, "GET / HTTP/1.1\r\nHost: google.com\r\nUser-Agent: curl/8.5.0\r\nAccept: */*\r\n\r\n", 75, 0, NULL, 0) = 75

The sendto() syscall does not send the packet across the internet immediately.

Instead, it copies those 75 bytes from curl's user space memory into the socket's kernel write buffer (sk_write_queue). The kernel packetizes the data, applies TCP segmentation, calculates checksums (often offloaded to the physical NIC via TCP Segmentation Offload, or TSO), and flushes the packet to the transmission ring buffer.

Curl now transitions from writing to reading. It calls poll() again, this time watching for POLLIN (incoming data readable).

8. Google's Edge and the Infamous 301 Redirect

Your request reaches Google's edge infrastructure on port 80.

Google does not serve its search engine over unencrypted HTTP. Serving plain HTTP would expose queries, cookies, and user sessions to eavesdropping and ISP injection.

However, Google cannot simply drop port 80 traffic. If it did, users typing google.com into their browsers would experience connection timeouts. Port 80 exists exclusively to bounce clients over to secure transport.

Google's server software processes your request and crafts an HTTP response:

HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Sun, 25 Oct 2026 05:14:22 GMT
Expires: Tue, 24 Nov 2026 05:14:22 GMT
Cache-Control: public, max-age=2592000
Server: gfe
Content-Length: 219

Take a close look at that Location header. It says:

Location: http://www.google.com/

This surprises almost everyone. Why did Google redirect plain http://google.com to http://www.google.com/ instead of jumping straight to https://?

Google maintains strict canonical domain rules. The apex domain google.com first normalizes to the www. subdomain. If you follow that redirect, http://www.google.com/ will issue a second redirect to https://www.google.com/.

Google also appends a tiny fallback HTML payload to the response body:

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

Why include HTML inside a 301 response? For ancient HTTP/1.0 clients or web crawlers that do not automatically follow Location headers, this embedded link allows a human or robot to manually click through to the new address.

9. Receiving the Bytes and Writing to stdout

The packet containing Google's 301 response travels back through your ISP, hits your router, enters your network card, and passes through the kernel's NAPI receive pipeline.

The kernel places the payload into the socket's receive buffer and wakes up curl's waiting poll() call.

Curl calls recv():

recv(3, "HTTP/1.1 301 Moved Permanently\r\nLocation: http://www.google.com/\r\n...", 16384, 0) = 542

Curl reads 542 bytes into its memory buffer.

Curl's HTTP state machine parses the response line: HTTP/1.1 301 Moved Permanently. It parses the headers line by line until it hits the double CRLF (\r\n\r\n).

Here is where standard curl behavior surprises people: curl does not follow redirects by default.

In a web browser, a 301 status code triggers an immediate internal navigation to the URL in the Location header. But curl is a UNIX command-line tool. UNIX tools do not make assumptions about what you want. Unless you explicitly pass the -L or --location flag, curl treats a 301 response as a finished transfer.

By default, curl hides HTTP response headers and only outputs the payload body.

Curl finds the start of the HTML body in its buffer. It prepares to print it. It calls the standard write() system call on file descriptor 1 (stdout):

write(1, "<HTML><HEAD><meta http-equiv=\"co"..., 219) = 219

Because file descriptor 1 points to your pseudo-terminal slave device (/dev/pts/X), the Linux line discipline receives the bytes. The line discipline passes the raw ASCII text across the pseudo-terminal master descriptor to your terminal emulator process.

Your terminal emulator rasterizes the fonts, maps the glyphs against its font cache, submits render draw calls to your graphics server (via Wayland or X11), and the HTML snippet appears on your monitor.

10. Connection Teardown and Process Destruction

The transfer is complete. Curl begins its shutdown routine.

First, it closes the network connection:

close(3) = 0

Inside the kernel, closing the socket file descriptor decrements its reference count to zero. The kernel initiates a clean TCP four-way handshake. It sends a FIN packet to Google's server. Google replies with an ACK, sends its own FIN, and your kernel replies with a final ACK.

Your local TCP socket enters the TIME_WAIT state.

Why TIME_WAIT? The kernel keeps the socket parameters alive for twice the Maximum Segment Lifetime (2 * MSL, typically 60 seconds on Linux) to ensure any delayed or retransmitted packets wandering through the internet do not accidentally collide with a newly opened socket using the same port.

Next, curl frees its memory structures. It deallocates the Curl_easy handle, unloads its protocol state machines, and calls exit_group(0):

exit_group(0)

The kernel reclaims the process's page tables, unmaps the shared libraries, and frees virtual memory pages. The process descriptor enters the EXIT_ZOMBIE state.

The Linux kernel delivers a SIGCHLD signal to the parent process: Bash.

Bash wakes up from its wait4() call:

wait4(28419, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 28419

Bash collects curl's exit status (0, indicating success). The kernel cleans up the remaining zombie process entry from the process table.

Bash issues another read() call to wait for your next command, prints its prompt string (PS1), and sits waiting for your next keystroke.

The entire sequence took less than fifty milliseconds.

The Practical Diagnostic Arsenal: Tracing It Yourself

You do not have to take any of this on faith. You can observe every single layer of this process on any Linux server using three standard diagnostic tools.

1. Tracing the System Calls (`strace`)

To see the exact boundary where curl interacts with the Linux kernel, run curl through strace:

strace -f -e trace=process,network,openat,read,write curl -s google.com > /dev/null

You will see the exact sequence: execve, dynamic library mapping, opening /etc/resolv.conf, creating the UDP socket for DNS, the non-blocking connect() returning EINPROGRESS, poll() waiting for readiness, and the final write() to file descriptor 1.

2. Measuring Internal Phase Latency with Curl

Curl contains a built-in formatting engine that extracts microsecond-level timing for every internal phase. Save this snippet in your shell:

curl -w "\
DNS Lookup Time   : %{time_namelookup}s\n\
TCP Connect Time  : %{time_connect}s\n\
App Connect (TLS) : %{time_appconnect}s\n\
Pre-Transfer Time : %{time_pretransfer}s\n\
Start Transfer    : %{time_starttransfer}s\n\
Total Time        : %{time_total}s\n\
HTTP Status Code  : %{http_code}\n\
Redirect URL      : %{redirect_url}\n" \
-o /dev/null -s google.com

Running this command against google.com displays the exact latency profile of your request:

DNS Lookup Time   : 0.004120s
TCP Connect Time  : 0.018450s
App Connect (TLS) : 0.000000s
Pre-Transfer Time : 0.018510s
Start Transfer    : 0.034210s
Total Time        : 0.034320s
HTTP Status Code  : 301
Redirect URL      : http://www.google.com/

Notice that time_appconnect is exactly 0.000000s. Because the request was plain HTTP, zero TLS handshakes occurred.

3. Watching the Wire with `tcpdump`

Open a second terminal window and run tcpdump on your active interface:

sudo tcpdump -i any -nn "port 80 or port 53"

Then run curl google.com in your primary window. You will see the raw UDP packets heading to port 53 for DNS, followed by the TCP SYN, SYN-ACK, ACK handshake on port 80, the HTTP GET push, the 301 response, and the FIN teardown.

Exactly What Happens When You Pass `-L`

If you want the real Google search homepage, you must instruct curl to follow redirects using the -L (or --location) flag:

curl -L google.com

When you pass -L, the internal process repeats twice more:

First, curl reads the Location: http://www.google.com/ header. It tears down the first TCP connection, resolves www.google.com, connects on port 80, and issues another GET.

Google's server on www.google.com:80 responds with a second redirect:

HTTP/1.1 302 Found
Location: https://www.google.com/

Curl catches the 302, notes the change in protocol scheme from http:// to https://, changes the target port to 443, resolves the address again, opens a brand new TCP connection, negotiates an OpenSSL TLS 1.3 handshake (exchanging ClientHello, ServerHello, cryptographic certificates, and session keys), and finally issues an encrypted HTTP/2 or HTTP/1.1 GET / request.

Only then does Google stream back the full, multi-kilobyte JavaScript and HTML payload of its actual search page.

An Interesting Fact About Curl

Curl was not created as a general-purpose web debugging tool. In 1996, Swedish developer Daniel Stenberg was maintaining an IRC bot called Fraggel. He wanted the bot to automatically fetch currency exchange rates from web pages once a day so IRC users could convert Swedish Kronor to US Dollars in chat rooms.

Finding no lightweight CLI tool that fit his needs, he adopted an open-source utility called httpget, patched it, added FTP support, and renamed it urlget. On March 20, 1998, after adding upload support, he officially released it under the name cURL (Client URL).

Today, Daniel Stenberg still actively maintains the project, and libcurl runs on an estimated twenty billion devices worldwide, including Mars rovers, smart televisions, cars, gaming consoles, and virtually every server running in the cloud.

Summary: The Complete Execution Path

When you strip away the abstraction, a single curl google.com command traverses ten clear boundaries:

Terminal: Readline captures keystrokes over /dev/pts/X.
Shell: Bash expands aliases, locates /usr/bin/curl via $PATH, forks, and calls execve().
Linker: ld.so maps libcurl, libssl, and libc, activating ASLR and page protections.
Initialization: Curl parses arguments, defaults to HTTP port 80, and allocates struct Curl_easy.
Resolver: glibc checks /etc/nsswitch.conf, skips /etc/hosts, and sends UDP queries to 127.0.0.53.
Network Layer: The kernel checks the FIB routing table, finds the default gateway, and resolves the gateway MAC via ARP.
Transport Layer: A non-blocking TCP socket sends a SYN packet, handles SYN-ACK via SoftIRQ, and reaches ESTABLISHED.
Application Layer: Curl formats a plaintext HTTP/1.1 GET request and writes it to the socket buffer.
Edge Response: Google Front End rejects plain HTTP and returns a 219-byte 301 Moved Permanently payload.
Terminal Output: Curl reads the payload, writes it to stdout (fd 1), closes the socket to enter TIME_WAIT, and exits to let Bash draw the next prompt.

Knowing these layers turns mystical network failures into predictable troubleshooting steps. If curl hangs on a server, you no longer guess. You test DNS with dig, check TCP routing with ip route get, inspect socket states with ss -t, and trace syscall bottlenecks with strace.

What is the most surprising layer you discovered when digging into network commands on Linux?

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

Scaling Next.js on a $0 VPS: 25ms Cloudflare Edge Shield Guide

Asep Sayyad — Fri, 23 Oct 2026 15:43:29 +0000

How combining lightweight /api healthchecks, Docker container limits, and Cloudflare edge caching protects dynamic servers from traffic spikes.

The alert triggered at 2:40 PM on a Friday.

A technical post discussing self-hosted infrastructure hit the front page of a popular aggregator. Within six minutes, traffic surged from a comfortable baseline of five requests per minute to more than two hundred requests per second.

On our single-core, 1GB RAM cloud server, the system degraded rapidly.

CPU utilization jumped from 4% to 100%. Node.js event loop latency climbed past 2,500 milliseconds. Within ninety seconds, memory climbed until the Linux kernel Out-Of-Memory (OOM) killer stepped in, terminated the Next.js container, and killed the active SSH daemon. The collapse was total.

Visitors saw blank pages and 502 Bad Gateway errors. Everything froze up.

HTTP/2 502 Bad Gateway
server: cloudflare
cf-ray: 8e5f12a9b8cd9a41-EWR
content-type: text/html

When your web server runs on modest hardware, raw compute capacity is not your friend. Dynamic Next.js Server-Side Rendering (SSR) evaluates React component trees, builds hydration scripts, and executes runtime logic for every single visitor. Under a sudden surge, an unshielded single-core server will fold in seconds.

Here is how we redesigned our deployment pipeline, locked down container resource limits, fixed our monitoring bottlenecks, and built a Cloudflare Edge Shield that slashed Time-to-First-Byte (TTFB) to 25ms while handling traffic spikes without breaking our zero-dollar budget.

The Anatomy of an SSR CPU Collapse

To fix a server under load, you first have to understand why Next.js behaves differently than static file servers like Nginx or Caddy.

When Nginx serves an HTML file, it issues a sendfile() system call. The kernel transfers file bytes directly from the page cache to the network socket. The CPU barely wakes up. Nginx can easily push thousands of static pages per second on modest hardware.

Next.js SSR does not work that way.

Every time a user visits a dynamic route (like / or /blogs/post-slug), Node.js goes to work in user space. It parses route parameters, runs data-fetching functions, executes React code, serializes components into HTML strings, and builds JSON hydration payloads for the browser.

Client Request
      |
      v
+---------------+
| Next.js Node  | ---> Parse Route & Context
| Server Engine | ---> Execute Component Tree
|               | ---> Query Database (SQLite/Postgres)
|               | ---> Render HTML & Hydration Payloads
+---------------+
      |
      v
Output HTML String (Heavy CPU & Memory Cost)

On a desktop machine or an expensive 16-core cloud instance, rendering a React page in 40 milliseconds feels instantaneous.

On a shared 1-vCPU cloud instance, however, that 40ms CPU burst turns deadly. If thirty concurrent visitors request pages at the same instant, they require 1,200ms of raw CPU processing time. Because Node.js runs on a single main thread event loop, those requests queue up behind each other.

Incoming TCP connections stack up in the kernel accept queue. Memory usage climbs as active request buffers sit uncompleted in the V8 heap.

Then comes the garbage collector. Seeing memory pressure, V8 pauses execution to scavenge unused memory. This pushes CPU load to 100%, freezes response handling, and causes upstream proxies to time out.

The server did not crash because of a bug in your code. It crashed because full dynamic SSR is fundamentally too expensive to run on-demand for every untrusted web request. The math is brutal.

Docker Containment: Why Running Without Resource Limits Is Fatal

Most developers deploy their applications using a basic docker-compose.yml file that looks something like this:

services:
  web:
    image: my-nextjs-app:latest
    ports:
      - "3000:3000"
    restart: always

This configuration contains a hidden danger: by default, Docker imposes zero limits on CPU and memory consumption.

If your Next.js process enters a heavy garbage collection loop, runs a slow database query, or gets hit by an aggressive search crawler indexing fifty pages concurrently, the container will consume every byte of RAM and every CPU cycle available on the host.

When host memory runs out, the Linux kernel invokes oom-killer. The kernel looks for processes with high memory usage and badness scores.

All too often, the kernel terminates the container, but occasionally it kills your logging daemon, your reverse proxy, or your SSH daemon (sshd). When sshd dies, you lose console access and cannot log in to troubleshoot the box.

To prevent this, you must define strict cgroups limits directly in your compose file:

services:
  web:
    image: my-nextjs-app:latest
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      - NODE_ENV=production
      - NODE_OPTIONS=--max-old-space-size=384
    deploy:
      resources:
        limits:
          cpus: '0.85'
          memory: 512M
        reservations:
          cpus: '0.20'
          memory: 256M

Two key defenses exist in this configuration:

First, limits.cpus: '0.85' guarantees that the Next.js container can never consume more than 85% of the host CPU time within any scheduling period. The Linux CFS (Completely Fair Scheduler) throttles container execution if it exceeds this threshold, leaving at least 15% of compute capacity reserved for the host operating system, SSH sessions, and background maintenance tasks.

Second, limits.memory: 512M combined with NODE_OPTIONS=--max-old-space-size=384 establishes a memory ceiling.

By telling Node.js that its V8 heap limit is 384MB, Node.js triggers aggressive garbage collection when heap usage approaches that boundary, well before the container reaches its 512MB Docker limit.

If a catastrophic memory leak occurs, Docker terminates only that specific container rather than destabilizing the host server.

You can inspect whether your limits are taking effect in real time with docker stats:

docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.BlockIO}}"

NAME                CPU %     MEM USAGE / LIMIT   MEM %     BLOCK I/O
production-web-1    2.15%     184.2MiB / 512MiB   35.98%    14.2MB / 1.1MB

The Healthcheck CPU Death Spiral

Once we placed resource boundaries around our container, we encountered a second, subtler issue: an internal denial of service caused by healthchecks.

To ensure high availability, uptime monitors and Docker healthchecks inspect containers on recurring intervals. A common pattern is writing a healthcheck like this:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3000/"]
  interval: 30s
  timeout: 5s
  retries: 3

Notice what that check targets: the root URL /.

As established earlier, / is an SSR page. Every 30 seconds, Docker fires up a subshell, runs curl, and hits the root page. Node.js renders the full React tree, queries the database, generates HTML, and returns a 200 OK status.

Now add external monitoring: an Uptime Kuma instance, a Better Stack probe, and a status check from an external dashboard, each pinging your root URL every 30 to 60 seconds.

Suddenly, your server is rendering 10 to 15 full SSR pages every single minute, even when zero real human visitors are browsing your website. Your baseline CPU utilization sits permanently around 25%.

Monitoring Probes (Uptime Kuma, Better Stack, Docker Engine)
                          |
                          v
             Hitting http://localhost:3000/
                          |
                          v
           Runs Full React SSR + DB Queries
                          |
                          v
    Baseline CPU Consumed Permanently: 25% to 35%

Even worse: if a legitimate traffic surge arrives and pushes CPU to 80%, the SSR page takes four seconds to render. The Docker healthcheck hits its 5-second timeout, logs a failure, and after three retries marks the container as unhealthy and restarts it.

Your monitoring system just turned a temporary latency spike into an unrecoverable crash loop.

The solution is simple: eliminate React execution from your healthchecks. Create a dedicated, ultra-lean API route that returns a static JSON status in sub-millisecond time.

In Next.js App Router, create app/api/health/route.ts:

import { NextResponse } from 'next/server';

export const dynamic = 'force-dynamic';
export const runtime = 'nodejs';

export async function GET() {
  return NextResponse.json(
    {
      status: 'healthy',
      timestamp: Date.now(),
      uptime: Math.floor(process.uptime()),
    },
    {
      status: 200,
      headers: {
        'Cache-Control': 'no-store, max-age=0',
      },
    }
  );
}

Then update your Docker healthcheck:

healthcheck:
  test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3000/api/health | grep -q 'healthy' || exit 1"]
  interval: 30s
  timeout: 3s
  start_period: 15s
  retries: 3

This endpoint requires zero React rendering, executes zero database queries, and uses built-in wget rather than heavy external tools. Response time drops to less than 2 milliseconds. Monitoring overhead drops to zero. That solved the problem.

Building the Edge Shield: Cloudflare Cache Rules

Restricting container resources and streamlining healthchecks ensures our server remains stable, but it does not solve the root problem: dynamic rendering cannot handle thousands of concurrent users on a 1-core VPS.

To solve that, we must prevent traffic from reaching our server IP in the first place.

Here is the common misconception about Cloudflare: many developers point their domain names to Cloudflare, enable the orange cloud proxy, and assume their entire website is automatically cached.

It is not.

By default, Cloudflare operates as a reverse proxy that caches only static asset extensions: .css, .js, .jpg, .png, .svg, .ico, and .woff2.

When an incoming request asks for https://example.com or https://example.com/blogs/edge-caching, Cloudflare inspects the content type. Because the response is text/html, Cloudflare bypasses its cache and sends the request straight to your origin server.

Every visitor, bot, and crawler still hits your VPS directly.

Default Cloudflare Setup:
User Request ---> Cloudflare (HTML Detected) ---> Origin VPS (Renders SSR) ---> Latency: 350ms

Cloudflare Edge Shield Setup:
User Request ---> Cloudflare Edge (HTML Cached in RAM) ---> Serves User ---> Latency: 25ms
                          |
                     (98% of requests stopped at the edge)

To turn Cloudflare into a true edge shield, we use Cloudflare Cache Rules (the modern replacement for legacy Page Rules).

Here is the exact rule structure we configure in the Cloudflare dashboard under Caching -> Cache Rules:

Rule Name: Edge Shield - Public Pages

Filter Expression:

(http.request.uri.path eq "/" or http.request.uri.path starts_with "/blogs") 
and not (http.request.uri.path starts_with "/api/") 
and not (http.request.uri.path starts_with "/admin")

Cache Settings:

Cache Eligibility: Eligible for cache
Edge TTL: Override origin -> 4 hours (or 24 hours depending on update frequency)
Browser TTL: Override origin -> 10 minutes
Serve Stale Content: Enabled (Serve stale content while revalidating)

What happens with this rule in place?

When the first visitor requests /, Cloudflare forwards the request to your origin VPS. Your Next.js container renders the page in 280ms and sends the HTML back.

Cloudflare stores that HTML response in the memory of the nearest data center.

When the next 5,000 visitors arrive from that region over the next four hours, Cloudflare serves the cached HTML directly from its global Edge network.

The request never touches your server. It consumes 0% CPU on your VPS, zero megabytes of host RAM, and zero bytes of origin bandwidth.

And because Cloudflare delivers the response from an edge node located geographically close to the user, the Time-to-First-Byte (TTFB) drops from 300ms down to 25ms.

Verifying Cache Hits with HTTP Headers

You should never guess whether your edge cache is working. You can verify it directly with curl:

curl -Iv https://example.com/ 2>&1 | grep -iE '^(< HTTP/|< cf-|< age:|< cache-control:)'

On your first request, you will see a cache miss:

< HTTP/2 200
< cache-control: s-maxage=14400, stale-while-revalidate=86400
< age: 0
< cf-cache-status: MISS
< cf-ray: 8e5f210a4c9b2e10-EWR

Run the exact same command two seconds later:

curl -Iv https://example.com/ 2>&1 | grep -iE '^(< HTTP/|< cf-|< age:|< cache-control:)'

The output changes completely:

< HTTP/2 200
< cache-control: s-maxage=14400, stale-while-revalidate=86400
< age: 2
< cf-cache-status: HIT
< cf-ray: 8e5f212f0a8d2e10-EWR

Look at cf-cache-status: HIT and age: 2. It served instantly.

That status confirms the page was served directly out of Cloudflare edge memory. The response was delivered in 22 milliseconds. Your origin VPS did not execute a single line of JavaScript.

Managing Dynamic vs Static Content: The Isolation Matrix

Caching HTML at the edge is powerful, but caching the wrong content is disastrous.

If you cache an API endpoint that handles contact form submissions, dynamic authentication tokens, or an administrative dashboard, one user might receive another user's private data or see a stale form submission token.

You must maintain strict isolation between edge-cached public pages and dynamic routes.

Here is how we partition our routes:

Public Pages (/, /blogs/*, /projects):

Cache Status: cf-cache-status: HIT
Edge TTL: 4 to 24 hours
Behavior: Fully cached at edge, background revalidation via stale-while-revalidate.

API Endpoints (/api/*):

Cache Status: cf-cache-status: DYNAMIC
Edge TTL: Bypassed
Behavior: Always forwarded to origin Next.js container.

Admin & Auth Routes (/admin/*, /dashboard/*):

Cache Status: cf-cache-status: BYPASS
Edge TTL: Bypassed
Behavior: Never cached under any circumstances.

To guarantee that Next.js itself enforces this boundary, configure your route handlers with explicit cache-control headers.

For dynamic API routes like /api/contact/route.ts:

import { NextResponse } from 'next/server';

export const dynamic = 'force-dynamic';

export async function POST(request: Request) {
  const body = await request.json();

  // Process contact form...

  return NextResponse.json(
    { success: true, message: 'Message received' },
    {
      status: 200,
      headers: {
        'Cache-Control': 'private, no-cache, no-store, must-revalidate',
        'Pragma': 'no-cache',
      },
    }
  );
}

In your Cloudflare dashboard, add a second Cache Rule placed with higher priority (Rule 1):

Rule Name: Bypass Cache for APIs and Admin

Filter Expression:

(http.request.uri.path starts_with "/api/") 
or (http.request.uri.path starts_with "/admin") 
or (http.cookie contains "admin_session")

Cache Settings:

Cache Eligibility: Bypass cache

Notice the cookie check: http.cookie contains "admin_session".

If an administrator logs in and holds an active session cookie, Cloudflare bypasses edge caching even when they browse public pages. They always see real-time, live content, while standard anonymous visitors receive fast edge-cached responses.

Zero-Downtime Deployments with SQLite Data Protection

Many full-stack Next.js applications running on small instances use SQLite (via Prisma, Drizzle, or better-sqlite3) because it requires no separate database process, consumes minimal RAM, and performs reads in microseconds.

However, running SQLite inside Docker creates a major deployment hazard: how do you deploy new application code without overwriting your live database or corrupting active transactions?

If you run docker compose down and docker compose up -d, you introduce downtime.

If your database file lives inside the container filesystem rather than an external volume, pulling a new Docker image permanently deletes all your production data.

Here is our battle-tested deployment setup that combines persistent host volume mounts, SQLite Write-Ahead Logging (WAL) mode, and safe CI/CD updates.

First, enable WAL mode on your SQLite database during initialization:

PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
PRAGMA busy_timeout=5000;

WAL mode allows concurrent readers to query the database while a writer is writing, preventing read locks during traffic spikes.

Second, mount the database directory from the host filesystem in docker-compose.yml:

services:
  web:
    image: ghcr.io/username/portfolio:latest
    volumes:
      - /opt/portfolio/data:/app/data
      - /opt/portfolio/uploads:/app/public/uploads

Third, automate deployments through a GitHub Actions workflow that creates an atomic SQLite backup before updating the container.

Here is the exact deploy script executed on the server over SSH:

#!/usr/bin/env bash
set -euo pipefail

APP_DIR="/opt/portfolio"
BACKUP_DIR="${APP_DIR}/backups"
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")

echo "Starting deployment at ${TIMESTAMP}..."
cd "${APP_DIR}"

# 1. Create atomic SQLite backup using the official sqlite3 CLI tool
mkdir -p "${BACKUP_DIR}"
if [ -f "${APP_DIR}/data/production.db" ]; then
  echo "Backing up SQLite database..."
  sqlite3 "${APP_DIR}/data/production.db" ".backup '${BACKUP_DIR}/db_backup_${TIMESTAMP}.db'"
  # Keep only the last 7 daily backups
  ls -tp "${BACKUP_DIR}"/db_backup_*.db | grep -v '/$' | tail -n +8 | xargs -I {} rm -- {} || true
fi

# 2. Pull the latest Docker image built by GitHub Actions
echo "Pulling new image..."
docker compose pull web

# 3. Recreate the container with zero downtime for sister services
echo "Recreating web container..."
docker compose up -d --no-deps web

# 4. Wait for healthcheck validation
echo "Waiting for container health..."
for i in {1..15}; do
  STATUS=$(docker inspect --format='{{json .State.Health.Status}}' portfolio-web-1 2>/dev/null || echo '"unknown"')
  if [ "${STATUS}" == '"healthy"' ]; then
    echo "Deployment successful: container is healthy."
    break
  fi
  echo "Container health status: ${STATUS}... waiting 2s"
  sleep 2
done

# 5. Clean up stale Docker images to save precious disk space
echo "Pruning dangling images..."
docker image prune -f

echo "Deployment complete."

Notice step 1: sqlite3 ... ".backup".

Never use cp production.db backup.db while an application is running. Standard cp reads bytes in chunks. If Next.js writes a transaction while cp is midway through the file, your backup file becomes corrupt.

The sqlite3 .backup command uses the SQLite online backup API, holding an internal shared lock to produce a consistent point-in-time snapshot.

Load Testing: Proving the Difference with Autocannon

To evaluate the effectiveness of this edge setup, we ran stepped HTTP benchmarks using autocannon, comparing direct origin access against the Cloudflare Edge Shield.

We pointed the load tester at our Next.js blog index page (/blogs/post-slug).

Test 1: Direct Origin Access (No Edge Caching)

Target: Origin IP directly
Concurrency: 80 connections
Duration: 20 seconds

Running 20s test @ http://origin-ip:3000/blogs/post-slug
80 connections

Stat         Avg       Stdev     Max
Latency      784ms     312ms     2140ms
Req/Sec      101.4     24.1      142
Bytes/Sec    1.42MB    340KB     2.01MB

2028 requests in 20.05s, 28.4MB read
Errors: 142 connection timeouts, 68 non-2xx responses

During this direct test, the server core ran at 100% CPU. Event loops froze, and 210 requests either timed out or returned 502 errors.

Test 2: Cloudflare Edge Shield Enabled

Target: Public domain via Cloudflare with Cache Rule active
Concurrency: 250 connections (over 3x higher load)
Duration: 20 seconds

Running 20s test @ https://example.com/blogs/post-slug
250 connections

Stat         Avg       Stdev     Max
Latency      24.6ms    8.2ms     82.4ms
Req/Sec      2,410     180       2,720
Bytes/Sec    33.8MB    2.5MB     38.1MB

48,200 requests in 20.02s, 676MB read
Errors: 0 timeouts, 0 non-2xx responses

The difference is night and day. Zero dropped packets.

Throughput jumped from 101 requests per second to 2,410 requests per second. Average latency plummeted from 784ms down to 24.6ms. Most importantly, errors dropped to zero.

And what was the origin server doing during those 48,200 requests?

Running top on the host revealed CPU utilization sitting calmly at 1.8%. The Next.js container served exactly one request to populate the edge cache. Cloudflare handled the remaining 48,199 requests from memory. It held up perfectly.

An Interesting Fact About Edge Caching

When Akamai invented commercial content delivery networks in 1998, edge caching was exclusively used for static images and media files.

For nearly two decades, caching dynamic HTML pages at the edge was considered impractical because content management systems lacked reliable cache invalidation mechanisms.

Until recently, Cloudflare reserved granular edge HTML caching for Enterprise-tier customers paying thousands of dollars per month under custom Page Rules.

With the introduction of modern Cloudflare Cache Rules and HTTP standard stale-while-revalidate directives, individual developers running single-core cloud instances can now deploy caching configurations that rival Fortune 500 media networks for zero dollars.

My Default Rule of Thumb for Small Next.js Deployments

When running production Next.js apps on small cloud servers:

Put a hard ceiling on containers: Always define limits.cpus (e.g. 0.85) and limits.memory in docker-compose, and configure NODE_OPTIONS=--max-old-space-size to prevent OOM panics.
Never monitor full SSR pages: Keep uptime bots and Docker healthchecks pointed at a lightweight /api/health route that returns in under 2ms.
Cache HTML at the edge: Use Cloudflare Cache Rules to cache public pages (/, /blogs/*) for hours while serving stale content during background revalidation.
Strictly bypass APIs: Explicitly exclude /api/* and admin routes from edge caching with Cache-Control: private, no-store.
Protect your SQLite database: Run SQLite in WAL mode, mount data to external Docker host volumes, and use sqlite3 .backup before every automated container restart.
Verify with headers: Run curl -Iv and confirm you see cf-cache-status: HIT and single-digit millisecond TTFB before launching to the public.

Have you ever had an unshielded server crash under unexpected viral traffic or crawler bots? What was the bottleneck that took it down?

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

Tuning the Linux Kernel for 10x Traffic: How We Prepared a Modest Cloud Server for the Hug of Death

Asep Sayyad — Thu, 22 Oct 2026 04:34:58 +0000

Somaxconn, SYN backlogs, TCP syncookies, and finding our breaking point with stepped Autocannon load benchmarks.

The alert hit our channel at 11:14 AM on a Tuesday.

Our product had just been picked up on a popular tech newsletter, and inbound traffic surged from our normal baseline of 8 requests per second to over 180 requests per second within three minutes.

On the surface, our 2-vCPU, 4GB RAM cloud instance looked relaxed. CPU utilization hovered around 32%. Memory had nearly 2GB free. Disk I/O was barely moving.

Yet visitors were seeing 502 Bad Gateway errors, and API clients reported connection timeouts:

curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to api.example.com:443

Why would a server drop incoming web traffic when more than half of its compute capacity was sitting idle?

The answer was not in the application code, the database queries, or the cloud provider firewall. The bottleneck was inside the Linux kernel network stack. Out of the box, Linux distributions ship with network socket limits designed for the hardware realities of twenty years ago. When an unexpected traffic surge hits your front door, the kernel drops incoming TCP handshakes before your reverse proxy or web application even knows they exist.

Here is the exact diagnostic workflow, the kernel parameters we tuned to handle ten times our normal traffic, and the stepped Autocannon load tests we used to find our server breaking point.

The Silent Killer: The Two TCP Queues

To understand why a server with an idle CPU rejects incoming web requests, you have to look at how the Linux kernel handles incoming TCP connections.

Before an HTTP request reaches Nginx, Node.js, Go, or Python, the kernel network subsystem must complete the standard TCP three-way handshake:

The client sends a SYN packet to request a connection.
The server responds with SYN-ACK.
The client replies with an ACK packet.

The connection is now established. Only then does the application receive the socket through the accept() system call.

What most engineers miss is that the kernel maintains two completely separate queues in memory during this process:

The SYN Queue (Half-Open Sockets)

When a client sends a SYN, the kernel creates a lightweight request socket in memory and parks it in the SYN queue (also called the half-open connection queue). The connection stays here while waiting for the client final ACK.

The size of this queue is bounded by the kernel parameter net.ipv4.tcp_max_syn_backlog.

The Accept Queue (Fully Established Sockets)

Once the client returns the final ACK, the handshake is complete. The kernel promotes the connection from the SYN queue to the Accept queue.

The connection sits in the Accept queue until your application process calls accept() to pick it up and assign it a file descriptor.

The size of this queue is bounded by the system-wide limit net.core.somaxconn and the backlog parameter passed by your application to the listen() system call.

Here is the problem: on many standard cloud images, net.core.somaxconn defaults to 128 (or 4096 on newer kernels, while application servers still request 128 or 511). And net.ipv4.tcp_max_syn_backlog often defaults to 128 or 512.

If a sudden burst of 200 users attempts to connect simultaneously, the Accept queue fills up instantly.

Once the queue is full, the kernel has a stark choice. By default, it silently drops newly arriving ACK packets. The client assumes the packet was lost in transit and attempts TCP retransmissions, creating an artificial latency spike of one to three seconds.

If the queue remains congested, the client connection times out or the server resets it with a RST packet.

Your application never logged an error because the request was killed before user space ever saw it.

How to Prove Your Sockets Are Dropping Packets

Never guess when debugging the Linux network stack. You can inspect queue depths and drop counters directly from the command line.

Inspecting Live Socket Backlogs with ss

The ss command is the modern replacement for netstat. Most engineers know how to use it to check open ports, but few know how to interpret its queue output on listening sockets.

Run this command on your server:

ss -lnt '( sport = :80 or sport = :443 )'

You will see output formatted like this:

State      Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN     0      511          0.0.0.0:80        0.0.0.0:*
LISTEN     128    128          0.0.0.0:443       0.0.0.0:*

On an established socket (a live data connection), Recv-Q represents bytes in the receive buffer waiting to be read by the application, and Send-Q represents bytes waiting to be sent over the wire.

On a LISTEN socket, those columns mean something completely different:

Send-Q: The maximum backlog capacity of the Accept queue.
Recv-Q: The current number of established connections waiting in the Accept queue to be accepted by the application.

Look at port 443 in the sample output above. Recv-Q is 128 and Send-Q is 128.

The Accept queue is completely full. Every single incoming connection arriving at that exact microsecond is being dropped by the kernel.

Checking Kernel Drop Counters with nstat and netstat

To verify whether your server has suffered queue overflows historically, query the kernel SNMP network statistics:

nstat -az TcpExtListenOverflows TcpExtListenDrops

Or using netstat:

netstat -s | grep -i "listen"

Sample output:

14820 times the listen queue of a socket overflowed
14820 SYNs to LISTEN sockets dropped

If TcpExtListenOverflows is greater than zero, your server has dropped real customer connections because the Accept queue was saturated.

If TcpExtListenDrops is incrementing, the kernel had to discard incoming connection attempts because either the Accept queue was full, the SYN backlog was exhausted, or memory pressure prevented socket allocation.

To check if SYN flood protection or queue exhaustion has been triggered recently, inspect the kernel ring buffer:

dmesg -T | grep -i -E "syn|drop|backlog"

If you see messages like TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters, the kernel SYN queue is overflowing.

The Double-Gatekeeper Bottleneck

A common mistake is changing net.core.somaxconn in /etc/sysctl.conf and assuming the problem is solved.

It is not. The Accept queue size is governed by a double gatekeeper:

Effective Accept Queue = min( net.core.somaxconn, application listen backlog )

When an application server initializes a socket, it executes the listen() system call:

int listen(int sockfd, int backlog);

The second argument tells the kernel how large of an Accept queue the application wants. But the kernel caps that request at the value of net.core.somaxconn.

If you raise net.core.somaxconn to 4096, but your application server requests a backlog of 128, your effective queue remains 128.

Conversely, if your Nginx configuration specifies listen 443 default_server backlog=4096;, but your kernel somaxconn is set to 512, the kernel truncates your queue to 512.

Both layers must be configured in unison.

Nginx Configuration

In your Nginx site configuration, explicitly set the backlog parameter on your listen directives:

server {
    listen 80 backlog=4096;
    listen 443 ssl backlog=4096;
    server_name api.example.com;

    # ... rest of configuration
}

Node.js / Express Configuration

In Node.js HTTP servers, the default backlog is 511. You can raise it by passing an options object to server.listen():

const express = require('express');
const app = express();

const server = app.listen({
    port: 3000,
    host: '0.0.0.0',
    backlog: 4096
}, () => {
    console.log('Server running with listen backlog 4096');
});

Python Gunicorn / Uvicorn

If you run Python workloads with Gunicorn, set the backlog parameter in your command line or config file:

gunicorn --workers 4 --backlog 4096 -b 0.0.0.0:8000 app:wsgi_app

The High-Performance Kernel Recipe

To prepare a production Linux host for high concurrency and unexpected traffic surges, create a dedicated sysctl configuration file at /etc/sysctl.d/99-network-tuning.conf.

Here is the exact configuration we deploy across our cloud nodes:

# /etc/sysctl.d/99-network-tuning.conf

# 1. Expand Socket Listen and SYN Backlogs
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 4096

# 2. Connection Exhaustion and SYN Defense
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2

# 3. Socket Lifecycle and Teardown Acceleration
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1

# 4. Ephemeral Port Range Expansion
net.ipv4.ip_local_port_range = 10240 65535

# 5. Network Buffer Allocation (Safe for 2GB - 8GB RAM nodes)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# 6. Anti-Spoofing and Path Verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

Let us break down why each parameter is set this way and what it changes in practice.

Socket Listen and SYN Queues

net.core.somaxconn = 4096 expands the maximum Accept queue limit from the legacy default to 4096 entries. This gives the application an order of magnitude more headroom during traffic spikes.

net.ipv4.tcp_max_syn_backlog = 4096 ensures the SYN queue can hold up to 4096 half-open connections while handshakes complete.

net.core.netdev_max_backlog = 4096 governs the incoming packet queue between the physical or virtual network interface card (NIC) and the kernel network stack. When the NIC receives packets faster than the CPU can process kernel interrupts, this buffer prevents packet drops at the driver layer.

SYN Cookies and Handshake Resiliency

net.ipv4.tcp_syncookies = 1 activates cryptographic SYN cookies when the SYN backlog overflows. Instead of dropping new connection requests, the kernel encodes the connection state into the initial TCP sequence number sent in the SYN-ACK. When the legitimate client responds with its ACK, the kernel reconstructs the socket without having stored anything in memory. This repels SYN flood attacks and absorbs unexpected spikes without degrading service.

net.ipv4.tcp_synack_retries = 2 lowers the number of times the kernel retransmits a SYN-ACK packet before abandoning the attempt. The default is 5, which ties up resources for over 30 seconds if a client disappears. Lowering it to 2 reclaims half-open sockets within 7 seconds.

Socket Teardown and TIME_WAIT Recycling

net.ipv4.tcp_fin_timeout = 15 reduces the time a socket stays in the FIN-WAIT-2 state from the default of 60 seconds down to 15 seconds. When clients disconnect abruptly without completing a clean four-way TCP shutdown, this ensures the kernel frees orphaned sockets four times faster.

net.ipv4.tcp_tw_reuse = 1 allows the kernel to safely reuse sockets in the TIME_WAIT state for new outgoing connections, provided the incoming timestamps are strictly greater than the previous connection. This is vital when running a reverse proxy (like Nginx) that opens and closes high-frequency TCP connections to local backend services.

Ephemeral Port Range

net.ipv4.ip_local_port_range = 10240 65535 expands the dynamic port range available for outbound client connections. The default Linux range is 32768 60999, offering approximately 28,231 ports. Setting it to 10240 65535 expands the pool to 55,295 ephemeral ports.

When an Nginx proxy handles 5,000 requests per second against an upstream Node.js application, ports in TIME_WAIT can rapidly exhaust the default range, leading to EADDRNOTAVAIL (Cannot assign requested address) errors. Doubling the port pool eliminates this ceiling.

Applying the Changes Safely

To load the new configuration into the running kernel without rebooting:

sudo sysctl --system

Verify that the values took effect:

sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_tw_reuse

Output:

net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_tw_reuse = 1

Don't Forget the Conntrack Table and File Descriptors

Two additional system limits frequently trip up engineers who tune socket backlogs but leave firewall or file descriptor defaults untouched.

1. The Netfilter Connection Tracking Table

If you use iptables, nftables, or ufw on your server, the kernel tracks every active stateful connection in a table called conntrack.

When traffic spikes, the connection tracking table can fill up completely. When that happens, the kernel drops incoming packets immediately, logging this terrifying message to dmesg:

nf_conntrack: table full, dropping packet

Check your current conntrack usage and maximum table limit:

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

On a system with 2GB to 4GB of RAM, raise the limit inside /etc/sysctl.d/99-network-tuning.conf:

net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 600

Lowering nf_conntrack_tcp_timeout_established from its default of 432,000 seconds (5 full days!) down to 600 seconds (10 minutes) purges dead tracking entries exponentially faster.

2. Systemd and File Descriptor Ceilings

Every open socket in Linux requires a file descriptor. The default per-process file limit in systemd is often 1024.

If your Nginx worker process hits 1,024 open connections, it logs:

2026/10/21 11:15:02 [alert] 1240#1240: *45100 socket() failed (24: Too many open files) while connecting to upstream

To permanently fix this for Nginx, create a systemd drop-in configuration:

sudo mkdir -p /etc/systemd/system/nginx.service.d/
sudo nano /etc/systemd/system/nginx.service.d/override.conf

Add these lines:

[Service]
LimitNOFILE=65535

Also add worker_rlimit_nofile 65535; to the top of /etc/nginx/nginx.conf:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
worker_rlimit_nofile 65535;

events {
    worker_connections 8192;
    multi_accept on;
}

Reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart nginx

Check the applied limit on the running process:

cat /proc/$(pgrep -o nginx)/limits | grep "Max open files"

Output:

Max open files            65535                65535                files

Real-World Socket Engineering: The Local Port Churn Lesson

When I was building AiroShare, a local DLNA streaming server and high-performance media engine, socket management was one of the earliest friction points I ran into.

In that project, the application broadcasts media manifests to DLNA receivers over SSDP multicast while servicing concurrent media chunk streaming requests over HTTP. During intense chunk streaming, ephemeral sockets opened and closed at an aggressive rate.

Without setting net.ipv4.tcp_tw_reuse = 1 and expanding the local port range, high-concurrency client requests quickly choked on dead sockets sitting in TIME_WAIT. The operating system refused to assign new client sockets, throwing EADDRNOTAVAIL errors even though the machine had negligible CPU load.

The exact same problem happens on cloud servers with reverse proxies. When Nginx proxies thousands of incoming connections to a local backend on 127.0.0.1:3000, it consumes an ephemeral port for every single upstream connection.

If keepalive connections are not enabled between Nginx and the backend, each request opens a new socket and leaves it in TIME_WAIT for 60 seconds.

Always configure upstream keepalive pools in Nginx:

upstream backend_cluster {
    server 127.0.0.1:3000;
    keepalive 64;
}

server {
    listen 443 ssl backlog=4096;
    server_name api.example.com;

    location / {
        proxy_pass http://backend_cluster;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Adding proxy_http_version 1.1 and clearing the Connection header allows Nginx to reuse established TCP connections to your application, slashing socket churn by over 80%.

Stress Testing Live: Finding the Breaking Point with Autocannon

Applying kernel parameters without testing them is wishful thinking. To verify our tuning and find the true breaking point of our 2-vCPU node, we ran stepped load benchmarks using Autocannon.

Autocannon is a fast, Node.js-based HTTP benchmarking tool that supports high concurrency, pipelining, and custom headers.

Instead of hitting the server with a static blast of traffic, we used a stepped concurrency testing model:

Step 1: 10 concurrent connections (moderate baseline load).
Step 2: 25 concurrent connections (medium-high production burst).
Step 3: 50 concurrent connections (extreme saturation test).

We ran tests against two separate endpoints: a heavy SSR/database-backed endpoint and an optimized, lightweight JSON telemetry endpoint.

Test 1: Baseline Load at 10 Concurrent Connections

Command executed:

autocannon -c 10 -d 30 -p 10 https://api.example.com/api/v1/telemetry

Output:

Running 30s test @ https://api.example.com/api/v1/telemetry
10 connections with 10 pipelined requests

Stat         2.5%     50%      97.5%    99%      Avg       Stdev     Max
Latency      78ms     135ms    210ms    245ms    138.4ms   34.2ms    312ms

Stat         1%       2.5%     50%      97.5%    Avg       Stdev     Min
Req/Sec      28       31       38       44       38.2      4.1       25

1.1k requests in 30.05s, 4.2 MB read
0 errors, 0 timeouts, 0 non-2xx responses

At 10 concurrent connections with pipelining, our server processed 38.2 requests per second at a median latency of 135ms. The CPU remained below 40%. Zero packet drops occurred. Socket errors stayed at zero.

Test 2: Scaling Up to 25 Concurrent Connections

Next, we tested the optimized API endpoint under higher concurrency:

autocannon -c 25 -d 30 https://api.example.com/api/v1/status

Output:

Running 30s test @ https://api.example.com/api/v1/status
25 connections

Stat         2.5%     50%      97.5%    99%      Avg       Stdev     Max
Latency      52ms     102ms    180ms    210ms    105.1ms   28.6ms    260ms

Stat         1%       2.5%     50%      97.5%    Avg       Stdev     Min
Req/Sec      62       68       75       84       75.3      5.8       58

2.3k requests in 30.04s, 8.9 MB read
0 errors, 0 timeouts, 0 non-2xx responses

On this endpoint, throughput climbed to 75.3 requests per second with a median latency of 102ms.

The contrast was stark. Before our kernel tuning, this exact test triggered TcpExtListenOverflows within four seconds. It dropped over 18% of all connection attempts. With the expanded backlog and tuned TCP syncookies, zero connections were dropped. Not one packet failed.

Test 3: The Latency Cliff at 50 Concurrent Connections

We pushed the 2-vCPU node further to find its ceiling:

autocannon -c 50 -d 30 https://api.example.com/api/v1/telemetry

Output:

Running 30s test @ https://api.example.com/api/v1/telemetry
50 connections

Stat         2.5%     50%      97.5%    99%      Avg       Stdev     Max
Latency      190ms    450ms    1620ms   1840ms   485.2ms   215.4ms   2410ms

Stat         1%       2.5%     50%      97.5%    Avg       Stdev     Min
Req/Sec      35       38       42       46       41.6      3.2       30

1.2k requests in 30.08s, 4.8 MB read
0 errors, 0 timeouts, 0 non-2xx responses

Look closely at what happened.

Throughput plateaued at 41.6 requests per second. Median latency jumped from 135ms to 450ms. The 99th percentile surged to 1,840ms.

During the test, running pidstat -u 1 on the server showed CPU utilization at 98.4%. The cores were pegged.

This is the classic latency cliff.

Crucially, because the kernel queue parameters were properly tuned, there were still zero connection drops and zero timeout errors (0 errors, 0 timeouts).

The server did not crash. It did not reset connections. Instead, requests queued orderly in the expanded Accept queue until CPU cycles were available to process them.

That distinction matters.

An untuned server crashes under load. It drops users with 502 Bad Gateway errors while CPU sits idle. A hardened server queues the burst gracefully. It trades latency for 100% request completion.

An Interesting Fact About somaxconn

The default value of 128 for somaxconn was not chosen through scientific analysis of modern networking.

It was first hardcoded into the 4.2BSD Unix kernel release in 1983.

At the time, typical Unix workstations had between 2MB and 16MB of total physical RAM. Network engineers feared that allowing more than 128 outstanding socket requests would exhaust kernel mbuf memory and trigger a system crash.

That 128-socket limit remained the default across mainstream Linux distributions for over two decades. Even today, many production cloud images default to conservative backlog values that can be saturated by a single office browser refresh.

My Default Rule of Thumb for High-Concurrency Nodes

When preparing a production Linux server for sudden traffic surges:

Match your backlogs: Set net.core.somaxconn = 4096 and net.ipv4.tcp_max_syn_backlog = 4096 in sysctl, and match them with backlog=4096 in your Nginx or application server.
Enable SYN cookies: net.ipv4.tcp_syncookies = 1 protects you from both deliberate SYN floods and legitimate viral traffic surges.
Speed up socket reclamation: Set net.ipv4.tcp_fin_timeout = 15 and net.ipv4.tcp_tw_reuse = 1 to prevent orphaned sockets from lingering.
Double your file limits: Add LimitNOFILE=65535 to your systemd overrides and configure worker_rlimit_nofile 65535; in Nginx.
Reuse proxy upstreams: Always configure keepalive in Nginx upstream blocks to avoid ephemeral port exhaustion on 127.0.0.1.
Monitor with ss -lnt and nstat: If Recv-Q matches Send-Q on a listening socket, or TcpExtListenOverflows is increasing, your Accept queue is overflowing.

Have you ever experienced an outage where your server was dropping traffic even though CPU and memory were barely breaking a sweat? What turned out to be the bottleneck?

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

The Mystery of the Dying 1GB Server

Asep Sayyad — Tue, 20 Oct 2026 03:29:54 +0000

How an unsuspected GUI daemon, zero swap, and unrotated Docker logs pushed our cloud node into kernel panics (and how we reclaimed 500MB of free RAM).

The incident report pinged my phone at 02:40 AM.

Our smallest cloud node, a 1-vCPU virtual machine with 1GB of RAM hosting two internal Node.js utilities and a lightweight reverse proxy, stopped responding to HTTP health checks. When I attempted to open an SSH session, the connection hung indefinitely before dropping with a TCP timeout:

ssh: connect to host 198.51.100.42 port 22: Connection timed out

I logged into the cloud provider's web console and opened the out-of-band VNC terminal. The display was frozen on a dump of kernel backtraces. The last four lines on the screen spelled out the entire disaster:

[ 14820.192041] Out of memory: Kill process 1824 (node) score 812 or sacrifice child
[ 14820.194210] Killed process 1824 (node) total-vm:492104kB, anon-rss:312480kB, file-rss:0kB
[ 14820.201844] kernel: oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[ 14820.204512] Kernel panic - not syncing: Out of memory and no killable processes...

The system was dead. The kernel had invoked its Out of Memory (OOM) killer, sacrificed our primary Node.js process, failed to reclaim enough contiguous pages to satisfy outstanding kernel memory allocations, and panicked.

After issuing a hard reboot through the provider API, the machine booted back up. I logged in over SSH and checked memory statistics:

free -h

               total        used        free      shared  buff/cache   available
Mem:           987Mi       642Mi        60Mi        12Mi       285Mi       285Mi
Swap:             0B          0B          0B

The server had 285 MiB of available memory right after booting. More critically, the swap row read exactly zero.

Running a 1GB cloud node with zero swap space under modern container workloads is like driving down a highway with no brakes. Any brief traffic spike, automated package update, or background memory allocation triggers an instant system collapse.

Here is the exact technical diagnosis and step-by-step remediation we used to turn this fragile node into a rock-solid server with over 530 MiB of free RAM and zero crashes over ninety days of continuous operation.

The Mechanics of Low-Memory Collapse

To understand why a 1GB server collapses so violently, you need to look at how the Linux kernel manages virtual memory.

Linux uses an optimistic memory allocation strategy called memory overcommit. When a process requests memory via malloc() or the brk() and mmap() system calls, the kernel grants virtual address space immediately without allocating physical RAM frames. The kernel assumes that processes will not use all the memory they request at the exact same instant.

Physical memory pages are only mapped when the process actually writes bytes to those virtual addresses, triggering a page fault.

Under normal conditions, physical memory is divided into two primary categories:

Anonymous memory: Data belonging to running processes, including the heap, stack, and private mapped memory. This data cannot be discarded because it has no backing file on disk.
Page cache: Copies of disk blocks kept in RAM, including executable binaries, shared libraries, and cached file reads.

When applications allocate more memory, the kernel must make room. It does this through page reclamation, coordinated by the background kernel thread kswapd0.

Here is where zero swap becomes fatal.

If a server has a swap device configured, kswapd0 can write cold, inactive anonymous pages to disk, freeing up physical RAM frames for active tasks. But if the server has zero swap, the kernel has only one option: it must evict the page cache.

The kernel begins throwing away cached disk files, shared library segments, and binary code pages. As the page cache shrinks toward zero, every single operation on the machine requires synchronous disk I/O. If the shell needs to run a command, it must fetch the binary from slow virtualized cloud storage block by block.

When the page cache hits rock bottom and anonymous memory fills remaining RAM, kswapd0 spins at 100% CPU in a desperate loop trying to find reclaimable pages. Because there is no swap to write anonymous memory to, the allocator fails.

The kernel invokes out_of_memory(). If the OOM killer selects a process that holds critical locks or cannot release memory fast enough, the entire operating system locks up or panics.

The Culprit You Never Suspect: Headless Desktop Daemons

After rebooting the box, my first priority was auditing where 642 MiB of RAM was going on a fresh boot.

We were running two Docker containers, an Nginx reverse proxy, and basic system utilities. Those workloads should have consumed no more than 250 MiB combined.

I sorted running processes by physical resident memory usage:

ps aux --sort=-%mem | head -n 15

The output delivered an unpleasant surprise:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       912  0.1  4.8 182400 48120 ?        Sl   02:45   0:01 /usr/lib/xorg/Xorg :10 -auth .Xauthority
appuser   1042  0.0  3.9 142100 39180 ?        Sl   02:45   0:00 xfce4-session
appuser   1108  0.0  3.4 118400 34200 ?        S    02:45   0:00 xfwm4
root       845  0.0  2.1  45100 21400 ?        S    02:45   0:00 /usr/sbin/xrdp
root       848  0.0  1.6  38200 16120 ?        S    02:45   0:00 /usr/sbin/xrdp-sesman
appuser   1124  0.0  1.2  89200 12400 ?        S    02:45   0:00 /usr/bin/pulseaudio
message+   612  0.0  0.8  12400  8120 ?        Ss   02:45   0:00 /usr/bin/dbus-daemon --system

A full graphical desktop environment was running quietly in the background on a headless cloud server.

Xorg, xfce4-session, xfwm4, xrdp, and even pulseaudio were initialized and running under systemd.

Why was a graphical desktop running on a production Linux node?

This happens far more often than engineers realize. Many generic hosting providers offer pre-baked OS templates labeled "Ubuntu Desktop" or "Developer Edition" that include XFCE and XRDP out of the box. In other cases, a junior engineer or contractor installs a GUI package like xrdp or xfce4 to troubleshoot a problem visually, leaves the services enabled, and forgets about them.

Let us add up the resident memory footprint of those idle desktop daemons:

Xorg: 48.1 MiB
xfce4-session: 39.1 MiB
xfwm4: 34.2 MiB
xrdp and xrdp-sesman: 37.5 MiB
pulseaudio: 12.4 MiB
Total wasted physical RAM: 171.3 MiB

On a machine with 987 MiB of total memory, 171 MiB represents over 17% of the entire hardware capacity. That memory was locked into physical RAM frames, completely useless, waiting for an RDP login that would never arrive.

Purging the Desktop Stack

If your server does not have a physical monitor attached, it has no business running X11, display managers, or audio daemons.

First, stop and disable the XRDP and display manager services immediately:

sudo systemctl stop xrdp xrdp-sesman lightdm 2>/dev/null || true
sudo systemctl disable xrdp xrdp-sesman lightdm 2>/dev/null || true

Next, purge the packages completely to reclaim disk space and prevent future package updates from re-enabling them:

sudo apt-get purge -y xrdp xfce4* lightdm pulseaudio x11-common
sudo apt-get autoremove --purge -y
sudo apt-get clean

Finally, tell systemd to boot strictly into the multi-user console target rather than the graphical target:

sudo systemctl set-default multi-user.target

Checking memory immediately afterward showed our available RAM jump from 285 MiB to 448 MiB.

We had recovered over 160 MiB of clean memory in two minutes, but the server was still missing its most critical safety feature: swap.

Building a 2GB Emergency Swap Architecture

There is a widespread misconception across cloud forums that modern Linux servers running on SSDs or NVMe drives should never use swap. People claim that swap causes SSD wear or introduces unacceptable latency.

That advice is completely wrong for low-memory cloud instances.

Swap is not meant to serve as cheap, slow RAM for active computing. If your active application working set exceeds physical RAM and constantly reads from swap, your performance will suffer regardless of the disk type.

The true purpose of swap is twofold:

An emergency escape valve: It gives the kernel a buffer to absorb memory spikes (such as an npm install, a database migration, or a sudden burst of HTTP requests) without triggering the OOM killer.
A parking lot for dormant memory: Operating system daemons, initialization code, and idle processes often allocate memory during startup that they never touch again. Moving those cold pages to swap frees up expensive physical RAM for your active application heap and filesystem cache.

On a 1GB cloud VPS, running without swap is negligence. We need a 2GB swapfile.

Creating the Swapfile Safely

You can allocate a swapfile using fallocate or dd. While fallocate is faster because it manipulates filesystem metadata directly, some virtualized filesystems or copy-on-write systems (like Btrfs) can create fragmented files or sparse allocation holes that the kernel refuses to swap to.

To ensure contiguous, reliable allocation on any cloud block storage, use dd:

sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress

This writes exactly 2,048 megabytes of zeroed blocks to /swapfile.

Locking Down Permissions

A swapfile contains raw dumps of kernel and process memory. That memory contains database credentials, API tokens, session cookies, and private encryption keys. If regular users can read this file, your security model is destroyed.

Set strict read-and-write permissions for the root user only:

sudo chmod 600 /swapfile

Verify that the permissions show -rw-------:

ls -l /swapfile

Formatting and Enabling Swap

Format the file with the Linux swap header structure:

sudo mkswap /swapfile

The command will output the swap space version and its unique UUID:

Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
no label, UUID=a4891b6f-7b89-4e89-a212-098276f512c0

Now, activate the swapfile:

sudo swapon /swapfile

Verify that the kernel has recognized the new swap capacity:

swapon --show

NAME      TYPE  SIZE USED PRIO
/swapfile file    2G   0B   -2

Ensuring Persistence Across Reboots

If the server reboots, /swapfile will not activate automatically unless it is registered in /etc/fstab.

Add the entry to /etc/fstab:

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Verify that /etc/fstab is syntactically correct and mounts cleanly:

sudo mount -a

Our low-memory instance now had a 2GB safety cushion. But leaving swap with default kernel parameters introduces another dangerous failure mode: disk thrashing.

Taming Kernel Swappiness and Cache Pressure

By default, the Linux kernel ships with vm.swappiness = 60.

Swappiness is a kernel parameter that controls the balance between reclaiming page cache (file-backed memory) and swapping out anonymous memory (application data). The value ranges from 0 to 200 on modern kernels (historically 0 to 100).

When vm.swappiness = 60, the kernel is relatively aggressive about pushing anonymous memory to swap even when there is plenty of physical RAM available.

On a desktop workstation or a database server with 64GB of RAM, this behavior helps keep the filesystem cache large and responsive. But on a 1GB cloud VPS running on budget shared block storage, an aggressive swappiness value is disastrous.

When a background task triggers minor disk I/O, the kernel eagerly moves active Node.js or Python runtime pages into swap. The next time an incoming HTTP request hits your web app, the process stalls while the kernel reads its pages back from virtualized disk into RAM.

Your API response times spike from 25 milliseconds to 1,800 milliseconds.

We do not want the kernel swapping application code during routine operations. We want swap utilized strictly as a safety reserve when physical memory is genuinely exhausted.

To achieve this, we set vm.swappiness = 10.

Tuning VFS Cache Pressure

Another critical sysctl parameter for low-memory systems is vm.vfs_cache_pressure.

This parameter controls the kernel's tendency to reclaim directory entries (dentries) and inode objects from memory compared to page cache and swap. The default value is 100.

On cloud instances with limited I/O operations per second (IOPS), constantly dropping directory and inode caches forces the operating system to re-read file metadata from slow storage over and over.

Lowering vm.vfs_cache_pressure to 50 tells the kernel to retain directory structures and inode metadata in memory longer, speeding up filesystem lookups and lowering CPU wait times.

Applying Persistent Sysctl Rules

Create a dedicated configuration file inside /etc/sysctl.d/:

sudo nano /etc/sysctl.d/99-swappiness.conf

Insert the following parameters:

# Reduce aggressive swapping to preserve latency on small cloud nodes
vm.swappiness = 10

# Retain directory and inode cache longer to prevent disk read stalls
vm.vfs_cache_pressure = 50

Apply the new parameters immediately without rebooting:

sudo sysctl --system

Verify the active values:

cat /proc/sys/vm/swappiness
cat /proc/sys/vm/vfs_cache_pressure

With vm.swappiness = 10, the kernel keeps your active processes pinned to fast physical RAM. It will only touch the 2GB swap partition when available RAM drops below 10-15%, preventing premature disk thrashing while still protecting the node from OOM panics.

The Silent Memory Killer: Unbounded Docker Logs

While inspecting our second low-memory VPS, I noticed a baffling symptom: physical RAM was stable, but swap usage had bloated to 538 MiB.

What was consuming hundreds of megabytes of swap on a system running just three small Docker containers?

The answer was hiding inside /var/lib/docker/containers/.

By default, Docker uses the json-file logging driver. Unless you explicitly configure log rotation flags, Docker has no maximum file size limit. Every console.log(), access record, and error stack trace output by your containers is appended to a JSON file on disk forever:

/var/lib/docker/containers/<container-id>/<container-id>-json.log

Let us check the sizes of those log files on disk:

sudo du -sh /var/lib/docker/containers/*/*-json.log

On our secondary node, a single container had generated an 8.4GB log file over four months:

8.4G    /var/lib/docker/containers/a7f10b28.../a7f10b28...-json.log

How does an 8GB log file on disk consume memory and swap?

First, whenever a monitoring agent, log shipper, or developer runs docker logs --tail 100 <container>, the Docker daemon attempts to read and parse the giant JSON file.

Second, the Linux kernel allocates page cache buffers to service these massive file reads. As these buffers expand, the kernel feels memory pressure and pushes dormant container memory into swap.

Third, if the disk space fills up completely, system daemons fail to write temporary files, SQLite databases lock up, and memory allocations fail across the board.

Configuring Global Log Rotation in Docker

You do not need to configure log limits manually inside every single docker-compose.yml file. You can enforce global rotation policies across all containers at the daemon level.

Edit or create /etc/docker/daemon.json:

sudo nano /etc/docker/daemon.json

Add the following configuration:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

This configuration tells Docker that no container log file may exceed 10 megabytes. Docker will keep a maximum of 3 rotated files (.log, .log.1, .log.2), capping total log storage per container at 30 megabytes.

Reload the Docker daemon configuration safely without stopping active containers:

sudo systemctl reload docker

Safely Truncating Existing Bloated Logs

Reloading Docker enforces the 10MB limit on newly rotated files, but it does not shrink existing multi-gigabyte log files already open on disk.

Do not delete the .log files with rm. If a running container has an open file descriptor pointing to that file, deleting it leaves the file descriptor open in an unlinked state. The disk space will not be freed until the container stops.

Instead, truncate the files in place:

sudo sh -c 'for f in /var/lib/docker/containers/*/*-json.log; do truncate -s 0 "$f"; done'

The truncate -s 0 command zeros out the contents instantly while keeping the existing file descriptor valid.

Within seconds, gigabytes of disk space were freed, kernel buffer pressure dissipated, and swap usage plummeted.

Enforcing Container Memory Limits and Protecting SSH

If you run Docker containers on a 1GB host without memory constraints, a single runaway process inside a container can consume every byte of RAM and crash the underlying host.

Every container running on a low-memory VPS must have explicit resource limits defined in its docker-compose.yml file.

Here is a hardened configuration for our Node.js service:

version: "3.8"

services:
  web-app:
    image: node:20-alpine
    restart: unless-stopped
    command: node server.js
    environment:
      - NODE_ENV=production
      # Limit V8 heap allocation to 192MB to avoid OOM crashes
      - NODE_OPTIONS=--max-old-space-size=192
    deploy:
      resources:
        limits:
          cpus: "0.75"
          memory: 256M
        reservations:
          memory: 128M

Notice the combination of --max-old-space-size=192 and memory: 256M.

If you only set the Docker memory limit without configuring Node.js, the V8 garbage collector will assume it has access to the full host memory. V8 will postpone garbage collection until the container hits the 256MB threshold, causing Docker to kill the container.

By setting max-old-space-size below the container limit, V8 aggressively runs garbage collection when heap usage reaches 192MB, keeping the process well within its 256MB boundary.

Protecting SSH from the OOM Killer

If an unexpected memory spike occurs despite your limits, you still need to access the server over SSH to troubleshoot.

By default, the Linux OOM killer can choose sshd as a victim if it needs to free memory. If sshd is terminated, you are locked out of your server and forced to perform a hard reboot.

You can protect the SSH daemon by adjusting its OOM score modifier: /proc/<pid>/oom_score_adj.

The score ranges from -1000 (never kill) to +1000 (always kill first).

Create a systemd service override for SSH:

sudo mkdir -p /etc/systemd/system/ssh.service.d/
sudo nano /etc/systemd/system/ssh.service.d/override.conf

Add the following configuration:

[Service]
OOMScoreAdjust=-500

Reload systemd and restart the SSH daemon:

sudo systemctl daemon-reload
sudo systemctl restart ssh

Verify that the active sshd process has the updated score:

cat /proc/$(pgrep -f "/usr/sbin/sshd" | head -n 1)/oom_score_adj

The output will confirm -500.

Even if your web application consumes memory and triggers the OOM killer, the kernel will sacrifice other processes before touching SSH, allowing you to log in and diagnose the issue cleanly.

The Results: Before vs. After Telemetry

After applying these four optimizations, the change in server stability and memory availability was immediate and dramatic.

Here is the operational comparison on our primary 1GB cloud node:

Initial State (Vulnerable to OOM):
Physical RAM Total: 987 MiB
Available Memory: 285 MiB (28.8% free)
Swap Space Total: 0 B
Swap Space Used: 0 B
Idle Background Load: 0.45
OOM Outages: 3 panics in 48 hours
Rogue Daemons: Xorg, XFCE4, XRDP consuming 171 MiB

Final Hardened State:
Physical RAM Total: 987 MiB
Available Memory: 531 MiB (53.8% free)
Swap Space Total: 2,048 MiB (2.0 GiB)
Swap Space Used: 18 MiB (dormant pages only)
Idle Background Load: 0.04
OOM Outages: 0 panics over 90+ consecutive days
Rogue Daemons: 0 (pure headless multi-user console)

Available memory jumped from 285 MiB to 531 MiB. We recovered 246 MiB of clean physical RAM, nearly doubling our available memory headroom on the exact same budget cloud hardware.

An Interesting Fact About the Linux OOM Killer

The Linux kernel does not choose its victims randomly. When memory exhaustion triggers the OOM killer, the kernel evaluates every runnable process using the badness() function located in mm/oom_kill.c.

The algorithm calculates an integer score between 0 and 1000 for each candidate process:

points = (process_rss + page_table_pages + swap_pages) * 1000 / total_ram_pages

The process using the highest percentage of memory receives the highest baseline score.

However, the kernel adds a protective heuristic for privileged tasks: if a process runs as root (has_capability_noaudit(p, CAP_SYS_ADMIN)), the algorithm subtracts 30 points from its score. The kernel gives system daemons a slight advantage over regular user processes.

Finally, the kernel adds the value of /proc/<pid>/oom_score_adj directly to the calculated points.

If a process has an oom_score_adj of -1000, the calculation returns 0 immediately, granting that process absolute immunity from the OOM killer. If you set it to +1000, that process becomes the immediate sacrifice the moment memory runs dry.

The 5-Minute Hardening Checklist for 1GB VPS Instances

Whenever you provision a new low-spec cloud VPS, run through this five-minute checklist before deploying your application workloads:

Check for graphical bloat: Run ps aux --sort=-%mem | head -n 10. If you see xorg, xrdp, or xfce, purge them immediately and switch to multi-user.target.
Provision a 2GB swapfile: Allocate with dd, set permissions to chmod 600, format with mkswap, and persist in /etc/fstab.
Set swappiness to 10: Create /etc/sysctl.d/99-swappiness.conf with vm.swappiness = 10 and vm.vfs_cache_pressure = 50.
Enforce Docker log rotation: Configure max-size: 10m and max-file: 3 in /etc/docker/daemon.json and reload the daemon.
Protect SSH: Set OOMScoreAdjust=-500 in a systemd override for ssh.service so you never get locked out during an incident.

Have you ever logged into a budget cloud VPS only to discover a rogue desktop environment or unrotated Docker logs consuming half your memory? What is the strangest daemon you have discovered running on a production node?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Sharing this article with fellow Linux and DevOps engineers.
Connecting on LinkedIn and GitHub to discuss cloud infrastructure and Linux internals.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

Hunting the Ghost in the Machine: Why Our Linux Servers Kept Spiking to 100% CPU (And Why the Monitoring Script Was Lying)

Asep Sayyad — Sun, 18 Oct 2026 05:16:28 +0000

A deep dive into CPU delta calculations, isolated Docker metric samplers, and disabling rogue systemd timers.

The Telegram notification arrived at 03:14 AM.

ALERT: [prod-app-02] CPU utilization is at 100.0% (Threshold: >90.0% for 1m)

I rolled out of bed, grabbed my laptop, opened a terminal, and authenticated into the instance over SSH. I expected to find a runaway process or a denial of service attack consuming every compute cycle.

I ran htop.

Nothing was happening. The 15-minute load average sat at 0.12. Idle CPU hovered around 98.4%. The memory graph was flat. Aside from the standard background noise of systemd-journald and our container runtime, the machine was basically asleep.

I closed my laptop and went back to sleep. Two hours later, my phone buzzed again with the exact same alert on the exact same server.

When you log into a server that your monitoring stack claims is on fire and find an operating system that is barely breathing, you have a ghost in your system. The problem was not that our workload was overwhelming the hardware. The problem was that our monitoring daemon was calculating CPU usage incorrectly, our Docker healthcheck was creating its own CPU spikes, and a rogue systemd timer was waking up in the background and locking the disk.

Here is how we tracked down every single one of those issues, fixed the underlying mathematics, and silenced the false alarms for good.

The Illusion of Point-in-Time CPU Usage

Most developers and sysadmins think of CPU usage like a speedometer in a car. You look at the dial right now, and it tells you that you are driving at 65 miles per hour.

A CPU does not work that way.

A modern CPU core is a digital state machine governed by a clock frequency. At any given nanosecond, a processor core is either executing an instruction or it is halted waiting for an interrupt. It is either 100% busy or 0% busy. There is no physical state where a CPU transistor is operating at "42% capacity."

When an operating system reports that CPU usage is at 42%, it is reporting a statistical calculation averaged over a specific window of time:

CPU Usage % = (Time spent executing tasks / Total elapsed time) * 100

Without an elapsed time window (a delta), CPU percentage is a mathematically meaningless number.

Yet countless custom monitoring scripts, bot daemons, and internal status checks make the mistake of attempting to query CPU usage instantaneously. When a script asks the operating system for current CPU load without measuring a time difference between two discrete samples, it ends up reading whatever transient spike or kernel accounting tick happened to coincide with that exact millisecond.

If that millisecond happens to catch a context switch, a JSON parsing loop, or an encrypted TLS handshake, the calculation returns 100%.

Your monitoring daemon sees 100%, panics, and fires an alert webhook to your phone.

How Linux Actually Tracks CPU Cycles

To understand why custom monitoring scripts report garbage numbers, you have to look directly at the pseudo-filesystem where the Linux kernel exposes CPU metrics: /proc/stat.

Inspect /proc/stat on any Linux box:

head -n 1 /proc/stat

The kernel outputs a single line containing raw numbers:

cpu  2450123 12450 678910 89234120 145020 452 12050 0 0 0

These values are not percentages, and they are not seconds. They are cumulative CPU clock ticks, known inside the Linux kernel as jiffies, counted since the system booted. On most Linux distributions (including Ubuntu and Debian on x86_64), the kernel clock runs at CONFIG_HZ=250 or USER_HZ=100, meaning each tick represents 10 milliseconds or 4 milliseconds of processor time.

The ten numeric fields represent where the processor cores spent their ticks:

user: Normal processes executing in user space (applications, web servers, scripts).
nice: Low-priority user processes whose scheduling priority was reduced via nice.
system: Kernel space execution (system calls, driver operations, context switching).
idle: No runnable tasks present; the processor was halted executing an idle loop.
iowait: CPU was idle while at least one I/O request was in flight to a block device.
irq: Servicing hardware interrupts.
softirq: Servicing software interrupts (network packet handling, timer queues).
steal: In virtualized cloud environments, time spent waiting for the hypervisor to allocate a physical CPU core to your virtual machine.
guest: Running a guest operating system inside a virtual machine managed by KVM.
guest_nice: Running a niced guest operating system.

Because these counters continuously increment from the moment the kernel boots, reading /proc/stat a single time tells you only one thing: the total CPU time spent since the server was turned on.

If you read /proc/stat once, sum the fields, and calculate:

Total = user + nice + system + idle + iowait + irq + softirq + steal
Busy = Total - idle
Usage % = (Busy / Total) * 100

You have just calculated the average CPU usage of your server over its entire lifetime since boot, not its current usage.

The Common Script Trap: `top -b -n 1`

When engineers realize they cannot easily parse /proc/stat with a single command, many take a dangerous shortcut in their Bash or Node.js monitoring daemons:

# DO NOT USE THIS FOR INSTANTANEOUS METRICS
CPU_USAGE=$(top -b -n 1 | grep "Cpu(s)" | awk '{print $2 + $4}')

This is one of the most common anti-patterns in Linux system administration.

If you check the man page for top, you will find a critical warning: the first iteration of top (-n 1) always displays values averaged since the last system boot.

It does not measure the current system load. To get a real measurement from top, you must run at least two iterations with a delay between them:

# Correct measurement using top
top -b -n 2 -d 1 | grep --line-buffered "Cpu(s)" | tail -n 1 | awk '{print 100 - $8 "%"}'

The second iteration computes the delta across the one-second delay (-d 1), producing a meaningful number. But running a full top process every five seconds inside a monitoring cron job is heavy, allocates process table slots, and wastes memory.

We needed a clean, native solution inside our Node.js monitoring daemon.

Fixing the Metric Sampler in Node.js

Our infrastructure monitoring service ran a lightweight Node.js daemon that reported health stats to our Telegram alerting pipeline.

When we opened the daemon codebase, we found the bug immediately. Whenever the reporting cron ticked, it called a function that looked roughly like this:

// THE BROKEN IMPLEMENTATION: Single-point snapshot
const os = require('os');

function getCpuUsageInstant() {
  const cpus = os.cpus();
  let totalIdle = 0;
  let totalTick = 0;

  for (const cpu of cpus) {
    for (const type in cpu.times) {
      totalTick += cpu.times[type];
    }
    totalIdle += cpu.times.idle;
  }

  // BUG: This compares cumulative ticks to cumulative ticks since boot!
  // Or worse, when run with a tiny micro-sleep, floating-point jitter creates 100% spikes.
  return (1 - totalIdle / totalTick) * 100;
}

Because os.cpus() in Node.js wraps the uv_cpu_info function from libuv (which reads /proc/stat directly), querying it without comparing it to an earlier sample produced wild, unreliable results.

Worse, when the previous engineer tried to fix this by adding a tiny 50-millisecond delay between two os.cpus() calls, the measurement window was so microscopic that any minor kernel context switch happening during those 50ms swung the math from 0% to 100%.

We rewrote the metric collector to run an isolated, continuous background sampling loop with a fixed 2000ms delta window.

Here is the exact production implementation in metrics.js:

// THE ROBUST DELTA SAMPLER: metrics.js
const os = require('os');

let lastCpuSnapshot = null;
let currentCpuPercent = 0.0;

function takeCpuSnapshot() {
  const cpus = os.cpus();
  let idle = 0;
  let total = 0;

  for (let i = 0; i < cpus.length; i++) {
    const times = cpus[i].times;
    idle += times.idle;
    total += (times.user + times.nice + times.sys + times.idle + times.irq);
  }

  return { idle, total };
}

// Background sampler running every 2000ms
function startCpuSampler(intervalMs = 2000) {
  lastCpuSnapshot = takeCpuSnapshot();

  setInterval(() => {
    const current = takeCpuSnapshot();

    const idleDelta = current.idle - lastCpuSnapshot.idle;
    const totalDelta = current.total - lastCpuSnapshot.total;

    if (totalDelta > 0) {
      const busyPercent = ((totalDelta - idleDelta) / totalDelta) * 100;
      // Clamp between 0.0 and 100.0 to prevent floating precision anomalies
      currentCpuPercent = Math.max(0.0, Math.min(100.0, parseFloat(busyPercent.toFixed(2))));
    }

    lastCpuSnapshot = current;
  }, intervalMs);
}

function getSystemMetrics() {
  return {
    cpuPercent: currentCpuPercent,
    freeMemMB: Math.round(os.freemem() / (1024 * 1024)),
    totalMemMB: Math.round(os.totalmem() / (1024 * 1024)),
    uptimeHours: (os.uptime() / 3600).toFixed(1)
  };
}

module.exports = {
  startCpuSampler,
  getSystemMetrics
};

By decoupling the measurement loop from the alert check, the daemon stopped taking ad-hoc, microscopic snapshots. The alert loop simply read the pre-computed, rolling 2-second moving average.

This eliminated the phantom mathematical spikes. But our investigation was only half finished.

Even with the new delta sampler running, we noticed that every two minutes, our CPU usage genuinely jumped from 2% to 85% for a brief 3-second pulse.

The monitoring math was now accurate. The remaining spikes were real.

The Docker Healthcheck Trap: Killing Next.js With SSR

We inspected running processes on the server using pidstat to catch whatever process was consuming CPU cycles:

pidstat -u 1 10

pidstat samples process activity every 1 second for 10 iterations. During the next micro-spike, the output lit up:

11:42:15 AM   UID      PID    %usr %system  %guest   %wait    %CPU   CPU  Command
11:42:16 AM  1001    24810   82.00   14.00    0.00    1.00   96.00     1  node /app/server.js

Our primary application container was a Next.js production build running under Node.js. Why would a web service with virtually zero traffic spike to 96% CPU like clockwork?

We opened docker-compose.yml and found the culprit under the healthcheck directive:

# THE WRONG WAY: Hammering the root SSR route
services:
  web:
    image: our-registry/web-frontend:v1.4.2
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:3000/"]
      interval: 15s
      timeout: 5s
      retries: 3

Notice what wget was requesting: http://127.0.0.1:3000/.

In Next.js, the root path (/) is not a static HTML file sitting on an Nginx webroot. It is a dynamic React server component. Every time Docker executed that healthcheck probe:

wget established a TCP handshake with Node.js on port 3000.
Next.js received the HTTP GET request for /.
The server runtime triggered a complete Server-Side Rendering (SSR) cycle.
Next.js rendered the React component tree into HTML strings.
It evaluated layout wrappers, serialized hydration payloads, and encoded headers.
It streamed the complete 85-kilobyte homepage response back to wget.
wget piped the payload to stdout (discarded) and exited.

On a modest cloud VPS with 1 or 2 virtual CPU cores, executing a full React SSR pipeline takes 200 to 400 milliseconds of concentrated CPU time.

Because Docker was repeating this healthcheck every 15 seconds, and because our monitoring script was checking utilization around the same time, the monitoring daemon routinely caught the server in the middle of rendering its own homepage for an internal health probe.

The server was not under load from real users. The server was spending all its energy proving to Docker that it was alive.

The Fix: Dedicated Zero-Overhead Healthcheck Route

Never point container healthchecks at server-rendered web pages, authenticated routes, or complex application handlers.

A healthcheck should do exactly one thing: verify that the HTTP process event loop is responsive and that socket connections are accepted.

In Next.js (using the App Router), we created a minimal API route at app/api/health/route.ts:

// app/api/health/route.ts
export const dynamic = 'force-dynamic';

export async function GET() {
  return new Response(JSON.stringify({ status: 'ok', uptime: process.uptime() }), {
    status: 200,
    headers: {
      'Content-Type': 'application/json',
      'Cache-Control': 'no-store, no-cache, must-revalidate',
    },
  });
}

If you are using the Pages Router, create pages/api/health.ts:

// pages/api/health.ts
import type { NextApiRequest, NextApiResponse } from 'next';

export default function handler(req: NextApiRequest, res: NextApiResponse) {
  res.status(200).json({ status: 'ok', uptime: process.uptime() });
}

Then we updated docker-compose.yml to target the lightweight API route:

# THE RIGHT WAY: Ultra-lightweight JSON endpoint
services:
  web:
    image: our-registry/web-frontend:v1.4.3
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3000/api/health | grep -q 'ok' || exit 1"]
      interval: 30s
      timeout: 3s
      start_period: 10s
      retries: 3

We also increased the interval from 15 seconds to 30 seconds. A container does not need to be interrogated four times every minute when a 30-second window is more than fast enough for load balancers.

The impact was immediate:

Before (SSR /): 320ms execution time, 94% single-core CPU utilization during request, 85 KB response payload.
After (/api/health): 1.2ms execution time, 0.01% CPU consumption, 42 bytes response payload.

The container was now healthy, quiet, and cool.

Killing Rogue Systemd Timers: The `fwupd` Nightmare

With the monitoring daemon patched and the Docker healthchecks optimized, false alerts dropped by 95%.

Yet once every day or two, an alert still slipped through. This time, however, pidstat showed that Node.js was completely blameless. The culprit was an operating system utility: fwupd-refresh.

What is fwupd?

It is the Linux Vendor Firmware Service (LVFS) daemon. Its job is to detect motherboard hardware, query online repositories for updated firmware (such as BIOS updates, Thunderbolt dock firmware, or SSD controller patches), and notify the system administrator.

On a physical Dell PowerEdge rack server or a ThinkPad laptop, fwupd is a useful utility.

On a virtualized cloud VPS running on AWS, DigitalOcean, Hetzner, or Linode, it is completely useless. A guest virtual machine running inside KVM or Xen does not have access to physical motherboard hardware. It cannot flash BIOS firmware, and it never will.

Yet on standard Ubuntu 22.04 and Debian cloud images, fwupd and its associated systemd timer are installed and enabled by default.

How `fwupd-refresh.timer` Causes Spikes

Let us see what timers are scheduled to run on your Linux system:

systemctl list-timers --all

Look for fwupd-refresh.timer. When this timer triggers, systemd launches fwupd-refresh.service.

On a resource-constrained 1 GB or 2 GB RAM cloud instance, fwupd-refresh does the following:

It awakens and initializes internal hardware bus inspection libraries.
It downloads remote firmware metadata archives (often 20MB to 50MB of compressed XML).
It decompresses these large cabinet archives in memory using libxml2.
It parses tens of thousands of XML nodes into an in-memory database to cross-reference against device GUIDs that do not exist on virtual machines.
It writes temporary cache databases to /var/lib/fwupd/.

On a virtual machine with single-thread CPU constraints and slow shared storage, this metadata decompression pins the CPU at 100% and saturates disk I/O for 15 to 45 seconds.

The Remediation: Disable and Mask Unnecessary Timers

If your server is a virtual cloud instance, disable and mask fwupd immediately:

# Stop running services
sudo systemctl stop fwupd-refresh.timer fwupd-refresh.service fwupd.service

# Disable them from starting on boot
sudo systemctl disable fwupd-refresh.timer fwupd-refresh.service fwupd.service

# Mask the services so other packages cannot trigger them
sudo systemctl mask fwupd-refresh.service fwupd.service

Masking points the systemd unit symlink to /dev/null. Even if another package (like an apt hook) tries to invoke fwupd, systemd refuses to launch it.

Other Background Services Worth Auditing

While inspecting systemd timers, audit these other common cloud culprits:

# Check all active timers
systemctl list-timers --no-pager

apt-daily.timer and apt-daily-upgrade.timer: Automates package index updates. On managed production nodes where updates are controlled via Ansible or CI/CD pipelines, unannounced background apt runs lock /var/lib/dpkg/lock-frontend and spike CPU. If you manage updates manually, disable these timers.
man-db.timer: Rebuilds the manual page index cache. On a headless container host, you do not need an automated job burning CPU cycles indexing man pages you will never read.
mlocate.timer or plocate.timer: Periodically indexes the entire filesystem for locate. On a server running Docker with hundreds of thousands of overlayfs files, this job causes severe disk thrashing.

To safely disable man-db indexing:

sudo systemctl stop man-db.timer
sudo systemctl disable man-db.timer
sudo systemctl mask man-db.service

Live Patching Inside Docker: Volume Mount Workarounds

During this debugging session, our monitoring daemon was running as a containerized service alongside our web apps. We needed to test changes to metrics.js and verify our delta calculations against live system ticks without tearing down our entire production stack or triggering false downtime.

In development environments, developers often use volume bind mounts. In this specific incident, having a volume bind mount configured in our docker-compose.yml allowed us to hot-patch the monitoring daemon live:

services:
  system-monitor:
    image: node:20-alpine
    working_dir: /app
    volumes:
      - ./monitor-src:/app/src:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
    environment:
      - NODE_ENV=production
      - HOST_PROC=/host/proc
    command: ["node", "src/index.js"]
    restart: always

Because ./monitor-src was mounted directly into /app/src, we were able to update the delta calculation logic directly on disk.

A simple docker kill -s SIGHUP <container_id> or a process manager restart inside the container reloaded the Node.js script in 80 milliseconds without dropping the underlying network namespace or interrupting surrounding containers.

Once verified over a 24-hour baseline, we baked the final code into a versioned container image and removed the host bind mount for production immutability.

The Linux Performance Diagnostic Toolbox

When a server claims to be at 100% CPU, do not restart it immediately. Restarting destroys the volatile process state and clears kernel counters, leaving you blind to the root cause.

Run through this diagnostic sequence in order:

1. Check System Wide Queue Pressure (`vmstat`)

vmstat 1 5

The output gives you immediate ground truth:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 812404  45120 412890    0    0     0    12  320  450  2  1 97  0  0

Focus on four specific columns:

r (runnable): Number of processes waiting for CPU time. If r exceeds your core count (e.g. r=8 on a 2-core machine), you have true CPU starvation.
b (blocked): Processes sleeping in uninterruptible disk sleep. If b is high, your "CPU spike" is actually disk I/O lockup.
id: Percentage of CPU time spent idle.
wa: Percentage of CPU time spent waiting on disk or network storage.

2. Identify the Culprit Process (`pidstat`)

# Track active CPU consumers every 1 second
pidstat -u 1 5

Unlike top, which sorts dynamically and rearranges lines while you are trying to read, pidstat outputs sequential timestamped lines. It shows you the exact command, PID, and whether the time was spent in user space (%usr) or kernel space (%system).

3. Check for Interrupted Hardware Events (`mpstat`)

If you have sysstat installed, inspect core distribution:

mpstat -P ALL 1 3

This reveals whether an interrupt storm (%irq or %soft) is pinning a single core while the remaining cores sit completely idle. A broken network driver or misconfigured packet queue often pins CPU 0 at 100% softirq while total system usage looks deceptively moderate.

An Interesting Fact About Linux CPU Accounting

Linux does not track CPU percentages anywhere in kernel memory. The kernel's scheduler tracks only elapsed time in nanoseconds per task via runtime structures (se.sum_exec_runtime inside the Completely Fair Scheduler / EEVDF).

When /proc/stat is read, the kernel translates those internal nanosecond clocks into legacy architecture ticks defined by USER_HZ (standardized to 100 ticks per second on modern POSIX systems).

If a 32-bit Linux system runs continuously without a reboot for approximately 497 days, a 32-bit jiffies counter wraps around from 4,294,967,295 back to 0. In the late 1990s and early 2000s, this integer overflow routinely caused unpatched monitoring daemons and uptime utilities to crash or report that systems were operating at negative 10,000% CPU utilization.

Summary Checklist: Eliminating False 100% CPU Spikes

Before you trust an alert claiming your Linux server is pegged at maximum capacity, verify these four fundamentals:

Audit Metric Math: Ensure monitoring scripts compare two discrete snapshots across a defined delta window ($\ge 1000\text{ms}$). Never calculate instantaneous usage from single-point readings.
Beware top -b -n 1: The first iteration of top reports lifetime averages since boot, not current usage. Always discard the first iteration or use /proc/stat deltas.
Inspect Docker Healthchecks: Never probe dynamic, server-rendered web routes like Next.js root (/). Build a dedicated, zero-dependency /api/health endpoint that returns static JSON in under 2ms.
Kill Rogue Cloud Timers: Mask fwupd-refresh.timer on cloud virtual machines where motherboard firmware cannot be flashed. Audit secondary timers like man-db and unmanaged apt-daily services.
Trust the Run Queue Over Percentages: Use vmstat 1 and inspect the r (runnable) queue. If the run queue is below your core count, your system is not suffering from computational exhaustion regardless of what a dashboard says.

Have you ever spent hours chasing a phantom 100% CPU alert on your servers only to discover the monitoring tool was measuring the wrong metric? What was the rogue script or misconfigured healthcheck in your environment?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

Zero-Trust Linux: How I Hardened 4 Cloud Servers Against Brute-Force Attacks in Under an Hour

Asep Sayyad — Sat, 17 Oct 2026 18:47:40 +0000

Zero-Trust Linux: How I Hardened 4 Cloud Servers Against Brute-Force Attacks in Under an Hour

No passwords, no root login, and aggressive Fail2ban jails: transforming noisy, attacked ports into impenetrable fortresses.

Spin up a fresh Linux virtual machine on any major cloud provider, assign it a public IPv4 address, and wait ten minutes.

You do not need to advertise the IP address. You do not need to register a domain name. Within moments, automated scanners, compromised routers, and botnets will locate port 22 and begin hammering your operating system with dictionary attacks.

I recently audited four cloud VPS instances deployed across different hosting providers and data centers. These servers ran various workloads, from reverse proxies to microservice APIs. When I opened the authentication logs on the first server, the volume of automated aggression was staggering.

In a twenty-four-hour window, a single machine recorded over fourteen thousand failed authentication attempts. Scripts from hundreds of distinct IP subnets were cycling through usernames like root, admin, ubuntu, test, postgres, and deploy. Every few seconds, the OpenSSH daemon was forced to spawn a new process, negotiate a cryptographic handshake, check authentication credentials, and log a failure.

While modern public-key cryptography protects you against brute-force guessing if passwords are disabled, leaving the default OpenSSH configuration untouched is reckless. Unthrottled connection attempts consume CPU cycles, exhaust connection state tables, clutter system journals, and leave the door open if an operator accidentally creates a user account with a weak password.

In less than sixty minutes, I applied a standardized, zero-trust hardening strategy across all four machines. We eliminated password authentication entirely, barred the root account from remote login, tuned OpenSSH connection boundaries, and deployed persistent Fail2ban jails to ban repeat offenders at the firewall layer.

Here is the exact technical blueprint, step by step, with zero fluff.

The Log Evidence: Quantifying the Attack Surface

Before modifying a single configuration file, inspect the scale of unauthorized traffic hitting your server. You can extract this data directly from the systemd journal or the traditional authentication log.

On Debian and Ubuntu systems running modern systemd, query the SSH service journal for failed password attempts recorded since midnight:

journalctl -u ssh -S today | grep "Failed password" | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head -n 10

On CentOS, RHEL, or Rocky Linux systems where the unit is named sshd, query the service like this:

journalctl -u sshd -S today | grep "Failed password" | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head -n 10

Sample output from one of our audited nodes:

   3412 185.220.101.5
   2189 194.26.29.112
   1874 45.148.10.74
   1205 103.152.220.44
    982 91.240.118.234
    841 167.99.142.12

Notice what is happening. A single external IP address attempted more than thirty-four hundred logins in less than twelve hours.

Next, check the usernames these bots are attempting:

journalctl -u ssh -S today | grep "Invalid user" | awk '{print $NF}' | sort | uniq -c | sort -nr | head -n 10

Output:

   4520 admin
   3110 root
   1890 test
   1420 user
   1105 ubuntu
    840 oracle
    610 ftpuser
    490 git

Every single one of these attempts represents an unauthenticated entity executing code against your server. If any local account has an empty password, a default cloud password, or a common dictionary phrase, your machine is compromised.

Why Direct Edits to sshd_config Are an Anti-Pattern

For years, the standard tutorial advice was simple: open /etc/ssh/sshd_config with an editor, change three lines, save the file, and restart the daemon.

In modern production environments, editing /etc/ssh/sshd_config directly is a bad practice.

Whenever your package manager updates the OpenSSH server package during regular security upgrades (apt upgrade or dnf update), the package manager detects that the default vendor configuration file was modified. It stops the automated upgrade or throws an interactive prompt asking whether you want to keep your modified file, accept the package maintainer version, or inspect a diff. In automated CI/CD pipelines, Ansible runs, or unattended upgrades, this conflict can stall updates or accidentally revert your security baseline.

OpenSSH version 8.2 and newer solved this problem by introducing modular configuration directories. Look at the very top of /etc/ssh/sshd_config:

# This is the sshd server system-wide configuration file.
Include /etc/ssh/sshd_config.d/*.conf

The daemon reads files inside /etc/ssh/sshd_config.d/ in alphabetical order before parsing the rest of /etc/ssh/sshd_config. Directives set in earlier drop-in files take precedence.

By isolating your security hardening rules inside a dedicated drop-in file like /etc/ssh/sshd_config.d/99-hardening.conf, you keep vendor defaults pristine. Your custom configuration survives package upgrades cleanly and can be deployed across hundreds of machines using a single file transfer.

The Four Core Hardening Directives

To secure our four cloud servers, we created a single drop-in file that implements four critical OpenSSH directives alongside strict protocol settings.

Here is the exact configuration we deployed to /etc/ssh/sshd_config.d/99-hardening.conf:

# /etc/ssh/sshd_config.d/99-hardening.conf
# Production OpenSSH Hardening Baseline

# 1. Disable remote root login completely
PermitRootLogin no

# 2. Enforce cryptographic public key authentication
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no

# 3. Limit authentication attempts per connection
MaxAuthTries 3

# 4. Limit unauthenticated connection concurrency
MaxStartups 10:30:60

# 5. Disable unused, high-risk protocol features
X11Forwarding no
AllowTcpForwarding no
AllowAgentForwarding no

# 6. Dead connection detection and idle session teardown
ClientAliveInterval 300
ClientAliveCountMax 2

Let us dissect each of these directives and understand how they alter daemon behavior under the hood.

1. PermitRootLogin no

In default cloud installations, providers often set PermitRootLogin prohibit-password. While this allows public-key access only and stops password guessing for the root account, it still allows external attackers to test whether an SSH key can authenticate directly as root.

Setting PermitRootLogin no enforces an absolute barrier. No user can log into the server as root over the network, regardless of the credentials they present.

To administer the machine, an engineer must first authenticate using a dedicated, unprivileged personal account with an SSH key, and run privileged commands with sudo. This creates a reliable audit trail in /var/log/auth.log that ties every privileged action to an individual identity rather than an anonymous root session.

2. PasswordAuthentication no & KbdInteractiveAuthentication no

Disabling password authentication shuts down dictionary attacks at the cryptographic negotiation phase.

When a client initiates an SSH connection, the server sends an SSH_MSG_USERAUTH_REQUEST listing the authentication methods it supports. With PasswordAuthentication no and KbdInteractiveAuthentication no, the server refuses to offer password or keyboard-interactive mechanisms.

If an attacker does not hold a private key corresponding to an authorized public key stored in ~/.ssh/authorized_keys, the server terminates the connection without ever prompting for an input string.

Use modern Ed25519 keys rather than legacy RSA keys. Generate them on your local workstation with:

ssh-keygen -t ed25519 -C "admin@company.internal"

Ed25519 keys are faster, provide stronger cryptographic security, and avoid the mathematical vulnerabilities that have plagued older RSA key lengths.

3. MaxAuthTries 3

By default, OpenSSH allows up to six failed authentication attempts per connection before terminating the TCP socket.

Why lower this to 3?

Many developers and operations engineers maintain dozens of SSH keys inside their local SSH agent (ssh-agent). When an engineer connects to a server, their SSH client automatically attempts to authenticate using every key loaded in memory, one after another. If the client offers six irrelevant keys before reaching the correct key, the server drops the connection with Too many authentication failures.

Setting MaxAuthTries 3 limits how many keys or credentials an incoming connection can test. This forces clients to use explicit key definitions (configured via ~/.ssh/config using the IdentitiesOnly yes directive) while instantly dropping automated scripts that attempt to cycle through multiple stolen credentials in a single TCP session.

4. MaxStartups 10:30:60

When an automated botnet targets port 22, it does not connect one session at a time. It opens hundreds of concurrent, unauthenticated TCP connections simultaneously to consume server memory and file descriptors.

The MaxStartups directive mitigates this behavior. The syntax 10:30:60 defines three operational thresholds:

10: The server allows up to ten unauthenticated connections without restriction.
30: Once unauthenticated connections exceed ten, the server begins dropping new connection attempts randomly with a thirty percent probability.
60: If unauthenticated connections reach sixty, the server drops all subsequent unauthenticated connection attempts immediately.

This simple parameter prevents connection exhaustion attacks from denying legitimate SSH access to engineers during an ongoing brute-force wave.

The Pre-Flight Safety Verification

Before applying these changes to a live remote server, follow the golden rule of remote system administration:

Never reload or restart the SSH service without testing the syntax first, and never close your active terminal session until you have verified a successful secondary login.

If you introduce a typo into an SSH configuration file and restart the daemon, the service will fail to bind to port 22. If your current session disconnects, you are locked out of the machine and must resort to your cloud provider's web console to fix the mistake.

Run the OpenSSH syntax validator:

sudo sshd -t

The -t flag runs OpenSSH in test mode. It reads all included files, validates configuration syntax, checks permission boundaries, and returns cleanly with exit code 0 if everything is valid. If there is a syntax error, it prints the exact file name and line number where the issue occurred.

Once sshd -t passes with zero errors, reload the daemon:

sudo systemctl reload ssh

Note that we use reload, not restart.

Reloading sends a SIGHUP signal to the parent OpenSSH process. The parent re-reads the configuration files and applies them to future incoming connections, without interrupting or terminating existing, active SSH sessions.

Now, keep your current terminal window open. Open a completely new terminal window on your local machine and test authenticating into the server:

ssh -i ~/.ssh/id_ed25519 deploy@server_ip

If the secondary terminal connects successfully without prompting for a password, your configuration is rock-solid.

Deploying Fail2ban for Persistent Firewall Jails

Eliminating password authentication secures your server from unauthorized entry, but it does not stop the noise.

Even with public key authentication enforced, unauthorized connection attempts still hit port 22, trigger TCP three-way handshakes, spawn child processes, and generate thousands of log lines.

To eliminate this noise, deploy Fail2ban. Fail2ban monitors log files for repeated authentication failures and dynamically adds firewall rules using nftables or iptables to drop traffic from offending IP addresses before the packets ever reach OpenSSH.

Install Fail2ban:

On Debian and Ubuntu:

sudo apt update && sudo apt install -y fail2ban

On RHEL, Rocky Linux, or AlmaLinux:

sudo dnf install -y epel-release && sudo dnf install -y fail2ban

Configuring Fail2ban Jails the Right Way

Fail2ban ships with a default configuration file located at /etc/fail2ban/jail.conf.

Just like OpenSSH, you should never edit jail.conf directly. Future package updates will overwrite your changes. Instead, create a local override file named /etc/fail2ban/jail.d/01-sshd.local or /etc/fail2ban/jail.local.

Create /etc/fail2ban/jail.d/01-sshd.local:

[DEFAULT]
# Whitelist local loopback and your trusted office IP
ignoreip = 127.0.0.1/8 ::1 203.0.113.50

# Default ban duration (1 hour)
bantime = 1h

# Window of time to measure failed attempts (10 minutes)
findtime = 10m

# Maximum failed attempts within findtime before a ban is issued
maxretry = 3

# Enable progressive, exponential bantime for repeat offenders
bantime.increment = true
bantime.factor = 2
bantime.maxtime = 48h

[sshd]
enabled = true
port = ssh
filter = sshd
backend = systemd
maxretry = 3

Notice the key configuration choices we made:

ignoreip: Crucial. Always add your static office IP, home VPN, or bastion host to this line. This prevents you or your team from accidentally banning yourselves if someone runs an automated deployment script with misconfigured keys.
backend = systemd: On modern Linux distributions where logging is handled by systemd-journald, using the systemd backend is significantly faster and more reliable than tailing disk-based flat files in /var/log/auth.log.
bantime.increment = true: This enables exponential backoff. If an IP gets banned once, it is blocked for one hour. If it returns and triggers another ban, the ban duration doubles to two hours, then four, up to forty-eight hours. This silences persistent botnets indefinitely.

Start and enable the Fail2ban service:

sudo systemctl enable --now fail2ban

Verifying Jail Status and Real-Time Bans

Once Fail2ban is running, verify that the SSH jail is actively monitoring the daemon:

sudo fail2ban-client status sshd

Sample output after fifteen minutes of operation on one of our cloud nodes:

Status for the jail: sshd
|- Filter
|  |- Currently failed: 4
|  |- Total failed:     284
|  `- File list:        systemd
`- Actions
   |- Currently banned: 18
   |- Total banned:     37
   `- Banned IP list:   45.148.10.74 91.240.118.234 185.220.101.5 194.26.29.112 ...

Notice the immediate impact. Eighteen abusive IP addresses were banned at the packet filter level.

You can inspect the exact kernel firewall rules created by Fail2ban. On systems using nftables:

sudo nft list ruleset | grep -A 10 "f2b-sshd"

Or on systems running legacy iptables:

sudo iptables -S | grep f2b-sshd

When packets arrive from any banned IP address, the Linux kernel drops them immediately at the network interface layer. The packets never trigger socket allocations, OpenSSH never spawns a process, and your authentication log remains completely quiet.

How to Unban an IP Address

If an engineer accidentally triggers a ban, do not reboot the server or restart the firewall. Unban the specific IP address cleanly using the fail2ban-client:

sudo fail2ban-client set sshd unbanip 198.51.100.25

The tool deletes the corresponding firewall rule instantly and restores network connectivity for that IP.

The 60-Second Hardening Script

To repeat this configuration across multiple machines without typing individual commands, you can assemble the steps into a simple, auditable bash script.

Here is the script we used to roll out this configuration across the remaining three nodes:

#!/usr/bin/env bash
set -euo pipefail

echo "==> Applying OpenSSH Hardening Baseline..."

cat <<'EOF' | sudo tee /etc/ssh/sshd_config.d/99-hardening.conf > /dev/null
# Production OpenSSH Hardening Baseline
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
MaxAuthTries 3
MaxStartups 10:30:60
X11Forwarding no
AllowTcpForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2
EOF

# Validate syntax
sudo sshd -t
echo "OpenSSH syntax valid."

# Reload OpenSSH cleanly
sudo systemctl reload ssh || sudo systemctl reload sshd
echo "OpenSSH reloaded."

echo "==> Configuring Fail2ban SSH Jail..."
cat <<'EOF' | sudo tee /etc/fail2ban/jail.d/01-sshd.local > /dev/null
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1
bantime = 1h
findtime = 10m
maxretry = 3
bantime.increment = true

[sshd]
enabled = true
port = ssh
filter = sshd
backend = systemd
maxretry = 3
EOF

sudo systemctl restart fail2ban
echo "Fail2ban deployed and active."

An Interesting Fact About SSH Port 22

Why does SSH run on port 22 by default?

In July 1995, Tatu Ylönen, a researcher at Helsinki University of Technology, designed the original Secure Shell protocol after a password-sniffing attack compromised his university network.

At the time, the dominant remote login tools were FTP (running on port 21) and Telnet (running on port 23). Ylönen wanted SSH to replace both insecure protocols. He noticed that port 22 between them was unassigned.

He sent an email to Jon Postel, the legendary Internet Assigned Numbers Authority (IANA) administrator who managed global port assignments. Postel approved the request and assigned port 22 to SSH in a single day.

Within weeks, the protocol spread across the nascent internet, and port 22 has been the standard target for network scanners ever since.

Summary and Operating Checklist

Hardening Linux infrastructure does not require complex, invasive tooling or heavy enterprise software agents.

By applying foundational zero-trust principles to OpenSSH, you eliminate the entire vector of credential guessing and brute-force intrusion:

Never edit /etc/ssh/sshd_config directly; use /etc/ssh/sshd_config.d/*.conf drop-ins to survive package upgrades cleanly.
Bar root logins with PermitRootLogin no and mandate unprivileged accounts with sudo.
Enforce cryptographic keys with PasswordAuthentication no and modern Ed25519 pairs.
Limit authentication attempts with MaxAuthTries 3 and throttle scanners with MaxStartups 10:30:60.
Test before applying with sshd -t and prefer systemctl reload ssh over destructive restarts.
Drop repeat offenders at the firewall layer using Fail2ban with incremental backoff.

After applying these settings across our four cloud instances, failed authentication attempts dropped from over fourteen thousand per day to zero. The system journals are readable, CPU usage is flat, and the perimeter is secure.

How many failed authentication attempts is your primary production server logging right now?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

Why a Deleted Linux File Can Still Consume Disk Space

Asep Sayyad — Fri, 16 Oct 2026 16:05:41 +0000

Your root partition is at 100%, you ran rm on a 50GB log file, and df still reports zero free bytes. Here is how inodes, unlinked file descriptors, and kernel references keep ghost data alive, and how to recover disk space without restarting production.

A critical disk alert fires on your phone.

A busy database node or API host just crossed 99% disk capacity on /var. Applications begin failing. MySQL refuses incoming transactions with disk-full errors. Docker containers start getting evicted by the kubelet. System loggers choke on write operations.

You SSH into the struggling server and run a quick scan to find the space hog:

ls -lhS /var/log/app/

-rw-r--r-- 1 appuser appuser  52G Oct 16 19:40 debug.log
-rw-r--r-- 1 appuser appuser 1.2G Oct 16 00:00 access.log
-rw-r--r-- 1 appuser appuser 210M Oct 15 00:00 app.log.1

A rogue debug flag was left enabled in production, and debug.log has blown up to 52 gigabytes.

Relieved that you found the culprit so fast, you run the obvious command:

rm -f /var/log/app/debug.log

The shell prompt returns instantly. You run ls -lh /var/log/app/debug.log to make sure it is gone. The terminal confirms the file no longer exists:

ls: cannot access '/var/log/app/debug.log': No such file or directory

Problem solved, right? You run df -h to verify how much breathing room you just recovered:

df -h /var

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        60G   60G     0 100% /var

The partition is still at 100% capacity. Available space is still 0 bytes.

Confused, you run du to inspect actual directory usage:

du -sh /var

4.8G    /var

du says /var is only consuming 4.8 gigabytes. df insists all 60 gigabytes are completely full.

Where did the other 55 gigabytes go? Did the filesystem corrupt itself? Is the storage hardware failing?

Neither. You just walked straight into one of the most common traps in Linux systems administration: the open unlinked file descriptor.

rm Does Not Delete Files

To understand where your disk space went, you have to unlearn what the rm command actually does.

On Linux, rm does not mean "delete file". It stands for remove, which invokes the unlink() or unlinkat() system call inside the Linux kernel.

Unix filesystems like ext4, XFS, and btrfs do not store file data inside filenames or directories. Storage architecture in Linux is split into three separate layers:

First, Directory Entries (dentries). A directory in Linux is simply a special structured file that maps human-readable names to numbers. When you look at /var/log/app/debug.log, the string debug.log is just a pointer sitting in a directory table. That pointer references an integer called an inode number.

Second, Inodes (Index Nodes). The inode is the actual metadata structure of the file. It stores file ownership (UID and GID), access permissions, file size, modification timestamps, and pointers to the physical data blocks or extents on the storage drive. Crucially, the inode also stores an internal counter named i_nlink (the hard link count). This tracks how many directory entries point to this specific inode.

Third, Data Blocks. These are the raw 4KB blocks on the physical drive where the file contents live.

When you execute rm -f /var/log/app/debug.log, the kernel only performs two specific actions:

// Simplified kernel logic during unlink()
unlink("/var/log/app/debug.log");
// 1. Remove the dentry "debug.log" from the directory
// 2. Decrement the inode link counter: inode->i_nlink--;

The directory entry is erased, so ls cannot find it. The i_nlink counter drops from 1 to 0.

However, the kernel will not free the allocated data blocks on disk back to the filesystem free block bitmap just because i_nlink reached zero.

The kernel tracks a second reference counter: the open file table references.

Whenever an active process opens a file with the open() system call, the kernel creates a struct file entry in its internal system-wide open file table and increments an in-memory reference counter (f_count).

The storage driver adheres to a strict rule:

Disk blocks and inode allocations are only released when both conditions are met:

The hard link count reaches zero (i_nlink == 0).
Every active process closes its file descriptor pointing to that inode (f_count == 0).

If an application (like your Node.js backend, a Java worker, Python daemon, or Nginx) opened /var/log/app/debug.log when it booted up and is still running, that process holds an active file descriptor pointing directly to the inode.

When you ran rm, you severed the directory entry. But the application still holds the open file handle.

The inode remains alive in kernel memory. The data blocks remain marked as active and allocated on your NVMe or SSD drive. The process keeps writing fresh bytes into those unlinked blocks.

You did not delete the file. You only removed its name from the phone book while the phone call was still active.

The Feud Between df and du

Now the conflicting numbers between df and du make complete sense.

These two utilities measure disk space from two completely different perspectives in the kernel.

du stands for Disk Usage. It is a directory tree crawler. When you run du -sh /var, the utility executes opendir("/var") and starts walking through every subdirectory with readdir(). For every filename it encounters, it calls the lstat() or stat() system call to ask the filesystem for the block count allocated to that path.

Because you ran rm, the directory entry for debug.log is gone. There is no filename for du to discover. It skips right past it. As far as du is concerned, those 52 gigabytes do not exist because they have no presence in the directory hierarchy.

df stands for Disk Free. It does not look at directories, filenames, or paths at all.

When you run df -h /var, df issues a single system call directly against the filesystem: statvfs() or statfs().

struct statvfs buf;
statvfs("/var", &buf);
// Reads total blocks, free blocks, and available blocks directly from the superblock

This system call queries the filesystem superblock and block group allocation bitmaps in kernel memory. The superblock tracks the real physical state of the storage partition:

Total block count on the partition
Total allocated blocks currently marked as "in use" in the allocation bitmap
Total free blocks available for new writes
Reserved blocks reserved strictly for the root user

Because your background daemon still has the file open, the filesystem driver continues to mark all 13 million 4KB blocks of that 52GB file as allocated. The superblock counters do not change.

df tells you the physical reality of your storage device. du tells you the visible contents of your directory tree.

When df and du disagree by gigabytes, it almost always means an active process is holding open a deleted file.

Tracking Down the Ghost Files

Finding these unlinked files does not require guessing which daemon is responsible. The Linux kernel exposes every open file handle in userspace.

There are two primary ways to track down these ghost files.

The fastest tool is lsof (List Open Files). You can instruct lsof to filter specifically for files whose hard link count is less than 1:

lsof +L1

The +L1 flag tells lsof to look for open files where link count < 1. If you want to restrict the search to a specific mount point like /var, add the filesystem path:

lsof +aL1 /var

Here is what the output looks like on a production host:

COMMAND     PID    USER   FD   TYPE DEVICE   SIZE/OFF NLINK      NODE NAME
node      24810 appuser   14w   REG    8,2 55834574848     0 144182931 /var/log/app/debug.log (deleted)

The columns lay out the exact crime scene. Under COMMAND and PID, you get the executable (node) and process ID (24810). The FD column shows descriptor 14w, meaning file descriptor 14 opened for writing. Most tellingly, NLINK shows 0, confirming the directory entry was unlinked from disk while SIZE/OFF reveals all 55.8GB sitting in limbo under /var/log/app/debug.log (deleted).

If lsof is not installed on a minimal container or hardened production bastion, you do not need to install anything. You can query the /proc virtual filesystem directly.

Every running process in Linux has a directory at /proc/<PID>/fd/. Inside this directory, the kernel creates symbolic links for every open file descriptor assigned to that process.

When a file is unlinked while still open, the kernel automatically appends the string (deleted) to the symlink target.

You can sweep the entire system in one command:

find /proc/*/fd -ls 2>/dev/null | grep '(deleted)'

144182931  54525952 -rw-r--r--   1 appuser appuser 55834574848 Oct 16 19:40 /proc/24810/fd/14 -> /var/log/app/debug.log\ (deleted)

The output reveals everything: Process ID 24810 holds file descriptor 14, which points to your unlinked 55GB log file.

To inspect what the process is currently doing with that descriptor, you can check its descriptor status in /proc:

cat /proc/24810/fdinfo/14

pos:    55834574848
flags:  0102001
mnt_id: 34
ino:    144182931

The pos field shows the current write offset in bytes. It is sitting at 55.8GB and steadily growing. The daemon is actively writing to a file that you thought you deleted twenty minutes ago.

Freeing the Space Without Killing the Process

Once engineers discover the PID holding the deleted file, their immediate reflex is to kill the process:

kill -9 24810

Do not do this unless you have no other choice.

Killing a production process drops active user websocket connections, terminates inflight database transactions, invalidates local cache pools, and can trigger cascading failure alerts in your monitoring stack. If the process is a core database or monolithic service, restarting it might take ten minutes of cache warm-up time.

There is a clean, zero-downtime way to reclaim every single byte immediately:

Truncate the open file descriptor directly through /proc.

Run this command as root or as the user owning the process:

> /proc/24810/fd/14

Alternatively, you can use the truncate utility:

truncate -s 0 /proc/24810/fd/14

Or redirect from /dev/null:

cat /dev/null > /proc/24810/fd/14

Why does this work?

/proc/<PID>/fd/14 is a magic symlink maintained by the Linux kernel. When you redirect output into this path using >, your shell opens the underlying inode referenced by descriptor 14 with the O_TRUNC flag.

The kernel receives the truncate request, bypasses the missing directory entry, and executes do_truncate() directly against the active inode.

The filesystem driver immediately deallocates the extents and data blocks mapped to that inode and writes them back into the free block bitmap. The file size resets to 0.

Now check df -h /var:

df -h /var

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        60G  4.8G   53G    9% /var

Instantly, 53 gigabytes of storage return to the available pool. The disk-full alerts resolve. And best of all, the application never stopped running.

The O_APPEND Caveat

There is an important technical nuance you must understand before truncating a running file descriptor.

How the application behaves after truncation depends on how it originally opened the file.

Most production logging libraries, syslog daemons, and web servers open log files with the O_APPEND flag:

int fd = open("/var/log/app/debug.log", O_WRONLY | O_CREAT | O_APPEND, 0644);

When a file descriptor has O_APPEND enabled, the Linux kernel guarantees that before every single write() call, the file offset is automatically reset to the current end of the file.

When you truncate the file to 0 bytes, the next write goes to offset 0. The file continues logging normally from byte zero.

However, if an unusual program opened the file without O_APPEND and maintains its own internal byte counter using lseek():

lseek(fd, current_offset, SEEK_SET);
write(fd, buffer, length);

If the internal offset was at 55GB, truncating the inode sets the length to 0, but the application's next write might seek forward to byte 55,000,000,000 and write 100 bytes there.

This creates a sparse file. On modern filesystems, sparse files do not consume physical disk space for unwritten zero-blocks, so your available disk space will still remain free. But reading the file will return gigabytes of null bytes (\0) before reaching the new log lines.

In practice, virtually all standard daemons (Nginx, Apache, Node.js Winston/Bunyan, Python logging handlers, Logstash) use append mode or standard streaming writes, making descriptor truncation completely safe.

Other Ways Linux Hides Deleted and Used Space

Open unlinked file descriptors account for roughly 90% of df vs du discrepancies. But when lsof +L1 returns completely empty and your disk is still full, one of three other subtle filesystem mechanisms is stealing your storage.

1. The Over-Mounted Directory Shadow

This is the sneakier cousin of the open file descriptor problem.

Suppose an automated backup script runs early during server initialization before external storage volumes are attached:

tar -czf /mnt/backups/system-backup.tar.gz /var/data

The backup script dumps 40 gigabytes into /mnt/backups/. At this moment, /mnt/backups/ is just a regular directory residing on your primary / root partition.

Twenty minutes later, a systemd automount unit or storage script mounts an external EBS volume, SAN LUN, or NFS share on top of that mount point:

mount /dev/sdb1 /mnt/backups

What happened to the 40GB archive?

The mount operation did not delete it. It shadowed it.

When a filesystem is mounted on a directory, the kernel redirects all directory lookups to the root inode of the newly mounted device. The original directory contents on the underlying filesystem become completely invisible to ls, du, and find.

du /mnt/backups now measures the secondary disk /dev/sdb1.

Meanwhile, your root filesystem /dev/sda1 is still holding those 40 gigabytes of data blocks, and df -h / reports that root is almost full.

How do you check for shadowed files without unmounting active storage?

Use a bind mount to inspect the underlying root filesystem in an unmasked state:

mkdir -p /mnt/root_inspect
mount --bind / /mnt/root_inspect

A bind mount creates an alternate view of the root filesystem without re-triggering the mount points stacked on top of it.

Now you can inspect the shadowed directory directly:

ls -lh /mnt/root_inspect/mnt/backups/

There is your missing 40GB file. You can safely remove it from /mnt/root_inspect/mnt/backups/, and df on your root partition will instantly drop.

When you are done, clean up the inspection mount:

umount /mnt/root_inspect
rmdir /mnt/root_inspect

2. The Hard Link Survival Trap

Another common misconception is assuming rm frees disk blocks when a file has hard links.

If another engineer or an automated snapshot script created a hard link to your log or data file:

ln /var/log/app/debug.log /opt/archive/debug_backup.log

Both filenames point to the exact same inode number. You can verify this with ls -l or stat:

stat -c "Links: %h Inode: %i File: %n" /var/log/app/debug.log /opt/archive/debug_backup.log

Links: 2 Inode: 8912345 File: /var/log/app/debug.log
Links: 2 Inode: 8912345 File: /opt/archive/debug_backup.log

Notice the link count is 2.

When you run rm /var/log/app/debug.log, the kernel decrements the inode link count from 2 to 1.

Because i_nlink is still greater than zero, the inode is not marked for deletion. Not a single data block is freed. The 50GB file remains fully allocated on disk under the name /opt/archive/debug_backup.log.

If you run rm on a large file and space is not reclaimed even though no process holds it open, check if the file had multiple hard links before deletion:

find / -inum 8912345 2>/dev/null

3. Reserved Filesystem Blocks (The 5% Buffer)

Have you ever seen df -h show that a filesystem is 100% full, but the math does not add up?

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   95G     0 100% /

Total size is 100G. Used space is 95G. Why is Available space sitting at 0 instead of 5G?

By default, the mke2fs utility reserves 5% of total filesystem blocks exclusively for the root user when creating ext2/ext3/ext4 partitions.

The reasoning dates back to early multi-user Unix systems:

It prevents unprivileged user processes from completely filling the disk, which would prevent system daemons (like sshd, syslog, or postfix) from writing state.
It mitigates filesystem fragmentation by ensuring the allocation engine always has contiguous blocks available.

On a 20GB disk created in 2005, a 5% reserve was only 1GB. On a modern 4TB NVMe drive, 5% is a staggering 200 gigabytes of wasted storage that regular service accounts cannot use.

When your application service account hits this 5% wall, its writes fail with ENOSPC: No space left on device, even though physical blocks are empty.

You can inspect and adjust this reserved percentage on a live, mounted ext4 filesystem with tune2fs:

tune2fs -l /dev/sda1 | grep "Reserved block count"

To reduce the reserve from 5% to 1% (or even 0% on a dedicated non-root data partition), run:

tune2fs -m 1 /dev/sda1

This immediately releases 4% of the partition's total capacity back to general users and background services without a reboot.

Exactly One Surprising Fact

The fact that unlink() keeps open files alive until process termination is not an accident, a bug, or an oversight of early filesystem design.

It was an intentional, foundational security and design pattern invented by Ken Thompson and Dennis Ritchie in the early 1970s for Unix.

In Unix, programs frequently needed temporary scratchpad files for sorting, staging intermediate compiler output, or buffering pipeline streams. If a program crashed or was terminated with kill, temporary files littered /tmp and required manual cleanup scripts.

Thompson and Ritchie designed unlink() so that a program could create a file, immediately call unlink() on it, and continue reading and writing to its file descriptor in private memory.

Because the file had no directory entry, no other user on the shared multi-user mainframe could see the file, guess its name, or tamper with its data. And the microsecond the program finished or crashed, the kernel automatically cleaned up every single allocated disk block.

Modern Linux filesystems formalized this exact pattern in Linux kernel 3.11 with the O_TMPFILE flag passed to open():

int fd = open("/tmp", O_TMPFILE | O_RDWR, 0600);

This creates an anonymous, completely invisible inode that never exists in any directory tree from the moment of creation. It consumes disk blocks while open, and vanishes without a trace the moment the descriptor closes.

How to Prevent the Ghost File Trap in Production

Now that you understand the mechanics, here is how to make sure this never catches your team off guard again.

1. Never rm an Active Log File

If a service is currently running, never use rm on its active log files. Always truncate them instead:

# Good: Frees disk space immediately while keeping the file descriptor valid
> /var/log/app/debug.log

# Also good:
truncate -s 0 /var/log/app/debug.log

Truncation preserves the directory entry, resets the file length to zero, and forces the filesystem to reclaim the physical disk blocks instantly. The application continues logging without missing a write or needing a restart.

2. Configure logrotate Correctly

Most ghost file incidents occur because of misconfigured log rotation.

There are two primary paradigms in logrotate:

Approach A: Copy and Truncate (copytruncate)

/var/log/app/*.log {
    daily
    rotate 7
    compress
    missingok
    copytruncate
}

The copytruncate directive instructs logrotate to copy the live log file to a backup name (e.g., debug.log.1), and then truncate the original file in place to 0 bytes.

This is ideal for applications that do not support reloading log file handles (such as third-party binaries or proprietary Java apps). Because the original file is never unlinked, its descriptor remains intact.

Approach B: Move and Signal (create + reload signal)

/var/log/nginx/*.log {
    daily
    rotate 14
    missingok
    compress
    delaycompress
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

In this model, logrotate renames access.log to access.log.1 and creates a fresh empty access.log.

At this exact instant, Nginx is still writing to descriptor pointing to access.log.1.

The postrotate block sends the USR1 signal to Nginx. Nginx intercepts USR1, closes its existing file descriptors, and reopens /var/log/nginx/access.log.

If someone deletes the postrotate section or the signal fails because of an incorrect PID file path, the daemon will hold the rotated or unlinked file open forever.

3. Monitor Unlinked File Descriptors

Do not wait for a 100% disk outage to discover deleted files.

If you use Prometheus and the node_exporter, configure alerts on the divergence between filesystem size and available bytes:

# Alert when unlinked files or reserved blocks consume more than 15% of partition
(node_filesystem_size_bytes - node_filesystem_free_bytes) - (node_filesystem_size_bytes - node_filesystem_avail_bytes)

Alternatively, add a simple periodic cron check or Datadog custom metric that counts the volume of open unlinked files:

lsof +aL1 /var -F s | grep '^s' | cut -c 2- | awk '{s+=$1} END {print s/1024/1024 " MB"}'

The 60-Second Ghost File Triage Playbook

When df reports that a disk partition is full but du cannot find the files, follow this sequence:

First, identify all open files with zero directory links on the affected mount:

lsof +aL1 /mount_point

Second, if lsof is not available, sweep the process descriptors in /proc:

find /proc/*/fd -ls 2>/dev/null | grep '(deleted)' | sort -k7 -n

Third, note the Process ID (PID) and File Descriptor (FD) of the largest unlinked file.

Fourth, reclaim the disk space immediately without killing the process:

> /proc/<PID>/fd/<FD>

Fifth, verify that disk space has returned:

df -h /mount_point

Sixth, if the application supports signal reloading, notify it to close and reopen logs cleanly:

kill -HUP <PID>
# Or for Nginx:
kill -USR1 <PID>

Seventh, if lsof showed zero deleted files, check for shadowed files trapped under an active mount point:

mkdir -p /mnt/temp_root && mount --bind / /mnt/temp_root
du -sh /mnt/temp_root/* | sort -h
umount /mnt/temp_root && rmdir /mnt/temp_root

Wrapping Up

Storage in Linux is governed by reference counting, not desktop trash cans.

When you run rm, you are only removing a label from a directory index. As long as any running process keeps that file handle open, the Linux kernel protects those data blocks from being overwritten.

The next time df and du tell you two conflicting stories during a storage incident, do not reboot the server and do not reach for kill -9. Look at the file descriptors in /proc, truncate the open handle, and get your disk space back in seconds without taking your services down.

What is the largest ghost file you have ever chased down in production, and which daemon was stubbornly refusing to let it go?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

Your Linux Service Is Running but the Application Is Down: Here's Why?

Asep Sayyad — Sun, 11 Oct 2026 05:46:17 +0000

Systemd says your daemon is active and healthy, yet every request fails. Here is what is actually broken inside the kernel and how to pinpoint the real failure in under two minutes.

The most deceptive line in Linux systems administration is Active: active (running).

You get paged in the middle of the night because an ingress proxy is returning 502 Bad Gateway and 504 Gateway Timeout errors to end users. You SSH into the affected node, run a status check on the background daemon, and see a comforting green indicator:

systemctl status payments-api.service

● payments-api.service - Payments Gateway Engine
     Loaded: loaded (/etc/systemd/system/payments-api.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-10-05 04:12:00 UTC; 6 days ago
   Main PID: 18492 (node)
      Tasks: 11 (limit: 4915)
     Memory: 412.4M
        CPU: 1min 22.418s
     CGroup: /system.slice/payments-api.service
             └─18492 /usr/bin/node /opt/payments/dist/server.js

The process ID is present. The memory footprint looks steady. The CPU counters tick upward. Systemd insists that everything is running smoothly.

Then you test the local HTTP endpoint directly on the box:

curl -v http://127.0.0.1:8080/health

The terminal hangs. Ten seconds pass. Thirty seconds pass. Then curl aborts with a connection timeout or an empty reply.

Why does systemd report that your service is running smoothly when the application is completely dead to incoming traffic?

Because systemd is a process supervisor, not an application health inspector.

To systemd and the Linux kernel, a service is active as long as its designated main process ID exists in the kernel task table and has not emitted an exit signal. The kernel does not know that your event loop is frozen, that your socket accept queue is dropping packets, that worker threads are deadlocked on database connections, or that the process hit its file descriptor ceiling three hours ago.

When an active service stops serving requests, one of several specific system-level failures is happening behind that green status message.

Process Liveness vs. Socket Readiness

To understand why this disconnect happens, look at how the Linux kernel separates process execution from network communication.

When systemd launches a unit configured with the default Type=simple, it issues a fork() system call followed by execve(). As soon as the kernel loads the binary into memory and assigns a process ID, systemd marks the unit as active.

Your application, however, is nowhere near ready to handle traffic.

Before an application can service a single network packet, it must execute a sequence of network system calls:

int fd = socket(AF_INET, SOCK_STREAM, 0);
bind(fd, (struct sockaddr *)&addr, sizeof(addr));
listen(fd, backlog);
int client_fd = accept(fd, (struct sockaddr *)&client_addr, &client_len);

Between execve() and the first invocation of accept(), an application often parses gigabytes of configuration data, connects to message queues, runs database schema checks, or pre-allocates memory buffers.

If any of those internal initialization steps block indefinitely, the process remains in memory forever. Systemd sees a running PID and reports green. But the socket either does not exist, was bound to the wrong interface, or has an accept queue that no thread is reading.

The green status tells you that a binary is taking up space in RAM. It tells you nothing about whether the process is capable of servicing work.

Failure 1: Listen Backlog Saturation and the Accept Queue Overflow

This is the most common reason an application stops responding while showing normal CPU usage.

When a client establishes a TCP connection with a server, the Linux kernel handles the initial three-way handshake entirely inside kernel space, without waiting for the application code.

The kernel maintains two distinct queues for every listening socket:

First, the SYN Queue (or incomplete connection queue). When a client sends a TCP SYN packet, the kernel stores the half-open connection here and replies with SYN-ACK. The maximum size of this queue is controlled by the kernel parameter net.ipv4.tcp_max_syn_backlog.

Second, the Accept Queue (or listen backlog). Once the client returns the final ACK packet, the three-way handshake is complete. The connection transitions to the ESTABLISHED state inside the kernel. The kernel then moves the socket into the Accept Queue.

The connection sits in this queue until your application calls accept() or accept4().

Now consider what happens when your application freezes. Maybe the Python Global Interpreter Lock is stuck, a synchronous database driver blocked the main Node.js event loop, or every worker thread in a Java pool is waiting on an external API call without a timeout.

The application stops calling accept().

New connections continue completing the TCP handshake in the kernel. The Accept Queue fills up until it hits the backlog limit defined in the application's listen(fd, backlog) system call.

Once the Accept Queue is completely full (sk_ack_backlog >= sk_max_ack_backlog), the kernel cannot queue any more connections.

What does the kernel do with new incoming connections?

By default, Linux silently drops the final ACK packet from the client. The kernel parameter /proc/sys/net/ipv4/tcp_abort_on_overflow defaults to 0. The kernel pretends it never received the ACK, hoping that the application will clear the backlog before the client gives up.

The client socket thinks the connection is established. It sends its HTTP request bytes. The server kernel ignores them. The client sits waiting until its read timeout fires, producing a 504 Gateway Timeout or a dropped connection.

If tcp_abort_on_overflow is set to 1, the server immediately responds with a TCP Reset (RST) packet, producing an instant Connection reset by peer or 502 Bad Gateway.

You can diagnose an accept queue overflow in five seconds with ss:

ss -lnt '( sport = :8080 )'

Look closely at the output columns:

State      Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN     129    128    0.0.0.0:8080       0.0.0.0:*

On an established TCP socket, Recv-Q represents the bytes received from the network that the application has not yet read from the socket buffer.

On a LISTEN socket, the meaning of these two columns changes completely:

Send-Q displays the maximum listen backlog configured for that socket (the second argument passed to listen(), capped by /proc/sys/net/core/somaxconn).

Recv-Q displays the exact number of connections currently sitting in the kernel Accept Queue waiting for the application to invoke accept().

In the output above, Recv-Q is 129 and Send-Q is 128. The Accept Queue has overflowed. The application is completely deaf.

You can verify how many connections the kernel has dropped due to listen queue overflow by querying the kernel network statistics:

nstat -az TcpExtListenOverflows TcpExtListenDrops

Sample output:

#metric                     value      type
TcpExtListenOverflows       48291      0.0
TcpExtListenDrops           48291      0.0

If those numbers are ticking upward, your service process is alive in the eyes of systemd, but its connection queue is a dead end.

Failure 2: The Binding Blindspot (Localhost, IPv6, and Interface Isolation)

Another common cause of the ghost outage is socket interface mismatch.

An application developer updates a configuration file or deploys a framework upgrade. The application starts cleanly, reports success, and listens on port 8080. But your Nginx reverse proxy running on the same server, or a cloud load balancer querying the node's private VPC IP, gets Connection refused on every request.

Why? The application bound to the loopback interface (127.0.0.1) instead of the wildcard interface (0.0.0.0).

When an application calls bind() with 127.0.0.1, the Linux kernel routes packets to that socket only if they originate from the local loopback adapter lo. If a proxy forwards traffic to the server's private network interface (for example, 10.0.4.12:8080), the kernel inspects the destination IP, finds no socket listening on that specific address, and returns a TCP RST packet immediately.

The reverse proxy logs:

[error] 1421#1421: *89 connect() failed (111: Connection refused) while connecting to upstream, client: 198.51.100.4, server: api.example.com, upstream: "http://10.0.4.12:8080/health"

Check the exact binding address using ss:

ss -tlpn '( sport = :8080 )'

Compare these two outputs:

# Broken: Only accepts connections directed to localhost
LISTEN 0 128 127.0.0.1:8080 0.0.0.0:* users:(("node",pid=18492,fd=19))

# Working: Accepts connections on all IPv4 interfaces
LISTEN 0 128 0.0.0.0:8080   0.0.0.0:* users:(("node",pid=18492,fd=19))

There is an even trickier variant involving IPv6 dual-stack sockets.

When an application binds to the IPv6 wildcard address [::]:8080, Linux can handle both IPv4 and IPv6 traffic over that single socket by mapping IPv4 addresses into the IPv6 space (::ffff:192.0.2.1).

However, this behavior depends on the sysctl setting net.ipv6.bindv6only. If that value is set to 1, or if the application explicitly passed the IPV6_V6ONLY socket option when creating the file descriptor:

int opt = 1;
setsockopt(fd, IPPROTO_IPV6, IPV6_V6ONLY, &opt, sizeof(opt));

The socket will accept IPv6 traffic only. Any client or local proxy attempting to connect via IPv4 127.0.0.1:8080 receives an instant Connection refused, even while ss shows the service listening on [::]:8080.

Verify this behavior by issuing curl calls explicitly over both protocol families:

curl -v -4 http://localhost:8080/health
curl -v -6 http://localhost:8080/health

If the IPv6 command succeeds while the IPv4 command fails, you have isolated the binding trap.

Failure 3: The Uninterruptible Sleep Lockup (D State and Kernel I/O Stalls)

Run a process check with ps and examine the process state column:

ps -eo pid,stat,wchan:20,comm | grep -E "(payments|PID)"

Look at the STAT output:

    PID STAT WCHAN                COMMAND
  18492 D    nfs_wait_bit_uninter payments-api

Notice the letter D.

In Linux, process state D stands for TASK_UNINTERRUPTIBLE.

When a process is in state S (TASK_INTERRUPTIBLE), it is simply sleeping, waiting for a timer or an event, and will wake up immediately if a signal arrives.

When a process enters state D, it is executing a system call inside the kernel and waiting on a hardware or filesystem event that cannot be interrupted. The kernel deliberately blocks all signals from reaching the process while it is in this state.

Even kill -9 (SIGKILL) cannot kill a process in state D. The kernel queues the signal, but it will not deliver it until the process returns from the kernel system call back into user space.

If the underlying resource never responds, the process stays frozen in D state indefinitely.

Common triggers for this state in production include:

A shared NFS, Ceph, or SMB network mount that became unresponsive or dropped network packets during a write operation.
A cloud block storage volume experiencing acute I/O throttling or hypervisor-level storage timeouts.
Memory allocation stalls where the kernel is attempting synchronous page reclaim while the swap device is saturated.

Because the process is blocked inside a kernel driver, its user space threads cannot execute. It cannot process incoming socket buffers. It cannot emit error logs. It cannot shut down.

To see exactly what kernel function is trapping your process, inspect the kernel stack file in /proc:

cat /proc/18492/stack

Sample output:

[<0>] nfs_wait_bit_uninterruptible+0x2c/0x40 [nfs]
[<0>] nfs_wait_on_request+0x35/0x40 [nfs]
[<0>] nfs_updatepage+0x180/0x8d0 [nfs]
[<0>] nfs_write_end+0x7b/0x2c0 [nfs]
[<0>] generic_perform_write+0xd2/0x190
[<0>] __generic_file_write_iter+0xda/0x1e0
[<0>] nfs_file_write+0xa1/0x1b0 [nfs]
[<0>] vfs_write+0x21c/0x3f0
[<0>] ksys_write+0x5f/0xe0
[<0>] do_syscall_64+0x5b/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

This stack trace pinpoints the issue instantly: the application thread attempted a write() system call against an NFS mount, and the NFS client driver is blocked waiting on an unacknowledged RPC request.

No amount of restarting via systemctl restart will fix this until the NFS server recovers, because the restart command will block indefinitely waiting for the existing PID to exit.

Failure 4: File Descriptor Exhaustion and the Silent EMFILE Loop

In Linux, virtually every input/output channel is represented as a file descriptor. Sockets, open log files, epoll instances, database connections, and pipes all consume slots in the process file descriptor table.

Every process has a maximum limit of open file descriptors, defined by the RLIMIT_NOFILE resource limit.

When a process attempts to accept a new network connection via accept4() or open a connection to an upstream service via socket() while its descriptor table is full, the kernel returns error code 24:

EMFILE: Too many open files

Well-behaved production servers should handle this cleanly. In reality, many application runtimes and frameworks handle EMFILE in an unexpected way:

They catch the exception in an internal network listener loop to prevent the process from crashing. Inside the catch block, they write a warning to a log file, sleep for 20 to 100 milliseconds, and try again.

If the file descriptor leak is permanent (for example, leaked HTTP client sockets that were never closed after an upstream timeout), the application enters an infinite loop:

The accept queue stays backed up. The application refuses to pull incoming connections off the queue because it has no free descriptors to allocate for them. The process CPU usage drops to zero.

The service is running. It just cannot communicate.

Check the limits of the running process:

cat /proc/18492/limits | grep "Max open files"

Max open files            1024                 1024                 files

Notice the soft and hard limit: 1024. This is the historic default limit on many Linux distributions when not explicitly raised.

Now count how many descriptors the process currently has allocated:

ls -1 /proc/18492/fd | wc -l

If that number reads 1024, your process has exhausted its quota.

To see what is consuming those descriptors, categorize them with lsof:

lsof -p 18492 | awk '{print $5}' | sort | uniq -c | sort -nr

Sample output:

    982 IPv4
     28 REG
     10 FIFO
      4 DIR

In this case, 982 descriptors are IPv4 sockets. You can drill down further to see which remote addresses those sockets are connected to:

lsof -p 18492 -a -i 4 | awk '{print $9}' | sort | uniq -c | sort -nr | head -10

If you see hundreds of sockets stuck in CLOSE_WAIT connected to an internal microservice, your application has a resource leak: the remote service closed the connection, but your application code never called close() on the socket descriptor.

To prevent systemd services from hitting the restrictive default descriptor limit, configure LimitNOFILE directly in the unit file:

[Service]
LimitNOFILE=65535

Apply the change and restart the daemon:

systemctl daemon-reload
systemctl restart payments-api.service

Failure 5: Worker Deadlocks and Thread Pool Starvation

Modern web applications rarely use a single monolithic process to service traffic. Architectures like Gunicorn (Python), Puma (Ruby), PHP-FPM, or cluster modules in Node.js follow a master-worker model.

The master process (PID 1200) runs as root or an unprivileged application user. Its sole responsibility is to spawn, monitor, and restart worker processes (PIDs 1201, 1202, 1203, 1204).

Systemd monitors the master process.

Now suppose your application pool runs four worker processes. A deployment introduces an unindexed database query, or an external billing API slows down from 50 milliseconds to 30 seconds per request.

Incoming HTTP requests hit worker 1, worker 2, worker 3, and worker 4.

Each worker issues a synchronous database query or an un-timed HTTP call. All four workers block on network input from their respective backends.

The fifth incoming request arrives. The master process is not designed to service HTTP requests directly. All four workers are fully saturated and blocked.

The incoming request sits in the socket listen queue until the client times out.

Systemd checks the master process PID 1200. The master process is completely healthy, consuming 0% CPU, waiting for a SIGCHLD signal that never arrives because none of the workers have crashed.

How do you diagnose this condition?

Trace what the worker processes are actively doing using strace:

strace -p 1201 -s 128

If the worker is responsive and servicing traffic, you will see a rapid stream of system calls: epoll_wait, read, write, close.

If the worker is deadlocked or waiting indefinitely, strace prints one line and halts:

recvfrom(12,

The process is stuck waiting for data on file descriptor 12. Check what file descriptor 12 points to:

ls -l /proc/1201/fd/12

lrwx------ 1 app app 64 Oct 11 05:20 /proc/1201/fd/12 -> socket:[382910]

Now match the socket inode 382910 against ss:

ss -tanp | grep 382910

ESTAB 0 0 10.0.4.12:44920 10.0.10.50:5432 users:(("payments-api",pid=1201,fd=12))

Port 5432 is PostgreSQL. Your application worker is waiting on a response from your database server.

By following the socket from strace through /proc/<pid>/fd to ss, you bypassed guesswork and identified that the application is hanging on database query latency.

Failure 6: The Shell Wrapper PID Illusion

This failure is a classic misconfiguration in homegrown systemd service units.

An engineer writes a shell wrapper script to handle environment variables or pre-launch checks:

#!/usr/bin/env bash
# /opt/payments/start.sh

source /opt/payments/env.sh
java -Xmx4g -jar /opt/payments/app.jar &
tail -f /dev/null

And configures the systemd service unit like this:

[Unit]
Description=Payments Service

[Service]
Type=simple
ExecStart=/opt/payments/start.sh
User=app

[Install]
WantedBy=multi-user.target

Look at what happens during execution:

Systemd launches /opt/payments/start.sh. The kernel assigns PID 5100 to the bash script.

Inside the script, the java binary is launched in the background with &. The kernel assigns PID 5101 to Java.

The shell script then executes tail -f /dev/null to stay open.

Three hours later, the Java process encounters an unhandled runtime error, hits an out-of-memory exception, and terminates.

PID 5101 is dead. The port is closed.

What does systemd report?

systemctl status checks PID 5100 (the bash shell script) or its child tail. Both are still executing happily.

Systemd displays Active: active (running). The application has been offline for hours, but systemd has no idea because it was never configured to track the Java process.

Fix this anti-pattern using one of two techniques:

First, replace the shell process with the application binary using the shell exec builtin:

#!/usr/bin/env bash
source /opt/payments/env.sh
exec java -Xmx4g -jar /opt/payments/app.jar

exec instructs the shell to replace its own process image in memory with the target binary. The PID remains 5100, but the process executing is now Java. When Java exits, PID 5100 terminates immediately, allowing systemd to detect the exit code and trigger restart policies.

Second, avoid shell wrappers entirely whenever possible. Set environment files and paths directly inside the unit file:

[Service]
Type=simple
EnvironmentFile=/opt/payments/env.conf
ExecStart=/usr/bin/java -Xmx4g -jar /opt/payments/app.jar
Restart=on-failure

Failure 7: Reverse Proxy Permission Mismatches on Unix Domain Sockets

Many high-performance web deployments connect Nginx or Envoy to application backends over Unix domain sockets rather than TCP loopback connections. Unix sockets avoid the overhead of TCP packet headers, checksums, and network buffers.

However, Unix domain sockets are subject to standard Linux filesystem permissions.

An application starts cleanly under its own system user account (payments:payments). It creates a socket at /run/payments/app.sock:

srwxr-xr-x 1 payments payments 0 Oct 11 05:00 /run/payments/app.sock

The service is active and listening.

Yet Nginx returns a 502 Bad Gateway on every user request. The Nginx error log reveals the problem:

[error] 2102#2102: *402 connect() to unix:/run/payments/app.sock failed (13: Permission denied) while connecting to upstream

The Nginx worker process runs as user www-data or nginx.

Even if the socket file itself has write permissions, Nginx must be able to search (execute) every parent directory in the path leading up to that socket file.

If /run/payments/ has directory permissions 0700 (rwx------) owned by payments:payments, any process running as www-data will be blocked from traversing the directory.

You can inspect permissions along an entire directory path with namei:

namei -om /run/payments/app.sock

Sample output:

f: /run/payments/app.sock
 drwxr-xr-x root     root     /
 drwxr-xr-x root     root     run
 drwx------ payments payments payments
 s-rwxr-xr-x payments payments app.sock

The third line reveals the barrier: drwx------ on payments. The www-data user cannot traverse into the directory to reach the socket.

To fix this cleanly in systemd, configure runtime directory permissions inside the service unit:

[Service]
RuntimeDirectory=payments
RuntimeDirectoryMode=0755

When systemd starts the service, it automatically creates /run/payments with mode 0755, ensuring reverse proxies can access the socket.

Failure 8: Garbage Collection Freezes and Stop-the-World Pauses

Managed memory runtimes (Java Virtual Machine, Node.js V8 engine, Go, .NET) periodically reclaim unused memory via garbage collection.

Under normal operation, modern concurrent collectors perform sweeps in background threads with minimal interruption.

However, when an application experiences heavy memory pressure and approaches its maximum heap allocation (-Xmx in Java or --max-old-space-size in Node), the runtime triggers a full garbage collection cycle.

During a full collection, the runtime halts all application threads. This is known as a Stop-the-World pause.

While the collector scans gigabytes of memory references, the process continues running from the perspective of the operating system. Its PID is active in /proc.

Yet every network thread is frozen. No new connections are accepted. Existing connections receive no data. Health check probes sent by load balancers exceed their timeouts.

To check whether a process is running on the CPU or frozen in threads, inspect per-thread activity:

top -H -p 18492

The -H flag instructs top to display individual threads instead of aggregating the entire process.

If you see a single GC thread pinning 100% of a CPU core while all application worker threads display 0.0% CPU and zero state changes, the runtime is caught in an intensive memory cleanup sweep.

For Java applications, check the garbage collection statistics directly:

jstat -gcutil 18492 1000 5

If the FGC (Full GC count) is incrementing rapidly and GCT (total GC time) accounts for the majority of the elapsed seconds, the service is thrashing in memory management rather than servicing user traffic.

The Story Behind systemd's NOTIFY_SOCKET

Here is a piece of Linux systems engineering history that explains why this problem exists in the first place.

When Lennart Poettering and Kay Sievers designed systemd to replace the legacy SysVinit system in 2010, one of their primary design goals was eliminating the race condition known as startup deception.

Under SysVinit, a startup script executed an init command, waited for the command to return exit code 0, and immediately reported [ OK ] to the boot console.

In reality, traditional Unix daemons called fork(), spawned a child into the background, and returned exit code 0 to the parent shell before the child had even opened a config file, let alone bound its network port. Dependent services that started immediately afterward would attempt to connect to the port, get Connection refused, and crash.

To solve this, systemd introduced the Type=notify unit type and the sd_notify() API.

Instead of guessing when a process is ready based on PID creation, systemd creates a private UNIX domain datagram socket and passes its path to the process in the environment variable NOTIFY_SOCKET.

The service starts, performs all internal initialization, loads its caches, opens its database pools, and binds its network listening sockets.

Only when the socket is truly ready to handle traffic does the application send a single datagram to systemd:

READY=1

Until systemd receives that explicit string over NOTIFY_SOCKET, the unit remains in the activating state. If the initialization deadlocks or fails, systemd triggers its TimeoutStartSec timer, aborts the startup, and alerts the operator.

If you maintain services written in Go, Python, C, or Rust, switching your critical units from Type=simple to Type=notify eliminates the startup ghost outage entirely.

The 60-Second Triage Checklist

When a service is marked active in systemd but users cannot connect, avoid speculative restarts. Follow this systematic, non-destructive sequence:

First, check the listen queue for backlog overflow:

ss -lnt '( sport = :PORT )'

If Recv-Q is greater than or equal to Send-Q, the accept queue is saturated. The application is frozen or deadlocked.

Second, verify the binding IP address:

Ensure the local address is 0.0.0.0:PORT or the correct private interface address, rather than strictly 127.0.0.1.

Third, check process state for kernel I/O locks:

ps -eo pid,stat,wchan:20,comm | grep <PID>

If the state is D, read /proc/<PID>/stack to see which filesystem, storage mount, or driver is blocking execution.

Fourth, check file descriptor consumption:

ls -1 /proc/<PID>/fd | wc -l
cat /proc/<PID>/limits | grep "Max open files"

If current descriptors match the maximum limit, the process is dropping new connections due to EMFILE.

Fifth, trace active system calls:

strace -p <PID> -c -w 3

Observe whether the process is actively executing system calls or blocked permanently inside futex, recvfrom, or epoll_wait.

Sixth, check socket file path traversal if using Unix domain sockets:

namei -om /path/to/socket.sock

Confirm that every parent directory grants execute (+x) permissions to the web server user.

Wrapping Up

A green status indicator from systemctl confirms that a process exists in the Linux kernel process table. Nothing more.

It does not verify that the socket accept queue is draining. It does not check whether worker threads are deadlocked on database locks. It does not ensure file descriptors are available.

The next time an application stops responding while its service status looks pristine, resist the urge to reboot or cycle the process blindly. Check the listen backlog with ss, inspect descriptor allocations in /proc, and trace what the threads are waiting on.

You will find the real breakdown in under two minutes, preserve the diagnostic evidence, and fix the actual system limitation before it strikes again.

What is the strangest root cause you have uncovered where a service was active in systemd but completely unreachable?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

Your Linux Server Is Slow. Don't Restart It Yet: Do This First

Asep Sayyad — Thu, 08 Oct 2026 18:08:12 +0000

Rebooting clears the crime scene and hides the real bottleneck. Here is the non-destructive 60-second triage sequence to run before you touch the reboot button.

You connect over SSH to a production node and press the Enter key.

Nothing happens for two full seconds.

Then the cursor stumbles across the terminal, paints your username, and lags on every keystroke. Your Slack channels are flashing red with high-latency alerts from the load balancer, error rates are climbing, and a colleague on the incident bridge is already saying the words: "Just reboot the box."

Do not do it.

Rebooting a sluggish Linux server is an expensive admission of defeat. Worse, it is often a trap.

When you issue a reboot on a live, degraded system, you wipe volatile RAM state. You flush kernel ring buffers (dmesg) that hold hardware controller and memory warnings. You drop ephemeral network connection tables, kill inflight process traces, and erase the memory-backed filesystems in /run and /tmp.

And what happens next? If the real cause was an exhausted socket listen backlog, an unresponsive NFS storage mount, a runaway cron job, or memory thrashing under heavy swap, the node reboots, takes two minutes to start its services, and falls flat on its face the moment production traffic swings back.

You haven't resolved the incident. You just turned a reproducible five-minute diagnostic problem into an intermittent ghost bug that will strike again during peak traffic tomorrow.

Triage under fire is not about typing loud, disruptive commands to look busy. It is about disciplined observation before intervention.

Before you touch the reboot button or send destructive signals, run this systematic, non-destructive triage sequence. It takes under sixty seconds, costs virtually zero CPU overhead, and pins down the exact subsystem choking your system.

The 10-Second Baseline: Uptime, Core Count, and Load Trajectory

Start by checking whether the machine agrees with your external monitoring alerts. Synthetic health checks often flag a node as down simply because an external router jittered or an ingress proxy timed out.

Open your shell and run two commands:

uptime
nproc

Sample output:

 14:22:08 up 88 days, 14:12,  2 users,  load average: 36.40, 18.20, 6.10
 8

This output immediately tells you two critical facts: how many processor cores exist to service work, and which direction the load is heading.

Many engineers misinterpret load average as raw CPU utilization. It is not.

On Linux, load average measures the exponential moving average of processes in two specific kernel states: TASK_RUNNING (state R, processes actively running on a CPU or waiting in the run queue) and TASK_UNINTERRUPTIBLE (state D, processes blocked waiting on uninterruptible disk, network I/O, or hardware kernel locks).

On an 8-core host, a load average of 8.0 means every core is fully occupied with zero tasks queuing. A load of 36.4 means an average of 28 tasks were queuing for service over the past sixty seconds.

Now read the trajectory across the three timestamps (one, five, and fifteen minutes):

When the numbers read 36.4, 18.2, 6.1, load is actively surging upward. The fire is spreading right now.
When the numbers read 4.2, 16.5, 32.0, the worst of the spike has already cleared. The sluggishness you feel in your terminal is the tail end of queued work or swap reclamation, not an active disaster.
When all three numbers hover around 28.0, 27.8, 28.2, you are dealing with chronic, steady-state saturation.

A high load average tells you that a queue is backing up, but it does not tell you where that queue lives. It could be CPU starvation, disk I/O wait, or memory paging.

To find out, check the kernel's pressure telemetry.

Modern Linux Telemetry: Pressure Stall Information (PSI)

If your server runs Linux kernel 4.20 or newer (standard on Ubuntu 20.04+, Debian 11+, and RHEL 8+), you have access to Pressure Stall Information (PSI).

PSI is the single most accurate telemetry system built into modern Linux. Traditional utilization numbers can mislead you: a server with 98% CPU utilization might be running perfectly healthy, high-throughput database queries with zero user-facing delay. Conversely, a server with only 15% CPU utilization might be completely frozen because every thread is stuck waiting for a choked disk.

PSI solves this contradiction by measuring stall time: the percentage of wall-clock time that tasks were delayed waiting for resources.

Inspect the three pressure files in /proc:

grep -H . /proc/pressure/{cpu,memory,io}

Sample output:

/proc/pressure/cpu:some avg10=2.10 avg60=1.40 avg300=0.80 total=1240182
/proc/pressure/memory:some avg10=64.20 avg60=42.10 avg300=12.00 total=8923011
/proc/pressure/memory:full avg10=38.50 avg60=18.30 avg300=4.10 total=4129841
/proc/pressure/io:some avg10=8.40 avg60=5.10 avg300=2.00 total=3120914
/proc/pressure/io:full avg10=1.20 avg60=0.80 avg300=0.20 total=410920

Notice the two distinct metric lines: some and full.

The some metric indicates the percentage of time during which at least one task was stalled waiting for that resource while other tasks continued executing.

The full metric indicates total starvation: all runnable tasks on the system were simultaneously stalled waiting for memory or storage. During those windows, zero productive CPU cycles occurred. Notice that CPU only has a some metric, because if any task is executing on a CPU, the CPU is making progress.

Look at the sample output above:

CPU pressure is minimal (some avg10=2.10). CPU cycles are not your problem.
Memory pressure is alarming (some avg10=64.20, full avg10=38.50). For almost 40% of the past ten seconds, every process on the host was stalled waiting on memory page allocation or swap transfer.

In five seconds, without installing third-party agents or guessing, PSI isolated the exact physical bottleneck: memory starvation.

Now let us inspect the four primary failure domains individually to verify the underlying process mechanisms.

Subsystem 1: CPU Starvation, System Call Overhead, and CPU Steal

When CPU pressure is high, you need to know who is burning the cycles. Are they user applications, kernel context switches, or a noisy neighbor in your cloud environment?

Run vmstat and mpstat:

vmstat 1 5

Sample output:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
14  0      0 120400  42100 812900    0    0     4    12 2840 4520 88 11  1  0  0
16  0      0 119800  42100 812900    0    0     0     0 2910 4810 90  9  1  0  0
15  0      0 120100  42100 812900    0    0     0     8 2880 4700 89 10  1  0  0

Read the r column under procs. That is your run queue: the number of processes currently executing or waiting for CPU time slices. If r consistently exceeds your core count, you have genuine CPU saturation.

Next, glance at the cpu columns on the far right:

The us (user) column reflects application code execution. If us is above 85%, your application processes (such as Python workers, Java JVM threads, or Node.js instances) are burning cycles doing compute-heavy work.

The sy (system) column tracks kernel execution time. If sy spikes above 30%, application code is not the direct culprit. The kernel is spending massive time servicing system calls, resolving lock contention, or executing page faults. High sy often points to inefficient software hammering system calls like gettimeofday, epoll_ctl, or thrashing on thread synchronization locks (futex).

The st (steal) column is critical for cloud virtual machines (such as AWS EC2 or GCP Compute Engine). Steal time measures the percentage of physical CPU cycles that the underlying hypervisor took away from your VM to service other tenants on the same physical hardware.

If st climbs above 15% or 20%, your virtual machine is starved by noisy neighbors on the physical host. No amount of software tuning, caching, or code optimization will solve high steal time. You must migrate, resize, or redeploy the instance to a new hypervisor.

To pinpoint the exact processes consuming CPU, use pidstat:

pidstat -u 1 3

Alternatively, run a targeted ps query sorted by CPU usage:

ps -eo pid,ppid,user,%cpu,%mem,comm --sort=-%cpu | head -n 10

If you notice a single multi-threaded process reporting 750% CPU on an 8-core machine, remember that Linux sums CPU usage across all threads. To expand that process and see which specific threads are burning cycles, inspect the threads directly:

ps -eLf -q <PID>

Subsystem 2: Memory Starvation and the Swap Thrashing Trap

When a server feels sluggish, junior engineers often open free -h, see 150MB in the free column, and assume the server is out of memory.

That is almost always a false alarm.

The Linux kernel treats unused memory as wasted memory. Any RAM not actively claimed by application heaps is aggressively allocated to page caches (buff/cache) to keep disk reads fast.

free -h

Sample output:

               total        used        free      shared  buff/cache   available
Mem:            15Gi        12Gi       210Mi       1.2Gi       2.8Gi       1.8Gi
Swap:          4.0Gi       3.2Gi       819Mi

Ignore the free column. Focus entirely on available.

The available metric is the kernel's estimate of how much memory can be handed to new or existing processes without forcing the system into swap. It includes easily reclaimable page cache and dentries.

The real disaster in memory management is not low free memory. It is swap thrashing.

When active memory demand exceeds physical RAM, the kernel starts writing anonymous memory pages to your swap device while simultaneously evicting file-backed page caches.

Look at vmstat 1 5 again, specifically the si (swap in) and so (swap out) columns:

procs -----------memory---------- ---swap-- -----io----
 r  b   swpd   free   buff  cache   si   so    bi    bo
 4  3 342100 110200  18200 142000 4820 5120  4890  5210
 5  4 349800 108400  18100 141200 5120 6210  5190  6300

When si and so are both nonzero and continuously ticking in the thousands of kilobytes per second, your server is thrashing.

Every time a thread needs a page of memory, the kernel has to stall that thread, issue a disk I/O request to read it back from swap, and write another page out to make room. Because NVMe and SSD storage are orders of magnitude slower than DDR4 or DDR5 RAM, every thread on the system slows to a crawl. Your shell freezes because the bash process itself is being paged in and out of swap.

Next, check whether the kernel's Out-Of-Memory (OOM) killer has already begun terminating processes in the background:

dmesg -T | grep -i -E "oom|out of memory|killed process"

If the OOM killer triggered, you will see output documenting the victim process name, PID, total virtual memory, and RSS footprint:

[Tue Oct  8 14:18:22 2026] Out of memory: Killed process 14210 (node) total-vm:4210984kB, anon-rss:3810240kB, file-rss:0kB, shmem-rss:0kB

When you see this, you do not have a mysterious system degradation. You have a specific application process that exceeded its memory boundaries.

Subsystem 3: Uninterruptible Sleep (The "D" State) and Storage Latency

Have you ever tried to terminate a sluggish process using kill -9 <PID>, only to watch the process continue sitting in ps completely unchanged?

You did not break kill. You encountered a process stuck in uninterruptible sleep, represented by state D.

When a process executes a system call that requires waiting on physical hardware—such as reading a block from an SSD, waiting on a SAN controller, or issuing an RPC over an NFS mount—the kernel places that task into TASK_UNINTERRUPTIBLE.

In this state, the kernel explicitly refuses to deliver asynchronous signals to the process. Even SIGKILL cannot touch it. The kernel does this to prevent internal driver state corruption. The process will remain frozen until the hardware I/O request returns or times out.

To identify processes trapped in D state:

ps -eo pid,user,state,wchan:20,comm | grep -E ' (D) '

Sample output:

 18420 www-data  D nfs4_wait_on_open    php-fpm
 18421 www-data  D nfs4_wait_on_open    php-fpm
 18429 www-data  D nfs4_wait_on_open    php-fpm

The wchan column reveals the kernel function where the process is currently sleeping. In the snippet above, three PHP workers are stuck waiting on an NFS storage lock (nfs4_wait_on_open). The web server is slow not because PHP is broken, but because the remote NFS file server became unresponsive.

To see the exact line of kernel execution where a stuck process is blocked, read its kernel stack directly:

cat /proc/<PID>/stack

Sample output:

[<0>] io_schedule+0x42/0x70
[<0>] nfs4_wait_on_open+0x98/0x120
[<0>] nfs4_file_open+0x140/0x210
[<0>] do_dentry_open+0x180/0x3a0
[<0>] vfs_open+0x2d/0x30
[<0>] path_openat+0x2c0/0x10a0
[<0>] do_filp_open+0x91/0x140
[<0>] do_sys_openat2+0x96/0x150
[<0>] __x64_sys_openat+0x54/0x90
[<0>] do_syscall_64+0x58/0x120
[<0>] entry_SYSCALL_64_after_hwframe+0x6e/0x0

Notice the top frame: io_schedule. The process surrendered its CPU time slice and is waiting for the storage stack to respond.

To inspect overall storage subsystem performance, check disk await latency with iostat:

iostat -xz 1 3

Sample output:

Device    r/s     w/s     rkB/s     wkB/s   r_await   w_await  aqu-sz  %util
nvme0n1  12.0  1450.0      48.0   98200.0      1.10    148.20   24.50  99.80

Look closely at two metrics: w_await (average write wait time in milliseconds) and aqu-sz (average queue length).

Here, writes are taking 148 milliseconds to complete, and almost 25 I/O requests are stacking up in the device queue. For modern solid-state storage, write latencies above 10 or 15 milliseconds indicate serious drive saturation, controller throttling, or write-buffer exhaustion.

Do not be misled by %util. On modern NVMe drives that support 64,000 parallel queues, a disk can show 100% %util while maintaining sub-millisecond latencies under concurrent access. Always cross-reference %util with r_await and w_await.

The Hidden Storage Killer: Open Deleted Files

There is another common storage trap: df -h reports your root filesystem is at 100% capacity, but when you run du -sh /*, you can only account for 35GB of files.

Where is the missing disk space?

In Linux, deleting a file using rm simply unlinks its directory entry from the filesystem tree. If a running process (like an Nginx access logger or a database worker) still holds an open file descriptor pointing to that inode, the kernel preserves the underlying disk blocks.

The file disappears from ls and du, but the space remains 100% allocated.

To locate these unlinked, space-consuming file handles:

find /proc/*/fd -ls 2>/dev/null | grep '(deleted)' | sort -k7 -nr | head -n 5

Sample output:

142091 10485760 -rw-r--r-- 1 root root 10737418240 Oct  8 12:10 /proc/2190/fd/3 (deleted)

Process ID 2190 is holding open a 10GB file that was deleted from disk.

You do not need to kill the process or restart the daemon to get that disk space back. You can safely truncate the file descriptor through /proc in a single command:

: > /proc/2190/fd/3

The colon operator (:) is the shell's built-in no-op. Redirecting it into the file descriptor instantly resets the file size to zero bytes, freeing the 10GB of storage space immediately without dropping active network connections or restarting the parent daemon.

Subsystem 4: Socket Queues, Connection Trackers, and Network Choke Points

Sometimes CPU usage is 10%, RAM is clear, and disks are idle, yet API requests take five seconds to connect.

When a server feels sluggish from the outside but shows empty system queues on the inside, inspect the network transport layer.

Run ss to inspect your TCP listen backlogs:

ss -lnt

Sample output:

State      Recv-Q Send-Q  Local Address:Port   Peer Address:Port
LISTEN        512    128        0.0.0.0:8080        0.0.0.0:*
LISTEN          0    511        0.0.0.0:443         0.0.0.0:*

For listening sockets (indicated by the LISTEN state), the numbers in Send-Q and Recv-Q take on a completely different meaning than for established data connections:

The Send-Q column indicates the maximum listen backlog limit configured for that socket (the backlog parameter passed to the listen() system call, capped by /proc/sys/net/core/somaxconn).

The Recv-Q column indicates the number of completed TCP handshakes currently waiting in the kernel's accept queue, waiting for the user-space application to call accept().

Look at port 8080 above: Recv-Q is 512, while Send-Q is 128.

The application's listen backlog is completely overflowing. Incoming client connection attempts have saturated the queue, and the Linux kernel is silently dropping new SYN or ACK packets. Clients perceive this as total server unresponsiveness, yet CPU and memory monitors show the box doing almost nothing.

When building network services, handling connection queues and socket backlogs is a frequent tuning requirement. If an application event loop blocks on synchronous disk operations or CPU-heavy JSON parsing, it stops calling accept(), causing Recv-Q to blow past Send-Q within milliseconds.

Next, check connection tracking table exhaustion:

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

If nf_conntrack_count approaches nf_conntrack_max, the Linux kernel's Netfilter firewall subsystem cannot track any new stateful network connections.

When the table fills, the kernel drops incoming packets immediately at the raw interface layer without sending a reset. External clients time out, while internal processes report connection failures.

Check kernel log buffers for confirmation:

dmesg -T | grep -i "nf_conntrack: table full"

If you see this error, you can raise the table capacity dynamically in memory without restarting:

sysctl -w net.netfilter.nf_conntrack_max=262144

Five Panic Reflexes That Make Outages Worse

During high-stress outages, human adrenaline is often more dangerous than the original bug.

Avoid these five destructive reflexes:

Blindly sending kill -9 (SIGKILL): When you issue SIGKILL, the kernel unceremoniously strips process memory without notifying the application. Inflight write buffers are never flushed to disk, open network connections are abruptly severed without sending TCP resets or FIN packets, and distributed locks (such as Redis or etcd leases) remain locked until they expire. Always send SIGTERM (kill -15) first and give the application ten seconds to drain its state.
Flushing page caches with drop_caches: Running echo 3 > /proc/sys/vm/drop_caches is a common misstep when memory looks low. Dropping clean page caches forces the kernel to purge in-memory caches of shared libraries, binaries, and frequently read indexes. The very next second, every running process must read those files directly from physical disk, creating an immediate, violent disk I/O spike that can push a struggling server into total lockup.
Running unthrottled search tools across the filesystem: Typing find / -name "*.log" or du -sh /* on an I/O-starved machine reads millions of metadata dentries and inodes from disk. If storage is already choked or an NFS share is unresponsive, your search command joins the queue, locks file descriptors, and compounds the I/O freeze.
Applying random sysctl snippets from forums: Changing kernel TCP buffer sizes, virtual memory ratios, or scheduler tunables in the middle of an incident introduces unknown variables. If you change three sysctl settings simultaneously, you will never know which one stabilized the machine or which one created a secondary memory leak.
Executing a panic reboot: A reboot destroys the live /proc metrics, socket queues, and kernel traces needed to diagnose the issue. You trade twenty minutes of temporary relief for another outage tomorrow.

Controlled Stabilization: How to Take Control Safely

Once you isolate the runaway process or saturated subsystem, intervene using controlled, non-destructive tools.

If a runaway background worker or batch script is consuming 100% of your CPU cores, do not kill it immediately. Freeze it:

kill -SIGSTOP <PID>

The SIGSTOP signal suspends process execution instantly. The kernel deschedules the process from the CPU run queue, and its CPU usage drops to 0% within microseconds.

Crucially, the process remains in memory: its memory allocations, network sockets, and file descriptors stay completely intact.

Freezing the process removes system pressure immediately, restoring responsiveness to your SSH session and giving your team time to inspect the process state:

lsof -p <PID>
cat /proc/<PID>/environ

If you confirm the process was a rogue script that cannot be saved, terminate it cleanly:

kill -SIGTERM <PID>

If you decide the process is legitimate but simply needs to run without starving interactive traffic, lower its scheduling priority using renice and ionice:

renice +19 -p <PID>
ionice -c 3 -p <PID>

Setting a nice value of +19 instructs the Linux completely fair scheduler (CFS) to grant the process minimum CPU priority, while ionice -c 3 assigns it to the idle I/O scheduling class. It will only receive disk I/O when no other process on the machine requests the disk.

Finally, resume the frozen process:

kill -SIGCONT <PID>

The process resumes execution in the background without starving production workloads.

An Interesting Historical Detail: The Origin of Uninterruptible Sleep

The TASK_UNINTERRUPTIBLE state (D state) in Linux dates back to the earliest days of Unix design and Linus Torvalds' early 0.9x kernel revisions in 1992.

Operating system designers faced a difficult physical reality: when a driver requests a raw sector transfer from a hard disk controller or programs a DMA bus channel, hardware registers are left in an active, transient state.

If the kernel allowed an asynchronous user-space signal (like an impatient engineer pressing Ctrl+C or sending SIGKILL) to interrupt the process mid-transfer, the kernel would have to unwind the system call while the hardware controller was still writing physical magnetic tracks. This risked bus locks, corrupted sector headers, and kernel panics.

By creating an uninterruptible sleep state, the kernel guaranteed that the hardware driver could always complete its physical hardware transaction safely before the calling process could be altered or killed.

The downside is the exact production behavior we experience today: when physical storage controllers lock up or network mounts stop responding, the kernel waits forever, creating processes that cannot be terminated until the machine is rebooted.

The 60-Second Non-Destructive Triage Cheatsheet

When your next server alert fires, resist the urge to reboot. Open your terminal and run this exact diagnostic sequence:

Step 1: Check baseline and trajectory
- Command: uptime && nproc
- Action: Compare 1-minute load against core count. Check if load is surging or recovering.
Step 2: Check resource stall pressure
- Command: grep -H . /proc/pressure/{cpu,memory,io}
- Action: Look for high avg10 numbers under some and full to isolate CPU, memory, or disk starvation.
Step 3: Check memory and swap thrashing
- Command: free -h && vmstat 1 3
- Action: Check available memory. If si and so in vmstat are actively ticking, swap is thrashing. Check OOM logs with dmesg -T | grep -i oom.
Step 4: Check disk wait latency and stuck processes
- Command: iostat -xz 1 2 && ps -eo pid,state,wchan:20,comm | grep ' D '
- Action: Watch for w_await > 20ms or processes stuck in D state. Read /proc/<pid>/stack on stuck tasks.
Step 5: Check network socket backlogs
- Command: ss -lnt
- Action: Verify that Recv-Q is not exceeding Send-Q on active application listening ports.
Step 6: Safely stabilize without killing
- Command: kill -SIGSTOP <pid>
- Action: Freeze rogue processes to restore terminal control before deciding whether to renice, truncate deleted file handles, or gracefully reload.

Wrapping Up

Rebooting a server feels productive because it temporarily resets running counters. But in production engineering, temporary relief without diagnosis is a liability.

The next time an incident bridge starts demanding a reboot, pause for sixty seconds. Measure load trajectory, inspect kernel pressure metrics, check for swap thrashing and D-state tasks, and verify socket backlogs.

You will find the real culprit, protect your production data, and save your team from troubleshooting the exact same outage tomorrow.

What is the strangest root cause you have ever uncovered on a sluggish Linux server that turned out not to be high CPU at all?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Portfolio: asepsayyad007.in
Blog: asepsayyad007.in/blogs
GitHub: github.com/asepsayyad007
LinkedIn: linkedin.com/in/asepsayyad
Medium: asepsayyad007.medium.com
Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
Starring my open-source projects on GitHub.
Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

DEV Community: Asep Sayyad

The ss Command Is More Powerful Than You Think: 10 Practical Examples

Modern socket diagnostics, TCP queue inspection, and kernel-level network troubleshooting for Linux engineers

1. Fast Socket Triage with ss -tulpn

2. Uncovering Backlog Bottlenecks with Recv-Q and Send-Q

3. Global Socket Summary Statistics with ss -s

4. Querying Sockets by State Names

5. Filtering Sockets by Source and Destination Ports

6. Filtering by CIDR Subnets and Specific IP Addresses

7. Deep TCP Telemetry and Retransmissions with ss -ti

8. Sizing Socket Memory Buffers with ss -tm

9. Inspecting UNIX Domain Sockets with ss -xlp

10. Combining State and Port Filters for Targeted Incident Response

Quick Reference: Essential ss Commands

An Interesting Fact About ss and Netlink

Practical Takeaways for Your Daily Workflow

About the Author

Connect with Me

Enjoyed this article?

Stop Using cat for Everything: 9 Better Ways to Read Linux Files

Practical tools to inspect, page, slice, and debug files without freezing your terminal

1. bat: Syntax Highlighting, Git Awareness, and Smart Paging

2. less with -FXRS: Zero-Memory Paging and Instant Regex Search

3. tail -F and head: Slicing Boundaries and Tracking Rotated Logs

4. tac: Bottom-Up Reverse Reading for Fast Crash Diagnostics

5. zless, zcat, and zstdcat: In-Place Compressed Log Inspection

6. view: Read-Only Structural Navigation Without Edit Accidents

7. nl: Structured Line Numbering with Logical Page Boundaries

8. column -t: Formatting Jagged Delimited Files into Clean Tables

9. od, hexdump, and xxd: Exposing Invisible Bytes and Encoding Traps

Quick Reference: Choosing the Right Tool

An Interesting Fact About cat

Practical Takeaways for Your Daily Workflow

About the Author

Connect with Me

Enjoyed this article?

How I Troubleshoot a Linux Server That Suddenly Loses Network Access

A systematic Layer 1 to Layer 4 diagnostic guide for recovering isolated production machines through an out-of-band console without rebooting blindly.

1. The Physical and Driver Link Layer

2. Kernel Hardware Events and Driver Watchdogs

3. Addressing and Routing: The Silent Disappearance of the Gateway

4. The Layer 2 Neighbor and ARP Cache

5. Netfilter, nftables, and Connection Tracking Drops

The Connection Tracking Table Full Bug

Reverse Path Filtering (rp_filter)

6. Live Packet Tracing with tcpdump

7. Name Resolution and Local DNS Stubs

Interesting Fact

Summary of the Triage Ladder

About the Author

Connect with Me

Enjoyed this article?

What Really Happens When You Run curl google.com on Linux?

1. The Terminal, the Shell, and the Execve Boundary

2. The ELF Loader and Dynamic Shared Libraries

3. Argument Parsing and the Easy Handle

4. The Domain Name Resolution (DNS) Pipeline

Step 4A: The Name Service Switch (/etc/nsswitch.conf)

Step 4B: Checking Static Files (/etc/hosts)

Step 4C: Reading the Resolv Configuration (/etc/resolv.conf)

Step 4D: The DNS Query on the Wire

Step 4E: The "Happy Eyeballs" Race (RFC 8305)

5. Routing, ARP, and Layer 2 Discovery

Resolving the Gateway MAC via ARP

6. The Non-Blocking TCP Three-Way Handshake

The SYN Packet

The SYN-ACK Arrival

Waking Curl

7. Sending the HTTP Request

8. Google's Edge and the Infamous 301 Redirect

9. Receiving the Bytes and Writing to stdout

10. Connection Teardown and Process Destruction

The Practical Diagnostic Arsenal: Tracing It Yourself

1. Tracing the System Calls (strace)

2. Measuring Internal Phase Latency with Curl

3. Watching the Wire with tcpdump

Exactly What Happens When You Pass -L

An Interesting Fact About Curl

Summary: The Complete Execution Path

About the Author

Step 4A: The Name Service Switch (`/etc/nsswitch.conf`)

Step 4B: Checking Static Files (`/etc/hosts`)

Step 4C: Reading the Resolv Configuration (`/etc/resolv.conf`)

1. Tracing the System Calls (`strace`)

3. Watching the Wire with `tcpdump`

Exactly What Happens When You Pass `-L`