DEV Community

Cover image for How I Troubleshoot a Linux Server That Suddenly Loses Network Access
Asep Sayyad
Asep Sayyad

Posted on Originally published at asepsayyad007.in

How I Troubleshoot a Linux Server That Suddenly Loses Network Access

#sre #linux #networking #infrastructure

A systematic Layer 1 to Layer 4 diagnostic guide for recovering isolated production machines through an out-of-band console without rebooting blindly.

An SSH session freezes in the middle of a command. Seconds later, your monitoring dashboard lights up with host-unreachable alerts. Pings to the server return 100 percent packet loss. When a production Linux machine drops off the network without warning, the worst thing you can do is pull the virtual power cord or trigger a hard reboot from your hypervisor console.

Rebooting clears the kernel ring buffer. It flushes the ARP cache, wipes interface error counters, resets firewall states, and discards volatile routing policy entries. If the problem stemmed from a corrupted state, a race condition during a dynamic lease renew, or a driver watchdog timeout, rebooting guarantees you will learn nothing. You will wait nervously for the exact same failure to strike again during peak hours.

Unless your physical hardware completely lost power or the motherboard failed, a sudden network drop is almost always traceable to a specific breakdown point. Link carrier dropped, a network driver stalled, a default route vanished, ARP resolution failed, or Netfilter silently discarded valid frames.

When network connectivity breaks, your only lifeline is out-of-band access: an IPMI serial-over-LAN session, iDRAC, AWS Serial Console, or a hypervisor VNC console. Once logged into the raw console, you can work upward through the network stack in a disciplined sequence.

Here is the exact diagnostic sequence to run when a server goes dark.

1. The Physical and Driver Link Layer

You must start by verifying whether the operating system still sees an active network link. If the operating system cannot confirm physical or virtual carrier signals, testing higher layers like DNS or routing is a waste of time.

Run ip -d link show to inspect the detailed state of your network interfaces:

ip -d link show eth0
Enter fullscreen mode Exit fullscreen mode

Look closely at two specific indicators in the output:

2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9000
Enter fullscreen mode Exit fullscreen mode

Notice the flags inside the angle brackets and the state field. If you see state DOWN, the interface administrative state was toggled off. A process, a configuration reload script, or a network daemon intentionally brought the link down.

If the output lists NO-CARRIER, the interface is administratively configured to be up, but the hardware sees no electrical or optical signal from the upstream switch port.

Next, query the link state directly from the network interface controller using ethtool:

ethtool eth0
Enter fullscreen mode Exit fullscreen mode

Inspect the link status, speed negotiation, and duplex settings:

Settings for eth0:
    Supported ports: [ TP ]
    Speed: Unknown!
    Duplex: Unknown! (Full)
    Auto-negotiation: on
    Link detected: no
Enter fullscreen mode Exit fullscreen mode

When Link detected: no appears on bare-metal servers, verify whether the physical optical transceiver is healthy by reading its internal diagnostic telemetry:

ethtool -m eth0
Enter fullscreen mode Exit fullscreen mode

This command queries the SFP module. It shows laser output power, receiver optical power, temperature, and operating voltage. If the receiver optical power is reading -40 dBm, the fiber strand is broken or the upstream switch port is shut down.

If Link detected: yes is present, check whether the network interface controller is dropping packets at the hardware ring level before the kernel can even process them:

ethtool -S eth0 | grep -E "drop|err|miss|fifo"
Enter fullscreen mode Exit fullscreen mode

Look for counters like rx_fifo_errors, rx_missed_errors, or rx_dropped. A sudden surge in these counters means incoming frames are hitting the physical card, but the card cannot allocate DMA ring buffers fast enough to deliver them to the operating system.

2. Kernel Hardware Events and Driver Watchdogs

If physical link carrier is active but network traffic stopped completely, check the kernel log. When network drivers hang or fail to transmit packets, the Linux kernel logs specific watchdog notifications.

Run dmesg with human-readable timestamps and filter for interface names and network subsystem events:

dmesg -T | grep -E -i "eth|e1000e|ixgbe|bnxt|mlx|link|down|reset|tx timeout" | tail -n 30
Enter fullscreen mode Exit fullscreen mode

You are hunting for three critical failure signatures:

First, look for Transmit Queue Timeouts. The message will look similar to this:

[Thu Oct 29 11:14:02 2026] NETDEV WATCHDOG: eth0 (ixgbe): transmit queue 2 timed out
[Thu Oct 29 11:14:02 2026] ixgbe 0000:03:00.0 eth0: TXDCTL.ENABLE not cleared within the polling period
Enter fullscreen mode Exit fullscreen mode

When a network driver queues packets into a hardware ring descriptor for transmission, it expects the network card to issue an interrupt indicating completion. If the hardware controller freezes or PCIe bus stalls occur, packets sit in the ring buffer indefinitely. The Linux kernel network watchdog detects that the queue has been blocked for several seconds and prints a NETDEV WATCHDOG warning.

Second, check for PCIe Aer errors or device resets. When high-bandwidth interfaces suffer uncorrectable bus errors, the kernel attempts an automatic PCI device reset. During the reset cycle, the interface drops off the bus for several seconds, terminating all active TCP sessions.

Third, look for link cycling loops:

[Thu Oct 29 11:15:20 2026] eth0: Link is Down
[Thu Oct 29 11:15:24 2026] eth0: Link is Up - 10Gbps/Full - flow control rx/tx
[Thu Oct 29 11:15:32 2026] eth0: Link is Down
Enter fullscreen mode Exit fullscreen mode

Rapid link flaps point to autonegotiation mismatches, faulty patch cables, or an upstream switch port bouncing because of spanning tree topology change notifications.

If a transmit queue timeout froze your interface, you can attempt to reset the driver ring buffer directly from the console:

ip link set dev eth0 down
ip link set dev eth0 up
Enter fullscreen mode Exit fullscreen mode

If the interface recovers after bouncing it down and up, the root cause is driver or offload related. Certain network offloads, specifically TCP Segmentation Offload (TSO) and Generic Segmentation Offload (GSO), can trigger driver ring buffer corruption on older or buggy firmware revisions. You can inspect active offloads:

ethtool -k eth0 | grep -E "segmentation|checksum"
Enter fullscreen mode Exit fullscreen mode

To prevent recurring driver hangs while troubleshooting, temporarily disable segmentation offloads:

ethtool -K eth0 tso off gso off
Enter fullscreen mode Exit fullscreen mode

3. Addressing and Routing: The Silent Disappearance of the Gateway

Once you confirm the link is up and the driver is passing frames, move up to Layer 3.

Check if the interface still owns its expected IP address:

ip -br addr show
Enter fullscreen mode Exit fullscreen mode

The output gives you a quick snapshot:

lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0             UP             192.168.1.50/24 fe80::5054:ff:fe12:3456/64
Enter fullscreen mode Exit fullscreen mode

If eth0 has no IP address assigned, your network management daemon either died, crashed during a configuration reload, or lost its DHCP lease.

Check the status of the managing daemon:

# If using systemd-networkd:
systemctl status systemd-networkd

# If using NetworkManager:
systemctl status NetworkManager
Enter fullscreen mode Exit fullscreen mode

In environments using DHCP, a lease renewal failure will quietly unbind the IP address after the lease expiration time elapses. The server remains powered on and running, but it has no Layer 3 presence.

If the IP address is intact, inspect the routing table:

ip route show
Enter fullscreen mode Exit fullscreen mode

A properly configured server will have an unambiguous default route pointing to your local subnet gateway:

default via 192.168.1.1 dev eth0 proto static metric 100 
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.50 metric 100
Enter fullscreen mode Exit fullscreen mode

One of the most frequent causes of sudden total network loss is the unexpected disappearance of the default route. When the default route vanishes, inbound traffic from external subnets reaches the server interface, but the Linux kernel drops all outbound replies instantly. The kernel cannot decide where to send response packets, resulting in Network is unreachable errors.

How does a default route vanish on an active server?

  • A second network interface came up, such as a temporary VPN tunnel, a Docker bridge, or a newly configured secondary VLAN. If that secondary interface acquired a default route with a lower metric, all external outbound packets are redirected into a dead path.
  • A dynamic network configuration tool reloaded and flushed routes because of an invalid DNS or search domain parameter in a netplan or ifcfg file.
  • An automation tool ran ip route flush as part of an incomplete configuration deployment.

If the default route is missing, restore it manually to verify reachability:

ip route add default via 192.168.1.1 dev eth0
Enter fullscreen mode Exit fullscreen mode

Also check routing policy database rules. Linux supports multiple routing tables. If custom routing rules were introduced, incoming packets might be evaluated against an alternate table that lacks valid egress routes:

ip rule show
Enter fullscreen mode Exit fullscreen mode

Normal outputs show rules pointing to the main routing table:

0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default
Enter fullscreen mode Exit fullscreen mode

If you see rules directing specific subnets to non-existent or unpopulated custom tables, traffic matching those rules will drop immediately.

4. The Layer 2 Neighbor and ARP Cache

Your server has an active link. The IP address is bound. The default route points to 192.168.1.1. Yet, when you attempt to ping the gateway from the console, every packet fails:

ping -c 3 192.168.1.1
Enter fullscreen mode Exit fullscreen mode 
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 192.168.1.50 icmp_seq=1 Destination Host Unreachable
Enter fullscreen mode Exit fullscreen mode

When Destination Host Unreachable appears, the Linux kernel is telling you that it cannot find the physical hardware MAC address for that IP address on the local network segment. Layer 3 routing is functional, but Layer 2 address resolution has failed.

Inspect your neighbor table:

ip neighbor show
Enter fullscreen mode Exit fullscreen mode

Review the state listed for your default gateway:

192.168.1.1 dev eth0 lladdr 00:1a:2b:3c:4d:5e REACHABLE
Enter fullscreen mode Exit fullscreen mode

The status column reveals the exact state of ARP resolution:

  • REACHABLE means the kernel knows the hardware address and recently verified it.
  • STALE means the cached entry is valid, but the kernel will re-verify it on the next outbound packet.
  • INCOMPLETE means the kernel sent broadcast ARP queries requesting the MAC address of 192.168.1.1, but has not received a response.
  • FAILED means multiple ARP broadcast requests timed out without a single answer from the network.

If the gateway entry is in a FAILED or INCOMPLETE state, bypass the higher networking layers and send a direct raw ARP query using arping:

arping -I eth0 -c 4 192.168.1.1
Enter fullscreen mode Exit fullscreen mode

If arping receives replies from the gateway MAC address while ping and ip neighbor continue to fail, check your kernel ARP sysctl parameters:

sysctl -a | grep -E "arp_ignore|arp_announce|arp_filter"
Enter fullscreen mode Exit fullscreen mode

A misconfigured net.ipv4.conf.all.arp_ignore setting causes the server to ignore ARP requests received on an interface if the target IP is not configured on that specific physical adapter. This setting is common in multi-interface setups or servers hosting virtual IP addresses for load balancing.

If an upstream network switch had its MAC address forwarding table cleared, or if another host accidentally assumed the gateway IP, the local ARP cache may hold a poisoned entry. Flush the neighbor table for that interface to force fresh discovery:

ip neigh flush dev eth0
Enter fullscreen mode Exit fullscreen mode

After flushing, send a single ping to the gateway and check ip neighbor show again to see if a valid MAC address resolves.

5. Netfilter, nftables, and Connection Tracking Drops

When Layer 2 and Layer 3 appear fully intact, the next suspect is the Linux packet filtering subsystem. Firewalls rarely break on their own, but automated host security tools, fail2ban daemons, container runtimes, and configuration updates can insert restrictive rules that sever communication without warning.

First, check the status of nftables or legacy iptables depending on your Linux distribution.

To inspect active nftables rulesets:

nft list ruleset
Enter fullscreen mode Exit fullscreen mode

To inspect iptables rules with line numbers and packet counters:

iptables -L -n -v --line-numbers
iptables -t nat -L -n -v --line-numbers
Enter fullscreen mode Exit fullscreen mode

Pay special attention to two details in the iptables output:

First, check the default policy of the INPUT and OUTPUT chains. If the default policy is set to DROP, verify whether an explicit rule exists permitting packets with the ESTABLISHED,RELATED state:

Chain INPUT (policy DROP 142 packets, 11840 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1      450 38000 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
Enter fullscreen mode Exit fullscreen mode

If the rule permitting established connections was inadvertently removed during an automated update, existing SSH sessions drop immediately, and no new connections can be formed.

Second, check packet and byte counters. If you see a rule targeting DROP or REJECT with rapidly incrementing packet numbers, that specific rule is intercepting and discarding your traffic.

If you suspect firewall rules locked you out, you can set the chain policies back to ACCEPT temporarily from your console to test connectivity:

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
Enter fullscreen mode Exit fullscreen mode

The Connection Tracking Table Full Bug

Another common silent network killer on high-traffic Linux systems is connection tracking table exhaustion. The Netfilter connection tracking module (conntrack) records state for every active network flow. If this table fills up, the Linux kernel begins dropping every new incoming and outgoing packet.

Check whether conntrack is currently dropping packets:

dmesg -T | grep -i "nf_conntrack: table full, dropping packet"
Enter fullscreen mode Exit fullscreen mode

You can check current table usage versus maximum capacity directly:

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
Enter fullscreen mode Exit fullscreen mode

If nf_conntrack_count equals or approaches nf_conntrack_max, the server will refuse all new TCP handshakes and UDP connections. You can increase the limit immediately without rebooting:

sysctl -w net.netfilter.nf_conntrack_max=262144
Enter fullscreen mode Exit fullscreen mode

Reverse Path Filtering (rp_filter)

If the server has multiple network interfaces, Docker bridge networks, or active WireGuard or OpenVPN tunnels, check reverse path filtering.

Reverse path filtering is a kernel security mechanism designed to prevent IP spoofing. When a packet enters an interface, the kernel checks its routing table to see if the return path for the packet source address would go back out of that exact same interface. If the return path uses a different route or interface, the kernel drops the packet silently.

Query the current rp_filter settings:

sysctl -a | grep '\.rp_filter'
Enter fullscreen mode Exit fullscreen mode

Values can be:

  • 0: No reverse path validation.
  • 1: Strict reverse path validation defined in RFC 3704. Each incoming packet is tested against the local routing table. If the interface is not the best reverse path, the packet is discarded.
  • 2: Loose reverse path validation. The incoming packet is accepted as long as the source address is reachable through any interface on the system.

If asymmetric routing exists on your network and rp_filter is set to 1, inbound traffic will hit the network card, but the kernel will drop it before applications or sockets ever see it. Change the parameter to loose mode to verify:

sysctl -w net.ipv4.conf.all.rp_filter=2
sysctl -w net.ipv4.conf.eth0.rp_filter=2
Enter fullscreen mode Exit fullscreen mode

6. Live Packet Tracing with tcpdump

When interface flags look healthy, routing tables contain the right gateways, and firewall counters are quiet, you need direct visibility into wire activity. tcpdump lets you watch packets enter and leave the network adapter in real time.

From your out-of-band console, start capturing ICMP packets on your primary interface:

tcpdump -nn -v -i eth0 icmp
Enter fullscreen mode Exit fullscreen mode

Now, initiate a continuous ping toward the server from an external machine or your local workstation. Observe the console output.

You will encounter one of three distinct diagnostic scenarios:

Scenario A: Inbound packets arrive, but no replies leave.

11:22:15.102341 IP 10.0.0.15 > 192.168.1.50: ICMP echo request, id 412, seq 1, length 64
11:22:16.104522 IP 10.0.0.15 > 192.168.1.50: ICMP echo request, id 412, seq 2, length 64
Enter fullscreen mode Exit fullscreen mode

You see the incoming echo request, but no corresponding echo reply is transmitted by eth0.
This proves that physical cabling, upstream switching, and NIC hardware are working. The problem is strictly inside the server. The packet was dropped by an iptables rule, rejected by rp_filter, or the kernel attempted to send the reply through a different interface because of a routing mismatch.

Scenario B: Outbound packets leave, but no incoming responses return.
Attempt to ping your default gateway from the console while watching tcpdump:

11:23:01.401120 IP 192.168.1.50 > 192.168.1.1: ICMP echo request, id 88, seq 1, length 64
11:23:02.403210 IP 192.168.1.50 > 192.168.1.1: ICMP echo request, id 88, seq 2, length 64
Enter fullscreen mode Exit fullscreen mode

The server sends the request, but the wire stays completely silent. The gateway is not responding. The fault lies outside the host: a misconfigured switch port VLAN, an upstream router failure, or an IP conflict on the subnet.

Scenario C: Complete silence on the wire.
No packets appear on the screen, even while external systems are actively attempting to connect.
The traffic is being stopped before reaching the Linux operating system. On virtual machines, this indicates that the hypervisor virtual switch disconnected the vNIC, the host security group blocked traffic, or the cloud provider route table was altered.

7. Name Resolution and Local DNS Stubs

Sometimes a server appears to have lost all external network access, but direct IP communication works without issue. The server can ping public IP addresses like 1.1.1.1, but any command relying on hostnames—such as curl, apt, or package updates—fails with errors like Could not resolve host or Temporary failure in name resolution.

Verify basic IP transport first:

ping -c 2 1.1.1.1
Enter fullscreen mode Exit fullscreen mode

If pinging a numerical IP succeeds, the network stack is functional. The breakdown is confined to DNS resolution.

Check the resolver configuration file:

cat /etc/resolv.conf
Enter fullscreen mode Exit fullscreen mode

On modern systemd-based Linux systems, /etc/resolv.conf is typically a symbolic link pointing to the systemd-resolved stub resolver:

nameserver 127.0.0.53
options edns0 trust-ad
Enter fullscreen mode Exit fullscreen mode

If systemd-resolved stopped running or crashed, local applications querying 127.0.0.53 receive no response. Check the resolver daemon status:

systemctl status systemd-resolved
Enter fullscreen mode Exit fullscreen mode

Inspect active upstream DNS servers configured in the resolver:

resolvectl status
Enter fullscreen mode Exit fullscreen mode

If the upstream nameservers point to unreachable addresses or invalid local forwarders, replace them temporarily in /etc/resolv.conf to test:

nameserver 1.1.1.1
nameserver 8.8.8.8
Enter fullscreen mode Exit fullscreen mode

Test resolution again using getent:

getent hosts google.com
Enter fullscreen mode Exit fullscreen mode

Interesting Fact

In the Linux kernel network subsystem, the NETDEV WATCHDOG timer does not run on a static timer. Instead, each network device driver defines a watchdog_timeo variable, typically set between 2 to 5 seconds. Whenever the driver queues a packet, it updates a trans_start timestamp on that transmission queue. During every kernel timer tick, dev_watchdog() scans active queues. If jiffies - trans_start exceeds the timeout while the hardware queue remains marked as stopped, the kernel declares a driver lockup and issues an interrupt to force a controller reset.

Summary of the Triage Ladder

When network connectivity drops, keep this structured diagnostic order in mind:

First, check the physical and driver link status with ip -d link show and ethtool. Confirm LOWER_UP and verify that ring buffer drop counters are zero.

Second, check dmesg -T for transmit queue timeouts, firmware crashes, or PCIe resets.

Third, verify your IP configuration with ip -br addr show and confirm that an unambiguous default route exists with ip route show.

Fourth, check your Layer 2 ARP cache with ip neighbor show. If entries are marked FAILED, use arping to test broadcast discovery.

Fifth, review firewall policies and connection tracking limits with nft list ruleset, iptables -L -n -v, and sysctl net.netfilter.nf_conntrack_count.

Sixth, trace raw wire traffic using tcpdump -nn -i eth0 to verify whether packets arrive and whether outbound replies leave the interface.

Working methodically from Layer 1 through Layer 4 lets you identify the exact breaking point without restarting the server or destroying forensic clues.

What was the most elusive network drop you ever diagnosed on a production Linux machine?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

Asep Sayyad is a Linux and DevOps engineer passionate about Linux administration, automation, cloud technologies, containers, and open-source software. He enjoys solving real-world infrastructure challenges and sharing practical knowledge through in-depth technical articles, tutorials, and hands-on guides.

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

  • Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
  • Starring my open-source projects on GitHub.
  • Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

© 2026 Asep Sayyad

Top comments (1)

Collapse
 
jun_liu_ca219b73a8f1489b5 profile image
jun liu •

Solid L1-to-L4 framing. In ops I've found the out-of-band console angle is the part most teams miss — by the time you realize the box is isolated you've already lost your only management path. One thing I'd add to the checklist: verify the hypervisor/virtual switch or switch port mirroring early, since many 'sudden loss' cases trace back to a port or VLAN change rather than the OS. Thanks for the systematic write-up.