Diagnosing Packet Loss: Telling Link Errors, Queue Drops, and Rate Limiting Apart

Disclosure up front: I run IPYE, a company that leases IP space. It gets one short paragraph at the end; the rest is plain networking.

"We are seeing packet loss" is a report, not a diagnosis. Loss has several distinct causes (corrupted frames on a bad cable, a full queue, a policer, an overloaded CPU, a failing path further away), and each needs a different fix. The most valuable first step is to answer one question: where are packets disappearing, and what kind of disappearance is it? This article gives a method that works with common tools on Linux hosts and routers. It builds on the earlier article about reading traceroute and mtr, and uses documentation addresses throughout: 192.0.2.1 as a gateway, 198.51.100.10 as a remote target, and 2001:db8:200::10 as an IPv6 target.

Step 1: Measure it properly

A handful of pings is anecdote. Run enough probes to see a rate, in both directions and with more than one packet size:

ping -c 200 -i 0.2 198.51.100.10
ping -c 200 -i 0.2 -s 1400 198.51.100.10
mtr -rwzc 200 198.51.100.10

Compare small and large packets. If only large ones are lost, suspect MTU or fragmentation problems rather than congestion. Run the same test over IPv6 with ping -6 or mtr -6, because a loss problem that appears in only one family points to a path or policy difference. Also note the time: loss that appears only at specific hours suggests congestion, while constant loss suggests a fault.

Step 2: Read the mtr output with care

In mtr output, loss shown at an intermediate hop only matters if it persists to the following hops. Many routers rate-limit replies to ICMP probes aimed at themselves while forwarding transit traffic perfectly, so a hop showing 40% loss with 0% at the final hop is almost always control-plane rate limiting and not real loss. Real loss on a forwarding path shows up at that hop and every hop after it, including the destination. The earlier traceroute article describes the reasoning; the rule is to trust the last hop and the pattern of continuity.

Step 3: Check your own host first

Before blaming the network, look at the local interface and the kernel:

ip -s link show eth0
ethtool -S eth0 | grep -Ei 'err|drop|miss|fifo|crc'
nstat -az | grep -Ei 'drop|retrans|overflow'

What the counters mean:

Counter family	Typical meaning
CRC / frame errors, alignment errors	Physical layer: bad cable, connector, optic, or duplex mismatch
RX missed / fifo / no buffer	The NIC or driver could not keep up; ring buffers too small or CPU overloaded
TX dropped / qdisc drops	The output queue overflowed: more traffic than the link can send
Softnet drops	The kernel's per-CPU backlog overflowed; see /proc/net/softnet_stat
Socket buffer overruns	The application could not read fast enough

Increasing counters during a test point at the culprit. A clean set of counters on both ends moves suspicion to the path between them.

Step 4: Distinguish congestion from errors

Corrupted frames and congestion drops look similar from the application, but their signatures differ:

• Errors (CRC) grow steadily regardless of load and are in the interface counters of one specific link, usually at one end of a cable or optic. Clean, reseat, or replace, and check optical power levels with ethtool -m eth0 where the module reports them.
• Congestion grows with load, appears at times of peak traffic, and shows as queue drops on a specific egress interface. Check utilisation, the queue discipline (tc -s qdisc show dev eth0), and whether bursts exceed the buffer. The earlier article on traffic control describes fq_codel and shaping options.
• Policing shows a hard ceiling: throughput plateaus at an exact rate and drops appear even when the link is not full. Policers appear at provider edges, cloud virtual NICs, and misconfigured shapers.

A stepped test helps. Send a fixed-rate UDP stream with iperf3 -u -b 50M and increase the rate; loss that begins abruptly at a specific bandwidth indicates a policer or a bottleneck at that rate. The iperf3 methodology article covers how to do this without misleading yourself.

Step 5: Localise along the path

If your host is clean and loss persists:

1. Capture on both ends and compare counts. tcpdump -n -i eth0 icmp and host 198.51.100.10 on source and destination shows exactly which packets disappeared and, with timestamps, whether they were delayed or lost.
2. Test to intermediate points you control. Loss between A and B but not A and C narrows the segment.
3. Ask neighbours and upstreams to check interface counters on the specific link, giving them timestamps, addresses, and the measured rate. A report with a time window and mtr evidence gets resolved faster than "the network is slow".
4. Use independent vantage points, such as the probes described in the active measurement article, to see whether other networks experience the same loss.

Step 6: Rule out the usual local culprits

• Firewall or conntrack limits. dmesg | grep conntrack and nstat may show table-full drops. The conntrack article explains what to adjust.
• Reverse path filtering. A strict rp_filter on a multihomed host silently discards asymmetric traffic.
• Duplex or speed mismatch. ethtool eth0 shows negotiated speed and duplex; unexpected 100 Mb/s links are common with a bad cable.
• CPU saturation. top or mpstat -P ALL 1 shows whether one core handling interrupts is pegged at 100%.

Record what you found

Write down the test method, the packet size, the time, the rate of loss, the interface counters before and after, and the hop where the pattern begins. That record lets you or a colleague repeat the test after a fix, and it is the evidence a carrier will ask for.

The pattern to remember: measure, check your own host, classify the loss as error, congestion, or policing, and then narrow the path. Most problems reveal themselves by step four.

About IPYE

IPYE (https://my.ipye.com) leases IPv4, IPv6, and ASN space; a /48 IPv6 block is free for the first year. LOA, ROA, rDNS, and WHOIS are handled by IPYE. Locations: Los Angeles, New York, Ashburn, Salt Lake City, London, Amsterdam, and Frankfurt. Questions: [e-mail].

Disclosure: I run IPYE