Network Troubleshooting as a Stack: Find Which Layer Is Broken First

The difference between a good infrastructure troubleshooter and someone who restarts services and hopes is a mental model. When "HTTPS times out" lands in your inbox, you don't guess — you know exactly which layer to interrogate first, and in what order.

The network is a stack, so treat it like one

Every request rides through the same layers, top to bottom:

Application → TLS → Port → DNS → Gateway → Route → Interface

That's the dependency order — TLS can't work if the port is closed, the port is meaningless if DNS resolved to the wrong host, and none of it matters if your interface has no IP. So you verify in the inverse order, from the ground up:

Interface → IP → Route → Gateway → DNS → Port → TLS → Application

Start at the bottom because a broken lower layer produces confusing symptoms higher up. Confirm each layer is healthy before you climb. The moment a layer fails, you've found your problem — everything above it is a red herring.

Walk it: "HTTPS to api.example.com times out"

1. Interface — do we have a link and an address?

ip addr show

Look for your primary interface (say eth0) in state UP with an inet line like 192.168.1.20/24. No inet? DHCP failed or the link is down — stop here, nothing above will work. If the address is present and sane, climb.

2. Route — is there a path to the destination?

ip route get 93.184.216.34

This shows the exact route the kernel would pick, including the source IP and gateway (via 192.168.1.1 dev eth0 src 192.168.1.20). If you get "Network is unreachable" or no default route, you've found it. This is also the signature behind the classic curl error "No route to host."

3. Gateway — can we reach the first hop?

ping -c3 192.168.1.1
ip neigh show

ping tests reachability; ip neigh shows the ARP table. A gateway entry in state REACHABLE with a MAC address means L2 is fine. FAILED or INCOMPLETE means the gateway isn't answering ARP — a VLAN, cabling, or firewall problem. Note that many hosts drop ICMP, so treat a failed ping as a hint, not a verdict — trust the neighbor state.

4. DNS — does the name resolve, and to the right thing?

dig +short api.example.com

An empty answer or "could not resolve host" from curl means DNS is your layer. Compare the returned IP against what you expect — resolving successfully to a stale address is a subtle failure that looks like an application bug. Add dig api.example.com (no +short) to see the SERVER: line and confirm which resolver answered.

5. Port — is the service actually listening/reachable?

nc -vz api.example.com 443

Three outcomes, three different meanings — this is the most diagnostic step:

• "succeeded" / "open" — TCP handshake completed, move up to TLS.
• "Connection refused" — you reached the host but nothing is listening on 443 (service down, or wrong port). The host answered fast with a RST.
• Timeout / hangs — a firewall or security group is silently dropping packets. The signature is the wait, not an immediate error.

Refused vs. timeout is the single most valuable distinction in network debugging: refused = reached the host, timeout = something ate the packet. On the server, ss -tlnp | grep :443 confirms the process is bound to the right address (a service on 127.0.0.1 won't accept external traffic).

6. TLS — does the handshake complete and the cert validate?

openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null

Read the Verify return code at the bottom. 0 (ok) means the chain is valid. An expired cert, wrong hostname, or missing intermediate shows up here as a nonzero code (e.g. 21 unable to verify the first certificate) — long before your application logs blame something vague. The -servername flag sends SNI, which matters on shared hosts serving multiple certs.

7. Application — now, finally, the app

curl -v https://api.example.com/health

Only when everything below is green does an app-layer error (HTTP 500, 401, a slow backend) actually mean the app. The -v output replays the whole climb — DNS, connect, TLS handshake, request, response headers — so it's also a great one-shot sanity check to confirm your layer-by-layer conclusion.

Why this beats guessing

Each command targets exactly one layer and each failure has a distinct signature: no route to host (route), could not resolve (DNS), connection refused (port, service down), timeout (port, firewall drop), verify return code != 0 (TLS). Learn the signatures and you skip straight to the broken layer instead of restarting things at random.

I put the full layer-by-layer walkthrough, with these tools built into a portable toolbox, in the Kali Linux Networking for DevOps series — a 15-lesson path that frames Kali as a disposable, throwaway diagnostics box, not a "learn to hack" course. The Linux routing lesson and the DNS troubleshooting lesson go deep on the two layers people misdiagnose most.

One ground rule: only probe, scan, or capture on systems you own or are explicitly authorized to assess — this is defensive infrastructure validation, not an excuse to poke at other people's networks.

Try it next time

Next incident, resist the restart reflex — climb the stack from ip addr upward and let the failure signature name the layer. If you want the guided version, work through the full 15-lesson networking series; the site has free hands-on Kali learning paths built specifically for DevOps engineers.