DEV Community

Xinyang Wu
Xinyang Wu

Posted on

Your SSH Key Isn’t Always the Problem: A Layer-by-Layer Debugging Guide

#ssh #networking #security #devops

I once ran:

ssh demo-server
Enter fullscreen mode Exit fullscreen mode

and immediately suspected my SSH key.

The actual error was:

ssh: Could not resolve hostname demo-server: Name or service not known
Enter fullscreen mode Exit fullscreen mode

My key was innocent. SSH had not opened a TCP connection, exchanged a single protocol message, checked the server's identity, or attempted user authentication. It did not even know which IP address to contact.

That failure taught me a more useful way to debug SSH: treat every error as a timestamp. It tells you how far the connection progressed before it stopped.

Instead of changing keys, restarting services, and deleting known_hosts entries at random, ask one question:

Which layer did I fail to reach?

Here is the model I now use.

The connection has seven checkpoints

When you type ssh demo-server, the useful mental model is:

configuration
    → name resolution
    → TCP connection
    → SSH transport and key exchange
    → server verification
    → user authentication
    → channel creation
Enter fullscreen mode Exit fullscreen mode

Each stage depends on the previous one. A failure in name resolution cannot be fixed by rotating an authentication key, because the authentication stage never happened.

These are diagnostic checkpoints, not seven separate wire-protocol layers. In the SSH protocol, server host-key authentication is part of transport key exchange; I separate verification here because it produces its own recognizable failures and fixes.

Stage 0: SSH resolves its configuration

Before contacting the network, the client combines command-line options with SSH configuration, usually from ~/.ssh/config and the system configuration.

Imagine this entry:

Host lab-*
    User dev
Enter fullscreen mode Exit fullscreen mode

Running ssh lab-gpu matches the pattern and sets the remote user to dev. It does not give lab-gpu an IP address. Without a HostName, the destination is still literally lab-gpu, so the operating system must resolve that name later.

A complete alias might look like this:

Host demo-server
    HostName 192.0.2.10
    User dev
    Port 22
    IdentityFile ~/.ssh/id_ed25519
Enter fullscreen mode Exit fullscreen mode

The fastest way to inspect what SSH actually derived is:

ssh -G demo-server
Enter fullscreen mode Exit fullscreen mode

For a compact view:

ssh -G demo-server | grep -E '^(hostname|user|port|identityfile) '
Enter fullscreen mode Exit fullscreen mode

This is safer than reasoning from one config block by eye. Multiple matching blocks may contribute settings, and for most options SSH uses the first value it obtains. “The most specific block wins” is therefore a misleading model.

If the resolved hostname, user, port, or identity file is wrong, stay at this stage. The network is not your problem yet.

Stage 1: The operating system resolves the hostname

SSH next needs to turn the resolved hostname into an address. Depending on the machine, that may involve:

  • DNS
  • /etc/hosts
  • a VPN or overlay network's DNS
  • other system name-service mechanisms

One subtle trap is confusing a deployment tool's inventory with the operating system's resolver. An inventory might contain:

lab-gpu ansible_host=192.0.2.10
Enter fullscreen mode Exit fullscreen mode

Ansible understands that mapping. OpenSSH does not automatically read Ansible's inventory. Unless lab-gpu is also defined through SSH configuration or a system resolver, ssh lab-gpu can still fail with:

Could not resolve hostname
Enter fullscreen mode Exit fullscreen mode

That message is unusually precise: configuration completed, but the client could not obtain an address. No packet reached an SSH server.

First use ssh -G to find the effective hostname; resolver tools do not read ~/.ssh/config aliases. If that effective value is a name such as server.example.com, check the Linux system name-service path with:

getent hosts server.example.com
Enter fullscreen mode Exit fullscreen mode

On macOS, the equivalent system lookup can be inspected with:

dscacheutil -q host -a name server.example.com
Enter fullscreen mode Exit fullscreen mode

To query DNS specifically rather than the full system resolver path, use:

dig +short server.example.com
Enter fullscreen mode Exit fullscreen mode

Also check whether the name only exists while a company VPN, mesh VPN, or private DNS service is active.

Stage 2: The client opens a TCP connection

Once the client has an address, it tries to connect to the configured port—normally port 22.

At this point, similar-looking messages describe different conditions:

  • Connection timed out: packets may be dropped by a firewall, the route may be broken, the VPN may be missing, or the host may be unavailable.
  • No route to host: the local networking stack cannot find a usable path, or an intermediate device reports that the destination is unreachable.
  • Connection refused: the client received an active rejection, usually because the destination has nothing listening on that port. An intermediary can also reject it.

A small TCP probe against the effective address and port from ssh -G helps isolate this layer:

nc -vz 192.0.2.10 22
Enter fullscreen mode Exit fullscreen mode

This does not prove that SSH authentication will work. It only answers a narrower and very useful question: can I establish direct TCP connectivity to this host and port? If SSH uses ProxyJump or ProxyCommand, a probe from your machine does not test the same path.

Stage 3: SSH negotiates a secure transport

After TCP succeeds, the client and server exchange SSH protocol versions and negotiate algorithms. They perform a key exchange to derive fresh session keys, and the server proves control of its host key by signing data from the exchange.

A successful exchange creates the encrypted, integrity-protected transport used by the rest of the session. Modern cipher suites often provide authenticated encryption directly, so it is better to think in terms of confidentiality and integrity than to assume every connection uses a separate MAC algorithm.

At the wire-protocol level, the host-key proof belongs to this exchange. The next checkpoint is the client's decision about whether to trust that key.

Failures here look different from network failures. Examples include protocol banners that never arrive, an early connection reset, or messages such as “no matching key exchange method found.” These often point to incompatible algorithm policies, a non-SSH service on the port, an intermediary, or a server-side SSH problem.

Stage 4: The client verifies the server

Encryption is not enough if you encrypt a session to the wrong machine. The client therefore checks the server's host key against ~/.ssh/known_hosts or another configured host-key database.

On a first connection, many setups use trust on first use: you verify and accept the fingerprint, and the client remembers it. On later connections, the key should match.

If you see:

REMOTE HOST IDENTIFICATION HAS CHANGED!
Enter fullscreen mode Exit fullscreen mode

the connection has already passed name resolution, TCP, and enough of the SSH handshake to receive a host key. The warning could mean a legitimate rebuild or address reassignment—but it could also indicate misrouting or an active attack.

Do not make “delete the known_hosts entry” your automatic response. First verify the new fingerprint through a trusted, independent channel. Remove or replace the old entry only after you understand why it changed.

Stage 5: The server authenticates the user

Only now does your personal SSH key enter the story.

With public-key authentication:

  • the private key remains on the client;
  • the corresponding public key is normally listed in the remote account's authorized_keys;
  • the client signs session-bound authentication data to prove that it controls the private key;
  • the server verifies that signature with the public key.

The private key is not uploaded to the server.

Two similarly named files have completely different jobs:

File Usually lives on Purpose
known_hosts client Verifies the server's identity
authorized_keys server Lists public keys allowed to authenticate as a user

This distinction explains why the following error is actually evidence of partial success:

Permission denied (publickey,password)
Enter fullscreen mode Exit fullscreen mode

The client resolved a destination, connected over TCP, negotiated SSH, and accepted the server identity. It then failed to authenticate the requested user.

Now—and only now—it makes sense to inspect the resolved username, offered identities, file permissions, account policy, SSH agent, and the remote authorized_keys entry.

Stage 6: SSH opens a channel

A successful authentication does not necessarily mean you will receive an interactive shell. SSH can open different channels for:

  • an interactive shell
  • a single remote command
  • SFTP
  • local, remote, or dynamic port forwarding

For example:

ssh -N -o ExitOnForwardFailure=yes \
  -L 127.0.0.1:8080:127.0.0.1:3000 demo-server
Enter fullscreen mode Exit fullscreen mode

asks the client to listen only on the local loopback interface at port 8080, without running a remote command. A connection to that port travels through the encrypted SSH session, then the remote SSH server connects to 127.0.0.1:3000 from its own network context. ExitOnForwardFailure catches failure to establish the requested listener, although it cannot guarantee that the target service will accept a later forwarded connection.

If authentication succeeds but forwarding reports “administratively prohibited” or “open failed,” the problem is at the channel or destination-policy layer—not with DNS or your key.

Read the error as a stage marker

Here is the compact version I wish I had earlier:

Error or symptom Last relevant stage Check next
Could not resolve hostname name resolution effective HostName, DNS, hosts file, VPN
No route to host network path routes, VPN, interface, gateway
Connection timed out TCP path reachability, firewall, correct address and port
Connection refused TCP destination SSH service and listening port
no matching key-exchange algorithm SSH transport client/server algorithm policies
host identification changed server verification independently verify the new fingerprint
Permission denied (publickey) user authentication resolved user, offered key, authorized_keys, account policy
forwarding is prohibited or open fails channel creation SSH server policy and target reachability

The table is not a replacement for logs, but it prevents category errors. It tells you where to begin.

My three-step first response

When an SSH connection fails, I now start with these steps.

First, inspect the effective configuration:

ssh -G demo-server
Enter fullscreen mode Exit fullscreen mode

Second, on a Unix-like server, get a stage-by-stage trace without password prompts and make a successful probe exit immediately:

ssh -vvv -T -o BatchMode=yes -o ConnectTimeout=5 demo-server true
Enter fullscreen mode Exit fullscreen mode

Verbose output usually makes the boundary visible: configuration applied, address resolved, connection attempted, host key checked, identities offered, or channel rejected.

Be careful before sharing that output. Sanitize hostnames, usernames, addresses, paths, key fingerprints, and other infrastructure details.

Third, test the suspected boundary independently. Resolve the effective hostname if resolution failed; probe the effective address and port if they exist; inspect offered keys only if authentication was actually reached. Remember that resolver and TCP tools do not apply your SSH alias or proxy configuration for you.

Debug the earliest failure, not the loudest theory

SSH combines configuration, naming, networking, cryptography, identity, and multiplexed channels behind one short command. That makes it feel mysterious—but it also makes its errors surprisingly informative.

The next time ssh fails, read the error as a timestamp, identify the earliest incomplete checkpoint, and test that boundary directly. A DNS problem does not need a new key. A refused TCP connection does not need a cleared known_hosts. And an authentication error is proof that several lower layers already worked.

That one mental model turns SSH debugging from guesswork into a sequence of small, testable questions.

Top comments (2)

Collapse
 
voltagegpu profile image
VoltageGPU •

Great breakdown of SSH troubleshooting! I've spent hours stuck at the same "Connection closed" screen, only to later realize it was a misconfigured ~/.ssh/authorized_keys permission (0644 instead of 0600). Also, sometimes the server's MaxStartups or MaxSessions can silently drop connections without a clear error. Always check the logs on both sides.

Collapse
 
mridul_it_is profile image
Mridul Tiwari •

thank you so much , this will help me next time i face any such issue i will be sure of what to look for.