A quick guide to SSH & Solutions to some Common SSH errors

Imagine you are debugging a Prod issue, you try to ssh into the Linux server where the services are running, and the connection fails, throwing an error "Permission denied!" despite providing the right key/credentials to access it.!
and we will be like "Idella ivagle agbekitta!?/ abhi hi hona tha kya".
we the Op's guys have faced it several times, so here in this article, I've covered the concepts such as, How SSH works, followed by what are some of the common connectivity errors, How to debug it, & what are the best practices to work with remote servers.

What is SSH? and How does it work?

SSH (short for Secure Shell) is a network protocol that provides a secure way for two computers to connect remotely. SSH employs encryption to ensure that hackers cannot interpret the traffic between two connected devices. Main use cases for SSH are: Remote access & File Transfer(SFTP)
If you’re using Linux or Mac, SSH comes preinstlled(OpenSSH), & for windows it requires an SSH client like PuTTY.

What are SSH keys?

SSH keys always come in pairs, and every pair is made up of a private key and a public key. Who or what possesses these keys determines the type of SSH key pair. If the private key and the public key remain with the user, this set of SSH keys is referred to as user keys.
If the private and public keys are on a remote system, then this key pair is referred to as host keys. Another type of SSH key is a session key. When a large amount of data is being transmitted, session keys are used to encrypt this information.
Functionally SSH keys resemble passwords.
An SSH key is an access credential in the SSH protocol. Its function is similar to that of user names and passwords, but the keys are primarily used for automated processes and for implementing single sign-on by system administrators and power users.
The authorized_keys file in SSH specifies the SSH keys that can be used for logging into the user account for which the file is configured(public key of host). This file is usually found in the user's home directory under /. ssh/authorized_keys

What do SSH keys look like

An authorized key can look like this:
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN+Mh3U/3We4VYtV1QmWUFIzFLTUeegl1Ao5/QGtCRGAZn8bxX9KlCrrWISIjSYAwCajIEGSPEZwPNMBoK8XD8Q= CN@CI-serv

An identity key can look like this:
-----BEGIN EC PRIVATE KEY----- MHcCAQEEIJWbvSW7h50HPwG+bWR3DXgQ6YhOxYbe0ifr1rRUvsUuoAoGCCqGSM49 AwEHoUQDQgAE34yHdT/dZ7hVi1XVCZZQUjMUtNR56CXUCjn9Aa0JEYBmfxvFf0qU KutYhIiNJgDAJqMgQZI8RnA80wGgrxcPxA== -----END EC PRIVATE KEY-----

Knownhost key in detail

The memorized host keys are called known host keys and they are stored in a file called known_hosts in OpenSSH. As long as host keys don't change, this appoach is very easy to use and provides fairly good security.

Session key in detail

A session key in SSH is an encryption key used for encrypting the bulk of the data in a connection. The session key is negotiated during the connection and then used with a symmetric encryption algorithm and a message authentication code algorithm to protect the data

The SSH command consists of 3 distinct parts:

ssh {user}@{host}

The SSH protocol is based on the client-server model. Therefore, an SSH client must initiate an SSH session with an SSH server. The host (server) listens on port 22 (or any other SSH assigned port) for incoming connections. Most of the connection setup is conducted by the SSH client itself. Public key is used to verify the identity of the SSH server, and then symmetric key encryption and hashing algorithms are used to maintain data transmission in ciphertext. That way, privacy and integrity of data transmission in both directions between the client and server is assured, man-in-the-middle attacks will be mitigated.

The steps involved in creating an SSH session go like this:

1.Connection establishment: The SSH server listens to a connection request sent by the client on a specific port. After the client sends a connection request to the server, a TCP connection is set up between the client and server.
2.Version negotiation: The SSH server and client negotiate with each other to determine an SSH version to be used.
3.Key exchange: The server and client use a key exchange algorithm to dynamically generate a shared session key and session ID used to establish an encrypted channel. The session key is used to encrypt subsequent data for transmission, and the session ID is used to identify the related SSH connection during authentication.
4.User authentication: The client sends an authentication request to the server, and then the server authenticates the client. SSH supports the following authentication modes:
•Password authentication: The client sends the encrypted username and password to the server. The server decrypts the username and password, compares them with the locally stored username and password, respectively, and returns an authentication success or failure message to the client.
•Public key authentication: The client uses the username, public key, and public key algorithm to exchange data with the server for authentication.
•Password+public key authentication: The client can log in to the system only after being authenticated by the server using both password authentication and public key authentication.
5.Session request: After the authentication succeeds, the SSH client sends a session request to the server, requesting the server to provide a certain type of service. That is, the SSH client requests to establish a session with the server.
6.Session interaction: After a session is established, the SSH server and client can exchange data.

I suggest you to try the verbose mode -v which will show the step-by-step process for connecting to a remote computer,(use -vvv for more details from both ends),This further helps in debugging the connectivoty issues.

---

Some common ssh errors and solutions:

1. "Permission denied (publickey)"
   This error comes when you messed up with the authorized_keys of the User in the Host, May the entry is not exist, or the permission & ownership to the .ssh directory isnt correct
   sol: login as root > switch to user account > check for the above said file.
   Second possibility is when you are using password-based authentication, you must set PasswordAuthentication to yes
   Log as as a root to the Host
   vi /etc/ssh/sshd_config
   make changes as said above and restart the sshd service
   service sshd reload

2. "Host key verification failed"
   this error means that the host key of the remote host was changed.
   SSH stores the host keys of the remote hosts in ~/.ssh/known_hosts. You can either edit that text file manually and remove the old key (you can see the line number in the error message), or use
   ssh-keygen -R hostname

3. "No route to host"
   possibilities would be,
   •host is unreachable, check using ping command
   •If you have a firewall service running on your host machine,check if is blocking the ssh port

4. "connection refused"
   possibilities would be,
   •Your SSH Service Is Down
   •You Have the Wrong Credentials
   •The Port You’re Trying to Use Is Closed
   use this command to check sudo lsof -i -n -P | grep LISTEN
   •Firewall Settings Are Preventing an SSH Connection

5. "Too many authentication failures"
   if you are trying to login to root user, Check sshd_config and verify that root login is permitted. sshd will need to be restarted if the setting changes.
   another possibility is that Your SSH server's MaxAuthTries limit was exceeded. It happens so that Your client is trying to authenticate with all possible keys stored in /home/USER/.ssh/ .
   This situation can be solved by these ways:
   •ssh -i /path/to/id_rsa root@host
   •Specify Host/IdentityFile pair in /home/USER/.ssh/config

---

Troubleshooting:

1. use "-vvv" option
2. Make sure your IdentiyFile points to your right PRIVATE key.
3. Suppose you are able to login to the Host by any other means then switch to the required User (using sudo su - username) and check if it contains the .ssh directory,authorized_keys, and right public key in it.
4. Sometimes the issue comes from permissions and ownership. Verify the ownership of .ssh directory under the home folder, if you have messed up with it, change the ownership back, chown -R your_user:your_user .ssh Permissions should be 700 for .ssh and 600 for the files within (authorized_keys) chmod 700 .ssh chmod 600 .ssh/authorized_keys (ssh-keygen will create files and directories for you with the proper permissions)
5. tail -f /var/log/auth.log (on the server) and monitor errors when you attempt to login (if you were able to login via any other methods, such as aws ssm)
6. If you have many key files, try IdentitiesOnly yes to limit the authentication to use the single, specified key.
7. check value of PasswordAuthentication in /etc/ssh/sshd_config if you require password-based authentication(not recommended) then it to yes. Don't forget to restart ssh service after that. service sshd reload

This ends up the oobjective of this article, :) If you wants to read further, Here are details of some more ssh commands and their functionalities.

In addition to the ssh executable, SSH has other executable commands used for additional functions, including the following:
-sshd initiates the SSH server, which waits for incoming SSH connection requests and enables authorized systems to connect to the local host.
-ssh-keygen is a program to create a new authentication key pair for SSH, which can be used to automate logins, to implement SSO and to authenticate hosts.
-ssh-copy-id is a program used to copy, install and configure an SSH key on a server to automate passwordless logins and SSO.
-ssh-add is used to add a key to the SSH authentication agent and is used with ssh-agent to implement SSO using SSH.
-scp is a program used for copying files from one computer to another and is an SSH-secured version of rcp.
-sftp is a program used to copy files from one computer to another and is an SSH-secured version of ftp, the original File Transfer Protocol.

Thanks! for reading, leave a comment and let me know what you have felt about the article, and feedbacks on the same. :)