Create a reverse SSH tunnel for remote access to a restricted Linux machine

I did not find a clear modern description of this set-up so have written this post to help others.

Say you have a Linux machine to which you want to ssh to but that machine is behind corporate or other firewalls etc which you have no control over, and thus you can not forward an external port for ssh etc. Let's call that the restricted machine and we assume you can make changes to it, e.g. locally at that machine. This post describes how you can set up a reverse ssh tunnel initiated and maintained from that restricted machine to a port on a Linux server elsewhere on the internet that you control. With the reverse tunnel in place, you can simply ssh to that port on your server to get a tunnelled ssh session back to the restricted machine through the restricted firewalls. Of course you should always check you have permission to do this with whoever controls the restricted network.

Note that autossh was previously often used for this (and nearly all existing online tutorials about reverse ssh tunnels use autossh) but autossh is redundant nowadays since modern openssh versions can monitor link health and exit if the link fails, and systemd can then be used to restart the ssh tunnel. So autossh is not required, we can implement an automatically maintained ssh tunnel using standard ssh and systemd alone.

In summary the steps are:

1. Set up the restricted machine.
2. Set up the server.
3. Final set-up and test.

Set up the restricted machine

On the restricted machine, create an ssh key pair in /etc/sshtunnel/ with an empty/no pass-phrase:

$ sudo mkdir -p /etc/sshtunnel
$ sudo ssh-keygen -qN "" -f /etc/sshtunnel/id_rsa

Create/paste the following systemd service file in /etc/systemd/system/sshtunnel.service and then edit it as needed for your specific server and ports:

[Unit]
Description=Service to maintain an ssh reverse tunnel
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
ExecStart=/usr/bin/ssh -qNn \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o ExitOnForwardFailure=yes \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null \
  -i /etc/sshtunnel/id_rsa \
  -R 9001:localhost:22 \
  [e-mail] -p 443
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target

You may likely want to change some of the unlabelled ssh parameters above so they are described below:

Parameter	Description
9001	Port on your server to which ssh clients will connect to.
22	Port you are running the ssh server on the restricted machine. Nearly always the default ssh port 22.
server.net	Internet hostname of your server. I actually use a dynamic DNS alias here (e.g. a hostname at duckdns.org) so I can easily re-select a different target server.
443	Port on your server where the ssh server is running. Often the default 22 but 443 is recommended as outgoing connections are less likely to be blocked by the restricted firewalls. However, using 443 requires that your server does not have a https server running on that port. Of course, if you use 443 then you must change your sshd configuration to listen on that port.

Set up the server

On the server, create an account for the sshtunnel user but restrict it so that shell logins and remote commands are not allowed. The server will only allow the remote sshtunnel user to set up port forwarding:

$ sudo useradd -m -s /bin/true sshtunnel
$ sudo mkdir -p ~sshtunnel/.ssh

Copy the public ssh key you created above on the restricted machine (/etc/sshtunnel/id_rsa.pub) to the sshtunnel user's authorized keys file on the server (~sshtunnel/.ssh/authorized_keys).

Ensure the owner/permissions are strictly set as ssh requires them.

$ sudo chown -R sshtunnel:sshtunnel ~sshtunnel/.ssh
$ sudo chmod 700 ~sshtunnel/.ssh
$ sudo chmod 600 ~sshtunnel/.ssh/authorized_keys

Final set-up and test

On the restricted machine, enable and start the service:

$ sudo systemctl enable --now sshtunnel

Check the status and logs with:

$ sudo systemctl status sshtunnel
$ sudo journalctl -u sshtunnel

Then on your server test the tunnel to the restricted machine with the following:

$ ssh -p 9001 localhost

To diagnose problems, check the sshd logs on both the server and the restricted machine:

$ sudo journalctl -u sshd

External public access via server (optional)

For most users, the above configuration is completely sufficient. The tunnel is available by making an ssh connection to the port on the server localhost interface. Either you connect to this locally from within that server, or if you are connecting from another machine then you use the ssh ProxyJump configuration keyword to automatically redirect to the restricted machine via the server proxy, again via the localhost interface.

Some users may want to make the tunnel port also externally accessible on the server public network interfaces, e.g. for use by users who don't have ssh access to the server itself. Be aware you are opening a public port directly connecting to the restricted machine if you do this so this approach should be avoided, or at least disable password authentication and only allow ssh key authentication on the restricted machine.

There are 2 extra steps to add this:

1. On the server, enable GateWayPorts yes in /etc/ssh/sshd_config. Then sudo systemctl reload sshd if you changed this.

2. On the restricted machine, prepend a colon (':') on the port number in the /etc/systemd/system/sshtunnel.service file, so the ssh option is -R :9001:localhost:22 rather than -R 9001:localhost:22. Then do sudo systemctl daemon-reload and sudo systemctl restart sshtunnel.

Removal

To remove all the changes described here do the following.

On the restricted machine:

sudo systemctl disable --now sshtunnel
sudo rm /etc/systemd/system/sshtunnel.service
sudo rm -rf /etc/sshtunnel

On the server:

$ sudo userdel -r sshtunnel

Disable GateWayPorts yes in /etc/ssh/sshd_config if you added it then sudo systemctl reload sshd.

Comments

[Frankenlist]

Great tutorial Mark! Some additions:

• Interesting history: autossh was released in 2002, OpenSSH added support for ServerAliveInterval/ServerAliveCountMax in 2004.

• If you want to bind to ports below 1024 your ssh user will either need to be root or use the usual tricks to allow it.

• ExitOnForwardFailure=yes will produce a hard fail if ssh fails to bind to the port (when a previous session is still hanging around), something like Error: remote port forwarding failed for listen port 9001. In practice it may take ~3 minutes before the zombie session is killed.
  Setting ClientAliveInterval/ClientAliveCountMax may reduce the time, but with systemd we can just use ExecStartPre= to kill any hanging ssh connections.

kill_bad_ssh.sh

#!/bin/bash
set -euf -o pipefail

PORT=$1;

SSH_PID=$(lsof -i 4:"$PORT" | grep '(LISTEN)' | awk '{print $2}');

kill "$SSH_PID"

If you are binding to an arbitrary port (eg 9001), you can simplify the script with SSH_PID=$(lsof -t -i:"$PORT"). But if you are binding to say 443/80 ports, and some other service happens to make an http request at the same time, it will also show up in lsof.

ExecStartPre=-/usr/bin/ssh [e-mail] '/root/kill_bad_ssh.sh 1000'

(the - allows the script to fail, eg when there are no previous ssh sessions to kill)

[Danny Diaz]

Hey! Great tutorial. The one thing that kicked our butt was the fact that we weren't running an SSH server on the restricted machine - we completely didn't realize it wasn't running. Mentioning that and how to start the ssh.service might be a good add to this tutorial (and maybe adding a user if you don't want to log in with the same user). Anyway, again, great tutorial thanks.