ssh -J Passes Through the Bastion, Not Into Its Records

Line 88 of my ~/.ssh/config reads:

Host vps5j
    HostName x.x.x.x
    ProxyCommand ssh vps3 nc %h %p

That pattern went obsolete in 2010. OpenSSH 5.4 introduced ssh -W, announced as "netcat mode" and written precisely to replace the nc on that line; in 2016, 7.3 added ProxyJump and -J, and the nc in the middle became unnecessary. Ten years on, I am still running a netcat process on the intermediate server. I never asked myself why, because it worked. I found the answer this week while looking at something else entirely, and the answer was not forgetfulness. ProxyJump could not get through that server. And it was writing the fact that it couldn't into the logs of two separate machines, every ten seconds.

The first half of this post is that story. The second half puts -J and nc side by side in a three-container lab and compares the traces each leaves on the bastion: process list, socket table, sshd log, known_hosts, agent socket. The short version: a ProxyJump connection passes through the bastion, but with default settings it does not pass into the bastion's records. You can read that as a security feature or as an audit gap; which one it is depends on what the bastion was built for.

12,342 attempts

On my Mac there is a small launchd agent that opens a tunnel to a management panel; it binds local 19443 to a remote 8443 and routes through VPS3. I had set it up with KeepAlive so it would come back if it dropped. This week I looked at the launchctl print output and saw this:

program = /usr/bin/ssh
arguments = {
    ...
    ProxyCommand=/usr/bin/ssh -4 -S none -o ControlMaster=no -o BatchMode=yes ... -W %h:%p vps3
}
runs = 12342
last exit code = 255
state = spawn scheduled

The counter started on 10 September, the day I set the agent up: twelve-thousand-odd launches, last exit code 255. By default launchd does not spawn a job more than once every ten seconds (man launchd.plist, ThrottleInterval); so in every period it was down, it had been opening and closing an ssh every ten seconds. The stderr file was 106,229 lines. I counted the errors: 11,116 times "administratively prohibited", 711 timeouts connecting to VPS3 (13 September, a day I could not reach the server at all), 430 times "connect failed: No route to host" (the bastion allowed it but could not reach the target itself). The majority is these three lines:

channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
Connection closed by UNKNOWN port 65535

I went to the other side, VPS3. journalctl -u ssh said the same thing 7,148 times in the last twenty-four hours:

sshd[998383]: refused local port forward: originator 127.0.0.1 port 65535, target x.x.x.x port 1051

I measured the gap between consecutive lines: fifteen of twenty were 10-11 seconds apart. launchd's rhythm. The latest uninterrupted stretch starts at 10:21 on 14 September; from then until I wrote this line at 14:00 on 15 September, 8,290 refusals in twenty-eight hours.

The cause was in two files. First, /etc/ssh/sshd_config.d/99-hardening.conf, the hardening set I wrote in June: AllowTcpForwarding no. Second, a drop-in in the same directory whose name starts with zzz- so it sorts last; the exception I added at 19:36 on 12 September:

Match User root Address 5.x.x.75
    AllowTcpForwarding local
    PermitOpen x.x.x.x:1051
Match all

My home connection's address ended in 75 that night, and the exception worked: the refusal lines stopped at 19:36:40 and the tunnel ran for a day and a half. At 10:21 on 14 September the Mac started coming from a different address; I used two separate networks that day, and by evening the home connection had picked up a new address ending in 169. Match Address no longer matched, sshd fell back to the global rule, and the global rule said no. I confirmed this with sshd's own test mode:

$ sshd -T -C user=root,addr=5.x.x.75,host=x,laddr=127.0.0.1,lport=52022 | grep -i "^allowtcpforwarding\|^permitopen"
allowtcpforwarding local
permitopen x.x.x.x:1051
$ sshd -T -C user=root,addr=5.x.x.169,host=x,laddr=127.0.0.1,lport=52022 | grep -i "^allowtcpforwarding\|^permitopen"
allowtcpforwarding no
permitopen any

The real lesson comes after that. On the same Mac, at the same time, there was a second ssh process going to the same panel, and that one was working. The difference was a single word: ProxyCommand=ssh ... root@vps3 /usr/bin/nc %h %p. AllowTcpForwarding no turns off port forwarding; it does not turn off nc. The sshd_config man page has said this plainly for years: disabling TCP forwarding does not improve security unless users are also denied shell access, as they can always install their own forwarders. The nc on line 88 walks around the same prohibition. I do not know when or why that line was written with nc; today I changed it to ProxyJump vps3 and tried, and got the same refusal. I understood that I had punched a hole in my own hardening rule with my own config while reading the log of the tunnel that rule was refusing.

What ProxyJump actually runs

-J is not a protocol feature; it is a small template inside the ssh client. In ssh.c, if options.jump_host is set, the client builds itself this ProxyCommand:

ssh [-l user] [-p port] [-J further,hops] [-F config] [-vvv] -W '[%h]:%p' jump-host

Run it with -v and it shows you the line: debug1: Setting implicit ProxyCommand from ProxyJump: ssh -v -W '[%h]:%p' bastion. ProxyJump is a shortcut built on top of ProxyCommand. If you write both, you get no error; readconf.c takes whichever comes first and silently ignores the later one (in the man page's words, whichever is specified first will prevent later instances of the other from taking effect). The inconsistent options: ProxyCommand+ProxyJump check in ssh.c exists, but it is a defensive line not reached through normal use.

-W host:port is this: the client opens a normal SSH session to the jump host, authenticates, and then, instead of asking for a shell, asks it to open a single direct-tcpip channel (RFC 4254 §7.2, the same channel type local port forwarding uses). One end of the channel is the TCP connection the bastion opens to target:22; the other end is the client's stdin/stdout. The outer ssh takes that stdin/stdout and sets up a second, entirely separate SSH session with the target. The bastion carries an encrypted stream in the middle; it cannot open it.

There is one more detail in the source. In the direct-tcpip request the client also sends "on whose behalf am I connecting"; since stdio has no socket, channels.c writes a fixed value here, 127.0.0.1 and 65535, with a comment saying it is a fake address/port to appease peers that validate it (Tectia). That is why VPS3's log says originator 127.0.0.1 port 65535; wherever you see that line, you are looking at a -W, most likely a ProxyJump.

And a trap: if you put ProxyJump bastion in a Host * block, the inner ssh heading to the bastion also falls into that block and wants to jump to itself. The client catches this and exits with jumphost loop via bastion; but it only catches it when target, port and user are all identical. The fix is a Host bastion block with ProxyJump none before Host *. Also worth knowing: the config that applies to the jump host comes from the bastion's own Host block, not the target's; options you pass with -o on the command line go to the target, not the bastion.

Lab: three containers, two ways through

Three Debian 13 based containers on my Mac with Docker: client, bastion, target; the second and third running openssh-server 1:10.0p1-7+deb13u4. The sshd-session in the process names arrived with 9.8; on VPS3's 9.6 the same lines show as sshd: ops. Two definitions on the client:

Host target
    User ops
    ProxyJump bastion
Host target-nc
    HostName target
    User ops
    ProxyCommand ssh ops@bastion nc %h %p

I ran ssh ... "sleep 6; hostname" with each and looked inside the bastion at the third second.

With ProxyJump:

[bastion ps]
     81 root     sshd-session: ops [priv]
     88 ops      sshd-session: ops
[bastion ss established]
172.23.0.2:45062 172.23.0.3:22
172.23.0.2:22    172.23.0.4:53750

With nc:

[bastion ps]
    109 root     sshd-session: ops [priv]
    116 ops      sshd-session: ops@notty
    117 ops      nc target 22
[bastion ss established]
172.23.0.2:22    172.23.0.4:33286
172.23.0.2:41400 172.23.0.3:22

The socket table is the same in both: one connection in, one out to target:22. The process list differs. On the nc path there is sshd-session: ops@notty; "notty" says a session was opened but no terminal was allocated, and below it the child process nc target 22 carries the target in plain sight. On the ProxyJump path there is sshd-session: ops, with neither @notty nor a child process; no session was opened, no shell ran. Someone looking with ps sees that the user is connected, and cannot see where to.

Then the log. The bastion's sshd, at the default LogLevel INFO, wrote three lines for each traversal, and all three were identical:

Accepted publickey for ops from 172.23.0.4 port 53750 ssh2: ED25519 SHA256:Zu4W...
Received disconnect from 172.23.0.4 port 53750:11: disconnected by user
Disconnected from user ops 172.23.0.4 port 53750

I raised it to LogLevel VERBOSE. The nc path gained one line: Starting session: command for ops from 172.23.0.4 port 38848 id 0. Not which command; session.c has a comment on that: we don't log unforced commands to preserve privacy. The ProxyJump path said not a word about the target even at VERBOSE. The serverloop.c function that services the direct-tcpip request writes the target only with debug_f; that line appears at DEBUG level, and the sshd_config man page says of DEBUG that it violates the privacy of users and is not recommended.

The target server's log is a separate matter: in Accepted publickey for ops from 172.23.0.2 the source address is the bastion's. The target never sees the client's real address; everyone who comes through the bastion arrives at the target from the same IP. Who came can be read from the key fingerprint; where they came from is known only on the bastion, and the bastion does not write it to the log with default settings.

Making one function talk

There is a way to get that one line without turning on DEBUG wholesale. LogVerbose in sshd_config (OpenSSH 8.5, March 2021) takes patterns by source file, function and line number and forces matching log calls to be written regardless of level. log.c builds the tag as file:function():line (bin=..., pid=...), so a wildcard at the end is needed:

LogLevel INFO
LogVerbose serverloop.c:server_request_direct_tcpip():*

The result, on a bastion that stays at INFO:

Accepted publickey for ops from 172.23.0.4 port 38858 ssh2: ED25519 SHA256:Zu4W...
debug1: serverloop.c:server_request_direct_tcpip():424 (bin=sshd-session, pid=197): originator 127.0.0.1 port 65535, target target port 22
Received disconnect from 172.23.0.4 port 38858:11: disconnected by user

I ran the nc path with the same setting: no line, because there is no direct-tcpip request on the nc path. This is the only log line that separates the two traversals. Two warnings: the line carries no username and no source address, only a pid; to know who went to which target you have to join it with the Accepted line on the same pid. And log.c hands this line to syslog at LOG_DEBUG priority; a filter like journalctl -p info, or an rsyslog rule that drops debug, swallows it silently.

One more counting trap. If the client has ControlMaster auto and ControlPersist for the bastion, three consecutive ssh target calls reach the bastion as one TCP connection and one authentication:

Accepted publickey for ops from 172.23.0.4 port 52388 ssh2: ...
debug1: ...server_request_direct_tcpip():424 (... pid=525): originator 127.0.0.1 port 65535, target target port 22
debug1: ...server_request_direct_tcpip():424 (... pid=525): originator 127.0.0.1 port 65535, target target port 22
debug1: ...server_request_direct_tcpip():424 (... pid=525): originator 127.0.0.1 port 65535, target target port 22
Received disconnect from 172.23.0.4 port 52388:11: disconnected by user

One Accepted, three channels, same pid. On a bastion you cannot reach the login count by counting Accepted lines; and without LogVerbose there is no line that gives the channel count either. VPS3 has 11,883 Accepted publickey lines in the last thirty days; how many channels ran underneath them is not something I can extract from today's log.

The log of refusal, the silence of acceptance

In the lab I set AllowTcpForwarding no on the bastion. ProxyJump produced the same three lines as the agent on VPS3 and died; the nc path went through. Look at the bastion log:

refused local port forward: originator 127.0.0.1 port 65535, target target port 22

sshd writes the target address at INFO level only when it refuses; the target of an accepted traversal stays at debug level. All 7,148 lines on VPS3 are refusal lines, which is why I can read the target address at all.

I tried narrowing with PermitOpen. PermitOpen target:22 let ProxyJump through; typing ssh -J ops@bastion ops@172.23.0.3 against the same bastion got refused: "Received request ... to connect to host 172.23.0.3 port 22, but the request was denied." The man page warns: no pattern matching or address lookups are performed on supplied names. Whatever text the client wrote into -W is what PermitOpen compares against (a plain strcmp in channels.c); target and 172.23.0.3 are the same machine but not the same string. You have to list the name and the address both; otherwise the bastion refuses when someone types ssh -J bastion 172.23.0.3, or, the other way round, if only the address is listed, whoever comes by name is refused.

The last experiment is the other half of the man page sentence. On the bastion I created a transit user with /usr/sbin/nologin as its shell and put restrict,port-forwarding on its authorized_keys line. It is nologin that closes the shell; restrict closes pty, agent, X11 and ~/.ssh/rc, not command execution, which is why both are needed. port-forwarding reopens forwarding alone. ssh -J transit@bastion ops@target went through. The nc path died with This account is currently not available., because there is no shell to run nc. The same authorized_keys line can also take permitopen="target:22", the per-key version of PermitOpen, with the same "no name resolution" warning. There is also DisableForwarding for switching forwarding off wholesale, X11 and agent included; but as long as the shell stays open it has no say over nc either. The version of hardening that holds is closing the shell and opening forwarding with a list. On VPS3 I had done the exact opposite.

The price of bringing your agent onto the bastion

The common habit before ProxyJump was ssh -A bastion, then ssh target from inside the bastion. I did that in the lab too and looked at what stayed behind on the bastion:

SOCK_ON_BASTION=/tmp/ssh-gcxGWhx5oW/agent.432
Warning: Permanently added 'target' (ED25519) to the list of known hosts.

For the duration of the session there is an agent socket on the bastion; anyone who is root on the bastion can have my key sign for them through that socket. The ssh.1 man page says this under -A and closes with: a safer alternative may be to use a jump host (-J). Also, the target's key got written into the bastion user's ~/.ssh/known_hosts; the list of targets now accumulates on the bastion.

I went to the same target with ProxyJump and looked: SSH_AUTH_SOCK empty on the target, no /tmp/ssh-* on the bastion, no line added to the bastion's known_hosts. The target's key landed in the client's own known_hosts under the name target; since the client sets up the session with the target itself, the client does the key verification too. If the bastion was built purely as a carrier, ProxyJump is exactly what you want. If the bastion was built to record sessions, the pipeline in the tlog session recording post does not see ProxyJump traffic: the recorder runs as the user's shell, and -W opens no shell. Leaving the shell open and recording is not enough either, because on an account with a shell nc target 22 runs, and tlog records that binary stream without ever writing the target.

Questions before you ship it

1. What is the bastion's job: carrier or recording point? If carrier, -J and a shell-less account. If recording point, write targets to the log with LogVerbose serverloop.c:server_request_direct_tcpip():* and accept that an account with a shell can escape via nc; closing that escape takes ForceCommand or a restricted shell that runs nothing but permitted binaries.
2. If you wrote AllowTcpForwarding no, is the shell closed too? If not, nc %h %p walks past that rule; search your own config for it.
3. Does the PermitOpen list have both the name and the address? Whatever string the client writes into -W is what matches.
4. Is the exception you opened with Match Address tied to a dynamic IP? Mine lasted a day and a half. Tie it to a separate user with Match User, or to the key with permitopen= in authorized_keys.
5. Do you take the Accepted publickey count as the login count? With ControlMaster clients, one acceptance, N channels.
6. The target server sees the bastion as the source address; "who" should come from the key fingerprint, "from where" from the bastion log. The bastion log does not give that by default.
7. Is there a ForwardAgent yes left in ~/.ssh/config? With -J you don't need it.

Closing

I did not fix the Match Address line on VPS3 today; the agent is approaching 12,400 as this paragraph is written. First I have to decide which side to change, and two options came into focus over the course of the post. Either I leave the panel tunnel without forwarding and stay on nc, in which case the AllowTcpForwarding no line stays on paper and I have accepted that. Or I create a shell-less transit account on VPS3, write the panel's address into PermitOpen in both forms, drop the channels into the log with LogVerbose, and convert the nc on line 88 to ProxyJump vps3, ten years late. The second is more work; the first has been reminding me of itself every ten seconds for twenty-eight hours.

For source lines I used the OpenSSH master branch; the lab version is Debian 13's 10.0p1-7+deb13u4 package, VPS3 is Ubuntu 24.04 9.6p1. For the history of -W and ProxyJump, the 5.4 and 7.3 release notes. The client error text on VPS3 differed from the lab's ("Stdio forwarding request failed: Session open refused by peer"); that text comes from mux.c, because I was reaching the bastion through ControlMaster. For sshd's new source-based penalty mechanism, the PerSourcePenalties post; for protecting the key itself, the FIDO2 post.

Official Sources

• ssh.c — the ProxyCommand generated from ProxyJump and the jumphost loop check
• serverloop.c — server_request_direct_tcpip: debug on accept, logit on refuse
• session.c — "we don't log unforced commands to preserve privacy"
• channels.c — stdio-forward and the 127.0.0.1:65535 placeholder
• log.c — LogVerbose tag format and forcing
• sshd_config(5) — AllowTcpForwarding, PermitOpen, LogLevel, LogVerbose, DisableForwarding
• ssh_config(5) — ProxyJump
• ssh(1) — -J, -W, -A
• RFC 4254 §7.2 — the direct-tcpip channel
• Debian trixie openssh-server package