DEV Community

asif naeem
asif naeem

Posted on

Probing SSH without getting penalised

My SSH probe gave different answers for the same server on every run. The server was my own test box: an Open Quantum Safe build of OpenSSH 10.2, offering 71 key exchange algorithms. One run said a post-quantum algorithm worked; the next run marked it as unsupported, or as ?, meaning no answer at all. Nothing on the server had changed between runs.

The server was not misbehaving. It was defending itself, exactly as designed, against a client that looked like an attack: mine. This post is about PerSourcePenalties, the OpenSSH feature behind it, and how to probe a server without tripping it.

Part 3 of a series on building pqc-probe. Earlier: part 1, part 2.

What the server log said

The answer was in the server's own log, captured while a probe ran (addresses replaced with documentation addresses):

srclimit_penalise: ipv4: new 198.51.100.7/32 deferred penalty of 1 seconds for penalty: connections without attempting authentication
drop connection #0 from [198.51.100.7]:45330 on [10.0.0.5]:2222 penalty: connections without attempting authentication
drop connection #0 from [198.51.100.7]:45348 on [10.0.0.5]:2222 penalty: connections without attempting authentication
...
srclimit_penalise: 198.51.100.7/32: activating ipv4 penalty of 16 seconds for penalty: connections without attempting authentication
drop connection #3 from [198.51.100.7]:35034 on [10.0.0.5]:2222 penalty: connections without attempting authentication
PerSourcePenalties logging rate-limited: additional 90 connections dropped
Enter fullscreen mode Exit fullscreen mode

Three things happen in that excerpt:

  1. Each of my connections that ended without logging in earned a 1-second penalty, at first only deferred.
  2. Once enough of them had piled up, sshd activated a 16-second penalty for my address.
  3. While it was active, sshd refused every new connection from me. It dropped so many that it stopped logging them individually: "additional 90 connections dropped".

Every one of those refused connections was a test my probe then reported as unsupported or unknown.

How PerSourcePenalties works

OpenSSH 9.8 added PerSourcePenalties to slow down password guessing and crash-probing. It is on by default. sshd notes what each client address does and, when the behaviour looks hostile, refuses that address for a while. The defaults:

Behaviour Penalty
Disconnects without attempting authentication (noauth) 1 s
Fails authentication (authfail) or tries an invalid user (invaliduser) 5 s
Runs out of LoginGraceTime (grace-exceeded) 10 s
Crashes sshd (crash) 90 s
Minimum accumulated before anything is enforced (min) 15 s
Maximum a source can be refused for (max) 10 min

A penalty applies to the whole IPv4 /32 (IPv6 /128) by default, configurable with PerSourceNetBlockSize.

The interesting part is how penalties add up. In srclimit.c, the first offence creates a record that expires penalty seconds from now. A record below min is only deferred: nothing is refused yet. Each further offence while that record is still alive pushes its expiry later by another penalty seconds. When the time left on the record exceeds min, the record turns active, and every new connection from the address is refused until it expires.

So the rule is about rate, not count. Each noauth adds one second and each second of real time removes one. A client that leaves without logging in no faster than once a second never builds up a balance; a client that does it faster banks the difference and crosses 15 seconds quickly. The manual even warns that the noauth penalty "may penalise legitimate scanning tools such as ssh-keyscan".

Rate, not count

[embedded content: Computed from the accumulation rule in OpenSSH's srclimit.c with the default noauth penalty (1 s) and min (15 s)]

The same rule, played out for four test rates. Every line makes the same kind of connection; only the spacing differs. Eight tests a second are refused after 2 seconds, which is what my first version ran into. One test every 1.1 seconds never holds more than a single second of penalty, however long it runs.

Why a probe trips it

To find out whether a key exchange really works, a probe has to perform one. pqc-probe starts ssh with a single algorithm allowed, waits until the key exchange completes (SSH2_MSG_NEWKEYS received), and then kills the client. It never logs in, because it has no account and does not want one. To sshd, that is a textbook noauth disconnect.

My first version did that for every algorithm in the server's offer, for both clients, several connections at a time. Against a server offering 71 algorithms, that meant well over a hundred noauth disconnects in a few seconds. The balance passed 15 seconds almost immediately, and from then on sshd refused my connections, while my probe dutifully recorded each refusal as a result.

The outcome was worse than a crash. A crash is obvious; this produced a plausible report in which real support looked like failure, and the pattern changed from run to run depending on timing.

The fix: ask first, test little, test slowly

The fix was to stop using connections to learn things that need no connection:

  1. Ask the client, not the server. ssh -Q kex lists every key exchange the local client implements, with no network traffic at all. Intersected with the server's offer (read from its first packet, as in part 2), that already answers "could these two negotiate X?" for every algorithm.
  2. Test only what matters. Real handshakes are reserved for the post-quantum algorithms, ML-KEM first, then sntrup, at most 8 per client (--ssh-verify, 0 turns them off). Classical algorithms are reported as supported without a test.
  3. One at a time, 1.1 seconds apart. Sequential tests at a little under one per second keep the penalty balance from ever growing.
  4. Say "don't know" honestly. Each test ends in one of three states. A completed exchange is ok (tested). A definite refusal of the algorithm ("no matching key exchange method found") is fails. A timeout, reset or dropped connection gets one retry after 300 ms and is otherwise ?, inconclusive, and left out of the comparison instead of counted as a no.

On the same test server, the SSH half of a probe now takes about 20 seconds. The server log shows only deferred penalty of 1 seconds lines during a run, with no activating and no drop connection. That is slower than the first version, and correct, which the first version was not.

If you scan servers, or run them

If you run a scanner for other people, pacing each probe is not enough. Ten users probing the same server from your service all share your address, and sshd counts them together. pqc-probe's web service therefore limits probes per server address across all users (6 at once, then 12 a minute), and a repeat request for the same target within a minute gets the earlier result instead of a new probe. Without that, any server your users like would end up refusing your service.

If you run SSH servers, PerSourcePenalties is worth keeping; it costs attackers far more than it costs you. Two practical notes:

  • Your own monitoring, inventory or ssh-keyscan jobs can trip it too. Exempt their addresses with PerSourcePenaltyExemptList instead of turning the feature off.
  • If a scanner reports your post-quantum support as missing or inconsistent, check your sshd log for activating ... penalty before believing it. The scanner may have measured your defences rather than your configuration.

No answer is not the same as no

The deeper bug was in how I read results. My first version treated "the connection failed" as "the algorithm is not supported". Those are different statements, and modern servers make the difference matter: a refused connection tells you about the server's mood towards your address, not about its key exchange.

A probe that wants to be trusted needs three answers, not two: yes, no, and "I could not tell". It also needs to behave like a polite client, because the server is watching how it behaves. Both lessons came from a server log that explained everything once I looked at it.

Next in the series: the stock client is already post-quantum, or why the "vanilla versus experimental" comparison is shrinking.

Top comments (0)