Mustafa ERBAY

Posted on Sep 12 Originally published at mustafaerbay.com.tr

sshd's New Reflex: Who Does PerSourcePenalties Lock Out?

#linux #ssh #openssh #security

On the server that runs this blog, fail2ban-client status sshd prints four lines and all four are zero: nobody banned, nobody failing. That is not the result of a tight defence; it is the result of sshd listening on a five-digit port instead of 22. Of the last 24 hours' 2,234 sshd log lines, exactly two belong to connections that closed without authenticating; the rest are my own automation's 484 key-based logins and a reverse tunnel's refused port forwards. Anyone looking at that picture would say "brute force is not my problem" and be right. But that server runs Ubuntu 24.04 with OpenSSH 9.6, and the mechanism this article is about does not exist there yet. Everyone moving to Debian 13, RHEL 10 or Ubuntu 26.04 has it, and it is on by default.

My thesis: PerSourcePenalties, which arrived with OpenSSH 9.8, moved part of what fail2ban has done for years by reading logs into sshd itself, right behind the accept() call. That is a good thing. But in teams that upgrade without knowing whom the penalty table targets, the first victim is easily not an attacker but an ssh-agent holding seven keys, or a health check that connects once a second. I read the mechanism in the source, then deliberately locked myself out of a Debian 13 container on my Mac. The numbers are below.

I have written about SSH here before from the angle of post-quantum key exchange and short-lived certificates; both were about who gets in. This one is about who never reaches the door.

At the door, or in the hall?

To see the difference you need to recall another 9.8 change: the server was split into two binaries. sshd is now just a small listener; every connection gets its own sshd-session binary, and the protocol lives there (10.0 moved the authentication phase into a third binary, sshd-auth; on Debian 13 both sit under /usr/lib/openssh/). The penalty table lives in the listener. That has two consequences. First, a blocked client never reaches key exchange, authentication, or even the version-string handshake; as soon as the socket is accepted, Not allowed at this time is written and the socket is closed. Second, the table is process memory: systemctl restart ssh or a SIGHUP (the listener re-executes itself with execv) wipes every penalty. It is not a persistent blacklist; it is a short-lived reflex.

When the penalty is written is also surprising: not during the connection, but when sshd-session exits. The listener reads the child's exit code and decides from that. The branch in sshd.c (as of the master branch; the invaliduser class arrived in 10.3 and is absent from the 10.0 used in the experiment), condensed:

EXIT_LOGIN_GRACE      -> grace-exceeded   (LoginGraceTime expired)
EXIT_CHILD_CRASH      -> crash            (session process crashed / killed by signal)
EXIT_AUTH_ATTEMPTED   -> authfail         (an attempt was made, closed without success)
EXIT_INVALID_USER     -> invaliduser      (nonexistent username)
EXIT_CONFIG_REFUSED   -> refuseconnection (rejected via RefuseConnection)
any other non-zero    -> noauth           (closed without attempting at all)

There is a detail here worth pausing on. The "attempt was made" flag is raised in the server's monitor process whenever any authentication-related request arrives, and that includes merely offering a public key. A successful login clears the flag. So a client that offers four wrong keys and gets in with the fifth is not penalised; a user who tries three keys and hits Ctrl-C at the password prompt is. The penalty is charged not for failing but for leaving without success.

The default table and the 15-second threshold

sshd -T | grep -i persource shows today's values. On Debian 13's OpenSSH 10.0 (ssh -V: OpenSSH_10.0p2 Debian-7+deb13u4) it printed exactly this:

persourcepenalties crash:90 authfail:5 noauth:1 grace-exceeded:10 refuseconnection:10 max:600 min:15 max-sources4:65536 max-sources6:65536 overflow:permissive overflow6:permissive
persourcepenaltyexemptlist none
persourcenetblocksize 32:128

The meaning is in sshd_config(5): every event from a source adds its own duration, durations accumulate up to max (10 minutes), and blocking only starts once the accumulated time exceeds min, 15 seconds. That last rule is the least understood part of the mechanism. A single wrong key creates a 5-second "deferred" penalty, and if no second event arrives within those 5 seconds it simply expires. A lock needs the events to catch up with each other: 5 + 5 + 5 = 15 does not exceed the threshold; the fourth 5 makes 20 and the door closes.

As far as I can follow from the release notes, the table has been revised several times since 2024. 9.9 lowered the grace-exceeded penalty so that a single forgotten password prompt cannot cross the threshold alone, and added the refuseconnection class. 10.0 fixed a LoginGraceTime expiry being wrongly charged as a 90-second crash penalty, and rate-limited the block logging. 10.1 raised the line that says a source's penalty has crossed the threshold and activated from VERBOSE to INFO; the actual block line, drop connection ... penalty:, has been at INFO since 9.8, so it is visible at the default LogLevel on every version. 10.3 added the invaliduser class and fractional durations below one second. Do not say "the default is X" without knowing where your distribution stands; look at sshd -T.

Locking myself out in a container

Reading the rules is one thing; watching the clock actually count 20 seconds is another. I installed openssh-server on debian:trixie-slim (10.0 came with it), enabled LogLevel VERBOSE, created a user named deneme and, from a second container, tried to connect repeatedly with an unauthorised ed25519 key. Four attempts fit into the same second; the fifth found the door closed:

--- deneme 1 06:36:46
deneme@hedef: Permission denied (publickey,password).
--- deneme 2 06:36:46
deneme@hedef: Permission denied (publickey,password).
--- deneme 3 06:36:46
deneme@hedef: Permission denied (publickey,password).
--- deneme 4 06:36:46
deneme@hedef: Permission denied (publickey,password).
--- deneme 5 06:36:47
Connection reset by 172.17.0.3 port 22
--- deneme 6 06:36:47
Connection reset by 172.17.0.3 port 22

The server's record of the same moments tells the mechanism's story in its own words:

srclimit_penalise: ipv4: new 172.17.0.2/32 deferred penalty of 5 seconds for penalty: failed authentication
...
srclimit_penalise: 172.17.0.2/32: activating ipv4 penalty of 20 seconds for penalty: failed authentication
drop connection #0 from [172.17.0.2]:45510 on [172.17.0.3]:22 penalty: failed authentication
drop connection #0 from [172.17.0.2]:45514 on [172.17.0.3]:22 penalty: failed authentication

The first penalty being "deferred" and the one after the fourth being "activating ... 20 seconds" is exactly the arithmetic above. Let me also note what the client sees, because this is the line that will show up in support tickets: a blocked connection dies with kex_exchange_identification: read: Connection reset by peer. The Not allowed at this time text the server sends is only visible with ssh -v:

debug1: kex_exchange_identification: banner line 0: Not allowed at this time
kex_exchange_identification: read: Connection reset by peer
Connection reset by 172.17.0.3 port 22

I waited twenty seconds and logged in with the right key; no problem. Then I moved to the scenario I was actually curious about.

The seven-key agent and MaxAuthTries

The classic state of a developer machine: five or six keys left over from client projects in ssh-agent, and the key that matches the server at the end of the list. I reproduced it: six useless keys in the agent, the right one as the seventh. The result, because the server's MaxAuthTries default is 6, is a connection cut off before the right key gets its turn:

Received disconnect from 172.17.0.3 port 22:2: Too many authentication failures

That is an old story; we have been fixing it with IdentitiesOnly yes for years. What is new is the line added to the server log:

maximum authentication attempts exceeded for deneme from 172.17.0.2 port 55594 ssh2 [preauth]
Disconnecting authenticating user deneme 172.17.0.2 port 55594: Too many authentication failures [preauth]
srclimit_penalise: ipv4: new 172.17.0.2/32 deferred penalty of 5 seconds for penalty: failed authentication

Every "Too many authentication failures" is now a 5-second penalty. Think about that on a CI runner, in an office where 20 parallel jobs leave through the same NAT address, or in a deployment script whose key was rotated yesterday: the moment four failed departures fit into 15 seconds, that entire address is outside the door, and the legitimate clients behind it see Connection reset too. When I passed -i /root/dogru with IdentitiesOnly=yes, I got in on the first attempt; no penalty was written.

Monitoring that connects once a second

The measurement that surprised me most came from the noauth class. That penalty is for connections that leave without ever attempting authentication, and it is only 1 second; the man page itself warns that it "may penalise legitimate scanning tools such as ssh-keyscan". I tested the warning: I ran ssh-keyscan -t ed25519 20 times in a loop (a single key type, so one connection per run), all of it finished in under a second. Then I tried to log in with the right key:

20 keyscan 0 sn surdu                      (20 keyscans took 0 s)
--- ardindan dogru anahtarla giris:        (then, login with the right key:)
Connection reset by 172.17.0.3 port 22

The log says the threshold was crossed on the sixteenth connection (15 seconds accumulated, the sixteenth second exceeds the threshold):

srclimit_penalise: 172.17.0.2/32: activating ipv4 penalty of 16 seconds for penalty: connections without attempting authentication

The culprit here is not ssh-keyscan; it is my loop. Although a keyscan run without -t opens five separate connections on 10.0, one per key type, so the threshold is crossed on the fourth run. The same pattern is produced in real life by a load balancer's TCP health check or an Ansible wait_for task. The difference is tempo: a check that opens a single connection less often than once a second never accumulates, because a 1-second penalty ends in 1 second. When checks get denser, when each run opens several connections, or when several monitors look from the same address at the same time, the threshold is crossed. If it were me, I would put monitoring sources on the exempt list from day one; the next section does exactly that.

The exempt list and a sane configuration

PerSourcePenaltyExemptList takes addresses that penalties will never apply to; it accepts wildcards and CIDR ranges. I exempted the container network, sent a SIGHUP, tried five times in a row with the wrong key, and then logged in with the right one:

persourcepenaltyexemptlist 172.17.0.0/16
deneme@hedef: Permission denied (publickey,password).   (x5)
--- dogru:
GIRDIM

No penalty line was written. The minimal block I would recommend for my own setup:

# /etc/ssh/sshd_config.d/10-penalties.conf
PerSourcePenaltyExemptList 10.0.60.0/24,203.0.113.10
PerSourceNetBlockSize 32:64
PerSourcePenalties authfail:5 noauth:1 max:15m

The reasoning behind the three lines. Put your management network, the bastion and the monitoring servers on the exempt list; it is the only certain cure for locking yourself out. PerSourceNetBlockSize defaults to /32 for IPv4 and /128 for IPv6; the latter is too optimistic in my view, because a typical IPv6 client arrives holding an entire /64 and changing address on every attempt costs it nothing. Tying the penalty entry to the /64 closes that escape route. It has a cost, too: some VPS providers and mobile carriers split one /64 across several customers, and there one tenant's penalty locks out the neighbour. Do not write that line without knowing where your users come from. max:15m on the third line is personal preference: the default 10 minutes is enough against password guessing, but I keep it a bit longer. None of these options is valid inside a Match block; they operate at the listener level and can only be written in the main configuration.

The no keyword must appear alone. PerSourcePenalties no disables the mechanism entirely; a line like PerSourcePenalties no authfail:5 prevents sshd from starting. And if, as on 24.04, the option does not exist yet, the line simply does not appear in sshd -T output; the "what do we have" question before an upgrade is that easy to answer. For the same reason, do not push the drop-in file above to the whole fleet through configuration management: a pre-9.8 host rejects the PerSourcePenalties line as Bad configuration option and sshd does not start at all. Gate the file on the version.

When the table fills up: overflow and the deny-all trap

The penalty table is not unbounded; there is a separate max-sources limit for IPv4 and IPv6, 65,536 entries by default. "Deferred" penalties that have not crossed the threshold count towards that number too. What happens when the limit is reached is decided by overflow, and the difference between its two modes is less a security setting than an availability decision. The default, permissive, expires the soonest-to-expire penalties early to make room for new connections; so if a large botnet fills the table, sshd starts forgetting penalties but leaves nobody outside. deny-all is the opposite: while the table is full, everyone not on the exempt list is refused, penalised or not, until a penalty expires.

deny-all looks "safer" on paper; do not be surprised to see it recommended in hardening guides. I would only consider it with a complete exempt list and console access, because in that mode the attacker's target is no longer your password but the table itself: one failed attempt from each of 65 thousand source addresses is enough to lock you out of your own server. On IPv6 that many addresses come for free inside a single /64, which is the second reason for my /64 recommendation for PerSourceNetBlockSize. The good part of the code is that the two address families are counted separately: a full IPv4 table does not affect the IPv6 side, and overflow6 lets you give IPv6 a different mode.

MaxStartups sits on the same code path. The listener checks the penalty table first; if there is no penalty it moves on to the MaxStartups limit (default 10:30:100: once the number of unauthenticated connections reaches 10, random early drop begins). Both mechanisms log the same drop connection #N from [...] line; the reason at the end of the line tells them apart: if it starts with penalty: ... it is a penalty, if it says Maxstartups it is load. Look at that last word when diagnosing.

Should fail2ban be retired?

No, but its job changes. The two mechanisms work in different places and at different scales. fail2ban reads the log, so the attacker at least reaches authentication a few times; in return the ban can last hours or days and is enforced in the kernel via nftables. PerSourcePenalties works on a scale of seconds to minutes, in process memory, and evaporates on restart; in exchange it cuts the connection right after accept() without sshd spending CPU. CVE-2025-26466 in the 9.9p2 release notes shows exactly that difference: as the mitigation for a pre-authentication memory and CPU exhaustion via PING packets, the notes pointed to PerSourcePenalties, not fail2ban. A tool that reads logs cannot see an attack that has not yet produced a log line.

On my server both looked unnecessary; moving the port had already cut the noise. But when that server moves to 26.04, the first thing I will do is not remove fail2ban but write the address my own automation comes from onto the exempt list. Because one day I will rotate the key of that script that logs in 484 times a day, and on that day the script will leave four times in a row with the wrong key.

Pre-upgrade checklist

Run sshd -T | grep -i persource on the target version; if the line is missing the mechanism is absent, if it is there note the values.
List everything that connects to the server without authenticating: health checks, port-probing monitors, wait_for tasks. Put their source addresses into PerSourcePenaltyExemptList.
For CI runners and office NAT addresses, make sure only one key is offered, via IdentitiesOnly yes; "Too many authentication failures" is now a penalty, not just an error.
If you use IPv6, try PerSourceNetBlockSize 32:64 and check beforehand that your IPv6 management range is on the exempt list.
When diagnosing, look for the drop connection ... penalty: line; it is at INFO on every version. The "who crossed the threshold when" line is VERBOSE before 10.1, and the "deferred" line where the penalty is first written is VERBOSE even today. From 10.0 the drop lines are rate-limited (under a burst you get a "logging rate-limited: additional N connections dropped" summary); if you build metrics by counting log lines, collect that summary line too.
When you do get locked out, do not panic: the default ceiling is 10 minutes and systemctl reload ssh clears it instantly. Keep another way in (console, another address).
Keep fail2ban, but do not confuse its bantime with sshd's max; they are different layers.

The reflex inside the black box

For years we thought of sshd as a passive door: it let you in if the key was right and did not if it was wrong; managing the crowd in front of the door was the job of outside tools. From 9.8 on, the door has a reflex of its own: it counts who stumbled how many times and ignores whoever crosses the threshold. Security-wise, that is a step in the right direction. But reflexes are not conscious decisions; when the kneecap is tapped, the leg does not check whose leg it is. The 20 seconds I saw in the container were designed to slow down an attacker; the same 20 seconds also fall to a deployment script whose key was just rotated. The question to ask before upgrading is not "is this mechanism good" but "do I know who on my network will trigger the reflex".

Sources used in the body but not in the list: OpenSSH release notes (changes to the penalty table from 9.8 to 10.5), man.openbsd.org sshd_config and the RHEL 10.0 release notes (OpenSSH 9.9 and the PerSourcePenalties entry).

DEV Community