In August an external penetration test listed twenty three of our Linux servers as accepting SSH password logins. Our hardening playbook had set PasswordAuthentication no on every one of them since the day they were built, and its compliance run, in check mode every night, reported nothing to change.
The playbook used lineinfile on /etc/ssh/sshd_config, with a regular expression that matched the commented default and replaced it with PasswordAuthentication no, then restarted sshd. I logged in to one of the reported hosts and the line was there, exactly as written. Then I ran sshd -T, which prints the configuration the daemon actually uses, and it said passwordauthentication yes.
Every reported server had been built from our hosting provider's newer Ubuntu image. Near the top of its sshd_config is the line Include /etc/ssh/sshd_config.d/*.conf, and in that directory cloud-init had written 50-cloud-init.conf, containing PasswordAuthentication yes, because the image offers a password at first boot. The sshd_config manual is plain about what happens next: for most keywords, the first value obtained is the one used. The include is read before the rest of the main file, so the drop-in won, and our line further down was read and ignored. Servers built from the older image had no such drop-in, which is why the playbook had worked for years.
Nobody had lied. The file said no, the playbook truthfully reported that the file said no, and check mode compares files with files. Nothing in our process ever asked the daemon.
We fixed it the same afternoon. The playbook now writes its settings to /etc/ssh/sshd_config.d/00-hardening.conf, which sorts first and so is read first, and removes PasswordAuthentication from any other drop-in. Every change is validated with sshd -t before it is put in place. After the restart a task runs sshd -T and fails the play unless the effective values are the ones we asked for, so the compliance run checks behaviour, not text. The image we build on is pinned, and a new one goes through the playbook in CI on a throwaway machine before any host uses it. And a nightly scan from outside our network asks each server which authentication methods it offers, which is all the penetration testers had done.
Configuration management sets what a file says. A program decides what that file means, and ours read another file first.
– Sergey Shinder
Top comments (0)