One firewall command, one locked server

#linux #security #devops #sysadmin

One firewall command, one locked server

Fresh VPS. Doing everything right — hardening the box before it went live. SSH keys installed, updates applied, firewall package ready. Time to lock it down, so I ran the firewall enable command. The session froze mid-command.

I reconnected. Connection refused. Tried again. Refused. The server was up, healthy, fully reachable — and completely closed to me. One command, and I had locked myself out of a machine I'd owned for under an hour.

What the firewall did

The firewall did exactly what it was told. Enabling it applied the default policy — deny everything inbound — and at that moment, my allow list was empty. I hadn't added the SSH rule yet. The very first rule to take effect was "deny all," and my own access wasn't on the allow list because the allow list didn't exist.

The trap: the firewall isn't judging your intentions. It doesn't know you're the administrator mid-hardening. It reads the rules as they exist right now, and right now the rules said nobody gets in. There is no grace period, no "but I was about to add the exception." Deny-all is deny-all.

The policy was right; the sequence was wrong

"Default deny, then allow" is the correct firewall policy. Every security guide says so, and every security guide is right. My mistake wasn't the policy — it was treating the policy as the first step instead of the last one.

Deny-all-then-allow is a policy, not a sequence. As a sequence, it's: build the allow list first, then flip the switch. When your own access isn't in the list yet, enabling the firewall is pulling up the drawbridge while you're standing outside the castle. The order is the entire difference between "hardened" and "bricked."

There is no undo from the outside

Here's what makes this mistake so expensive: once the firewall is up and SSH is blocked, you have no command channel left. You can't SSH in to fix the rule that blocks SSH. The box is not broken — it's faithfully enforcing your instructions, and your instructions say "go away."

The escape hatch is the provider console — a shell that doesn't go through the network firewall. If you've confirmed it works before you touch the rules, a lockout is a two-minute recovery. I hadn't. So the only fix available was rebuilding the server from scratch: wipe, reinstall, redo the entire setup I'd already finished. An hour of work, gone, because of the order of two commands.

Checklist

Allow SSH before enabling the firewall. Add the port 22 rule first — it's the first command, not the second.
Verify the rule is actually active before enabling. Read back the ruleset; don't trust your memory of what you typed.
Confirm break-glass access works. Log into the provider console and prove you can get a shell without SSH — before you touch firewall rules, not after.
Then enable the firewall. Default-deny is the last step, applied on top of a complete allow list.
Never close your only session. Make rule changes from a second SSH session; keep the first one open until you've proven the new rules let you back in.