DEV Community

Ramer Labs
Ramer Labs

Posted on

7 Tips for Securing SSH Key Management on Linux Servers

#linux #security #devops

Introduction

If you’re a DevOps lead responsible for a fleet of Linux servers, SSH keys are the lifeline of your day‑to‑day operations. They’re convenient, but a single leaked private key can open the doors to your entire infrastructure. This tutorial walks you through seven practical steps to harden SSH key management, from generation to rotation and audit.

1. Generate Strong, Unique Keys per User

Never reuse the same key across multiple accounts or servers. Use a modern algorithm and a sufficient key size:

# Generate an Ed25519 key (recommended) with a 100‑character comment
ssh-keygen -t ed25519 -a 100 -C "alice@prod‑bastion"
Enter fullscreen mode Exit fullscreen mode
  • Why Ed25519? It offers comparable security to RSA‑4096 with a much smaller footprint and faster operations.
  • Passphrase protection adds a second factor. Store the passphrase in a secure password manager, not in plain text.

2. Centralize Private Keys with an SSH Agent or Vault

Keeping private keys scattered on workstations is risky. Two approaches are common:

  • ssh‑agent: Load the key once per session and let the agent handle authentication.
  • HashiCorp Vault (or similar): Store the private key in a sealed secret backend and retrieve it on demand.

Example of adding a key to the agent:

ssh-add ~/.ssh/id_ed25519
Enter fullscreen mode Exit fullscreen mode

If you prefer Vault, a minimal policy might look like:

path "ssh/creds/readonly" {
  capabilities = ["read"]
}
Enter fullscreen mode Exit fullscreen mode

3. Harden authorized_keys with Options

The authorized_keys file can enforce restrictions per key. Add options before the key material:

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="/usr/local/bin/readonly-shell" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE... alice@prod-bastion
Enter fullscreen mode Exit fullscreen mode

Common options:

  • no-pty – prevents interactive shells.
  • restrict – a shorthand for a set of safe defaults.
  • from="10.0.0.0/8" – limits the source IP range.

These constraints limit what an attacker can do even if they obtain a private key.

4. Deploy a Bastion Host (Jump Box)

Instead of allowing direct SSH to production boxes, funnel all traffic through a hardened bastion. The bastion can:

  • Enforce MFA (e.g., Duo, Google Authenticator).
  • Log every session with auditd or tlog.
  • Apply rate‑limiting and intrusion‑detection rules.

A minimal sshd_config for the bastion might include:

Port 22
Protocol 2
PermitRootLogin no
PasswordAuthentication no
AllowTcpForwarding no
LogLevel VERBOSE
Enter fullscreen mode Exit fullscreen mode

5. Rotate Keys Regularly

Treat SSH keys like passwords: rotate them on a schedule (quarterly is a good baseline) or immediately after any personnel change. Automate rotation with a simple script:

#!/usr/bin/env bash
# Rotate a user’s key and push to all servers via Ansible
USER=alice
NEWKEY=$(ssh-keygen -t ed25519 -a 100 -f /tmp/${USER}_id -N '' -C "${USER}@$(hostname)" && cat /tmp/${USER}_id.pub)
ansible all -m authorized_key -a "user=${USER} key='${NEWKEY}' state=present"
Enter fullscreen mode Exit fullscreen mode

Make sure to revoke the old key (state=absent) after the new one is verified.

6. Use Hardware Security Modules (HSM) or YubiKeys

For high‑value accounts, store the private key on a hardware token that never leaves the device. The workflow changes slightly:

# Use the YubiKey as a smartcard for SSH
ssh-add -K /usr/local/lib/opensc-pkcs11.so
Enter fullscreen mode Exit fullscreen mode

The private key never touches the filesystem, dramatically reducing exposure.

7. Audit and Monitor SSH Activity

Visibility is the final line of defense. Enable verbose logging on the SSH daemon and ship logs to a centralized SIEM.

# /etc/ssh/sshd_config additions
LogLevel VERBOSE
# Forward logs to syslog
SyslogFacility AUTH
Enter fullscreen mode Exit fullscreen mode

Set up alerts for:

  • Failed login spikes (possible brute‑force).
  • Key usage from unexpected IPs.
  • Execution of restricted commands.

Tools like fail2ban can automatically ban offending IPs, while auditd can capture command execution details.

Quick Checklist

  • [ ] Generate unique Ed25519 keys with passphrases.
  • [ ] Store private keys in an SSH agent or Vault.
  • [ ] Apply restrictive options in authorized_keys.
  • [ ] Route all SSH through a bastion host.
  • [ ] Schedule quarterly key rotations.
  • [ ] Use hardware tokens for privileged accounts.
  • [ ] Enable VERBOSE logging and forward to a SIEM.

Conclusion

Securing SSH key management is a blend of disciplined processes and solid configuration. By generating strong keys, centralizing secrets, tightening authorized_keys, funneling traffic through a bastion, rotating keys, leveraging hardware tokens, and continuously auditing activity, you dramatically shrink the attack surface of your Linux fleet.

For more practical guides on hardening infrastructure and best‑practice workflows, check out https://lacidaweb.com.

Top comments (0)