If you’re not using SSH certificates you’re doing SSH wrong | Episode 1: Keys versus Certificates

SSH is ubiquitous. It's the de-facto solution for remote administration of *nix systems. But SSH has some pretty gnarly issues when it comes to usability, operability, and security.

You're probably familiar with these issues:

• SSH user experience is terrible. SSH user on-boarding is slow and manual. Connecting to new hosts produces confusing security warnings. You're left with weird new credentials to manage with little guidance on how to do so.
• Operating SSH at scale is a disaster. Key approval & distribution is a silly waste of time. Host names can't be reused. Homegrown tools scatter key material across your fleet that must be cleaned up later to off-board users.
• SSH encourages bad security practices. Rekeying is hard, so it's not done. Users are exposed to key material and encouraged to reuse keys across devices. Keys are trusted permanently, so mistakes are fail-open.

The good news is this is all easy to fix.

None of these issues are actually inherent to SSH. They're actually problems with SSH public key authentication. The solution is to switch to certificate authentication.

SSH certificate authentication makes SSH easier to use, easier to operate, and more secure.

Background

At smallstep, certificates are kind of our jam. We build open source software that lets you run your own private certificate authority and manage X.509 (TLS/HTTPS) certificates.

SSH certificates have been on our radar for a while. From our perspective they're all pros, no cons. But no one uses them. Why not? We've asked hundreds of people that question. Here's what we found out:

• People do use SSH certificates. In fact, pretty much everyone operating at scale that knows what they're doing does (Facebook, Uber, Google, Netflix, Intercom, Lyft, etc), but...
• Certificates and public key infrastructure (PKI) are hard to grok. People don't immediately understand the benefits.
• There's a (small) tooling gap that exacerbates this knowledge gap. It's not hard to fill, but people are wary to do so themselves without a deeper understanding of PKI concepts.
• More than anything, SSH certificates haven't gotten the press they deserve. Most people we asked hadn't heard of them at all.

We're convinced that SSH certificates are the right way to do SSH. They're not that hard to understand, and it's well worth the effort. SSH certificates deserve more press, and broader use.

Public key authentication

Most SSH deployments use public key authentication, which uses asymmetric (public key) cryptography with a public / private key pair generated for each user & host to authenticate.

The magic of asymmetric cryptography is the special correspondence between a public and private key. You can sign data with your private key and someone else can verify your signature with the corresponding public key. Like a hash, it's computationally infeasible to forge a signature. Thus, if you can verify a signature, and you know who owns the private key, you know who generated the signature.

Simple authentication can be implemented by challenging someone to sign a big random number. If I open a socket to you and send a random number, and you respond with a valid signature over that number, I must be talking to you.

This is an oversimplification, but it's more or less how SSH public key authentication works. Certificate authentication works the same way, but with an important twist that we'll get to in a moment.

To SSH to a host using public key authentication the host needs to know your public key. By default, your public key must be added to ~/.ssh/authorized_keys. Maintaining this file for every user across a fleet is operationally challenging and error prone.

SSH user onboarding with public key authentication usually starts with some baroque incantation of ssh-keygen, hopefully pulled from a runbook, but more likely cribbed from stack overflow. Next you'll be asked to submit your public key for approval and distribution. This process is typically manual and opaque. You might be asked to email an administrator or open a JIRA ticket. Then you wait. While you're doing that, some poor operator gets interrupted and told to add your key to a manifest in some repo and trigger a deploy. Once that's done you can SSH. Since key bindings are permanent, your SSH access will continue in perpetuity until someone reverses this process.

Certificate authentication

Certificate authentication eliminates key approval and distribution. Instead of scattering public keys across static files, you bind a public key to a name with a certificate. A certificate is just a data structure that includes a public key, name, and ancillary data like an expiration date and permissions. The data structure is signed by a certificate authority (CA).

$ ssh-keygen -L -f id_ecdsa-cert.pub
id_ecdsa-cert.pub:
        Type: [e-mail] user certificate
        Public key: ECDSA-CERT SHA256:O6M6oIjDm5gPm1/aTY619BgC3KSpS4c3aHVWxYh/uGQ
        Signing CA: ECDSA SHA256:EY2EXJGoPv2LA6yEbjH+sf9JjG9Rd45FH1Wt/6H1k7Y
        Key ID: "[e-mail]"
        Serial: 4309995459650363134
        Valid: from 2019-09-11T14:50:01 to 2019-09-11T18:50:01
        Principals:
                mike
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

To enable certificate authentication simply configure clients and hosts to trust any certificates signed by your CA's public key.

On each host, edit /etc/ssh/sshd_config, specifying the CA public key for verifying user certificates, the host's private key, and the host's certificate:

# Path to the CA public key for verifying user certificates
TrustedUserCAKeys /etc/ssh/ssh_user_key.pub

# Path to this host's private key and certificate
HostKey /etc/ssh/ssh_host_ecdsa_key
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub

On each client, add a line to ~/.ssh/known_hosts specifying the CA public key for verifying host certificates:

@cert-authority *.example.com ecdsa-sha2-nistp256 AAAAE...=

That's it. That's literally all that you need to do to start using certificate authentication. You can even use it alongside public key authentication to make transitioning easier.

Static keys in ~/.ssh/authorized_keys are no longer needed. Instead, peers learn one another's public keys on demand, when connections are established, by exchanging certificates. Once certificates have been exchanged the protocol proceeds as it would with public key authentication.

To learn more about SSH certificates visit the Smallstep website. You can even try our free hosted offering and experience the value of SSH certificates in under five minutes!