If you’re not using SSH certificates you’re doing SSH wrong | Episode 3: An ideal SSH flow

In our previous two episodes, we covered the benefits of SSH certificates and how they can help developers, operators, and security teams better manage privileged access. Today we will look at an ideal SSH flow using certificates and your existing identity provider for single sign-on access.

An ideal SSH flow

SSH certificate authentication is the foundation of what I think is the ideal SSH flow.

To SSH, users first run a login command in their terminal (e.g., step ssh login):

$ step ssh login
Your default web browser has been opened for you to login: https://accounts.google.com/...

A browser is opened and an SSO flow is initiated at your organization's identity provider:

A web-based SSO flow makes it easy to leverage strong MFA (e.g., FIDO U2F) and any other advanced authentication capabilities your identity provider offers. Users login with a familiar flow, and removing a user from your canonical identity provider ensures prompt termination of SSH access.

Once the user completes SSO, a bearer token (e.g., an OIDC identity token) is returned to the login utility. The utility generates a new key pair and requests a signed certificate from the CA, using the bearer token to authenticate and authorize the certificate request.

The CA returns a certificate with an expiry long enough for a work day (e.g., 16-20 hours). The login utility automatically adds the signed certificate and corresponding private key to the user's ssh-agent.

Users needn't be aware of any of this detail. All they need to know is that, in order to use SSH, they must first run step ssh login. Once that's done they can use SSH like normal:

$ ssh prod01.example.com
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1036-gcp x86_64)

Last login: Wed Sep 11 04:04:51 2019 from 98.210.132.79
mmalone@prod01:~$

Like browser cookies, short-lived certificates issued by this flow are ephemeral credentials, lasting just long enough for one work day. Like logging into a website, logging into SSH creates a session. It's a simple process that must be completed, at most, once per day. This is infrequent enough that strong MFA can be used without frustrating or desensitizing users.

New private keys and certificates are generated automatically every time the user logs in, and they never touch disk. Inserting directly into ssh-agent insulates users from sensitive credentials. If a user wants to connect from a different device it's easier for them to run step ssh login there than it is to exfiltrate keys from ssh-agent and reuse them.

There are lots of possible variations of this flow. You can adjust the certificate expiry, use PAM authentication at the CA instead of SSO, generate the private key on a smart card or TPM, opt not to use ssh-agent, or move MFA to the actual SSH connection. Personally, I think this combination offers the best balance of security and usability. Indeed, relative to most existing SSH deployments it's operationally simpler, more secure, and more usable.

Critics of SSH certificate authentication say that it's new, not well supported, and the tooling doesn't exist to use certificates in practice. The truth is, certificate authentication was added in OpenSSH 5.4 almost a decade ago. It's battle tested and used in production by massive operations. And the tooling required to build this ideal SSH flow is available today.

Tools

There are lots of existing tools for managing SSH certificates. Here are a few:

• step-ca is now an SSH CA (in addition to being an X.509 CA)
• step makes it easy for users and hosts to get certificates from step-ca
• ssh-keygen can generate root certificates and sign user & host certificates
• netflix/bless is Netflix's SSH CA that runs in AWS Lambda and uses IAM
• nsheridan/cashier is Intercom's SSH CA
• uber/pam-ussh lets you use certificates to authorize sudo use
• hashicorp/vault has an SSH secrets engine

With the appropriate configuration of step-ca you can use step to:

Get a host certificate automatically at startup

To demonstrate, let's create a new EC2 instance with the aws CLI tool. The interesting bits are tucked in some light configuration (using a user-data startup script) that gets a host certificate and enables certificate authentication for users:

$ aws ec2 run-instances --image-id ami-07d0cf3af28718ef8 \
    --count 1 --instance-type t2.micro \
    --key-name mike-test \
    --security-group-ids launch-wizard-7 \
    --user-data file://startup.sh

Note: you should be able to use our instance identity document support here, but we've got a few kinks to work out. Stay tuned.

Get a user certificate using SSO (OAuth OIDC)

Now we'll use step ssh certificate locally (you can brew install step) to generate a new key pair, get a certificate from the CA using SSO, and automatically add the certificate and private key to ssh-agent.

That sounds like a lot, but it's just one command:

$ step ssh certificate [e-mail] id_ecdsa
✔ Provisioner: Google (OIDC)
Your default web browser has been opened to visit:

https://accounts.google.com/o/oauth2/v2/auth?client_id=650...

✔ CA: https://ca.internal
✔ Private Key: id_ecdsa
✔ Public Key: id_ecdsa.pub
✔ Certificate: id_ecdsa-cert.pub
✔ SSH Agent: yes

Once that's done we can SSH to the instance we just created, using certificate authentication, with no TOFU!

$ ssh [e-mail]
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1044-aws x86_64)

Last login: Thu Sep 12 02:25:43 2019 from 98.210.132.79
mike@ip-172-31-70-94:~$

For more info check out our getting started guide and SSH example repo. Make sure you pass the --ssh flag to step ca init when you're setting up the CA (the getting started guide doesn't do this).

There's a lot more that can be done to make SSH certificate authentication even more awesome. We're working on that. If you have any ideas, let us know!

Use SSH certificates

SSH certificate authentication does a lot to improve SSH. It eliminates spurious TOFU warnings and host key verification failures. It lets you drop complex key approval & distribution processes and extend SSO to SSH. It makes rekeying possible for hosts and easier than key reuse for users. It makes SSH keys ephemeral, making key management oversights fail-secure.

You can deploy an SSH CA and reconfigure hosts in a matter of minutes. It's easy to transition — you can continue supporting public key authentication at the same time.

SSH certificate authentication is the right way to do SSH.

To learn more about SSH certificates visit the Smallstep website. You can even try our free hosted offering and experience the value of SSH certificates in under five minutes!

Comments

[Billie Hilton]

Wow! I wish GitHub and Circle CI supported this out of the box. And DigitalOcean, and AWS, and...

[Mike Maxey]

Actually, GitHub enterprise does support this out of the box:
docs.github.com/en/free-pro-team@l...
We are building an integration to connect it to your identity provider:
youtube.com/watch?v=jt4O-HplX0k
Best,
Maxey
VP Products at Smallstep