Use AWS Certificate Manager (ACM) to simplify UI certification management for CockroachDB

Recently a customer asked me how to use AWS Certificate Manager(ACM) to manage certifications for self-hosted CockroachDB clusters. I looked into it and would like to share tips and tricks below. Please feel free to comment below to let me know know what other topics you like to see!

Problem statement:

CockroachDB secure cluster requires TLS/HTTPS to access DBConsole, and we currently recommend using Let’s Encrypt and upload UI Certs to cockroach nodes. When certificates expires, this requires additional administrative effort to rotate and maintain the certs.

Solution:

AWS ACM can issue and auto renew certificates when using DNS validation and thus can reduce administrative overhead. AWS NLB TLS termination can handle TLS decryption between browser and NLB, and re-encryption between NLB and CockroachDB nodes to meet the requirement of CockroachDB. AWS NLB document isn’t very clear on how to configure it, so we’re adding additional screenshots here.

The other benefit is we now only need to manage Node Certs on the server side.

Walkthrough:

1. Request a public certificate in ACM for a domain name. I used jessielin.xxxx.dev in this case.

1. Create Network Load Balancer with 2 listeners. One for DBConsole, and one for SQL access.

a. For DBConsole access, add TLS listener and Target Group. Use the certs issued by ACM. Target Group port should be the port number specified in --http-addr. By using TLS listener and TLS target group AWS NLB will decrypt and re-encrypt.

Target Group

Healthcheck

b. For SQL access, add TCP listener and forward it to sql port. The official document explains it very well.

1. Add LB hostname and ip address to Node certs

a. use openssl x509 -in certs/node.crt -text to find out existing nodes' hostname, ip addresses

b. LB hostname is required to Common Name or Subject Alternative Names fields of the certificate, documented here. To Add LB hostname and ip addr to the list, use cockroach cert create-node to create new certs as documented and redistribute to all nodes. Validate the new certs are loaded correctly from DBConsole

1. Create an A record in Route 53 to redirect the jessielin.xxxx.dev to the NLB

1. Voila there you have it!