How to update your terraformed RDS / Aurora certificates

This post was prompted by Jeff Barr's recent Urgent & Important blog post

If you use RDS or Aurora you have probably received an email with this subject recently:

Important Reminder: Update Your Amazon RDS SSL/TLS Certificates by February 5, 2020

If you operate many AWS accounts or have put this change on the back burner your inbox might look like this:

I was a bit concerned when I first saw this communication in October. A lot more people are now concerned as the deadline is looming and the AWS social media / email campaign has ramped up. You even have to dismiss a modal upon logging in to AWS RDS Console at the minute to acknowledge these changes!

Does this affect you?

The first thing you have to do is figure out if this issue affects you. If you don't make SSL connections to your database this issue probably doesn't affect you.

If you use IAM Database Authentication this definitely affects you!

The first thing to do is to update your application to support the new certificate.

Update your Application

Note: If you do not have some kind of sandbox / staging environment it is worth spinning up an RDS / Aurora cluster with your existing terraform code so you can test all this safely

There are 100s of possible languages / clients you may be using to interact with RDS, so this part is up to you. I will give a lowest common denominator example in Bash towards the end of this post. But you need to update the certificate you use to connect to RDS.

AWS have updated their combined ca bundle to include both the old and new certificate to avoid a chicken and egg scenario. i.e the

• rds-ca-2015-root.pem will only work with RDS instances configured to use the old 2015 certificate
• rds-ca-2019-root.pem will only work with RDS instances configured to use the new 2019 certificate

By using the rds-combined-ca-bundle.pem certificate in our application we can update the underlying RDS certificate without issue. Those certificates are available in S3. Links as well as additional documentation are available in the official AWS Documentation

Terraform Changes

Typically when I get a notification like this from AWS my initial thought is "I hope I can achieve this with Terraform" I have found when you start using Infrastructure as Code you want every single change to be orchestrated by that tool. Out of bounds changes feel dirty and you want everything to be driven by version controlled code.

Both the aws_db_instance and aws_rds_cluster_instance resources support the ca_cert_identifier argument. You can update each resource to have the ca_cert_identifier argument. You probably want to expose this as a variable so that other consumers of this module have a choice. You may still be responsible for this code in 5 years when the 2024 cert is rolled out!

resource "aws_db_instance" "db" {
  # truncated for brevity
  tags                    = "${var.tags}"
  apply_immediately       = "${var.apply_immediately}"
  ca_cert_identifier      = "${var.ca_cert_identifier}"
}

Let's update our variable to rds-ca-2019 so that our aws_rds_cluster_instance resource is configured to use the new certificate.

variable "ca_cert_identifier" {
  default     = "rds-ca-2019"
  description = "Allows you to modify the underlying RDS certificate"
}

We can now run a terraform plan

At the point you may get an error like this:

Error: module.your_module.aws_db_instance.db: "ca_cert_identifier": this field cannot be set

Thats because you write best practices terraform and have pinned your provider version!

provider "aws" {
  version = "~> 1.52"
  region  = "${var.aws_region}"
}

Your current provider does not have support for updating the ca_cert_identifier. We pin terraform providers for the same reason we pin software libraries. To avoid breaking changes and other surprises from upstream. Typically you can head to the GitHub repo for the AWS provider and find out A) If what you are trying to do is supported. B) What provider version it was introduced in.

In this case (and most cases) the provider maintainers are on top of things and support for updating the certificate was introduced in v2.37.0 of the provider. We can simply bump our minimum provider version

provider "aws" {
  version = "~> 2.37"
  region  = "${var.aws_region}"
}

Lets do another plan.

~ module.your_module.aws_db_instance.db
    ca_cert_identifier:   "rds-ca-2015" => "rds-ca-2019"

Plan: 0 to add, 1 to change, 0 to destroy.

Now we get the expected plan.

However this change would not be applied immediately. Many changes to RDS resources are applied at the next maintenance window. Depending on the workload this may be the desired approach. We can run terraform apply
and make the change.

You may notice that if you do another plan it looks like our change did nothing. That's because the certificate won't be changed until a reboot during the maintenance window. To verify our work we can use the aws cli

aws rds describe-db-instances --db-instance-id your-db-identifier | jq '.DBInstances[].PendingModifiedValues'


{
  "CACertificateIdentifier": "rds-ca-2019"
}

Here we can see this instance has pending modifications.

Applying immediately

In this particular case I want to make the change immediately so I can make a connection from my client and ensure the certificate chain works as expected

I recommend exposing an apply_immediately variable on modules that create RDS resources.

variable "apply_immediately" {
  default     = false
  description = "Should modifications be applied immediately? Otherwise they are applied at next maintenance window"
}

It has a sensible default of false, but we still have the flexibility to apply changes immediately. In a production environment maybe somebody else on your team has to approve a Pull Request to make this change and your PR body would explain your rationale for the change. After the change is made you would set apply_immediately back to false

After setting our apply_immediately variable to true our plan looks like this:

~ module.your_module.aws_db_instance.db
    apply_immediately:    "false" => "true"
    ca_cert_identifier:   "rds-ca-2015" => "rds-ca-2019"

Plan: 0 to add, 1 to change, 0 to destroy.

For me this apply took 90 seconds. The RDS instance rebooted and now has the new certificate.

Bash example of connecting to an RDS instance with SSL and IAM authentication. The following code should work before and after your certificate upgrade

wget https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
TOKEN="$(aws rds generate-db-auth-token --hostname the_host --port 3306 --region eu-west-1 --username db_user)"
mysql -h the_host --port=3306 --ssl-ca=rds-combined-ca-bundle.pem --enable-cleartext-plugin -u db_user --password=$TOKEN"

I hope this helps anyone that has terraformed RDS workloads.

Comments

[Gopikrishna Sathyamurthy]

the apply_immediately saved my bacon, thank you!!