Blue/Green Deployments for RDS: How Fast is a Switchover?

AWS recently announced the general availability of RDS Blue/Green Deployments, a new feature for RDS and Aurora to perform blue/green database updates. One of the aspects that caught my eye is how fast a switchover is.

In as fast as a minute, you can promote the staging environment to be the new production environment with no data loss.

Let’s give it a try! How can we check how long a switchover really takes? We will perform the simplest test, using only a MySQL client and a while loop in Bash:

while true; do mysql -s -N -h renato-cluster.***.eu-west-1.rds.amazonaws.com -u renato 
-e "select now()"; sleep 1; done

(...)
2022-12-09 07:58:55
2022-12-09 07:58:56
2022-12-09 07:58:57
2022-12-09 07:58:58
(...)

We keep asking every second for the current date and time to the RDS server and we will check what happens when we trigger a switchover. We will compare the results with a reboot and a reboot with failover. To perform the test we will use an m6g.large Multi-AZ instance with some production traffic, a cluster where the Seconds_Behind_Master value of a replica hardly goes above a single-digit number. 

Rebooting an instance

Using the CLI, we can perform a reboot of an RDS instance without forcing a failover of the database server.

aws rds reboot-db-instance --db-instance-identifier renato-cluster --no-force-failover

(...)
2022-12-09 08:00:01
2022-12-09 08:00:02
2022-12-09 08:00:03
2022-12-09 08:00:23
2022-12-09 08:00:24
2022-12-09 08:00:25
(...)

That is just about 20 seconds for a reboot of an RDS instance.

Rebooting with failover

We can now repeat the test forcing a failover:

aws rds reboot-db-instance --db-instance-identifier renato-cluster --force-failover

(...)
2022-12-09 08:01:11
2022-12-09 08:01:12
2022-12-09 08:01:13
2022-12-09 08:03:28
2022-12-09 08:03:29
2022-12-09 08:03:30
(...)

This time it takes longer, including the time required to switch the CNAME of our database to the new IP address: 2 minutes and 15 seconds.

Switching a blue/green deployment

Finally, we can trigger a switchover in our blue/green deployment calling:

aws rds switchover-blue-green-deployment --blue-green-deployment-identifier renato-cluster-bg

(...)
2022-12-09 08:06:59
2022-12-09 08:07:00
2022-12-09 08:07:01
2022-12-09 08:08:07
2022-12-09 08:08:08
2022-12-09 08:08:09
(...)

The switchover time of the RDS Blue/Green Deployments is just about a minute: 1 minute and 6 seconds. 

Recap

• Reboot: 20 seconds
• Reboot with failover: 135 seconds
• Switchover: 66 seconds

The switchover of an RDS for MySQL instance is “as fast as a minute” and significantly faster than a reboot with failover of a Multi-AZ instance. Of course, the switchover can take significantly longer according to the write load of your production database. This was only a basic test to validate the minimum downtime for a switchover.

Questions? Comments? Contact or follow me.
Originally published on cloudiamo.com.

Comments

[Dennis Groß (he/him)]

Thanks Renato for the interesting post. :)

Does the switch over require a standby instance or will the Blue/Green deployment strategy promote a read-replica as well?

[Renato Losio 💭💥]

If the blue instance has read replicas, the read replicas are copied as read replicas of the green. Cascading and cross-region read replicas are not supported.

[Dennis Groß (he/him)]

thanks for explaining Renato!