Fail over RDS on purpose: a Multi-AZ restore drill, with the commands and the timings

The high-availability design says an RDS Multi-AZ failover is typically one to two minutes, and that committed data survives an Availability Zone loss. That number is a planning figure from AWS documentation. It is not a measurement of your instance, your DNS cache, or your connection pool.

This is the drill I run to replace that figure with three timestamps: when RDS finishes the failover, when a new connection succeeds, and how far back a point-in-time restore can actually reach. It uses a disposable Multi-AZ DB instance. The RDS API calls are the same for MySQL. The client commands below are PostgreSQL.

Companion to Designing a Highly Available and Scalable Architecture on AWS.

Who this is for: Cloud operations and SRE people who already have a Multi-AZ design and want a repeatable test, with the commands, before they trust it in production.

Production note: Run this on an instance created for the drill, or on a non-production copy restored from a snapshot. reboot-db-instance --force-failover interrupts the primary immediately. Open transactions roll back. AWS recommends stopping transactions before a reboot with failover. The open transaction later in this post exists only on the disposable instance, so the drill can show that rollback.

---

Two drills, two different promises

A Multi-AZ DB instance keeps a standby in a second Availability Zone and replicates storage synchronously. A reboot with failover promotes that standby and updates the DNS record of the same endpoint. Committed transactions are on both copies, so the recovery point for committed work is near zero. Anything not committed is gone.

A point-in-time restore builds a new instance from the daily backup plus transaction logs. RDS uploads those logs to S3 about every five minutes. LatestRestorableTime is usually within about five minutes of now. The new instance has a new endpoint. Nothing in your application moves until you change a DNS record you control.

	Multi-AZ failover	Point-in-time restore
Command	reboot-db-instance --force-failover	restore-db-instance-to-point-in-time
Endpoint	Unchanged. The IP behind it changes.	New hostname
Committed data	Kept	Kept up to the restore time
Open transactions	Rolled back	Not on the new instance
What it proves	RTO of an AZ failure, and that the app reconnects	RPO of the backup chain, and restore duration

If describe-db-instances returns a DBClusterIdentifier, stop. You are on Amazon Aurora or a Multi-AZ DB cluster, and the failover command is failover-db-cluster. The steps below are for a Multi-AZ DB instance: one primary, one standby, standby not readable.

---

1. Create an instance you can afford to break

Use a private subnet group that already exists in a non-production VPC. Pick the current PostgreSQL 16 minor, and the smallest instance class that Region offers for Multi-AZ. db.t4g.micro is the usual starting point. Multi-AZ bills the instance twice, and the restored copy in section 4 is a second instance until you delete it. Delete both at the end.

Run every command in this drill from a host inside the VPC, over SSH. This session is that host.

ENGINE_VERSION=$(aws rds describe-db-engine-versions \
  --engine postgres \
  --engine-version 16 \
  --default-only \
  --query 'DBEngineVersions[0].EngineVersion' \
  --output text)

aws rds create-db-instance \
  --db-instance-identifier drill-pg \
  --engine postgres \
  --engine-version "$ENGINE_VERSION" \
  --db-instance-class db.t4g.micro \
  --allocated-storage 20 \
  --storage-type gp3 \
  --master-username drilladmin \
  --manage-master-user-password \
  --multi-az \
  --backup-retention-period 7 \
  --no-publicly-accessible \
  --db-subnet-group-name drill-db-subnets \
  --vpc-security-group-ids sg-0123456789abcdef0

aws rds wait db-instance-available --db-instance-identifier drill-pg

--manage-master-user-password puts the password in Secrets Manager. Confirm the shape before going further. MultiAZ must be true, and DBClusterIdentifier must be null. In the console, the same check is the Configuration tab: Multi-AZ is Yes, and the primary and secondary Availability Zones are both filled in.

aws rds describe-db-instances \
  --db-instance-identifier drill-pg \
  --query 'DBInstances[0].{
    MultiAZ:MultiAZ,
    Cluster:DBClusterIdentifier,
    AZ:AvailabilityZone,
    Standby:SecondaryAvailabilityZone,
    Endpoint:Endpoint.Address,
    Latest:LatestRestorableTime
  }'

Write down the Availability Zone and the standby AZ. After the failover they swap. The console and the AvailabilityZone field can lag the real promotion by several minutes, so this field is a lagging clock, not the one you score.

Take one snapshot anyway. The drill instance is disposable, and the snapshot is the backstop if the instance itself fails.

aws rds create-db-snapshot \
  --db-instance-identifier drill-pg \
  --db-snapshot-identifier drill-pg-before

The password and the endpoint, for the client that runs inside the VPC:

SECRET_ARN=$(aws rds describe-db-instances \
  --db-instance-identifier drill-pg \
  --query 'DBInstances[0].MasterUserSecret.SecretArn' \
  --output text)

aws secretsmanager get-secret-value \
  --secret-id "$SECRET_ARN" \
  --query SecretString \
  --output text

aws rds describe-db-instances \
  --db-instance-identifier drill-pg \
  --query 'DBInstances[0].Endpoint.Address' \
  --output text

Run psql from an instance in that VPC, or through an SSM port-forward. A laptop on the public internet cannot reach an instance created with --no-publicly-accessible, and that is the point.

CREATE TABLE drill_marker (
  id         integer PRIMARY KEY,
  note       text NOT NULL,
  written_at timestamptz NOT NULL DEFAULT now()
);

INSERT INTO drill_marker (id, note) VALUES (1, 'committed-before-failover');

Leave a second transaction open in another session. Do not commit it.

BEGIN;
INSERT INTO drill_marker (id, note) VALUES (2, 'open-during-failover');
-- stay here until the failover finishes

---

2. Clock A — force the failover

A reboot without --force-failover restarts the primary in the same Availability Zone. That does not test promotion. The flag interrupts the primary, runs crash recovery if the engine needs it, promotes the standby, and updates the DNS record for the existing endpoint.

date -u +%Y-%m-%dT%H:%M:%SZ

aws rds reboot-db-instance \
  --db-instance-identifier drill-pg \
  --force-failover

The same action in the console is Actions → Reboot, with Reboot with failover checked. A reboot without that checkbox stays in the current Availability Zone and does not count as this drill.

Score the control plane from RDS events, not from the Availability Zone field. The messages you want are published by RDS:

• RDS-EVENT-0013 — Multi-AZ instance failover started.
• RDS-EVENT-0015 — Multi-AZ failover to standby complete. DNS propagation may take a few minutes.
• RDS-EVENT-0049 — Multi-AZ instance failover completed.

describe-events returns the message text, not the event id. Match the text. Ignore events whose timestamp is from before this reboot. If you run the drill twice in a row, RDS can refuse the second one with RDS-EVENT-0034: it abandons a user-requested failover because one just happened. Wait and retry.

aws rds describe-events \
  --source-identifier drill-pg \
  --source-type db-instance \
  --duration 60 \
  --query 'Events[].{Time:Date,Message:Message}' \
  --output table

On the instance, open Logs & events. Read the times from the bottom of the list upward. The middle row is the one that ends the database clock.

AWS documents the failover itself as typically 60–120 seconds. Large transactions or a long crash recovery make it longer. RDS-EVENT-0015 is the end of that clock. It is also explicit that DNS can take further minutes. Those are two measurements. Record both.

This loop prints events after the reboot and exits when the standby promotion is complete. Run it from anywhere the CLI works. It does not prove the application can connect.

python3 - << 'PY'
import json, subprocess, time
from datetime import datetime, timezone

ident = "drill-pg"
start = datetime.now(timezone.utc)
print("start", start.strftime("%Y-%m-%dT%H:%M:%SZ"))
subprocess.check_call([
    "aws", "rds", "reboot-db-instance",
    "--db-instance-identifier", ident,
    "--force-failover",
])
seen = set()
while (datetime.now(timezone.utc) - start).total_seconds() < 900:
    payload = json.loads(subprocess.check_output([
        "aws", "rds", "describe-events",
        "--source-identifier", ident,
        "--source-type", "db-instance",
        "--duration", "30",
        "--output", "json",
    ]))
    for event in payload["Events"]:
        when = datetime.fromisoformat(event["Date"].replace("Z", "+00:00"))
        if when < start or event["Date"] in seen:
            continue
        seen.add(event["Date"])
        elapsed = int((when - start).total_seconds())
        print(f"{elapsed:4d}s  {event['Message']}")
        if "failover to standby complete" in event["Message"]:
            raise SystemExit(0)
    time.sleep(10)
raise SystemExit("timed out waiting for failover to standby complete")
PY

Now the data-plane clock, from a host that can reach the instance. Use a new process each attempt. A connection that was already open is dead, and a connection pool that still holds it will keep reporting failure after RDS is fine.

HOST=$(aws rds describe-db-instances \
  --db-instance-identifier drill-pg \
  --query 'DBInstances[0].Endpoint.Address' \
  --output text)

# PGPASSWORD comes from the Secrets Manager secret above.
until psql "host=$HOST dbname=postgres user=drilladmin sslmode=require" \
      -c 'SELECT 1' >/dev/null
do
  date -u +%H:%M:%S
  sleep 5
done
date -u +%H:%M:%S
echo "new connection succeeded"

Then check the two rows:

SELECT id, note, written_at FROM drill_marker ORDER BY id;

Row 1, the committed insert, is still there. Row 2, the insert that was never committed, is not. Commit the session that held the open transaction and you will get a connection error, not a successful commit. That is the recovery point of a Multi-AZ failover: committed work survives, in-flight work does not.

Check the AZ last.

aws rds describe-db-instances \
  --db-instance-identifier drill-pg \
  --query 'DBInstances[0].{Status:DBInstanceStatus,AZ:AvailabilityZone,Standby:SecondaryAvailabilityZone}'

The primary AZ should now be the old standby. If the events and SELECT 1 already succeeded and this field still shows the old AZ, wait. The API lags the promotion. People who stop the stopwatch on this field report a worse failover time than their clients experienced, or they report success before DNS has moved.

---

3. What the application has to do

RDS updates its own DNS name. You do not change a connection string for a Multi-AZ failover. Clients that cached the old IP keep calling a node that is no longer the primary.

AWS best practices tell client applications to cache RDS DNS for less than 30 seconds. The JVM guidance on the failover page is a TTL of no more than 60 seconds, because some JVMs cache a lookup until the process restarts. Set it before the process opens connections:

networkaddress.cache.ttl=30

Or, in application startup code, before any socket is opened:

java.security.Security.setProperty("networkaddress.cache.ttl", "30");

The rest of the client behavior is what the drill is there to expose:

• The health check must open a new connection and run SELECT 1. A pooled connection from before the failover stays broken and will keep the target unhealthy.
• In-flight transactions roll back. Retries have to be safe, or the application has to surface the error.
• A connection pool will stampede the new primary the moment DNS flips. Amazon RDS Proxy does not remove the need to retry, and it does not keep an open transaction alive across the promotion. It does absorb the stampede. If the drill's SELECT 1 succeeds and the application is still down, the pool is the next place to look.
• On Aurora or a Multi-AZ DB cluster, the application has to use the cluster writer endpoint. An instance endpoint stays on the instance that used to be the writer.

---

4. Clock B — restore to a point in time

Failover never reads the backup. A database can fail over cleanly with a LatestRestorableTime that is an hour old. This second drill is the one that catches that.

Write down the latest restorable time, then write a marker you expect the restore to miss.

T0=$(aws rds describe-db-instances \
  --db-instance-identifier drill-pg \
  --query 'DBInstances[0].LatestRestorableTime' \
  --output text)
date -u +%Y-%m-%dT%H:%M:%SZ
echo "LatestRestorableTime $T0"

Subtract T0 from now. That gap is the recovery point the service is offering you at this moment. Transaction logs are uploaded about every five minutes, so a healthy instance is normally inside that window. A gap measured in tens of minutes means the backup chain is behind, and the failover you just passed did not reveal it.

Insert the marker after T0, on the source instance:

INSERT INTO drill_marker (id, note) VALUES (3, 'written-after-latest-restorable-time');

Restore to T0, into a new instance. Leave it Single-AZ. This copy exists to prove the backup, and Multi-AZ on a throwaway doubles the cost for a check you do not need here. A production cutover is a separate decision: you can convert it to Multi-AZ after the data check, or restore with --multi-az when the recovery time objective requires the new copy to already have a standby.

date -u +%Y-%m-%dT%H:%M:%SZ

aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier drill-pg \
  --target-db-instance-identifier drill-pg-restored \
  --restore-time "$T0" \
  --db-instance-class db.t4g.micro \
  --no-multi-az \
  --no-publicly-accessible \
  --db-subnet-group-name drill-db-subnets \
  --vpc-security-group-ids sg-0123456789abcdef0

aws rds wait db-instance-available \
  --db-instance-identifier drill-pg-restored

date -u +%Y-%m-%dT%H:%M:%SZ

In the console this is Actions → Restore to point in time. Pick Custom and the T0 you captured. Leave the new copy Single-AZ. The source instance is not replaced.

--restore-time and --use-latest-restorable-time are mutually exclusive. Use the timestamp you captured. Using "latest" restores a moving target and you can no longer say which marker should be present.

Duration grows with storage size and with how much log RDS has to replay. A 20 GiB drill instance only gives you the overhead of creating an instance and replaying a few minutes of log. Repeat the same command against a snapshot of a production-sized database if you need a restore time you can put in an RTO budget. Do that in the non-production account.

The restored instance has its own endpoint and its own master-user secret when the source used a managed password. Read that secret. Do not point the application at this host until the marker check passes.

SELECT id, note, written_at FROM drill_marker ORDER BY id;

On drill-pg-restored, row 3 is absent. Row 1 is present. On drill-pg, row 3 is present. That is the proof: the restore time you asked for is the data you got, and everything written after it was left behind on purpose.

The source instance is untouched. A restore does not fail the original over, and it does not rename anything.

To swing an application at a restored copy, put a Route 53 record you own in front of the RDS endpoint and set its TTL to 30 seconds or less. The cutover is an update of that record to the new endpoint, after the marker check. Renaming the RDS instance also changes the AWS-generated hostname and reboots the instance, so the CNAME you control is the shorter outage.

---

5. Scorecard

Fill this in from the drill. The right-hand column is the planning range from AWS documentation, for a Multi-AZ DB instance under ordinary load. Your numbers replace it.

Measurement	Result	Planning range
Reboot API to "failover to standby complete" (RDS-EVENT-0015)		Typically 60–120 seconds. Longer with a big transaction or a long recovery.
RDS-EVENT-0015 to a new SELECT 1		DNS can take additional minutes. A client TTL under 30 seconds is the target.
AvailabilityZone field shows the new primary		Often minutes behind the event. Do not score RTO on this field.
Committed marker (row 1) after failover	Present	Near-zero RPO for committed work.
Open transaction (row 2) after failover	Absent	Rolled back.
now minus LatestRestorableTime		Normally within about 5 minutes.
Restore API to available on the new instance		Depends on size and log replay. Keep the number with the database size beside it.
Marker written after T0 (row 3) on the restored copy	Absent	Restore stopped at T0.

If the first row is several minutes and the application row is much longer, the database failed over and the client did not follow. Fix DNS caching or the pool, then run the drill again. If LatestRestorableTime is far behind, fix backups before you discuss Availability Zones.

Aurora uses a different command and a different planning number. Fail over the cluster, and keep using the writer endpoint:

aws rds failover-db-cluster --db-cluster-identifier drill-aurora

AWS documents that cluster failover as typically completing within 30 seconds. Time it the same way, with a new connection after the failover event, and record your own number. Aurora's storage is shared across the cluster, so this still does not test a point-in-time restore. Run clock B against the cluster as well if the recovery point matters.

---

6. Clean up

Delete the restored copy first. --skip-final-snapshot is acceptable here because the copy was created to be thrown away. Do not put that flag on a production instance.

aws rds delete-db-instance \
  --db-instance-identifier drill-pg-restored \
  --skip-final-snapshot

aws rds delete-db-instance \
  --db-instance-identifier drill-pg \
  --skip-final-snapshot

aws rds delete-db-snapshot \
  --db-snapshot-identifier drill-pg-before

The managed secrets are deleted with the instances when RDS owns them. Confirm in Secrets Manager so a drill password is not left behind.

Run clock A after a material change to the instance class, the engine version, or the connection pool. Run clock B on a schedule, against a database close to production size, and keep the scorecard. A failover you have never timed is a planning number. A restore you have never finished is a backup you have not proved.