The finance lead forwarded the AWS bill on a Monday morning with three question marks in the subject line. The month had closed at $11,800 against a steady $3,200. The on-call engineer's first guess, sensible enough, was that a data scientist had left a cross-region Athena job running. It was not. It was an RDS read replica in a different AZ from its primary, provisioned five weeks earlier for a one-off load test, never decommissioned, and billing an oversized instance class plus its storage every hour since. Nobody had read from it in three weeks. Its replication had also been dead for six days, stopped on a fatal IO-thread error nobody was watching for, which cost no money at all and would have been the real incident on an account that actually depended on it.
Problem signals:
- AWS bill 2-4x its usual monthly baseline with no traffic or feature change
- Cost Explorer, grouped by usage type rather than by service, puts the movement on RDS instance-hours and provisioned storage rather than on any query or transfer meter
- An RDS read replica reports ReplicaLag = -1 (or a lag climbing without bound) and Replica Status: error
- No application config or BI tool actually points at the replica's endpoint
- A cross-AZ DataTransfer-Regional-Bytes charge you are about to blame on replication: same-Region replica traffic is not billed, and a cross-Region replica bills on the inter-Region AWS-Out-Bytes and AWS-In-Bytes usage types rather than this one, so the source is client traffic crossing AZs, NAT gateway, or inter-AZ service traffic
Chasing the analytics query that did not exist
What we thought it was first
Almost every cost spike I have seen in the last three years gets blamed on analytics first. There is usually a junior data person, a notebook, a forgotten SELECT *, and a story everyone tells themselves. So we did the natural thing. We pulled the Athena query history for the previous ten days. Nothing unusual. We checked Redshift, which the team barely uses. Idle. We checked the data warehouse cluster's autoscaling history. Flat.
The clue was in Cost Explorer, but only when we grouped by usage type instead of by service. Grouped by service, RDS was simply up. Grouped by usage type, the movement was almost entirely in instance-hours (the InstanceUsage:db.* meters) and provisioned storage. Not a query meter. Not a transfer meter. Hours and gigabyte-months, which is what you see when something is running and nobody is using it.
There was also a DataTransfer-Regional-Bytes line sitting in the same bill, and because a cross-AZ read replica turned up ten minutes later, we spent half a day trying to pin the spike on it. That was the second wrong turn, and it is worth naming because it is the one most teams make. AWS does not bill data transfer for replication between a source DB instance and its read replica in the same Region. The RDS User Guide states it directly: you are not charged for the data transfer incurred in replicating data between the source DB instance and a read replica within the same AWS Region. Same-Region replication is free regardless of which AZ each end sits in, so a same-Region replica cannot put a single byte on that meter.
Cross-AZ DataTransfer-Regional-Bytes comes from somewhere else, and only from these: client traffic crossing an AZ boundary (an app server, EC2 instance, Lambda, or BI tool in one AZ talking to an endpoint in another), NAT gateway traffic, or inter-AZ service traffic. A cross-Region replica is the only replica topology that carries a replication data-transfer charge, but it never lands on this meter; it is billed at the inter-Region rate on the AWS-Out-Bytes and AWS-In-Bytes usage types instead. Ours was a small, flat, unrelated line that had not moved at all.
The other habit worth building here is to read the usage quantity, not the dollars. Cost Explorer shows you GB and instance-hours next to the money, and the quantity is what tells you whether a mechanism is even physically possible. Cross-AZ transfer inside a Region is $0.01/GB in each direction, so an $8,600 transfer bill would mean roughly 430 TB moved in six days, about 71.7 TB/day or 830 MB/s sustained around the clock. A leftover load-test replica does not do that. The arithmetic rules the theory out before you spend a day on it.
How we found the orphan replica
The diagnostic turn
We listed every RDS instance in the account and compared the AZ of each replica to its primary. One read replica was in us-east-1b while its primary was in us-east-1a. That alone is not a problem; cross-AZ replicas exist for legitimate HA reasons. What was odd was that this replica was tagged with nothing. No Owner. No Purpose. No Environment. Just the default Name tag, which read load-test-replica-temp.
# List replicas with their AZ and their primary's AZ
aws rds describe-db-instances \
--query 'DBInstances[?ReadReplicaSourceDBInstanceIdentifier!=`null`].[DBInstanceIdentifier,AvailabilityZone,ReadReplicaSourceDBInstanceIdentifier,DBInstanceStatus]' \
--output table
# Then for each primary, get its AZ
aws rds describe-db-instances \
--db-instance-identifier <primary-id> \
--query 'DBInstances[0].AvailabilityZone'
The two commands that surfaced the orphan in about 30 seconds.
It was not a healthy idle replica either. CloudWatch ReplicaLag had been flat at -1 for six days. On RDS for MySQL and MariaDB, ReplicaLag reports the value of Seconds_Behind_Master, and when replication is not active that field is NULL, which RDS publishes as -1. The console agreed: Replica Status: error. There was no sawtooth. A sawtooth ReplicaLag, climbing to tens of seconds and dropping back toward zero over and over, is what a replica looks like when it is applying but falling behind under write bursts, which is healthy-but-lagging and the opposite diagnosis. A replica that is genuinely stuck reads -1, or climbs without bound and never comes back down. Ours had been pinned at one binlog position since the failure, which by day six would have been roughly 518,000 seconds of lag had the thread still been alive to measure it.
# What the replica itself reports, not CloudWatch's smoothed view
aws rds describe-db-instances \
--db-instance-identifier load-test-replica-temp \
--query 'DBInstances[0].StatusInfos'
# -> [{"StatusType": "read replication", "Normal": false,
# "Status": "error", "Message": "... Error 1236 ..."}]
-- On the replica (SHOW SLAVE STATUS before MySQL 8.0.22)
SHOW REPLICA STATUS\G
-- Replica_IO_Running: No
-- Last_IO_Errno: 1236
-- Seconds_Behind_Master: NULL -> published as ReplicaLag = -1
The two fields that actually identify a stuck replica: status error, and Seconds_Behind_Master NULL.
The error itself was latched on the replica: Last_IO_Errno 1236 from the source, log event entry exceeded max_allowed_packet. That error is fatal to the IO thread. The thread stopped, recorded the error, and stayed stopped. It did not retry in a loop. The reconnect interval set by SOURCE_CONNECT_RETRY (MASTER_CONNECT_RETRY on older versions) defaults to 60 seconds and applies only to retryable network errors, and RDS will not restart a stopped replication thread for you. Somebody had to run CALL mysql.rds_start_replication after fixing the cause. For six days nobody did, because nothing was watching.
The variable people reach for here is the wrong one. max_allowed_packet on the replica does not govern the replication stream; replica_max_allowed_packet does (slave_max_allowed_packet before 8.0.26), and it exists precisely so that a max_allowed_packet difference between a source and its replica cannot break replication. Its default and its maximum are both 1 GB, which is also the ceiling RDS allows for max_allowed_packet on the source, so a routine primary-side bump cannot emit an event a default replica is unable to carry. MySQL's own guidance is that replica_max_allowed_packet must be larger than the source's max_allowed_packet, and at the defaults it already is. A 1236 means something else: an event genuinely above the 1 GB replication ceiling, or a corrupted binlog position. Worth knowing too that error 1153, whose text reads almost identically, is a client/server packet error and not what a stalled replica reports.
And here is the part that matters for the bill: a stopped replica transfers nothing. It had not pulled a byte of binlog in six days, so it accrued no incremental transfer of any kind. It was expensive for the dullest possible reason, an instance class sized for a load test plus its provisioned storage and automated backups, billed every hour for five weeks whether or not anything was replicating or reading. Attributing the $8,600 to that one instance took the Cost and Usage Report with resource IDs enabled, not Cost Explorer: resource-level data there is opt-in and only covers the last 14 days, and its RESOURCE_ID grouping requires an EC2 service filter, so five weeks of an RDS orphan's spend is not retrievable that way at all. A cost allocation tag filter in Cost Explorer would have worked too, had anyone tagged the thing. Cost Explorer's job here was the usage-type grouping earlier, and that is where it should stay. The replication failure cost nothing. It only meant that if anything had depended on the replica, it had been serving six-day-old data the whole time.
The five-minute check that prevents the worse outcome
What we did before deleting anything
The instinct, when you have found the thing burning money, is to kill it immediately. We did not. The worse outcome here is not 'replica costs another hour of instance time'. The worse outcome is 'replica gets deleted, a quarterly BI dashboard breaks on Friday, and finance is back in your inbox with a different question'.
So we did the cheap verification first. We grepped the application monorepo for the replica's endpoint hostname. Zero hits. We checked the BI tool's data sources (Metabase in this case). Nothing pointed at it. We checked the data team's Airflow DAGs. Clean. We checked Terraform state to see how it had been created. It was in a workspace tagged load-test that had not been touched in a month, and the engineer who created it had left the company three weeks earlier.
| Step | What it does |
|---|---|
| If something had pointed at it | The right move would have been to keep the replica, compare replica_max_allowed_packet on its parameter group against the source's max_allowed_packet, and raise it only if someone had explicitly lowered it below 1 GB. Otherwise the remedy is to find the oversized or corrupted binlog event and either reposition the replica past it or rebuild it from a fresh snapshot, then call mysql.rds_start_replication and let it catch up before any client trusted its results again. Deletion would have caused a worse incident than the cost. |
| Nothing pointed at it | Delete with --skip-final-snapshot. The replica had been pinned at the same binlog position for six days, so a final snapshot would only have preserved six-day-old data. Billing stopped at the end of that hour. |
aws rds delete-db-instance \
--db-instance-identifier load-test-replica-temp \
--skip-final-snapshot
The actual delete, once we were confident nothing depended on the replica.
Tag hygiene, expiration sweeps, and a budget alarm that would have caught this in week one
What we changed afterwards
Forgotten resources are the largest single category of cloud waste I see in client accounts. Bigger than oversized instances. Bigger than reserved-instance gaps. The fix is mechanical. Every cost-generating resource needs three tags: Owner, Purpose, ExpiresAt. ExpiresAt is the one most teams skip and the one that does the work.
We deployed a small Lambda on a weekly schedule that walks RDS, EC2, ELB, ElastiCache, and OpenSearch, finds resources past their ExpiresAt date or missing tags entirely, and posts to a Slack channel pinging the Owner. The owner has two weeks to either re-tag with a new ExpiresAt or delete. Resources with no Owner go to the platform team's queue. The first sweep flagged 47 resources across the account. Six of them were costing real money.
The sweep logic. About 180 lines of Python in practice.
The second change was AWS Budgets with anomaly detection scoped per service. The team had a single account-wide budget set at $5,000/month, which is useless for catching this kind of incident because the account total only crossed it once the replica had already been running for weeks. RDS spend before the incident sat around $1,400/month; a per-service budget on RDS at $1,800 with a 20% variance threshold would have fired within days of the replica being created, five weeks before finance did. The alert that matters is the one that fires before you have spent the money, not after.
The third change was monitoring the thing that was actually broken. Every replica now carries a CloudWatch alarm on ReplicaLag configured to treat both missing data and -1 as breaching, because this failure mode publishes no useful datapoint at all, plus an RDS event subscription on the read replica event category. A replica can be dead for six days and still look perfectly quiet on a dashboard tuned only for a lag threshold.
The fourth is a checklist item. When anyone raises max_allowed_packet on a source, they now confirm replica_max_allowed_packet on each replica is at least as large. At the defaults it always is: replica_max_allowed_packet is 1 GB while max_allowed_packet defaults to 64 MB, and 1 GB is also the highest the source can ever be set to. That is exactly why nobody thinks about it until somebody has lowered one. It is not glamorous, and on its own it would not have saved a dollar. It would have saved six days of a replica quietly serving nothing, which on a different account is the incident that actually hurts.
Where cost spike triage gets stuck
If your AWS bill just jumped and you do not know why
The hard part of a cost spike is not finding the resource. It is being confident enough to delete it. Most teams we work with have at least one orphan RDS, ElastiCache, or NAT gateway they are afraid to touch because nobody remembers what depends on it. The triage takes a day; the courage to act takes a week of meetings. By then the bill has run another $2,000.
We run cost spike triage engagements every month. We have seen the orphan-replica case four times this year, the NAT-gateway-in-the-wrong-AZ case more often than that, and a half dozen variants of 'load test that never got cleaned up' across CloudWatch Logs, OpenSearch, and Aurora Serverless. The pattern is almost always the same: a resource that nobody owns, a tag policy that was never enforced, and a budget alert tuned too coarse to catch concentration in a single service. We have written more on the underlying patterns in the cloud cost spikes problem brief and across our services.
If your AWS bill jumped this month and you cannot point at the resource with confidence, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week. Cost stops accruing the day we find the orphan.
Originally published at https://infraforge.agency/insights/forgotten-rds-replica-cross-az-cost-spike/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)