The finance lead forwarded the AWS bill on a Monday morning with three question marks in the subject line. The month had closed at $11,800 against a steady $3,200. The on-call engineer's first guess, sensible enough, was that a data scientist had left a cross-region Athena job running. It was not. It was an RDS read replica in a different AZ from its primary, provisioned five weeks earlier for a one-off load test, never decommissioned, and billing an oversized instance class plus its storage every hour since. Nobody had read from it in three weeks. Its replication had also been dead for six days, stopped on a fatal IO-thread error nobody was watching for, which cost no money at all and would have been the real incident on an account that actually depended on it.
Problem signals:
- AWS bill 2-4x its usual monthly baseline with no traffic or feature change
- Cost Explorer, grouped by usage type rather than by service, puts the movement on RDS instance-hours and provisioned storage rather than on any query or transfer meter
- An RDS read replica reports ReplicaLag = -1 (or a lag climbing without bound) and Replica Status: error
- No application config or BI tool actually points at the replica's endpoint
- A DataTransfer-Regional-Bytes charge you are about to blame on replication: same-Region replication traffic is not billed, and a cross-Region replica bills on the source Region's inter-Region AWS-Out-Bytes usage type rather than this one, so the source is client traffic crossing AZs, Elastic IP traffic, or the cross-AZ legs of load balancer, NAT gateway and other service traffic
Chasing the analytics query that did not exist
What we thought it was first
Almost every cost spike I have seen in the last three years gets blamed on analytics first. There is usually a junior data person, a notebook, a forgotten SELECT *, and a story everyone tells themselves. So we did the natural thing. We pulled the Athena query history for the previous ten days. Nothing unusual. We checked Redshift, which the team barely uses. Idle. We checked the data warehouse cluster's autoscaling history. Flat.
The clue was in Cost Explorer, but only when we grouped by usage type instead of by service. Grouped by service, RDS was simply up. Grouped by usage type, the movement was almost entirely in instance-hours (the InstanceUsage:db.* meters) and provisioned storage. Not a query meter. Not a transfer meter. Hours and gigabyte-months, which is what you see when something is running and nobody is using it.
There was also a DataTransfer-Regional-Bytes line sitting in the same bill, and because a cross-AZ read replica turned up ten minutes later, we spent half a day trying to pin the spike on it before anyone read its usage quantity. That was the second wrong turn, and it is worth naming because it is the one most teams make. AWS does not bill data transfer for replication between a source DB instance and its read replica in the same Region. The RDS User Guide states it directly: you are not charged for the data transfer incurred in replicating data between the source DB instance and a read replica within the same AWS Region. Same-Region replication is free regardless of which AZ each end sits in, so a same-Region replica's replication cannot put a single byte on that meter; clients reading from it across an AZ boundary still can.
DataTransfer-Regional-Bytes comes from somewhere else. AWS's price list describes it as regional data transfer in, out or between EC2 Availability Zones, or using Elastic IPs or ELB. In practice that is client traffic crossing an AZ boundary (an app server, EC2 instance, Lambda, or BI tool in one AZ talking to an endpoint in another), traffic to public or Elastic IP addresses inside the Region, and the cross-AZ legs of load balancer, NAT gateway and other service traffic. A cross-Region replica is the only replica topology that carries a replication data-transfer charge, but it never lands on this meter; it is billed at the inter-Region rate on the source Region's AWS-Out-Bytes usage type instead (USE1-USW2-AWS-Out-Bytes, for example; the matching In-Bytes line is $0.00). Ours was a small, flat, unrelated line that had not moved at all.
The other habit worth building here is to read the usage quantity, not the dollars. Cost Explorer shows you GB and instance-hours next to the money, and the quantity is what tells you whether a mechanism is even physically possible. Cross-AZ transfer inside a Region is $0.01/GB in each direction, so an $8,600 transfer bill would mean roughly 430 TB moved in the month, about 14 TB a day or 166 MB/s sustained around the clock. A leftover load-test replica does not do that. The arithmetic rules the theory out before you spend a day on it.
How we found the orphan replica
The diagnostic turn
We listed every RDS instance in the account and compared the AZ of each replica to its primary. One read replica was in us-east-1b while its primary was in us-east-1a. That alone is not a problem; cross-AZ replicas exist for legitimate HA reasons. What was odd was that this replica was tagged with nothing. No Owner. No Purpose. No Environment. Just the default Name tag, which read load-test-replica-temp.
# List replicas with their AZ and their primary's AZ
aws rds describe-db-instances \
--query 'DBInstances[?ReadReplicaSourceDBInstanceIdentifier!=`null`].[DBInstanceIdentifier,AvailabilityZone,ReadReplicaSourceDBInstanceIdentifier,DBInstanceStatus]' \
--output table
# Then for each primary, get its AZ
aws rds describe-db-instances \
--db-instance-identifier <primary-id> \
--query 'DBInstances[0].AvailabilityZone'
The two commands that surfaced the orphan in about 30 seconds.
It was not a healthy idle replica either. CloudWatch ReplicaLag had been flat at -1 for six days. On RDS for MySQL and MariaDB, ReplicaLag reports the value of Seconds_Behind_Source (Seconds_Behind_Master on MariaDB and on MySQL before 8.0.22), and when replication is not active that field is NULL, which RDS publishes as -1. The console agreed: Replica Status: error. There was no sawtooth. A sawtooth ReplicaLag, climbing to tens of seconds and dropping back toward zero over and over, is what a replica looks like when it is applying but falling behind under write bursts, which is healthy-but-lagging and the opposite diagnosis. A replica that is genuinely stuck reads -1, or climbs without bound and never comes back down. Ours had been pinned at one binlog position since the failure, which by day six would have been roughly 518,000 seconds of lag had the thread still been alive to measure it.
# What the replica itself reports, not CloudWatch's smoothed view
aws rds describe-db-instances \
--db-instance-identifier load-test-replica-temp \
--query 'DBInstances[0].StatusInfos'
# -> [{"StatusType": "read replication", "Normal": false,
# "Status": "error", "Message": "... Error 1236 ..."}]
-- On the replica (SHOW SLAVE STATUS before MySQL 8.0.22)
SHOW REPLICA STATUS\G
-- Replica_IO_Running: No
-- Last_IO_Errno: 13114
-- Last_IO_Error: Got fatal error 1236 from source when reading data
-- from binary log: 'log event entry exceeded max_allowed_packet; ...'
-- Seconds_Behind_Source: NULL -> published as ReplicaLag = -1
The two fields that actually identify a stuck replica: status error, and Seconds_Behind_Source NULL (Seconds_Behind_Master on MariaDB and before MySQL 8.0.22). On 8.0 the errno is the replica's own 13114; the source's 1236 sits inside the message.
The error itself was latched on the replica: fatal error 1236 from the source, log event entry exceeded max_allowed_packet, recorded in Last_IO_Errno as 13114, MySQL 8.0's code for a fatal error reading the source's binary log. An alert keyed on Last_IO_Errno 1236 never fires on 8.0. AWS's own procedure pages still say Last_IO_Errno reports 1236; on 8.0 the MySQL code records 13114 and puts the 1236 in the message. That error is fatal to the IO thread. The thread stopped, recorded the error, and stayed stopped. It did not retry in a loop. The reconnect interval set by SOURCE_CONNECT_RETRY (MASTER_CONNECT_RETRY on older versions) defaults to 60 seconds and applies only to retryable network errors, and RDS will not restart a stopped replication thread for you. For this error a restart would not have helped anyway: CALL mysql.rds_start_replication has the replica ask the source again from where it stopped, and the source fails it the same way. Only a new replica would have, once any source-side cause was dealt with, and for six days nobody noticed it needed one, because nothing was watching.
The variable people reach for here is the wrong one, and AWS's read-replica troubleshooting page reaches for it too, advising the same max_allowed_packet on source and replica. On MySQL 8.0, max_allowed_packet on the replica does not govern the replication stream; the replication threads run with replica_max_allowed_packet instead (slave_max_allowed_packet before 8.0.26), and it exists precisely so that a max_allowed_packet difference between a source and its replica cannot break replication. Its default and its maximum are both 1 GB, which is also the ceiling RDS allows for max_allowed_packet on the source, so a routine primary-side bump cannot emit an event a default replica is unable to carry. MySQL's own guidance is that replica_max_allowed_packet must be larger than the source's max_allowed_packet, and at the defaults it already is. A 1236 means something else: an event genuinely above the 1 GB replication ceiling, or a corrupted binlog position. Worth knowing too that error 1153, whose text reads almost identically, is a client/server packet error and, on 8.0, not what a stalled replica reports.
And here is the part that matters for the bill: a stopped replica transfers nothing. It had not pulled a byte of binlog in six days, so it accrued no replication transfer, and nobody had read from it in three weeks, so there was no client transfer either. It was expensive for the dullest possible reason, an instance class sized for a load test plus its provisioned storage and automated backups, billed every hour for five weeks whether or not anything was replicating or reading. Attributing the $8,600 to that one instance took the Cost and Usage Report with resource IDs enabled, not Cost Explorer: resource-level data there is opt-in per service and only covers the last 14 days, so five weeks of an RDS orphan's spend is not retrievable that way at all. A cost allocation tag filter in Cost Explorer would have worked too, had anyone tagged the thing. Cost Explorer's job here was the usage-type grouping earlier, and that is where it should stay. The replication failure cost nothing. It only meant that if anything had depended on the replica, it had been serving six-day-old data the whole time. Left stopped for more than 30 consecutive days, it would also have had its replication terminated by RDS, which does that to spare the source the extra storage and long failover times.
The five-minute check that prevents the worse outcome
What we did before deleting anything
The instinct, when you have found the thing burning money, is to kill it immediately. We did not. The worse outcome here is not 'replica costs another hour of instance time'. The worse outcome is 'replica gets deleted, a quarterly BI dashboard breaks on Friday, and finance is back in your inbox with a different question'.
So we did the cheap verification first. We grepped the application monorepo for the replica's endpoint hostname. Zero hits. We checked the BI tool's data sources (Metabase in this case). Nothing pointed at it. We checked the data team's Airflow DAGs. Clean. We checked Terraform state to see how it had been created. It was in a workspace tagged load-test that had not been touched in a month, and the engineer who created it had left the company three weeks earlier.
| Step | What it does |
|---|---|
| If something had pointed at it | The right move would have been to keep a replica in service. Had someone lowered replica_max_allowed_packet, the replica would have stopped on its own packet error, "Got a packet bigger than 'replica_max_allowed_packet' bytes", and raising it and calling mysql.rds_start_replication would have been the fix. A 1236 from the source is not that. Find out which of the two causes it is first: if something on the source is writing events over the 1 GB ceiling, a new replica breaks at the next one, so that writer has to change. Then rebuild: create a new read replica, which RDS builds from a fresh snapshot of the source, and move clients to it once it has caught up. Skipping the old replica past the bad event is not a repair. A skipped event is never applied, so the replica stops being a faithful copy, and AWS documents its skip procedure (mysql.rds_next_master_log, or rds_next_source_log on 8.4) only for replication that fails after a Multi-AZ failover of the source. Deleting without a replacement would have caused a worse incident than the cost. |
| Nothing pointed at it | Delete with --skip-final-snapshot, which RDS requires when the instance being deleted is a read replica. Billing stopped once the deletion completed, because RDS bills partial instance-hours in one-second increments. |
aws rds delete-db-instance \
--db-instance-identifier load-test-replica-temp \
--skip-final-snapshot
The actual delete, once we were confident nothing depended on the replica.
Tag hygiene, expiration sweeps, and a budget alarm that would have caught this in week one
What we changed afterwards
Forgotten resources are the largest single category of cloud waste I see in client accounts. Bigger than oversized instances. Bigger than reserved-instance gaps. The fix is mechanical. Every cost-generating resource needs three tags: Owner, Purpose, ExpiresAt. ExpiresAt is the one most teams skip and the one that does the work.
We deployed a small Lambda on a weekly schedule that walks RDS, EC2, ELB, ElastiCache, and OpenSearch, finds resources past their ExpiresAt date or missing tags entirely, and posts to a Slack channel pinging the Owner. The owner has two weeks to either re-tag with a new ExpiresAt or delete. Resources with no Owner go to the platform team's queue. The first sweep flagged 47 resources across the account. Six of them were costing real money.
The sweep logic. About 180 lines of Python in practice.
The second change was per-service budgets. The team had a single account-wide budget set at $5,000/month, which is useless for catching this kind of incident because the account total only crossed it once the replica had already been running for weeks. RDS spend before the incident sat around $1,400/month; an AWS Budgets alert on RDS actual cost at $1,800 a month would have fired within days of the replica being created, about a month before finance did. The alert that matters is the one that fires before you have spent the money, not after.
The third change was monitoring the thing that was actually broken. Every replica now carries a CloudWatch alarm on ReplicaLag configured to treat both missing data and -1 as breaching, because this failure mode publishes no useful datapoint at all, plus an RDS event subscription on the read replica event category. A replica can be dead for six days and still look perfectly quiet on a dashboard tuned only for a lag threshold.
The fourth is a checklist item. When anyone raises max_allowed_packet on a source, they now confirm replica_max_allowed_packet on each replica is at least as large. MySQL's manual asks for larger, to leave room for an event's header, and the replica's receiver already adds that room itself. At the defaults it always is: replica_max_allowed_packet is 1 GB while max_allowed_packet defaults to 64 MB, and 1 GB is also the highest the source can ever be set to. That is exactly why nobody thinks about it until somebody has lowered one. It is not glamorous, it would not have saved a dollar, and it would not have stopped this 1236, which came from the source; the replica alarm above is what would have cut the six days short. The checklist closes the neighbouring door, a replica-side limit someone lowered, which fails just as quietly and on a different account is the incident that actually hurts.
Where cost spike triage gets stuck
If your AWS bill just jumped and you do not know why
The hard part of a cost spike is not finding the resource. It is being confident enough to delete it. Most teams we work with have at least one orphan RDS, ElastiCache, or NAT gateway they are afraid to touch because nobody remembers what depends on it. The triage takes a day; the courage to act takes a week of meetings. By then the bill has run another $2,000.
We run cost spike triage engagements every month. We have seen the orphan-replica case four times this year, the NAT-gateway-in-the-wrong-AZ case more often than that, and a half dozen variants of 'load test that never got cleaned up' across CloudWatch Logs, OpenSearch, and Aurora Serverless. The pattern is almost always the same: a resource that nobody owns, a tag policy that was never enforced, and a budget alert tuned too coarse to catch concentration in a single service. We have written more on the underlying patterns in the cloud cost spikes problem brief and across our services.
If your AWS bill jumped this month and you cannot point at the resource with confidence, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week. Cost stops accruing the day we find the orphan.
Originally published at https://infraforge.agency/insights/forgotten-rds-replica-cross-az-cost-spike/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)