In October we patched the host running the primary of our delivery tracking Redis, expecting Sentinel to promote the replica as it always had. It did not. The driver app could not sync for six minutes while the host rebooted, and when we looked at the replica it had not held a current copy of the data for three days.
Three days earlier there had been a forty second network interruption during switch maintenance. When a replica reconnects, the primary tries a partial resync from its replication backlog, a ring buffer of recent writes. Ours was at the default size of one megabyte. We write about fifteen megabytes a second at peak, so the backlog covered well under a second, and a forty second gap meant a full resync: the primary forks, produces a snapshot of the whole dataset and streams it to the replica.
While it does that, every new write for the replica is held in that replica's output buffer on the primary. The default limit for replica connections is 256 megabytes outright, or 64 megabytes held for sixty seconds. When the replica was first set up the dataset was four gigabytes and a full sync took under a minute. It was now twenty two gigabytes and took about four minutes, which at our write rate is far past both limits. The primary closed the connection, the replica reconnected, could not resync partially, and the whole thing began again. Roughly every four minutes for three days, each time with a fork of a twenty two gigabyte process.
Sentinel was right not to promote it. A replica whose link to the primary had been down that long had nothing current to offer.
The backlog is now 512 megabytes, enough for about thirty seconds at peak, so short interruptions resync partially. The replica output buffer limit is sized from the measured duration of a full sync multiplied by our peak write rate, with room to spare. We alert when master_link_status on a replica is down for more than two minutes and whenever the sync_full counter on the primary increases. And every quarter we force a full sync in staging with production's dataset size and time it.
Those limits were set for the database we had in 2023. The data kept growing and the limits stayed where they were.
– Sergey Shinder
Top comments (0)