The MySQL primary was still running. Applications could reach the database. Yet some write requests were taking too long and timing out.
The problem wasn't a missing primary. Both replicas had stopped replicating, and semi-synchronous replication had become part of the availability problem.
In Part 1, I covered deploying MySQL with MOCO. Part 2 covered backups, MinIO, and a davfs2 failure. This time, let's look at the replication behavior that matters when things go wrong.
The three-instance cluster
MOCO uses GTID-based semi-synchronous replication for its normal three-instance MySQL clusters: one writable primary and two read-only replicas.
Application
|
v
eks-db-1
PRIMARY
|
binary log events
/ \
v v
eks-db-0 eks-db-2
REPLICA REPLICA
\ /
replication ACK
With asynchronous replication, the primary can acknowledge a committed transaction to the application without waiting for any replica to confirm receipt.
With semi-synchronous replication, the primary waits for the configured number of replicas to acknowledge receiving the transaction's binary-log events. An ACK does not mean the replica has already applied the transaction.
With MOCO's normal three-instance configuration, one replica acknowledgement is required. Losing one replica does not necessarily stop writes; the other can still acknowledge them. Losing both removes that acknowledgement path.
The configuration that changes everything
On the primary, these commands reveal the relevant settings:
SHOW GLOBAL VARIABLES LIKE 'rpl_semi_sync_master%';
SHOW GLOBAL STATUS LIKE 'Rpl_semi_sync_master%';
A configuration captured during the investigation included:
| Setting | Value | Why it matters |
|---|---|---|
rpl_semi_sync_master_enabled |
ON |
Semi-sync is enabled |
rpl_semi_sync_master_wait_for_slave_count |
1 |
One replica ACK is required |
rpl_semi_sync_master_wait_no_slave |
ON |
Waiting can continue even when too few replicas are connected |
rpl_semi_sync_master_wait_point |
AFTER_SYNC |
The ACK wait occurs before completing the commit |
rpl_semi_sync_master_timeout |
86400000 |
The timeout is 24 hours, in milliseconds |
That last value surprised me. MySQL's usual default timeout is much shorter. MOCO deliberately configures a long timeout to avoid silently falling back to asynchronous replication and weakening its failover-safety assumptions.
It is a deliberate trade-off: protect acknowledged data, even when that means reduced write availability if replicas disappear.
These are MySQL 8.0-style variable names; names and implementation details differ across versions. Always inspect the version running in your cluster.
What we found
The failure state looked like this:
| Instance | Role | Replication state |
|---|---|---|
eks-db-0 |
Replica | I/O thread stopped |
eks-db-1 |
Primary | Running; replication ACKs unavailable |
eks-db-2 |
Replica | I/O thread stopped |
The primary didn't have to crash for applications to suffer. A transaction could reach MySQL and still spend longer than the application's timeout budget waiting for a replica acknowledgement.
A lightweight read-only probe such as SELECT 1 would not necessarily detect that failure. It doesn't exercise the commit path of a real write transaction.
MOCO also distinguishes a cluster with a ready primary from one whose replicas are healthy. Its replica readiness checks include replication status, so Running is not the same as Ready.
One caveat: this was a multi-symptom production incident, not a controlled experiment. Stopped replica threads and semi-sync waits were important evidence; they do not prove that every application timeout had exactly one cause.
What I took away
Semi-synchronous replication is not just a checkbox for high availability. It changes the conditions under which an application write can finish.
For a three-instance MOCO cluster, I'd monitor replica availability and semi-sync wait activity alongside ordinary MySQL uptime. If the last healthy replica disappears, a primary that is still running may no longer be enough.
In Part 4, I'll show the commands I use to investigate that situation: checking MOCO health, inspecting replica threads, reading semi-sync wait counters, and deciding whether a replica can safely rejoin.
Top comments (0)