DEV Community

Cover image for Getting Started with MOCO Part 4: Troubleshooting Failed MySQL Replicas
Mucheru Maina
Mucheru Maina

Posted on

Getting Started with MOCO Part 4: Troubleshooting Failed MySQL Replicas

#kubernetes #mysql #moco #devops

A MySQL pod can be Running even when replication is broken. I encountered this while investigating an incident in which both replicas stopped receiving transactions, the primary remained online, and application writes slowed down.

In Part 3, we covered how MOCO's semi-synchronous replication can make missing replica acknowledgements affect write availability. Here we'll focus on troubleshooting and recovery.

The examples use a MOCO cluster, eks-db, in the database namespace:

MySQL instance Kubernetes pod Role in this example
eks-db-0 moco-eks-db-0 Replica
eks-db-1 moco-eks-db-1 Primary
eks-db-2 moco-eks-db-2 Replica

The current primary must always be checked first. MOCO can change the primary during failover or switchover, so these roles are examples, not fixed assignments.

1. Check cluster and pod health 

kubectl get mysqlcluster eks-db -n database
kubectl describe mysqlcluster eks-db -n database
kubectl get pods -n database -o wide
Enter fullscreen mode Exit fullscreen mode

Check AVAILABLE, HEALTHY, PRIMARY, SYNCED REPLICAS and ERRANT REPLICAS. MOCO's replica readiness probe includes replication status: a pod may be Running but not Ready if replication is stopped or excessively delayed.

Confirm the current primary index directly:

kubectl get mysqlcluster eks-db -n database \
  -o jsonpath='{.status.currentPrimaryIndex}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

Do not assume eks-db-1 is still the primary just because it was the primary earlier.

2. Inspect replication on both replicas

Install the kubectl-moco plugin if it is not already available. Connect to replica 0:

kubectl moco -n database mysql --index 0 -it eks-db
Enter fullscreen mode Exit fullscreen mode

Run:

SHOW REPLICA STATUS\G
Enter fullscreen mode Exit fullscreen mode

Repeat for replica 2:

kubectl moco -n database mysql --index 2 -it eks-db
Enter fullscreen mode Exit fullscreen mode

Look for:

Field Meaning
Replica_IO_Running Is the receiver thread fetching events?
Replica_SQL_Running Is the applier thread executing events?
Last_IO_Error, Last_SQL_Error Why did either thread stop?
Retrieved_Gtid_Set, Executed_Gtid_Set Which transactions were retrieved and executed?
Seconds_Behind_Source Useful lag signal, not a complete health check

Inspect the affected MySQL error logs as well:

kubectl logs -n database moco-eks-db-0 -c mysqld --since=1h
kubectl logs -n database moco-eks-db-2 -c mysqld --since=1h
Enter fullscreen mode Exit fullscreen mode

3. Check the primary's semi-sync waits

The kubectl moco mysql command targets the current primary by default:

kubectl moco -n database mysql -it eks-db
Enter fullscreen mode Exit fullscreen mode 
SHOW GLOBAL VARIABLES LIKE 'rpl_semi_sync_master%';
SHOW GLOBAL STATUS LIKE 'Rpl_semi_sync_master%';
Enter fullscreen mode Exit fullscreen mode

Pay attention to Rpl_semi_sync_master_clients, Rpl_semi_sync_master_wait_sessions, and the transaction-wait counters. Cumulative counters need two readings over time; a high total alone does not prove writes are currently stalled.

The master variable names above apply to the MySQL 8.0-style semi-sync plugin. Check your MySQL version and replication plugin before copying the queries.

4. Decide whether the replica can be restarted or must be rebuilt

A stopped replication I/O thread is not automatically an errant replica. Check the MOCO cluster's errant-replica status, its events and the GTID histories first:

SELECT @@GLOBAL.gtid_executed;
Enter fullscreen mode Exit fullscreen mode

Compare the replica's GTID set with the current primary's, and check the associated replication errors. An errant replica has transactions that do not belong to the current primary's history. MOCO excludes it from the cluster for data safety.

There are two different recovery paths:

Path A — Resume a stopped I/O thread (only when safe). If the replica is not errant, the cause is understood, and the operation is approved, MySQL supports:

kubectl moco -n database mysql \
  -u moco-admin --index 0 -it eks-db
Enter fullscreen mode Exit fullscreen mode 
START REPLICA IO_THREAD;
SHOW REPLICA STATUS\G
Enter fullscreen mode Exit fullscreen mode

This command requires replication-administration privileges. Do not run it merely to override a replication thread deliberately stopped by MOCO; the operator may stop it again or the underlying inconsistency may remain.

Path B — Reinitialize an errant replica. When MOCO has excluded a replica because of errant transactions, the documented procedure is to delete that replica's PVC and pod so it can be recreated with fresh data.

Before deleting a PVC

Confirm all of the following:

  • The target is not the current primary; check the index again immediately before deletion.
  • There is a confirmed, valid data source from which MOCO can clone. Never delete the only good copy of your data.
  • You understand that deleting the claim destroys this replica's local data. Verify backups and the storage class's reclaim policy.
  • Rebuild one replica at a time, particularly when the cluster has already lost replication redundancy.

Inspect the exact resources and their backing storage:

kubectl get mysqlcluster eks-db -n database
kubectl get pod moco-eks-db-0 -n database -o wide
kubectl get pvc mysql-data-moco-eks-db-0 -n database
kubectl get pvc mysql-data-moco-eks-db-0 -n database \
  -o jsonpath='{.spec.volumeName}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

Use the returned PV name to inspect its reclaim policy with kubectl get pv <PV_NAME>. With a Delete reclaim policy, the underlying volume may be permanently deleted. With Retain, the old PV may need separate storage handling.

Rebuild eks-db-0 (destructive, replica only)

After confirming the target and approving the rebuild, MOCO's documented order is:

# Request deletion of the replica's data claim first.
kubectl delete pvc mysql-data-moco-eks-db-0 \
  -n database --wait=false

# Then delete the matching replica pod.
kubectl delete pod moco-eks-db-0 \
  -n database --grace-period=1
Enter fullscreen mode Exit fullscreen mode

Kubernetes protects a PVC that is still in use by a pod, which is why the deletion request may remain Terminating until the pod disappears. The StatefulSet should recreate the PVC and pod, allowing MOCO to reinitialize the instance by cloning data.

MOCO warns that, depending on the Kubernetes version, the StatefulSet may recreate a Pending pod before the old PVC has actually been removed. If that happens, verify the old claim is being deleted and follow MOCO's documented procedure for removing the premature pending pod. Do not remove PVC finalizers to force deletion.

Monitor the rebuild:

kubectl get pod,pvc -n database -w
Enter fullscreen mode Exit fullscreen mode

And inspect the recreated replica's logs:

kubectl logs -n database moco-eks-db-0 -c mysqld --since=30m
Enter fullscreen mode Exit fullscreen mode

Reinitialization may take substantial time for large databases. A recreated pod is not proof that cloning and replication have finished.

5. Verify recovery

Check MOCO's cluster status:

kubectl get mysqlcluster eks-db -n database
kubectl get pods -n database -o wide
Enter fullscreen mode Exit fullscreen mode

On the rebuilt replica, confirm:

SHOW REPLICA STATUS\G
Enter fullscreen mode Exit fullscreen mode

Look for running replication threads, no new replication errors, a catching-up/applied GTID set and acceptable lag. Also verify that MOCO no longer reports the instance as errant and that application write latency has recovered.

What I'd monitor next time

Signal Why
MOCO cluster health and errant replicas Detect degraded or divergent topology
Ready replicas and semi-sync clients Detect loss of acknowledgement sources
Semi-sync waiting sessions Identify transactions blocked on replication
Application write latency and timeouts Confirm user-visible impact

The lesson: don't confuse a running MySQL container with a healthy replication topology. Diagnose first, restart a replication thread only when justified, and rebuild a replica's data only when you have a verified source of truth.

References

Top comments (0)