Sev1: MySQL InnoDB Cluster lost quorum — writes blocked, how to restore a primary without split brain

#Symptoms

• Applications get errors like The MySQL server is running with the --super-read-only option or writes hang.
• cluster.status() shows NO_QUORUM or members UNREACHABLE.
• After a data-centre power event, no member is running Group Replication.

#1. Check the status

mysqlsh --uri clusteradmin@db1:3306 -- cluster status

or in MySQL Shell (JS mode):

var c = dba.getCluster();
c.status({extended: 1});

From SQL on any reachable member:

SELECT member_host, member_state, member_role FROM performance_schema.replication_group_members;

#2. Force quorum with the surviving partition

Only when you are certain the unreachable members are really down or isolated, not just unreachable from you:

var c = dba.getCluster();
c.forceQuorumUsingPartitionOf("clusteradmin@db1:3306");

Caution

If the "unreachable" members are actually running and serving clients on the other side of a network split, forcing quorum creates two groups accepting writes. Check those hosts directly (console, cloud provider) before forcing.

After forcing, the survivor becomes a working single-member group. Writes resume through MySQL Router.

#3. Reboot from complete outage

When all members were stopped (power loss, all VMs restarted), start mysqld on every node, then from one node:

dba.rebootClusterFromCompleteOutage();

MySQL Shell checks the GTID sets of reachable members and picks the most up-to-date one as the new primary. If it reports that another member has more transactions, connect to that member and run the command there instead. Don't override that check unless you accept losing those transactions.

#4. Check MySQL Router

mysqlrouter --version
journalctl -u mysqlrouter -n 50 --no-pager

Router reads metadata from the cluster; after recovery it should route to the new primary within seconds. If apps still fail, restart Router and check its metadata cache logs.

#5. Rejoin the failed members

When the other hosts are back:

var c = dba.getCluster();
c.rejoinInstance("clusteradmin@db2:3306");
c.rejoinInstance("clusteradmin@db3:3306");
c.status();

If a member has diverged (extra GTIDs not on the primary, for example after a split), it can't rejoin cleanly. Remove it and add it back with clone:

c.removeInstance("clusteradmin@db3:3306", {force: true});
c.addInstance("clusteradmin@db3:3306", {recoveryMethod: "clone"});

Extra transactions on a diverged member are discarded by clone. Compare and save them first if they matter.

#6. Confirm health

SELECT member_host, member_state, member_role FROM performance_schema.replication_group_members;
SELECT @@global.super_read_only, @@global.read_only;   -- 0 on primary

#Prevent it next time

• Three or five members in separate failure domains (zones/racks); never two.
• Alert on member state changes and on NO_QUORUM.
• Keep group_replication_exit_state_action and autorejoin_tries at sensible values for your recovery process.
• Practise rebootClusterFromCompleteOutage in staging after each version upgrade.