Sev1: Windows cluster lost quorum and every Availability Group database is offline — forcing quorum safely

#Symptoms

• Applications can't connect to the listener; AG role shows RESOLVING on every replica.
• Errors such as The availability group ... is not ready for automatic failover or error 35250/41131 in the SQL error log.
• Windows event log: event 1177 (The Cluster service is shutting down because quorum was lost) or 1135 (node removed from membership).

#1. Confirm the cluster state

Get-ClusterNode | Format-Table Name, State, NodeWeight, DynamicWeight
Get-ClusterQuorum | Format-List *
Get-WinEvent -LogName System -MaxEvents 200 |
  Where-Object { $_.ProviderName -eq 'Microsoft-Windows-FailoverClustering' } |
  Select-Object TimeCreated, Id, Message -First 20

Typical causes: a network split between data centres, the witness (file share or cloud witness) unreachable together with a node, or several nodes rebooted by patching at the same time.

#2. First try the normal path

If the lost nodes or the witness can come back in minutes (network fixed, VM restarted), do that. When enough votes return, the cluster regains quorum by itself and the AG comes online with no data loss. Forcing quorum is for when that is not possible.

#3. Decide which replica becomes primary

-- On each reachable replica
SELECT ag.name, ars.role_desc, ars.synchronization_health_desc,
       drs.database_id, drs.synchronization_state_desc, drs.last_hardened_lsn, drs.last_commit_time
FROM sys.dm_hadr_availability_replica_states ars
JOIN sys.availability_groups ag ON ag.group_id = ars.group_id
LEFT JOIN sys.dm_hadr_database_replica_states drs ON drs.replica_id = ars.replica_id AND drs.is_local = 1
WHERE ars.is_local = 1;

Choose the replica with the highest last_hardened_lsn (most data). A replica that was SYNCHRONIZED in synchronous-commit mode before the failure has no data loss.

#4. Force quorum on that node

Stop-Service ClusSvc -Force
Start-ClusterNode -Name SQLNODE2 -FixQuorum

Caution

Force quorum on one partition only. If nodes on the other side of a network split are also forced up, you get two clusters and two primaries (split brain). Make sure the other side is down or isolated, and tell the other data centre team not to start the cluster.

Remove votes from nodes that are gone so the remaining nodes can keep quorum:

(Get-ClusterNode -Name SQLNODE1).NodeWeight = 0
(Get-ClusterNode -Name SQLNODE3).NodeWeight = 0

#5. Bring the AG online

If the chosen replica was the primary, the AG may come online by itself. Otherwise force failover on it:

-- On the chosen secondary. Synchronous + SYNCHRONIZED replicas: try a normal failover first
ALTER AVAILABILITY GROUP [AG_Orders] FAILOVER;

-- If that is refused, accept possible data loss explicitly
ALTER AVAILABILITY GROUP [AG_Orders] FORCE_FAILOVER_ALLOW_DATA_LOSS;

Check the listener is online and test an application connection.

#6. When the other nodes return

• Secondary databases on the returning replicas are suspended after a forced failover. Compare data if needed, then resume:

ALTER DATABASE [Orders] SET HADR RESUME;

• Restore node weights and quorum configuration (witness) when all nodes are healthy:

(Get-ClusterNode -Name SQLNODE1).NodeWeight = 1
Set-ClusterQuorum -CloudWitness -AccountName stwitnessprod -AccessKey '<key>'

• Take a full backup on the new primary to start a clean chain.

#Prevent it next time

• Use a witness (cloud witness for multi-site) and keep an odd number of votes.
• Never patch more than one node at a time; drain roles before rebooting.
• Monitor event IDs 1135, 1177 and 1205 and AG health with alerts.
• Write down the forced-quorum decision process and who can approve data loss.