Home / Databases / SQL Server

DatabasesRunbookAdvanced

Sev1: after an Availability Group failover, applications cannot connect to the listener

The AG failed over and the new primary is healthy, but apps time out or reach the old node. Check listener resource state, DNS records and TTL, MultiSubnetFailover, cluster permissions and cloud load balancer probes, then restore connections quickly.

Deepak Bhardwaj4 min read

Applies toSQL Server 2016-2022 Always On Availability Groups on Windows Server Failover Clustering, on-premises and Azure/AWS VMs (multi-subnet and load-balancer listeners)

Symptoms

  • AG dashboard shows the new primary as healthy and synchronized.
  • Applications fail with Login timeout expired, A network-related or instance-specific error, or connect to the old primary and get The target database is in an availability group and is currently accessible for connections when the application intent is set to read only or read-only errors.
  • Some app servers work, others do not.

Step 1: is the listener online? (2 minutes)

SELECT ag.name AS ag, l.dns_name, l.port, ip.ip_address, ip.state_desc
FROM sys.availability_group_listeners l
JOIN sys.availability_groups ag ON ag.group_id = l.group_id
JOIN sys.availability_group_listener_ip_addresses ip ON ip.listener_id = l.listener_id;
Get-ClusterResource | Where-Object ResourceType -in 'Network Name','IP Address','SQL Server Availability Group' |
  Format-Table Name, State, OwnerNode, ResourceType -AutoSize
Finding Go to
Network Name or IP resource Failed Step 2
All online, clients still fail Step 3 (DNS / client)
Azure/AWS VMs, all online Step 4 (load balancer)

Step 2: listener resource failed

Check the cluster log for the reason:

Get-ClusterLog -Node SQL02 -TimeSpan 30 -Destination C:\Temp
Select-String -Path C:\Temp\*cluster.log -Pattern "Network Name|DNS|0x" | Select-Object -Last 40

Common causes:

  • Cluster name object (CNO) lacks rights to update the DNS record or the listener computer object in AD (errors about access denied when registering DNS). Grant the CNO "Full control" on the listener's computer object and on the DNS record, or have DNS admins fix record ownership.
  • IP address not valid on the new subnet (multi-subnet AG without an IP for that subnet). Add an IP for each subnet:
ALTER AVAILABILITY GROUP [AG1] MODIFY LISTENER N'AG1-LSN' (ADD IP ((N'10.20.2.50', N'255.255.255.0')));

Bring it online:

Start-ClusterResource -Name "AG1_AG1-LSN"

Step 3: DNS and client settings

In a multi-subnet AG the listener has one IP per subnet. Only the IP in the active subnet is online.

Resolve-DnsName AG1-LSN.corp.local
Get-ClusterResource "AG1_AG1-LSN" | Get-ClusterParameter RegisterAllProvidersIP, HostRecordTTL
Setting Effect
RegisterAllProvidersIP = 1 DNS returns all IPs; clients must use MultiSubnetFailover=True to try them in parallel
RegisterAllProvidersIP = 0 DNS returns only the online IP; clients wait for TTL (default 1200 s) after failover
HostRecordTTL How long clients cache the old IP

Clients that lack MultiSubnetFailover=True with RegisterAllProvidersIP=1 try IPs one at a time and time out. Fix in the connection string:

Server=tcp:AG1-LSN,1433;Database=Sales;MultiSubnetFailover=True;ApplicationIntent=ReadWrite;Encrypt=True

If you cannot change all clients now (legacy apps), reduce stale caching:

Get-ClusterResource "AG1_AG1-LSN" | Set-ClusterParameter RegisterAllProvidersIP 0
Get-ClusterResource "AG1_AG1-LSN" | Set-ClusterParameter HostRecordTTL 300
Stop-ClusterResource "AG1_AG1-LSN"; Start-ClusterResource "AG1_AG1-LSN"

Warning

Restarting the listener resource drops all current connections. Do it in a short, announced window, and restart the AG resource dependency chain correctly (the AG resource depends on the listener).

On client machines, flush caches: ipconfig /flushdns. Application connection pools may hold dead connections; recycle app pools or restart services.

Step 4: cloud VMs with a load balancer listener

On Azure VMs in a single subnet (and some AWS designs), the listener IP is on an internal load balancer that sends traffic to whichever node passes the health probe.

Get-ClusterResource "AG1_IP" | Get-ClusterParameter ProbePort, Address, SubnetMask
Test-NetConnection SQL02 -Port 59999      # probe port from a peer

Check:

  • Probe port set on the cluster IP resource and matches the load balancer probe.
  • Windows Firewall on every node allows the probe port and 1433.
  • Load balancer rule has floating IP (direct server return) enabled.
az network lb probe list -g rg-sql --lb-name ilb-sql -o table
az network lb rule list -g rg-sql --lb-name ilb-sql --query "[].{name:name, floatingIp:enableFloatingIp, port:frontendPort}" -o table

Azure VMs on Windows Server 2016+ can avoid the load balancer entirely with multi-subnet AGs or a distributed network name (DNN) listener on SQL Server 2019 CU8+.

Step 5: confirm

SELECT @@SERVERNAME AS connected_to, DATABASEPROPERTYEX('Sales','Updateability') AS mode;

Run from each app server through the listener. It should return the new primary and READ_WRITE.

Prevent it next time

  • All connection strings: listener name, MultiSubnetFailover=True; no node names.
  • Test a planned failover quarterly, including every application tier.
  • Pre-stage CNO permissions and DNS record ownership; document probe ports in cloud setups.
  • Alert on cluster resource state changes for the listener.