Sev1: region outage — run an Azure Site Recovery failover for production VMs, step by step

#Before you press the button

• Get an explicit decision to fail over. Failing back later is a second project.
• Confirm the outage is regional or the VMs are unrecoverable in place. A single VM issue is usually faster to fix in place.
• Know your RPO: the latest crash-consistent point is typically minutes old; app-consistent points are less frequent.

#1. Check replication health

In the vault in the target region, check that protected items are healthy and when the last recovery point was created.

$vault = Get-AzRecoveryServicesVault -Name rsv-dr-weu -ResourceGroupName rg-dr
Set-AzRecoveryServicesAsrVaultContext -Vault $vault

$fabric = Get-AzRecoveryServicesAsrFabric | Where-Object { $_.FabricSpecificDetails.Location -eq "northeurope" }
$container = Get-AzRecoveryServicesAsrProtectionContainer -Fabric $fabric
Get-AzRecoveryServicesAsrReplicationProtectedItem -ProtectionContainer $container |
  Select-Object FriendlyName, ProtectionState, ReplicationHealth, LastSuccessfulFailoverTime

Items should be Protected. Note any that are Critical or not replicating; they will fail over to an older point or not at all.

#2. Use a recovery plan, not single VMs

A recovery plan fails over groups in order: databases first, then app tier, then web tier, with scripts between.

$plan = Get-AzRecoveryServicesAsrRecoveryPlan -Name rp-orders-prod
$plan.Groups | ForEach-Object { $_.GroupType; $_.ReplicationProtectedItems.FriendlyName }

If no recovery plan exists, fail over databases and domain controllers first, then the rest.

#3. Pick the recovery point

#4. Run the failover

Because the source region is down, the "shut down source VMs" step is skipped or fails harmlessly.

$job = Start-AzRecoveryServicesAsrUnplannedFailoverJob -RecoveryPlan $plan `
  -Direction PrimaryToRecovery -RecoveryTag LatestProcessed
Get-AzRecoveryServicesAsrJob -Job $job | Select-Object State, StateDescription

Poll until the job finishes. Watch for per-VM errors in the job tasks.

Warning

Do not run the failover on a VM that a database is already handling through its own replication (for example SQL Always On or a geo-replica). You end up with two primaries. Fail those over with the database's own mechanism.

#5. Make the recovered VMs reachable

• DNS: update records or Traffic Manager / Front Door to point at the DR endpoints. Lower TTLs only help if set before the outage.
• Load balancers and public IPs: attach them if not handled by the recovery plan scripts.
• NSGs and firewall rules in the DR VNet must match production.
• Domain: make sure a domain controller is running in the DR region before app servers boot.
• Connection strings pointing at primary-region PaaS endpoints must move to the DR copies.

#6. Validate, then commit

Test the business transactions with app owners. When satisfied, commit, which removes the other recovery points:

Start-AzRecoveryServicesAsrCommitFailoverJob -RecoveryPlan $plan

#7. After the incident

• When the primary region is back, re-protect the VMs in the reverse direction, wait for initial replication, then plan a failback during a maintenance window.
• Record the actual RTO and RPO achieved against targets.

#Prevent it next time

• Run a test failover into an isolated VNet at least twice a year and after major changes.
• Keep recovery plans with ordered groups and automation runbooks for DNS and IPs.
• Monitor ASR replication health alerts and RPO breaches.
• Document who can declare a disaster and authorise failover.