Azure Monitor alert didn't fire, or nobody got the notification: a checklist

Something broke, the on-call phone stayed quiet, and the post-incident question is "why didn't we get an alert?" There are only three answers: the rule never fired, it fired but the notification was suppressed, or it fired and the action group didn't deliver. Work through this list in order and you'll know which one within fifteen minutes.

#1. Did the alert fire at all?

Every fired alert is recorded for about 30 days, whether or not anyone was notified. Query them with Azure Resource Graph (the CLI offers to install the resource-graph extension the first time):

az graph query -q "
alertsmanagementresources
| where type =~ 'microsoft.alertsmanagement/alerts'
| extend e = properties.essentials
| where todatetime(e.startDateTime) > ago(2d)
| project start = todatetime(e.startDateTime), rule = tostring(e.alertRule),
          sev = tostring(e.severity), condition = tostring(e.monitorCondition),
          suppressed = tobool(e.actionStatus.isSuppressed), target = tostring(e.targetResource)
| order by start desc" --first 200 -o table

#2. The rule didn't fire

#Is the rule enabled and pointing at the right thing?

# Metric alerts
az monitor metrics alert list -g rg-monitoring \
  --query "[].{name:name, enabled:enabled, freq:evaluationFrequency, window:windowSize, scopes:join(',', scopes)}" -o table

# Log search alerts (the CLI may prompt to install an extension for this group)
az monitor scheduled-query list -g rg-monitoring \
  --query "[].{name:name, enabled:enabled, freq:evaluationFrequency, window:windowSize, mute:muteActionsDuration, autoMitigate:autoMitigate}" -o table

# Activity log alerts (Service Health, resource deletes, etc.)
az monitor activity-log alert list -g rg-monitoring \
  --query "[].{name:name, enabled:enabled}" -o table

Checks:

• enabled is true. If not, find who disabled it: filter the Activity Log on the rule's resource ID.
• The scope still matches. A rule scoped to a VM that was rebuilt with a new resource ID watches nothing. A rule scoped to a resource group only covers resource types the metric applies to.
• The threshold, operator and aggregation make sense. "Average CPU > 90% over 15 minutes" won't fire on a 4-minute spike. "Total" vs "Average" vs "Maximum" changes everything for count-type metrics.
• Dimension filters still match. A rule filtered on ApiName = GetBlob or an instance name misses everything if names changed.

#Was there data to evaluate?

A metric alert can't fire on data that doesn't exist. A deallocated VM, a stopped agent or a deleted resource produces no data, not a breach. Check what the rule saw:

az monitor metrics list --resource <resource-id> --metric "Percentage CPU" \
  --interval PT5M --aggregation Average Maximum --offset 6h -o table

For "the server is down" scenarios, alert on absence of data instead, for example a log search alert on the Heartbeat table:

Heartbeat
| summarize LastSeen = max(TimeGenerated) by Computer
| where LastSeen < ago(10m)

#Did the log query work?

• Run the rule's query in the workspace over the incident window. If it returns nothing, the rule had nothing to fire on.
• Check the rule's identity still has read access to every workspace or resource the query touches. Cross-workspace queries break quietly when a workspace moves or access is removed.
• Look at the rule's Activity Log and the portal's rule history for query failures. A rule whose query keeps failing can end up disabled.
• Ingestion latency: if data arrives after the evaluation window has passed, the rule evaluates an empty window. Widen the window or lag the query.

#Was it already firing?

Metric alerts are stateful: they fire once and won't fire again until the condition resolves. Log search alerts behave the same when autoMitigate is true. If the alert fired yesterday and never resolved, today's outage won't generate a new notification. Check condition = Fired rows from earlier in section 1. Also check muteActionsDuration on log alerts, which deliberately silences repeats.

#3. It fired but was suppressed

Alert processing rules can remove all action groups during maintenance windows. A rule left on after a change window, or scoped wider than intended, silences everything under it.

# Uses the alertsmanagement extension, installed automatically on first run
az monitor alert-processing-rule list \
  --query "[].{name:name, enabled:properties.enabled, type:properties.actions[0].actionType, scopes:join(',', properties.scopes), schedule:properties.schedule}" -o json

• Any enabled rule with RemoveAllActionGroups whose scope covers the resource or subscription.
• Its schedule. A rule with no schedule applies all the time.
• Disable or delete it once you confirm nobody still needs it.

Warning

A suppression rule at subscription scope silences every alert in that subscription, including Service Health. Before you create one, give it an end date in the schedule, not a calendar reminder.

#4. It fired but nobody was notified

#Is the action group attached and enabled?

az monitor metrics alert show -g rg-monitoring -n alert-vm-cpu \
  --query "actions[].actionGroupId" -o tsv

az monitor action-group show -g rg-monitoring -n ag-oncall \
  --query "{enabled:enabled, email:emailReceivers[].{name:name, addr:emailAddress, status:status}, sms:smsReceivers[].{name:name, status:status}, webhooks:webhookReceivers[].{name:name, uri:serviceUri}}" -o json

• The rule actually references an action group. Rules created from templates or the portal "quick create" sometimes have none.
• The action group's enabled is true.
• No receiver shows status: Disabled. When someone clicks "unsubscribe" in an alert email, or replies STOP to an SMS, that receiver is disabled for the whole action group. Re-enable it:

az monitor action-group enable-receiver -g rg-monitoring \
  --action-group ag-oncall --name oncall-email

#Is the receiver reachable?

#Send a test notification

Test the path end to end without waiting for a real incident:

az monitor action-group test-notifications create -g rg-monitoring \
  --action-group ag-oncall --alert-type metricstaticthreshold \
  --add-action email oncall-test oncall@contoso.com usecommonalertschema

--alert-type controls the sample payload. Valid values include metricstaticthreshold, logalertv2, activitylog, servicehealth and resourcehealth. Run it for each receiver type the group uses, and check that the message arrives where the on-call person actually looks.

#Prevent it next time

• Alert on absence of data (heartbeat, no requests) for critical systems, not only on thresholds.
• Use one or two shared on-call action groups, defined in code, rather than a group per rule.
• Send email to a team list and page through your paging tool; don't rely on one person's inbox.
• Review alert processing rules monthly and give every suppression an end date.
• Run test-notifications against each on-call action group on a schedule, for example after every on-call rotation change.
• Add a "did we get alerted?" line to every incident review. If the answer is no, the fix goes on the action list like any other root cause.

Tip

Fired alerts age out of the alert store after about 30 days. If you need a longer history for incident reviews, have the action group send every alert to your ticketing tool, or to a Logic App or Event Hub that stores it.