Symptom
Hundreds of alert emails a day, the same CPU alert firing and resolving every hour, alerts during patch windows, and a real incident missed in the noise.
1. Measure the noise
Azure Resource Graph over fired alerts:
alertsmanagementresources
| where type == "microsoft.alertsmanagement/alerts"
| where properties.essentials.startDateTime > ago(30d)
| extend rule = tostring(properties.essentials.alertRule), sev = tostring(properties.essentials.severity),
state = tostring(properties.essentials.alertState)
| summarize fired = count(), acknowledged = countif(state != "New") by rule, sev
| sort by fired desc
| take 30
For each top rule ask: did anyone act on it? If not, it's noise, or a real problem nobody owns.
2. Fix the rule, not the inbox
| Problem | Fix |
|---|---|
| Static threshold too low for a busy server | Dynamic threshold (learns the normal pattern) |
| Spiky metric fires on 1-minute peaks | Longer aggregation window (15 min) and "failing periods" (e.g. 3 of 4) |
| Same condition on 200 VMs = 200 rules | One multi-resource metric alert scoped to a subscription/RG |
| Alert fires and resolves repeatedly | Stateful alerts with sensible auto-resolve; widen thresholds |
| Info-only alerts (disk at 70%) paging people | Lower severity, email/ticket only |
| Log alerts every 5 minutes on slow-changing data | Run every 15–60 minutes |
Dynamic threshold example:
az monitor metrics alert create -g rg-mon -n "vm-cpu-anomaly" \
--scopes /subscriptions/<sub>/resourceGroups/rg-app \
--region westeurope --target-resource-type Microsoft.Compute/virtualMachines \
--condition "avg Percentage CPU > dynamic medium 3 of 4 since 2026-10-01T00:00:00Z" \
--window-size 15m --evaluation-frequency 5m --severity 3 --action ag-ops-ticket
Dynamic thresholds need some history to learn and are poor for metrics with no pattern; static thresholds still fit things like "disk free < 10%".
3. Suppress during planned work
Alert processing rules remove notifications without deleting alerts:
az monitor alert-processing-rule create -g rg-mon -n "patch-window-suppress" \
--rule-type RemoveAllActionGroups \
--scopes /subscriptions/<sub>/resourceGroups/rg-app \
--schedule-recurrence-type Weekly --schedule-recurrence Saturday \
--schedule-recurrence-start-time 22:00:00 --schedule-recurrence-end-time 23:59:59 \
--schedule-time-zone "UTC" --description "Saturday patching"
Alerts still appear in the portal for later review; just nobody is woken up.
4. Route by severity
| Severity | Meaning | Route |
|---|---|---|
| Sev0–1 | Service down / data at risk | Page on-call (phone/SMS/incident tool) |
| Sev2 | Degraded, needs action today | Ticket + team channel |
| Sev3–4 | Warning, trend | Ticket or daily digest |
Use alert processing rules to attach action groups by severity at subscription level, instead of editing every rule.
5. Alert on symptoms first
Prefer alerts users would notice (availability tests failing, 5xx rate, latency SLO burn, queue age) over every cause (CPU, memory). Keep cause alerts as lower severity context.
6. Start from a baseline, then prune
The Azure Monitor Baseline Alerts (AMBA) policy sets give a consistent starting point for platform resources. Apply them, then run the noise query monthly and tune. Every rule should have an owner and a linked runbook.
7. Track improvement
| Metric | Target example |
|---|---|
| Alerts per week | Down 50–80% after first clean-up |
| % alerts acted on | > 70% for Sev0–2 |
| Pages per on-call shift | A handful, not dozens |
| Missed incidents (detected by users first) | Trending to zero |
Warning
Don't fix noise by disabling rules wholesale. For each removed rule, confirm another alert (usually a symptom alert) still covers the failure it was meant to catch.
Checklist
- Top firing rules and action rate measured
- Dynamic thresholds / longer windows where needed
- Per-resource rules consolidated
- Maintenance suppression with processing rules
- Severity-based routing
- Owners and runbooks on remaining rules; monthly review