Home / AWS

AWSHow-toIntermediate

Optimization: fewer, better CloudWatch alarms - anomaly detection, composite alarms and M-of-N evaluation

Noisy CloudWatch alarms train people to ignore pages. Measure which alarms flap and which lead to action, fix evaluation settings and missing-data handling, use anomaly detection for metrics with daily patterns, combine symptoms with composite alarms and suppress dependent alarms during known outages.

Deepak Bhardwaj3 min read

Applies toAmazon CloudWatch metric alarms, composite alarms, anomaly detection, SNS / Incident Manager / chat integrations

1. Find the noisy alarms

Alarm state changes over 30 days:

aws cloudwatch describe-alarm-history --history-item-type StateUpdate \
  --start-date "$(date -u -d '-30 days' +%FT%TZ)" --max-items 5000 \
  --query 'AlarmHistoryItems[].AlarmName' --output text | tr '\t' '\n' | sort | uniq -c | sort -rn | head -25

For each top alarm: how many times did it go to ALARM, and how many times did someone do something?

2. Evaluation settings

Problem Fix
Fires on single spikes DatapointsToAlarm M of N (e.g. 3 of 5 one-minute periods)
Flaps around the threshold Longer period or wider gap; separate warning vs critical alarms
ALARM on gaps (instance stopped, no traffic at night) TreatMissingData: notBreaching or ignore, deliberately chosen per alarm
Percent metrics on tiny volumes (1 error in 2 requests = 50%) Add a minimum-volume condition via metric math

Metric math example: error rate only when there's real traffic:

[
  {"Id": "e", "MetricStat": {"Metric": {"Namespace": "AWS/ApplicationELB", "MetricName": "HTTPCode_Target_5XX_Count",
     "Dimensions": [{"Name": "LoadBalancer", "Value": "app/web-alb/50dc6c495c0c9188"}]}, "Period": 60, "Stat": "Sum"}, "ReturnData": false},
  {"Id": "r", "MetricStat": {"Metric": {"Namespace": "AWS/ApplicationELB", "MetricName": "RequestCount",
     "Dimensions": [{"Name": "LoadBalancer", "Value": "app/web-alb/50dc6c495c0c9188"}]}, "Period": 60, "Stat": "Sum"}, "ReturnData": false},
  {"Id": "rate", "Expression": "IF(r > 100, 100 * e / r, 0)", "Label": "5xx rate when > 100 req/min", "ReturnData": true}
]

3. Anomaly detection for patterned metrics

For metrics with daily/weekly cycles (requests, latency, queue depth), a static threshold is either too tight at peak or too loose at night. An anomaly detection band follows the pattern:

aws cloudwatch put-metric-alarm --alarm-name web-requests-anomaly \
  --comparison-operator LessThanLowerOrGreaterThanUpperThreshold --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold-metric-id ad1 --treat-missing-data ignore \
  --metrics '[{"Id":"m1","ReturnData":true,"MetricStat":{"Metric":{"Namespace":"AWS/ApplicationELB","MetricName":"RequestCount","Dimensions":[{"Name":"LoadBalancer","Value":"app/web-alb/50dc6c495c0c9188"}]},"Period":300,"Stat":"Sum"}},
              {"Id":"ad1","Expression":"ANOMALY_DETECTION_BAND(m1, 3)","Label":"band"}]'

The model needs history (about two weeks for good bands). Exclude known abnormal periods (incidents, migrations) from training.

4. Composite alarms

Page on a combination that means real trouble, keep the inputs as non-paging:

aws cloudwatch put-composite-alarm --alarm-name orders-service-degraded \
  --alarm-rule 'ALARM("orders-5xx-rate") AND (ALARM("orders-latency-p99") OR ALARM("orders-healthy-hosts-low"))' \
  --alarm-actions arn:aws:sns:eu-west-1:123456789012:oncall-page

Use the ActionsSuppressor option to silence a composite while a parent alarm (e.g. "region network issue" or "maintenance mode") is in ALARM, so dependent alarms don't flood during a known event.

5. Route by severity

Severity Route
Critical (customer impact) Page (Incident Manager / paging tool)
Warning Team chat + ticket
Info Dashboard only (no alarm action)

Every paging alarm has an owner and a runbook link in its description.

6. Symptom first

Prefer user-facing signals (error rate, latency, synthetic canary failures, queue age) for paging, and keep cause metrics (CPU, memory) as warnings or dashboard context.

7. Review monthly

Top flapping alarms, % of pages acted on, and alarms without owners. Delete or downgrade what nobody acts on, after confirming a symptom alarm covers it.

Tip

CloudWatch charges per alarm metric (and anomaly detection alarms count as several). Consolidating noisy per-instance alarms into a few composite or metric-math alarms often lowers cost too.

Checklist

  • Flapping alarms and action rate measured
  • M-of-N and missing-data treatment set deliberately
  • Minimum-volume conditions on rate alarms
  • Anomaly detection on patterned metrics
  • Composite alarms page; inputs don't; suppressors for known events
  • Severity routing, owners and runbooks

Related fixes