CloudWatch alarm stuck in INSUFFICIENT_DATA or never firing: dimensions, periods, missing data and SNS delivery

The outage happened, the metric spiked, and nobody was paged. Or the alarm you created last week has shown INSUFFICIENT_DATA ever since. Both come down to a short list of causes: the alarm is watching a metric that doesn't exist exactly as written, the period doesn't match how often data arrives, missing data is treated as "fine", or the alarm fired and the notification went nowhere.

Work through the checks in order. Set these first:

export AWS_REGION=eu-west-1
ALARM="prod-api-5xx"

#Symptom to cause

#1. Read what the alarm actually watches

aws cloudwatch describe-alarms --alarm-names "$ALARM" \
  --query "MetricAlarms[0].{ns:Namespace,metric:MetricName,dims:Dimensions,stat:Statistic,ext:ExtendedStatistic,period:Period,evals:EvaluationPeriods,dta:DatapointsToAlarm,missing:TreatMissingData,threshold:Threshold,op:ComparisonOperator,actionsEnabled:ActionsEnabled,alarmActions:AlarmActions,state:StateValue,reason:StateReason}"

StateReason often tells you directly. "Insufficient Data: 0 datapoints were received" means the alarm found nothing to evaluate at all. That is almost always a check 2 problem.

List every alarm currently in that state:

aws cloudwatch describe-alarms --state-value INSUFFICIENT_DATA \
  --query "MetricAlarms[].[AlarmName,Namespace,MetricName]" --output table

#2. Confirm the metric exists with exactly those dimensions

A CloudWatch metric is identified by namespace, name and the complete set of dimensions. An alarm with LoadBalancer only won't match a metric published with LoadBalancer + TargetGroup. Values are case-sensitive, and for ALB the value is app/name/id, not the ARN or the DNS name.

aws cloudwatch list-metrics --namespace AWS/ApplicationELB \
  --metric-name HTTPCode_Target_5XX_Count \
  --query "Metrics[].Dimensions" --output json

Compare the output with dims from check 1, character for character. The usual mismatches:

• ALB/NLB: ARN pasted instead of the app/... or net/... suffix.
• EC2 via the CloudWatch agent: the agent adds InstanceId, ImageId, InstanceType, or path, device and fstype for disk metrics. The alarm has to include every one of them, or you need to change append_dimensions/aggregation_dimensions in the agent config.
• Auto Scaling replaced the instance: an alarm on InstanceId=i-old goes to INSUFFICIENT_DATA for good. Alarm on AutoScalingGroupName, or use a Metrics Insights query alarm.
• Wrong region: alarms evaluate metrics in their own region. Billing metrics (AWS/Billing) are only in us-east-1. CloudFront metrics are also in us-east-1.

Tip

list-metrics returns only metrics that received data in roughly the last two weeks. An empty result means either the dimensions are wrong or nothing is being published. Check the source before you edit the alarm again.

#3. Pull the raw datapoints at the alarm's period

aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB \
  --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/prod-api/0123456789abcdef \
  --start-time "$(date -u -d '-3 hours' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 60 --statistics Sum --output table

Look for gaps. Many AWS metrics are only published when something happens. HTTPCode_Target_5XX_Count has no datapoint in a minute with zero 5xx, and Lambda Errors has none when nothing was invoked. Other metrics arrive every 5 minutes (basic EC2 monitoring) or once a day (S3 BucketSizeBytes). A 60-second alarm on a 5-minute metric shows INSUFFICIENT_DATA four minutes out of five.

#4. Check how missing data and the statistic are handled

TreatMissingData has four values: missing (the default; the alarm can drift to INSUFFICIENT_DATA), notBreaching, breaching and ignore (keep the current state).

Warning

notBreaching on a metric that stops reporting during an outage hides the outage. If the instance, agent or app dies, a CPU or custom heartbeat metric just stops, and the alarm reports OK. For anything that must always report, use breaching, or add a second alarm on the source's health (for example StatusCheckFailed).

Check the statistic too. Average of a 0/1 error metric over 5 minutes hides a single failure, and Sum of a latency metric means nothing. For latency, use a percentile (p99). For percentile alarms on low-traffic metrics, decide what EvaluateLowSampleCountPercentile should do (evaluate or ignore).

Fix an alarm in place. put-metric-alarm replaces the whole definition, so pass every field again:

aws cloudwatch put-metric-alarm --alarm-name "$ALARM" \
  --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/prod-api/0123456789abcdef \
  --statistic Sum --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 20 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall \
  --ok-actions arn:aws:sns:eu-west-1:111122223333:oncall

#5. The alarm fired, but nobody was told

Read the history. Action failures are recorded as Action items:

aws cloudwatch describe-alarm-history --alarm-name "$ALARM" \
  --history-item-type Action --max-records 20 \
  --query "AlarmHistoryItems[].[Timestamp,HistorySummary]" --output table

aws cloudwatch describe-alarm-history --alarm-name "$ALARM" \
  --history-item-type StateUpdate --max-records 20 \
  --query "AlarmHistoryItems[].[Timestamp,HistorySummary]" --output table

If there are state changes but no action entries, check ActionsEnabled and whether an action is set for that state. --alarm-actions covers only ALARM. A disabled alarm still changes state; it just doesn't act.

aws cloudwatch enable-alarm-actions --alarm-names "$ALARM"

Then check the SNS side:

TOPIC=arn:aws:sns:eu-west-1:111122223333:oncall
aws sns list-subscriptions-by-topic --topic-arn $TOPIC \
  --query "Subscriptions[].[Protocol,Endpoint,SubscriptionArn]" --output table
aws sns get-topic-attributes --topic-arn $TOPIC \
  --query "Attributes.{policy:Policy,kms:KmsMasterKeyId}"

• SubscriptionArn of PendingConfirmation means the email was never confirmed.
• The topic policy must let cloudwatch.amazonaws.com call sns:Publish if someone replaced the default policy.
• If the topic is encrypted with alias/aws/sns, CloudWatch can't publish to it. Use a customer managed key whose key policy allows cloudwatch.amazonaws.com to use kms:Decrypt and kms:GenerateDataKey*.
• The topic must be in the same region as the alarm, unless you have deliberately set up cross-account or cross-region delivery.

#6. Test end to end

Force a state change. The alarm returns to its real state at the next evaluation:

aws cloudwatch set-alarm-state --alarm-name "$ALARM" \
  --state-value ALARM --state-reason "Paging test by on-call"

Caution

set-alarm-state triggers every action on the alarm, including Auto Scaling policies, EC2 stop/terminate/reboot actions and Systems Manager automation. Check AlarmActions before you test on a production alarm that does more than notify.

#Prevent it next time

• Build alarms from list-metrics output, not from the console's free-text fields or memory.
• Alarm on stable dimensions (AutoScalingGroupName, TargetGroup, ServiceName), not instance IDs.
• Give each alarm a TreatMissingData that you have chosen on purpose, and add a heartbeat alarm with breaching for every critical data source.
• Run a weekly report of describe-alarms --state-value INSUFFICIENT_DATA and fix or delete what it finds.
• Put set-alarm-state paging tests into your on-call handover for alarms that only send notifications.