Home / OCI

OCIHow-toIntermediate

Optimization: reducing OCI Monitoring alarm noise - MQL tuning, pending duration and suppressions

OCI alarms that fire on every short spike or during maintenance teach teams to ignore notifications. Measure which alarms fire most, tune MQL intervals and statistics, use pending duration, split notifications by severity, suppress during planned work and alert on symptoms that matter.

Deepak Bhardwaj3 min read

Applies toOCI Monitoring alarms (MQL), Notifications topics, alarm suppressions, Events, OCI Logging-based alerting

1. Find the noisy alarms

Alarm history per alarm (last 30 days):

oci monitoring alarm list --compartment-id $C --query 'data[].{name:"display-name",id:id}' --output table
oci monitoring alarm-history-collection get-alarm-history --alarm-id $ALARM \
  --alarm-historytype STATE_TRANSITION_HISTORY --timestamp-greater-than-or-equal-to "$(date -u -d '-30 days' +%FT%TZ)" \
  --query 'length(data.entries)'

For the top alarms: how many fired, and how many led to action?

2. Tune the query

Problem Fix
Fires on a 1-minute spike Longer interval or statistic over a window: CpuUtilization[5m].mean() > 85
Average hides problems Use .percentile(0.95) or .max() where peaks matter
One alarm per instance (hundreds) One alarm with a dimension filter/grouping across the compartment, notifications per dimension
Fires when there's no data Check "no data" handling; alert on absence only where it means failure 
# CPU sustained high on any production instance (per-instance notifications)
CpuUtilization[5m]{freeformTags.environment = "prod"}.groupBy(resourceId).mean() > 85

Grouping and dimension support in MQL varies by metric. Test the query in Metrics Explorer first.

3. Pending duration

Pending duration makes the alarm wait until the condition stays true:

oci monitoring alarm update --alarm-id $ALARM --pending-duration PT15M

15 minutes stops flapping on short spikes; keep it short (1–5 minutes) for availability alarms where every minute counts.

4. Severity and routing

Severity Notification topic
CRITICAL Paging integration (PagerDuty/Opsgenie via HTTPS, or SMS)
ERROR / WARNING Team chat (Slack/Teams via Function or HTTPS) + ticket
INFO Email digest or dashboard only

Set severity on each alarm and send it to the right topic. Enable per-dimension notifications only where a separate message per resource is useful. Set repeat notification frequency so unresolved critical alarms remind people, while warnings don't repeat every minute.

5. Suppress during planned work

Use alarm suppression for maintenance windows (console or CLI) with a start and end time and a description, so the alarm still evaluates but notifications are held:

oci monitoring alarm update --alarm-id $ALARM \
  --suppression '{"description": "Saturday patch window CHG1234", "timeSuppressFrom": "2026-11-07T20:00:00Z", "timeSuppressUntil": "2026-11-07T23:00:00Z"}'

Automate suppressions from your change calendar where possible, and remove them when the window ends.

6. Alert on symptoms

Prefer alarms on what users notice: load balancer 5xx rate and latency, health check failures, API Gateway errors, queue age, synthetic checks (Health Checks / APM synthetic monitors). Keep CPU/memory as warnings.

7. Review monthly

  • Alarms fired vs acted on.
  • Alarms without owners (use defined tags owner on alarms).
  • Alarms whose topics have no subscribers (silent alarms are worse than noisy ones).

Warning

Before deleting a noisy alarm, confirm another alarm (usually a symptom alarm) would catch the failure it was meant to detect.

Checklist

  • Top firing alarms and action rate measured
  • MQL intervals, statistics and grouping tuned
  • Pending duration set per alarm type
  • Severity-based topics and repeat settings
  • Suppressions for planned work
  • Symptom alarms; owners; monthly review

Related fixes