OCI Notifications not delivering: subscriptions stuck Pending, alarms firing but nobody paged

#Symptom

The alarm shows Firing in the console, or an Events rule matched, but the on-call channel stayed quiet. Sometimes only one channel gets the message: email arrives but Slack does not, or the other way round.

Notifications has two sides, and either can fail:

1. Publisher → topic. An alarm, an Events rule, Connector Hub or your own code publishes a message to a topic.
2. Topic → subscriptions. The topic delivers to every Active subscription. Pending subscriptions get nothing.

Test the topic directly first. That tells you which side is broken.

#Quick check: publish a test message

TOPIC=ocid1.onstopic.oc1..xxxx

oci ons message publish --topic-id $TOPIC \
  --title "ONS delivery test" \
  --body "Test from $(hostname) at $(date -u +%FT%TZ). Ignore."

• Test arrives everywhere: the topic and subscriptions work. The problem is on the publisher side (alarm, rule, connector). Skip to the publisher checks.
• Test is missing on some or all channels: the problem is on the subscription side. Continue below.
• The command fails with NotAuthorizedOrNotFound: you lack use ons-topics on that compartment, or the OCID or region is wrong. Topics are regional, so check --region.

#Check the subscriptions

List every subscription on the topic with its protocol and state:

C=ocid1.compartment.oc1..xxxx

oci ons subscription list --compartment-id $C --topic-id $TOPIC \
  --query 'data[].{id:id,protocol:protocol,endpoint:endpoint,state:"lifecycle-state"}' \
  --output table

#Resend a confirmation

SUB=ocid1.onssubscription.oc1..xxxx
oci ons subscription resend-confirmation --id $SUB

Where the confirmation shows up depends on the protocol:

• Email: a message with a confirmation link. Shared mailboxes and distribution lists often quarantine it, so ask the mailbox owner to look.
• Slack: search the channel for "To confirm the subscription".
• PagerDuty: an incident titled "Oracle Notification Service Subscription Confirmation".
• HTTPS (custom URL): the URL is in the X-OCI-NS-ConfirmationURL request header and in the ConfirmationURL field of the JSON body. Your endpoint, or a person reading its logs, must open that URL with a GET request.
• SMS: reply with the keyword in the message. The sender and the wording depend on the recipient's country.
• Function: no confirmation needed.

Warning

Every notification email has an unsubscribe link. If someone on a distribution list clicks it, the subscription is deleted for the whole list and alerts stop with no warning. Use per-team subscriptions. Watch for subscription deletions with an Events rule (next section).

#Webhook endpoints that confirm themselves

If you run your own HTTPS receiver, make it handle the confirmation message automatically:

# Minimal Flask handler: auto-confirm ONS subscriptions, then process alerts
import requests
from flask import Flask, request

app = Flask(__name__)

@app.post("/oci-alerts")
def oci_alerts():
    confirm_url = request.headers.get("X-OCI-NS-ConfirmationURL")
    if not confirm_url and request.is_json:
        confirm_url = (request.get_json(silent=True) or {}).get("ConfirmationURL")
    if confirm_url:
        requests.get(confirm_url, timeout=10)
        return "", 200
    payload = request.get_data(as_text=True)
    # forward payload to your paging or ticketing system here
    return "", 200

Return a 2xx quickly. Slow or failing endpoints cause retries and delayed alerts.

#Check the publisher

The test message arrived, so the topic works. Find out why the real publisher didn't send anything.

#Monitoring alarms

ALARM=ocid1.alarm.oc1..xxxx

oci monitoring alarm get --alarm-id $ALARM \
  --query 'data.{enabled:"is-enabled",destinations:destinations,pending:"pending-duration",repeat:"repeat-notification-duration",suppression:suppression}'

oci monitoring alarm-status list-alarms-status --compartment-id $C \
  --query 'data[].{name:"display-name",status:status,severity:severity}' --output table

#Events rules

oci events rule list --compartment-id $C \
  --query 'data[].{name:"display-name",state:"lifecycle-state",enabled:"is-enabled"}' --output table

oci events rule get --rule-id ocid1.eventrule.oc1..xxxx \
  --query 'data.{condition:condition,actions:actions.actions[].{type:"action-type",topic:"topic-id",enabled:"is-enabled"}}'

Check that the rule and its ONS action are both enabled and that the rule lives in a compartment that contains the event source. Rules see events from their own compartment and its children. Object Storage events also need Emit Object Events turned on for the bucket.

#Connector Hub

A connector that writes to Notifications needs a policy that lets the connector publish to the topic. If the policy is missing, the connector shows errors and messages are dropped:

Allow any-user to use ons-topics in compartment <topic-compartment>
  where all {request.principal.type = 'serviceconnector', request.principal.compartment.id = '<connector-compartment-ocid>'}

#Prevent it next time

• Heartbeat alarm. Create an always-firing low-severity alarm on a dedicated topic, or publish a scheduled test message, and check it arrives. Silence on the heartbeat means delivery is broken.
• Alert on subscription changes. Add an Events rule for Notifications subscription delete and update events (event type names vary by version; pick them from the Events rule builder under the Notifications service) and send it to a second topic with a different channel.
• Two channels for Sev1. Subscribe critical topics to both a paging tool and email, so one failing endpoint doesn't hide an outage.
• Review Pending weekly. Run the subscription list query above across compartments and fix anything not ACTIVE.
• Put topics under Terraform. Recreating a deleted subscription is then a plan and apply. Confirmation is still a manual step, so document who confirms each endpoint.

Tip

Use oci ons message publish as the last step of every alarm change, so you know the path works while you still remember what you changed.