OCI Connector Hub not moving data: policies, source and target errors, and deactivated connectors

#Symptom

A connector that used to ship data has gone quiet. Typical reports:

• The audit or VCN flow log archive bucket has no new objects since a certain date.
• The SIEM reading from a stream sees no new messages.
• A function that should run on every log batch hasn't been invoked.
• The connector shows Inactive in the console and nobody remembers turning it off.

Connector Hub reads from a source (Logging, Streaming, Monitoring, Queue), optionally runs a task (a log filter or a function), and writes to a target. A failure at any of those three stages stops the flow. The connector's own metrics tell you which stage.

#Quick triage

#1. Read the connector state

oci sch service-connector list \
  --compartment-id ocid1.compartment.oc1..xxxx --all \
  --query 'data.items[].{name:"display-name",state:"lifecycle-state",id:id}' \
  --output table

Then the full definition of the broken one:

oci sch service-connector get \
  --service-connector-id ocid1.serviceconnector.oc1..xxxx \
  --query 'data.{state:"lifecycle-state",details:"lifecycle-details",source:source,tasks:tasks,target:target}'

Write down the source log group or stream, the task, and the target OCIDs or bucket name. You'll check each one.

Also check the Announcements in the console. Connector Hub sends a warning announcement after about 4 days of continuous failure and another when it deactivates the connector.

#2. Find the failing stage with metrics

All connector metrics are in the oci_service_connector_hub namespace. Pull the last 24 hours for one connector:

CID=ocid1.serviceconnector.oc1..xxxx
for M in BytesReadFromSource MessagesWrittenToTarget ErrorsAtSource ErrorsAtTask ErrorsAtTarget; do
  echo "== $M"
  oci monitoring metric-data summarize-metrics-data \
    --compartment-id ocid1.compartment.oc1..xxxx \
    --namespace oci_service_connector_hub \
    --query-text "${M}[1h]{connectorId = \"$CID\"}.sum()" \
    --start-time "$(date -u -d '-24 hours' +%Y-%m-%dT%H:%M:%SZ)" \
    --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
    --query 'data[].{dims:dimensions,points:"aggregated-datapoints"[-3:]}'
done

The error metrics carry errorCode and errorType dimensions, so the output tells you whether you have 404 NotAuthorizedOrNotFound, 429 TooManyRequests, -1 or a 5xx.

Read the result like this:

• Bytes read is 0 and no errors. Nothing matches the source. Run the connector's log query yourself in Logging → Search over the same log group. A log that was disabled, moved to a new log group or renamed is the usual cause. Connectors can't follow a log that moved.
• Bytes read is non-zero, written is 0. The task or target is failing. Steps 3 and 4.
• DataFreshness keeps climbing. The connector is running but falling behind. For Logging sources, a connector reads at most 1 GB of log data per hour, and data undelivered for more than 24 hours is lost. Split high-volume logs across connectors.

Warning

Log data that sits undelivered for more than 24 hours is dropped and won't be backfilled when you fix the connector. If the archive has to be complete for compliance, export the gap from Logging Search now, before you start fixing.

#3. Fix target authorization (404)

Connectors don't use your identity. They act as a serviceconnector principal and need a policy for each target. When you create a connector, the console offers to create these policies. If someone cleaned up "unused" policies, renamed the bucket or moved the target to a different compartment, the policy no longer matches.

The default statements, by target:

# Object Storage
Allow any-user to manage objects in compartment id <bucket_compartment_ocid>
  where all {request.principal.type='serviceconnector', target.bucket.name='<bucket_name>',
             request.principal.compartment.id='<connector_compartment_ocid>'}

# Streaming
Allow any-user to use stream-push in compartment id <stream_compartment_ocid>
  where all {request.principal.type='serviceconnector', target.stream.id='<stream_ocid>',
             request.principal.compartment.id='<connector_compartment_ocid>'}

# Functions (task or target)
Allow any-user to use fn-function in compartment id <function_compartment_ocid>
  where all {request.principal.type='serviceconnector', request.principal.compartment.id='<connector_compartment_ocid>'}
Allow any-user to use fn-invocation in compartment id <function_compartment_ocid>
  where all {request.principal.type='serviceconnector', request.principal.compartment.id='<connector_compartment_ocid>'}

# Monitoring
Allow any-user to use metrics in compartment id <metric_compartment_ocid>
  where all {request.principal.type='serviceconnector', target.metrics.namespace='<namespace>',
             request.principal.compartment.id='<connector_compartment_ocid>'}

# Notifications
Allow any-user to use ons-topics in compartment id <topic_compartment_ocid>
  where all {request.principal.type='serviceconnector', request.principal.compartment.id='<connector_compartment_ocid>'}

Find what exists today:

oci iam policy list --compartment-id ocid1.compartment.oc1..xxxx --all \
  --query "data[?contains(to_string(statements), 'serviceconnector')].{name:name,statements:statements}"

Run it against the target's compartment and its parents, up to the tenancy. Check three things in each statement: the compartment OCID is the target's compartment, target.bucket.name or target.stream.id matches the current target exactly, and request.principal.compartment.id is the connector's compartment.

If you'd rather not use any-user, the documented alternative is a dynamic group with the rule All {resource.type = 'serviceconnector', resource.compartment.id = '<connector_compartment_ocid>'} and normal Allow dynamic-group ... statements.

New policies can take a few minutes to reach regions other than the home region. The connector won't move data until they do.

#4. Fix task and target errors

Function task or target (502, "Expected JSON response from function"). Check the function's own logs and invocation metrics first:

oci fn function get --function-id ocid1.fnfunc.oc1..xxxx \
  --query 'data.{state:"lifecycle-state",memory:"memory-in-mbs",timeout:"timeout-in-seconds"}'

Common causes are the function hitting its timeout on large batches, an image that no longer exists in OCIR, or a function that sets Content-Type: application/json but returns something else. Fix the function, or reduce batch size in the task settings so each invocation does less work.

Stream target with a private endpoint (-1). The connector can't reach the stream. Check that the VCN doesn't use a custom DNS resolver, the private endpoint subnet has free IP addresses, and the NSG or security list allows the connector traffic described in the Streaming documentation.

Object Storage target. Confirm the bucket still exists and is in the same name and namespace the connector expects:

oci os bucket get --bucket-name audit-archive --query 'data.{name:name,compartment:"compartment-id"}'

A retention rule doesn't stop new objects being written, but a deleted or recreated bucket in another compartment breaks the policy condition.

#5. Reactivate and prove it works

Updating a connector's source, task or target resets it internally. Oracle recommends creating a new connector for big changes. For a policy-only fix, reactivate the existing one:

oci sch service-connector activate \
  --service-connector-id ocid1.serviceconnector.oc1..xxxx

oci sch service-connector get --service-connector-id ocid1.serviceconnector.oc1..xxxx \
  --query 'data."lifecycle-state"'      # expect ACTIVE

Then enable the connector's service log in the console (Connector → Logs → Enable log) and search it. A healthy run logs a line like:

Service connector run succeeded - <n> messages (<n> bytes) written to target

A failed run logs Connector run failed due to <type> error, Error Code : <code>. Finally, check the target itself: a new object in the bucket, messages on the stream, or invocations on the function.

Caution

Don't delete and recreate a connector to "reset" it unless you've confirmed the source data window. A new connector starts from current data. Anything not delivered by the old one is not replayed.

#Prevent it next time

Create alarms in the oci_service_connector_hub namespace for every connector that matters. The documented queries are a good start:

DataFreshness[1h].mean() > 43200000
ErrorsAtSource[15m]{errorCode = "404"}.groupby(connectorId).sum() > 0
BytesReadFromSource[15m].groupby(connectorId).sum() == 0

Use a pending duration of 30 minutes or more so a single failed run doesn't page anyone.

• Keep connector policies in Terraform next to the connector, in the same module, so a policy cleanup can't remove them on its own.
• Put a comment or freeform tag on target buckets and streams saying a connector writes to them.
• Subscribe the on-call distribution list to Connector Hub announcements, so the 4-day deactivation warning reaches someone.
• After any rename of a bucket, stream, log group or compartment, run step 2 for the connectors that touch it.