Publishing custom metrics to OCI Monitoring at scale - ingestion endpoint, batching, dimension cardinality and partial failures

Getting a custom metric into OCI Monitoring takes one call. Running a publisher across hundreds of hosts or pods, without dropped data, cardinality blow-ups or alarms that never fire, takes more design. This post covers the ingestion contract, a publisher design that holds up at scale, and the queries and alarms on top.

#The ingestion contract

PostMetricData behaves differently from the rest of the Monitoring API:

Name lengths, datapoints per stream per call and request size are also limited. Check the current Monitoring limits page before you size batches, and treat these numbers as documented limits to design around, not guesses to push against.

#IAM: scope publishers to their namespace

Publishers need use metrics, and a condition can restrict them to one namespace:

Allow dynamic-group orders-hosts to use metrics in compartment orders-prod
  where target.metrics.namespace = 'orders_app'

With this condition, a bug or a compromised host can't write into another team's namespace. Readers (dashboards, alarm authors) need read metrics.

#Test the payload with the CLI first

metrics.json:

[
  {
    "namespace": "orders_app",
    "compartmentId": "ocid1.compartment.oc1..aaaa...",
    "resourceGroup": "checkout",
    "name": "queue_depth",
    "dimensions": { "env": "prod", "queue": "payments", "fd": "FAULT-DOMAIN-2" },
    "metadata": { "unit": "messages" },
    "datapoints": [
      { "timestamp": "2026-11-13T10:15:00Z", "value": 142 }
    ]
  }
]

oci monitoring metric-data post \
  --metric-data file://metrics.json \
  --batch-atomicity NON_ATOMIC \
  --endpoint https://telemetry-ingestion.eu-frankfurt-1.oraclecloud.com

The response has failed-metrics-count and a failed-metrics list with a message per stream. A 200 response with failures in the body is normal under NON_ATOMIC. A publisher that only checks the HTTP status loses data without noticing.

#Cardinality: the design decision that matters

Each unique combination of metric name and dimension values is a separate stream. Streams drive ingestion cost, query speed and how many streams an alarm evaluates.

Tip

Before you add a dimension, estimate the streams: (values of dim A) x (values of dim B) x ... x metric names. If the number grows with traffic or deployments instead of with infrastructure, it doesn't belong in a dimension.

#Publisher design

Don't post one call per measurement. Aggregate locally, flush on an interval, and chunk to the per-call stream limit.

1. Aggregate per minute in process. Monitoring queries work at one-minute or coarser intervals, so sub-minute posts add cost without adding detail for most use cases. Send one datapoint per stream per minute, as a sum, max or last value depending on the metric.
2. Flush every 60 seconds, chunked to at most 50 streams per request.
3. Use NON_ATOMIC and re-queue only the failed streams, after fixing them or dropping them if they're invalid.
4. Retry 429 and 5xx with exponential backoff and jitter. Drop datapoints older than about 110 minutes, because the service rejects anything older than two hours.
5. Cap the buffer. When ingestion is down, keep the most recent data and drop the oldest. A publisher that grows without limit takes the application down with it.

A Python publisher running on an instance with instance principals:

import datetime as dt
import oci

REGION = "eu-frankfurt-1"
signer = oci.auth.signers.InstancePrincipalsSecurityTokenSigner()
client = oci.monitoring.MonitoringClient(
    config={},
    signer=signer,
    service_endpoint=f"https://telemetry-ingestion.{REGION}.oraclecloud.com",
    retry_strategy=oci.retry.DEFAULT_RETRY_STRATEGY,
)

def flush(aggregates, compartment_id):
    """aggregates: {(name, frozenset(dims.items())): value} for the last minute"""
    ts = dt.datetime.now(dt.timezone.utc).replace(second=0, microsecond=0)
    items = [
        oci.monitoring.models.MetricDataDetails(
            namespace="orders_app",
            compartment_id=compartment_id,
            name=name,
            dimensions=dict(dims),
            datapoints=[oci.monitoring.models.Datapoint(timestamp=ts, value=float(v))],
        )
        for (name, dims), v in aggregates.items()
    ]
    failed = []
    for i in range(0, len(items), 50):
        resp = client.post_metric_data(
            oci.monitoring.models.PostMetricDataDetails(
                metric_data=items[i:i + 50], batch_atomicity="NON_ATOMIC"
            )
        )
        if resp.data.failed_metrics_count:
            failed.extend(resp.data.failed_metrics)
    return failed  # log these with their messages; don't silently retry invalid ones

Stagger flush times across hosts by adding a random offset at startup. A thousand publishers flushing at second :00 make bursts that are more likely to be throttled.

#Kubernetes and Prometheus-style workloads

For OKE workloads that already expose Prometheus metrics, run one collector per cluster that scrapes, aggregates and posts, rather than a publisher in every pod. That removes pod-name cardinality at the source and keeps instance-principal or workload-identity permissions on one component. The OCI Management Agent also has a Prometheus plugin that can forward scraped metrics to Monitoring. Check whether it fits before you write your own collector.

#Query what you published

Queries use the normal telemetry. endpoint, which is the CLI default:

oci monitoring metric-data summarize-metrics-data \
  --compartment-id ocid1.compartment.oc1..aaaa... \
  --namespace orders_app \
  --resource-group checkout \
  --query-text 'queue_depth[1m]{env = "prod"}.groupBy(queue).max()' \
  --start-time 2026-11-13T09:00:00Z --end-time 2026-11-13T11:00:00Z

# What streams exist? Use this to spot cardinality creep.
oci monitoring metric list \
  --compartment-id ocid1.compartment.oc1..aaaa... \
  --namespace orders_app --all \
  --query 'length(data)'

Run the metric list count weekly. A count that jumps after a release means somebody added an unbounded dimension.

#Alarm on missing data, not just on values

A publisher that dies stops sending data, and a threshold alarm on that metric never fires. Pair each important metric with an absence alarm:

queue_depth[5m]{env = "prod"}.groupBy(queue).absent()

Alarms on custom metrics evaluate only after data arrives. Data that lands late, close to the two-hour window, never triggers alarms for the minutes it covers. Keep publisher lag low and alarm on publisher health, for example on a heartbeat metric that each publisher posts every minute.

Warning

Don't build paging alarms on a metric whose publisher retries for up to two hours. When the backlog drains, the alarm evaluates old datapoints out of order and can page well after the problem has cleared. Drop stale data or publish a separate backlog metric.

#Prevent it next time

• Keep one published schema per namespace: metric names, units, allowed dimensions and their expected value counts. Review changes like an API change.
• Restrict each publisher to its namespace with target.metrics.namespace in the policy.
• Monitor the publisher itself: failed-stream count, buffer size, flush latency and last successful post.
• Track stream counts per namespace and alarm when they grow faster than the fleet.
• Load-test the publisher against a scratch namespace before a fleet-wide rollout.