Goal
You need a metric now, either to answer "is it this instance?" during an incident or to write an alarm. Paste a query into Metrics Explorer or run it from the CLI. Metric and dimension names here come from Oracle's metric references. Names can change, so confirm anything unexpected with metric list (below).
MQL in one line
MetricName[interval]{dimension = "value"}.groupBy(dimension).statistic() > threshold
| Part | Example | Notes |
|---|---|---|
| Interval | [5m] | 1m-60m, 1h-24h, 1d |
| Filter | {resourceDisplayName = "web-01"} | Separate multiple filters with commas |
| Fuzzy filter | {resourceDisplayName =~ "web-*"} | * is a wildcard, and a pipe between values means OR |
| Grouping | .groupBy(resourceId) | One line per value |
| Aggregate | .grouping() | One line for everything |
| Statistic | .mean() | Always specify one |
| Predicate | > 85 | Used for alarms and filtering results |
Statistics
| Statistic | Use it for |
|---|---|
mean() / avg() | Normal level, such as CPU |
max() / min() | Worst point, up/down state |
sum() | Counts, such as errors per interval |
count() | Number of data points received |
rate() | Per-second rate of change for cumulative counters |
increment() | Change over the interval |
percentile(0.95) | Latency. p is between 0.0 and 1.0, exclusive |
first() / last() | First or last value in the interval |
absent() | Metric stopped reporting |
Operators
Comparison operators are >, >=, ==, !=, <, <=, in and not in. Both in and not in are inclusive: CpuUtilization[1m].mean() in (60, 80).
Arithmetic with a number works:
100 - CpuUtilization[1m].mean()
TotalRequestLatency[1m].mean() / 1000
Join whole queries with && or ||, never between dimension sets:
CpuUtilization[1m]{faultDomain = "FAULT-DOMAIN-1"}.mean() || MemoryUtilization[1m]{faultDomain = "FAULT-DOMAIN-1"}.mean()
Run queries from the CLI
export C=<compartment-ocid>
oci monitoring metric-data summarize-metrics-data -c "$C" \
--namespace oci_computeagent \
--query-text 'CpuUtilization[5m].mean()' \
--start-time "$(date -u -d '-1 hour' +%FT%TZ)" \
--query 'reverse(sort_by(data[].{name:dimensions.resourceDisplayName,last:"aggregated-datapoints"[-1].value}, &last))' \
--output table
This lists instances by their latest CPU, highest first. MQL has no top-N function, so sorting happens in --query.
| Flag | Default | Notes |
|---|---|---|
--start-time | 3 hours ago | RFC 3339, date only or epoch seconds |
--end-time | Now | The end point itself is excluded |
--resolution | 1m on the CLI | Must be equal to or less than the query interval |
--compartment-id-in-subtree | Off | Only valid with the tenancy OCID, and needs tenancy-level permissions |
Find metric names and dimensions
# Which metrics does a namespace publish in this compartment?
oci monitoring metric list -c "$C" --namespace oci_lbaas \
--group-by '["name"]' --all --query 'data[].name'
# Which resources report this metric, and with what dimensions?
oci monitoring metric list -c "$C" --namespace oci_computeagent \
--name CpuUtilization --all --query 'data[].dimensions'
# What data exists for one instance?
oci monitoring metric list -c "$C" --namespace oci_computeagent \
--dimension-filters '{"resourceId": "<instance-ocid>"}' --all --query 'data[].name'
--dimension-filters is ignored when you use --group-by.
Compute (oci_computeagent)
Dimensions: availabilityDomain, faultDomain, imageId, instancePoolId, region, resourceDisplayName, resourceId, shape.
# CPU and memory per instance
CpuUtilization[5m].groupBy(resourceDisplayName).mean()
MemoryUtilization[5m].groupBy(resourceDisplayName).max()
# Whole instance pool, 95th percentile
CpuUtilization[5m]{instancePoolId = "<pool-ocid>"}.grouping().percentile(0.95)
# Load and memory pressure
LoadAverage[1m]{resourceDisplayName = "web-01"}.mean()
MemoryAllocationStalls[5m]{resourceDisplayName = "web-01"}.sum()
# Disk and network: cumulative counters, so use rate()
DiskBytesRead[1m]{resourceDisplayName = "web-01"}.rate()
DiskBytesWritten[1m]{resourceDisplayName = "web-01"}.rate()
DiskIopsRead[1m]{resourceDisplayName = "web-01"}.rate()
DiskIopsWritten[1m]{resourceDisplayName = "web-01"}.rate()
NetworksBytesIn[1m]{resourceDisplayName = "web-01"}.rate()
NetworksBytesOut[1m]{resourceDisplayName = "web-01"}.rate()
# Instance stopped reporting
CpuUtilization[1m].groupBy(resourceId).absent()
The network metrics are spelled NetworksBytesIn and NetworksBytesOut, with an "s". Those counters reset when the OS restarts.
Load Balancer (oci_lbaas)
The lbComponent dimension takes backendSet, listener or loadBalancer. Other dimensions include lbName, backendSetName, listenerName and resourceId.
# Backends down, per backend set
unhealthyBackendServers[1m]{lbComponent = "backendSet", lbName = "prod-lb"}.groupBy(backendSetName).max() > 0
# Errors from backends
httpResponses5xx[1m]{lbComponent = "backendSet", lbName = "prod-lb"}.groupBy(backendSetName).sum()
httpResponses502[1m]{lbComponent = "backendSet", lbName = "prod-lb"}.sum()
httpResponses504[1m]{lbComponent = "backendSet", lbName = "prod-lb"}.sum()
backendTimeouts[1m]{lbComponent = "backendSet", lbName = "prod-lb"}.sum()
# Latency: HTTP backends use the header metric, TCP uses first byte
responseTimeHttpHeader[5m]{lbComponent = "backendSet", lbName = "prod-lb"}.percentile(0.95)
responseTimeFirstByte[5m]{lbComponent = "backendSet", lbName = "prod-lb"}.percentile(0.95)
# Capacity and TLS
peakBandwidth[1m]{lbComponent = "loadBalancer", lbName = "prod-lb"}.max()
activeConnections[1m]{lbComponent = "loadBalancer", lbName = "prod-lb"}.max()
FailedSslHandshake[5m]{lbComponent = "loadBalancer", lbName = "prod-lb"}.sum()
Oracle's reference shows listener response metrics with inconsistent spacing, so list the real names with metric list before you use them. Network Load Balancers publish under a separate namespace.
Block Volume (oci_blockstore)
Dimensions: resourceId (volume OCID) and attachmentId.
VolumeReadThroughput[1m]{resourceId = "<volume-ocid>"}.sum()
VolumeWriteThroughput[1m]{resourceId = "<volume-ocid>"}.sum()
VolumeReadOps[1m]{resourceId = "<volume-ocid>"}.sum()
VolumeWriteOps[1m]{resourceId = "<volume-ocid>"}.sum()
# Volume is hitting its performance limit
VolumeThrottledIOs[1m]{resourceId = "<volume-ocid>"}.sum() > 0
# Current guaranteed performance
VolumeGuaranteedIOPS[5m]{resourceId = "<volume-ocid>"}.max()
VolumeGuaranteedThroughput[5m]{resourceId = "<volume-ocid>"}.max()
# Cross-region replication freshness, in seconds. Set the threshold from your RPO
VolumeReplicationSecondsSinceLastUpload[5m]{resourceId = "<volume-ocid>"}.max()
VolumeReplicationSecondsSinceLastSync[5m]{resourceId = "<replica-ocid>"}.max()
Object Storage (oci_objectstorage)
ObjectCount, StoredBytes and UncommittedParts are emitted hourly, so use a long interval and a time range of at least several hours. Buckets with no objects emit no data.
StoredBytes[1h]{resourceDisplayName = "app-logs"}.max()
ObjectCount[1h]{resourceDisplayName = "app-logs"}.max()
# Space held by incomplete multipart uploads
UncommittedParts[1h].groupBy(resourceDisplayName).max()
# Requests, client errors and latency in milliseconds
AllRequests[5m]{resourceDisplayName = "app-logs"}.sum()
ClientErrors[5m]{resourceDisplayName = "app-logs"}.sum()
TotalRequestLatency[5m]{resourceDisplayName = "app-logs"}.percentile(0.95)
FastConnect (oci_fastconnect)
The component dimension takes crossconnectgroup, crossconnect or virtualcircuit. Partner connections report only virtualcircuit. State metrics are 1 for up and 0 for down.
ConnectionState[1m]{component = "virtualcircuit"}.groupBy(resourceName).min() < 1
Ipv4BgpSessionState[1m]{component = "virtualcircuit"}.groupBy(resourceName).min() < 1
BitRateReceived[5m]{resourceName = "dc1-vc1"}.max()
BitRateSent[5m]{resourceName = "dc1-vc1"}.max()
PacketsError[5m]{resourceName = "dc1-vc1"}.sum()
PacketsDiscarded[5m]{resourceName = "dc1-vc1"}.sum()
Functions (oci_faas)
FunctionInvocationCount[5m].groupBy(resourceDisplayName).sum()
FunctionResponseCount[5m]{responseType = "Error"}.groupBy(ErrorCode).sum()
FunctionResponseCount[5m]{responseType = "Throttled"}.grouping().sum()
FunctionExecutionDuration[5m].groupBy(resourceDisplayName).percentile(0.95)
Turn a query into an alarm
Alarm queries need a predicate, and their only valid resolution is 1m. Use groupBy when you want to see which resource tripped.
oci monitoring alarm create -c "$C" \
--display-name "prod-lb backends unhealthy" \
--metric-compartment-id "$C" \
--namespace oci_lbaas \
--query-text 'unhealthyBackendServers[1m]{lbComponent = "backendSet", lbName = "prod-lb"}.groupBy(backendSetName).max() > 0' \
--severity CRITICAL \
--pending-duration PT5M \
--destinations '["<topic-ocid>"]' \
--is-enabled true
Warning
An alarm whose query matches no data has nothing to evaluate, so it may never fire. Run the query first and confirm it returns data. Pair critical alarms with an
absent()alarm where missing data means failure.
Query returns nothing
| Symptom | Likely cause | Fix |
|---|---|---|
| No data for any instance | Wrong compartment | Use the compartment of the resource, or the tenancy OCID with --compartment-id-in-subtree true |
| Memory or disk missing on one instance | Oracle Cloud Agent or its monitoring plugin not running | Check the plugin on the instance |
| Empty load balancer query | Wrong lbComponent for that metric | List the metric's dimensions with metric list |
| Object Storage metric empty | Bucket has no objects, or the time range is under an hour | Use a longer range |
| Disk or network values only climb | Reading a cumulative counter as a level | Use .rate() |
absent() fires late | Default absence period is two hours | Check the docs and test a shorter period before relying on it |
Maximum time range by resolution for metric queries:
| Resolution | Maximum range |
|---|---|
1m | 7 days |
5m | 30 days |
1h | 90 days |
1d | 90 days |
The maximum counts back from the current time, not from your end time.
Prevent it next time
- Save your most-used queries in a shared dashboard.
- Test every new alarm's query in Metrics Explorer and trigger the alarm once.