Azure Monitor managed Prometheus for AKS - PromQL and cardinality cheat sheet for operators

#Who this is for

You already have the metrics add-on running and dashboards in Grafana. Now the bill or the workspace limits are climbing, queries time out on large clusters, and a recording rule you need is missing. These are the commands and queries we keep at hand.

#Wiring and access

# Is the add-on enabled, and which workspace does it write to?
az aks show -g rg-aks -n aks-prod --query azureMonitorProfile -o json

# Enable it against an existing Azure Monitor workspace
az aks update -g rg-aks -n aks-prod --enable-azure-monitor-metrics \
  --azure-monitor-workspace-resource-id "$AMW_ID"

# Workspace query endpoint
AMW_QUERY=$(az monitor account show -g rg-monitor -n amw-prod \
  --query metrics.prometheusQueryEndpoint -o tsv)

# Collector pods
kubectl -n kube-system get pods -l rsName=ama-metrics -o wide
kubectl -n kube-system get pods | grep ama-metrics

Querying from the CLI needs the Monitoring Data Reader role on the workspace and a token for the Prometheus audience:

TOKEN=$(az account get-access-token --resource https://prometheus.monitor.azure.com \
  --query accessToken -o tsv)

promq() {
  curl -s -G "$AMW_QUERY/api/v1/query" \
    -H "Authorization: Bearer $TOKEN" --data-urlencode "query=$1" | jq '.data.result'
}

promq 'up{job="kube-state-metrics"}'

The endpoint speaks the standard Prometheus HTTP API, so /api/v1/query_range, /api/v1/series and /api/v1/labels work the same way.

#Cardinality: find what is inflating active series

Active time series drive both cost and the workspace limits. Check the workspace's ingestion and active-series utilization metrics in the portal, or list them with:

az monitor metrics list-definitions --resource "$AMW_ID" -o table

Then find the source with PromQL. Run these against a short window; on a large workspace they are expensive.

Usual offenders:

#Controlling what gets scraped

The add-on is configured through ConfigMaps in kube-system. The settings ConfigMap (ama-metrics-settings-configmap) turns default targets on or off and holds per-target keep-lists; custom scrape jobs go in ama-metrics-prometheus-config. Neither exists until you create it, and the settings schema has changed between versions (key names such as default-scrape-settings-enabled versus default-targets-scrape-enabled), so start from the template in the current Microsoft documentation for your add-on version.

kubectl -n kube-system get configmap | grep ama-metrics
kubectl -n kube-system get configmap ama-metrics-settings-configmap -o yaml

By default the add-on uses a minimal ingestion profile that keeps a curated set of metrics per default target. Turning that profile off can multiply ingestion for that target; add the specific metrics you need to the keep-list instead.

In custom scrape jobs, drop at ingestion with standard relabeling:

scrape_configs:
  - job_name: app-metrics
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
    metric_relabel_configs:
      - source_labels: [__name__]
        action: drop
        regex: "http_request_duration_seconds_bucket|go_gc_.*"
      - action: labeldrop
        regex: "request_id|user_id"

After applying, the ama-metrics pods restart and pick up the change within a few minutes. Validate with the collector logs:

kubectl -n kube-system logs -l rsName=ama-metrics -c prometheus-collector --tail=100 | grep -iE "error|invalid|config"

Container and label names in the add-on can vary by version; check kubectl describe pod if the selector returns nothing.

Warning

A typo in metric_relabel_configs with action: keep keeps only what matches, which can silently drop every metric for that job. Compare count by (job) before and after each change.

#PromQL operators actually use

#Saturation and throttling

#Health

Metric availability depends on the keep-lists above: with the minimal profile some cAdvisor and kube-state-metrics series are not ingested. If a query returns nothing, check the keep-list before assuming the workload is healthy.

#Keep queries fast at scale

• Aggregate early. sum by (namespace)(rate(x[5m])) scans the same data as rate(x[5m]) but returns far fewer series to Grafana.
• Use recording rules for anything on a shared dashboard or an alert. Managed Prometheus rule groups are Azure resources:

az alerts-management prometheus-rule-group create -g rg-monitor -n aks-prod-recording \
  -l westeurope --scopes "$AMW_ID" --cluster-name aks-prod --interval PT1M \
  --rules '[{"record":"namespace:container_cpu_usage:rate5m","expression":"sum by (namespace)(rate(container_cpu_usage_seconds_total{container!=\"\"}[5m]))","enabled":true}]'

That command lives in a CLI extension (az extension add --name alertsmanagement); check az alerts-management prometheus-rule-group create --help for the flags in your version, or deploy rule groups with Bicep (Microsoft.AlertsManagement/prometheusRuleGroups).

• Match rate() windows to the scrape interval. A window at least four times the scrape interval avoids gaps; with 30-second scrapes, [2m] is the practical minimum.
• Limit time ranges on dashboards that use high-cardinality metrics. Query limits on the workspace apply per query; check the current Azure Monitor workspace limits page rather than guessing.

#Prevent it next time

• Add a weekly "top 20 metrics by series" check and compare against last week.
• Require new scrape jobs to come with a metric_relabel_configs block and an expected series count.
• Alert on the workspace's active-series utilization before it reaches its limit.
• Put recording rules and scrape config in the same repo as the cluster IaC so changes are reviewed together.