Azure Machine Learning online endpoints CLI cheat sheet: deployments, traffic, scaling, logs, metrics and errors

#Goal

You need to check, call, scale or debug a managed online endpoint without opening the studio. An endpoint is the stable URL and auth. A deployment is the model, code and compute behind it. One endpoint can hold several deployments.

Commands below assume these defaults:

az extension add --name ml --upgrade
az account set --subscription <subscription-id>
az configure --defaults group=<resource-group> workspace=<workspace-name>

E=fraud-scoring
D=blue

#Check status

az ml online-endpoint list -o table

az ml online-endpoint show -n $E \
  --query '{state:provisioning_state, traffic:traffic, mirror:mirror_traffic, auth:auth_mode, uri:scoring_uri}'

az ml online-deployment list -e $E -o table

az ml online-deployment show -e $E -n $D \
  --query '{state:provisioning_state, sku:instance_type, count:instance_count, model:model}'

#Call the endpoint

az ml online-endpoint invoke -n $E --request-file sample.json
az ml online-endpoint invoke -n $E -d green --request-file sample.json

-d targets a single deployment and skips the traffic split. From curl or a client, send the azureml-model-deployment: green header. Get the key with az ml online-endpoint get-credentials -n $E.

#Create, update, delete

az ml online-endpoint create -f endpoint.yml
az ml online-deployment create -f deployment.yml -e $E -n green          # 0% traffic
az ml online-deployment create -f deployment.yml -e $E -n $D --all-traffic
az ml online-deployment update -f deployment.yml -e $E -n $D
az ml online-deployment delete -e $E -n green --yes

create fails if the deployment already exists, so use update. --all-traffic doesn't work with --no-wait. A deployment that still has live traffic can't be deleted, so set its traffic to 0 first.

Caution

az ml online-endpoint delete removes the endpoint, its scoring URL and all its deployments. Clients break straight away, and recreating them takes as long as a new deployment. Check the name and traffic first.

#Traffic

az ml online-endpoint update -n $E --traffic "blue=90 green=10"
az ml online-endpoint update -n $E --traffic "blue=100 green=0"     # roll back
az ml online-endpoint update -n $E --mirror-traffic "green=10"
az ml online-endpoint update -n $E --mirror-traffic "green=0"       # stop mirroring
az ml online-endpoint show -n $E --query '{traffic:traffic, mirror:mirror_traffic}'

The total must be 100 to serve traffic, or 0 to disable it. For the full rollout procedure, see the blue-green runbook for Azure ML online endpoints.

#Scale and tune

az ml online-deployment update -e $E -n $D --set instance_count=4

Probes and request settings go in the deployment YAML:

request_settings:
  request_timeout_ms: 30000
  max_concurrent_requests_per_instance: 2
liveness_probe:
  initial_delay: 300
  period: 10
  timeout: 5
  failure_threshold: 30

Allowed ranges vary, so check the current deployment YAML schema. Slow model loads usually need a longer initial_delay.

Check quota before scaling or choosing a SKU:

az ml compute list-sizes -o table
az ml compute list-usage -o table

Warning

Only one operation runs at a time on an endpoint or deployment. Don't send a traffic change while a scale operation is still running, or you may see OperationCanceled or a conflict. Check provisioning_state first, and use --no-wait only if your script polls it.

#Logs

# Inference server: print and logging output from score.py
az ml online-deployment get-logs -e $E -n $D --lines 200

# Storage initializer: model and code download
az ml online-deployment get-logs -e $E -n $D --container storage-initializer --lines 200

If the script logs through Python logging, set the level to INFO or your messages won't be published. If get-logs returns nothing, the container probably didn't start. Reproduce it locally with Docker:

az ml online-deployment create -f deployment.yml -e $E -n $D --local
az ml online-deployment get-logs -e $E -n $D --local
az ml online-endpoint invoke -n $E --local --request-file sample.json

#Send logs to Log Analytics

get-logs only shows recent console output, so turn this on before you need history:

EID=$(az ml online-endpoint show -n $E --query id -o tsv)

az monitor diagnostic-settings create --name aml-endpoint-logs --resource "$EID" \
  --workspace <log-analytics-resource-id> \
  --logs '[{"category":"AmlOnlineEndpointTrafficLog","enabled":true},
           {"category":"AmlOnlineEndpointConsoleLog","enabled":true},
           {"category":"AmlOnlineEndpointEventLog","enabled":true}]'

The first logs can take up to an hour to appear.

// Errors and latency by deployment
AmlOnlineEndpointTrafficLog
| where TimeGenerated > ago(1h)
| summarize requests = count(),
            errors = countif(toint(ResponseCode) >= 400),
            p95_ms = percentile(TotalDurationMs, 95)
          by DeploymentName, ResponseCode, ModelStatusCode
| order by errors desc

// Exceptions from the scoring script
AmlOnlineEndpointConsoleLog
| where TimeGenerated > ago(1h)
| where Message has_any ("Traceback", "Exception", "ERROR")
| project TimeGenerated, DeploymentName, InstanceId, ContainerName, Message
| order by TimeGenerated desc

AmlOnlineEndpointEventLog holds container lifecycle events, in the Name and Message columns. Table name casing differs across docs and some schemas are in preview. If a query fails, check the table list in your workspace.

#Metrics

EID=$(az ml online-endpoint show -n $E --query id -o tsv)
DID=$(az ml online-deployment show -e $E -n $D --query id -o tsv)

# Requests split by status code class
az monitor metrics list --resource "$EID" --metrics RequestsPerMinute \
  --filter "statusCodeClass eq '*'" --aggregation Average --interval 5m --offset 1h -o table

# P95 latency
az monitor metrics list --resource "$EID" --metrics RequestLatency_P95 \
  --aggregation Average --interval 5m --offset 1h -o table

# Instance health and pressure
az monitor metrics list --resource "$DID" \
  --metrics CpuUtilizationPercentage CpuMemoryUtilizationPercentage DiskUtilization \
  --aggregation Average --interval 5m --offset 1h -o table

#Deployment errors

#Scoring errors

#Rotate keys

az ml online-endpoint regenerate-keys -n $E --key-type secondary
az ml online-endpoint get-credentials -n $E

Warning

Regenerating a key invalidates it straight away. Move clients to the other key first, then regenerate the one no longer in use.

#Prevent it next time

• Keep endpoint and deployment YAML in source control, and don't leave --set changes only on the live resource.
• Set up diagnostic settings as part of creating each endpoint.
• Alert on RequestsPerMinute split by statusCodeClass, RequestLatency_P95 and DeploymentCapacity.
• Keep spare quota for your largest deployment so a scale-out or rollback doesn't hit OutOfQuota.