Runbook: blue-green model rollout on Azure Machine Learning online endpoints with mirrored traffic and instant rollback

#Goal

You have a model serving production traffic from a managed online endpoint. A new version (new model, new scoring script, new environment or new instance type) has to go live with no URL change, no downtime and a rollback path that takes seconds.

Managed online endpoints support this directly. One endpoint can hold several deployments, and the endpoint's traffic setting decides how live requests are split between them. In this runbook:

• blue is the current deployment and takes 100% of traffic.
• green is the new deployment.

Clients keep calling the same scoring URI the whole time.

#Prerequisites and checks

az extension add --name ml --upgrade
az version --query '"extensions"'

az account set --subscription <subscription-id>
az configure --defaults workspace=<workspace-name> group=<resource-group>

ENDPOINT_NAME=fraud-scoring
az ml online-endpoint show --name $ENDPOINT_NAME \
  --query '{state:provisioning_state, traffic:traffic, mirror:mirror_traffic, uri:scoring_uri}'
az ml online-deployment list --endpoint-name $ENDPOINT_NAME -o table

Before you start, confirm:

Warning

Both deployments run at full instance count during the rollout, so compute cost roughly doubles until you delete blue. Check quota for the instance type in the workspace region before creating green. For GPU SKUs this is the most common blocker.

#Step 1: Create green with no live traffic

Create the deployment without --all-traffic. It gets 0% of live traffic:

az ml online-deployment create \
  --name green \
  --endpoint-name $ENDPOINT_NAME \
  --file green-deployment.yml

A minimal green-deployment.yml:

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: green
endpoint_name: fraud-scoring
model: azureml:fraud-model:7
code_configuration:
  code: ./src
  scoring_script: score.py
environment: azureml:fraud-env:12
instance_type: Standard_DS3_v2
instance_count: 2

If creation fails, read the deployment logs before retrying:

az ml online-deployment get-logs --name green --endpoint-name $ENDPOINT_NAME --lines 200
az ml online-deployment get-logs --name green --endpoint-name $ENDPOINT_NAME \
  --container storage-initializer

The storage-initializer container logs show model download failures. The default inference server logs show errors from init() in the scoring script.

#Step 2: Test green in isolation

Call green directly. A request to a named deployment bypasses the traffic split, so no customer sees this:

az ml online-endpoint invoke \
  --name $ENDPOINT_NAME \
  --deployment-name green \
  --request-file tests/sample-request.json

From an application or load-test tool, use the scoring URI with the azureml-model-deployment header:

KEY=$(az ml online-endpoint get-credentials --name $ENDPOINT_NAME --query primaryKey -o tsv)
URI=$(az ml online-endpoint show --name $ENDPOINT_NAME --query scoring_uri -o tsv)

curl -s -X POST "$URI" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -H "azureml-model-deployment: green" \
  --data @tests/sample-request.json

get-credentials returns keys for key-auth endpoints. For token-auth endpoints it returns a token. Run your full regression set against green here, plus a short load test at expected peak request rate. Cold paths (first request after init()) and large payloads are where new versions usually break.

#Step 3: Mirror live traffic to green

Mirroring sends a copy of a percentage of live requests to green. Clients still get their answer from blue, and green's responses are discarded. You see how green behaves on real traffic without risk to users.

az ml online-endpoint update --name $ENDPOINT_NAME --mirror-traffic "green=10"
az ml online-endpoint show --name $ENDPOINT_NAME --query '{traffic:traffic, mirror:mirror_traffic}'

Rules to know (check the current Azure ML docs for limits, as they can change):

• Only one deployment can receive mirrored traffic, and it can't receive live traffic at the same time.
• Mirrored traffic is capped (50% at the time of writing) because it counts against the endpoint's bandwidth quota.
• Requests that name a deployment directly are not mirrored.
• Mirroring isn't supported on Kubernetes online endpoints.
• Older CLI versions drop the mirror setting when you update the endpoint, so use CLI v2 2.4.0 or later for every update during the rollout.

Caution

Mirrored requests are real requests. If your scoring script writes to a database, sends events or calls downstream APIs, green will do that too, for every mirrored request. Disable side effects in green (for example with an environment variable checked in score.py) before you turn mirroring on.

#What to compare during mirroring

If the workspace sends endpoint logs to Log Analytics (diagnostic setting on the endpoint), compare the two deployments over the same window. Column names vary by schema version, so run AmlOnlineEndpointTrafficLog | take 10 first and adjust:

AmlOnlineEndpointTrafficLog
| where TimeGenerated > ago(1h)
| summarize requests = count(),
            errors = countif(toint(ResponseCode) >= 500),
            p95_ms = percentile(TotalDurationMs, 95)
          by DeploymentName, bin(TimeGenerated, 5m)
| order by TimeGenerated asc

Also watch green's console logs for scoring exceptions:

az ml online-deployment get-logs --name green --endpoint-name $ENDPOINT_NAME --lines 300 | grep -i -E "error|exception|traceback"

When green has handled enough mirrored traffic with an error rate and latency at least as good as blue, stop mirroring:

az ml online-endpoint update --name $ENDPOINT_NAME --mirror-traffic "green=0"

#Step 4: Shift live traffic in steps

Live traffic percentages must add up to 100. Move in steps and hold at each one long enough to see real errors (15–60 minutes is typical, longer for low-volume endpoints):

az ml online-endpoint update --name $ENDPOINT_NAME --traffic "blue=90 green=10"
# hold, compare metrics
az ml online-endpoint update --name $ENDPOINT_NAME --traffic "blue=50 green=50"
# hold, compare metrics
az ml online-endpoint update --name $ENDPOINT_NAME --traffic "blue=0 green=100"

Before going to 100%, scale green to the instance count blue runs at peak, or confirm its autoscale rule matches:

az ml online-deployment update --name green --endpoint-name $ENDPOINT_NAME --set instance_count=4

#Go/no-go per step

#Step 5: Roll back

Rollback is one traffic update. The URL doesn't change, and blue is still running, so recovery takes effect as soon as the update finishes:

az ml online-endpoint update --name $ENDPOINT_NAME --traffic "blue=100 green=0"
az ml online-endpoint show --name $ENDPOINT_NAME --query traffic

Warning

Don't scale blue down or delete it until green has run at 100% through at least one full business cycle (a day, or a peak period). Once blue is gone, rollback means redeploying it, which takes as long as any other deployment and needs free quota.

#Step 6: Clean up

After the observation period:

az ml online-deployment delete --name blue --endpoint-name $ENDPOINT_NAME --yes
az ml online-endpoint show --name $ENDPOINT_NAME --query '{traffic:traffic, mirror:mirror_traffic}'

For the next release, green is the live deployment and the new one is called blue. Alternate names, or use version-based names such as v7 and v8.

#Prevent it next time

• Put each step in a pipeline (GitHub Actions or Azure DevOps) with a manual approval between traffic steps, so nobody jumps from 0 to 100.
• Keep the regression request set and expected outputs in the repo next to score.py, and run it against the new deployment automatically in Step 2.
• Make side effects in score.py switchable with an environment variable, so mirroring is always safe.
• Send endpoint traffic and console logs to Log Analytics before the rollout, not during an incident.
• Reserve quota for two full deployments of your largest endpoint. Without it, you can't do a safe rollout when you need one.