Running Azure Spot VMs without surprises - eviction policy, max price, eviction rates and restore

#The goal

Spot VMs run on spare Azure capacity at a discount, and Azure can take them back at any time. Teams usually get hurt in one of three ways: VMs that were "deallocated" by eviction and never came back, a max price that caused evictions nobody expected, or a VM size picked without checking how often it is evicted in that region.

This post covers the settings that decide how a Spot VM behaves, how to check eviction rates and price history before you commit to a size, and how to test evictions so the workload is proven to survive them.

Warning

Spot VMs have no SLA and can be evicted with about 30 seconds' notice. Don't run anything that can't be interrupted: single-instance databases, stateful services without replication, or jobs with no checkpointing.

#The three settings that matter

#Eviction policy: Deallocate or Delete

• Deallocate stops the VM and keeps its disks. You keep paying for the disks, and the deallocated VM's cores still count against your Spot vCPU quota. When you start it again, it only runs if Spot capacity is available.
• Delete removes the VM and its disks on eviction. Nothing is left to pay for, and quota is released. This fits stateless scale set instances that are rebuilt from an image.

The common failure is a fleet of single Spot VMs with Deallocate, where nobody restarts them after an eviction. They sit deallocated for weeks, using quota and disk spend while doing nothing.

#Max price

With --max-price -1 the VM is evicted only when Azure needs the capacity back, and you pay the current Spot price, never more than the pay-as-you-go price. If you set a fixed price, the VM is also evicted when the Spot price goes above it. Unless you have a hard cost ceiling, use -1.

#Create a Spot VM and a Spot scale set

RG=rg-batch-prod
LOC=westeurope

# Single Spot VM, evicted only for capacity, disks kept on eviction
az vm create -g $RG -n vm-spot-worker01 \
  --image Ubuntu2204 --size Standard_D4s_v5 \
  --priority Spot --max-price -1 --eviction-policy Deallocate \
  --admin-username azureuser --generate-ssh-keys

# Stateless scale set: delete on eviction, try to restore capacity for up to 1 hour
az vmss create -g $RG -n vmss-spot-workers \
  --image Ubuntu2204 --vm-sku Standard_D4s_v5 \
  --orchestration-mode Flexible --instance-count 6 \
  --priority Spot --max-price -1 --eviction-policy Delete \
  --enable-spot-restore true --spot-restore-timeout PT1H \
  --admin-username azureuser --generate-ssh-keys

Spot restore makes the scale set keep trying to bring evicted instances back, up to the timeout (an ISO 8601 duration), so the instance count returns to target when capacity frees up.

For Flexible scale sets, Spot Priority Mix lets you keep a base number of regular VMs and a percentage split above that base, so a capacity eviction never takes the set to zero. The CLI flag names for priority mix have changed between versions; check az vmss create --help on your build before you script it.

#Changing max price or priority on an existing VM

You can't change the max price of a running Spot VM. Deallocate it first:

az vm deallocate -g $RG -n vm-spot-worker01
az vm update -g $RG -n vm-spot-worker01 --max-price -1
az vm start -g $RG -n vm-spot-worker01

Switching an existing VM between Regular and Spot priority isn't supported. Recreate it from its disks or an image.

#Check eviction rates and price history before you pick a size

Azure Resource Graph exposes eviction rates and Spot price history in the SpotResources table. Run these in Resource Graph Explorer or with az graph query.

// Eviction rate band for candidate sizes in one region
SpotResources
| where type =~ 'microsoft.compute/skuspotevictionrate/location'
| where location =~ 'westeurope'
| where sku.name in~ ('standard_d4s_v5', 'standard_d4as_v5', 'standard_e4s_v5')
| project skuName = tostring(sku.name), location, evictionRate = tostring(properties.evictionRate)
| order by skuName asc

// Recent Spot price points for one size and OS
SpotResources
| where type =~ 'microsoft.compute/skuspotpricehistory/ostype/location'
| where sku.name =~ 'standard_d4s_v5' and location =~ 'westeurope'
| where properties.osType =~ 'linux'
| project skuName = tostring(sku.name), location, spotPrices = properties.spotPrices

az graph query -q "SpotResources | where type =~ 'microsoft.compute/skuspotevictionrate/location' | where location =~ 'westeurope' | project sku.name, properties.evictionRate" --first 200

The eviction rate is a band (for example 0-5% or 5-10%) over a recent trailing window, not a forecast. The exact property layout can change; inspect properties on a raw row if a projection comes back empty.

Tip

Don't commit a whole workload to one size. Pick two or three sizes with similar shapes and low eviction bands, and spread across them. A capacity crunch on one size family rarely hits all of them at the same moment in the same region.

#Find out when and why a VM was evicted

Inside the VM, the only reliable signal is the Preempt event in Scheduled Events, issued about 30 seconds before eviction:

curl -s -H Metadata:true \
  "http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01" \
  | jq '.Events[] | select(.EventType=="Preempt")'

Use that window to stop taking work, flush state and deregister from a load balancer. See the Scheduled Events post for a full handler.

From outside, look for eviction entries in the Activity Log. They are often recorded with an operation name containing EvictSpotVM, but Microsoft notes that not every scheduled event appears in the Activity Log or Resource Health. Check what your own subscription records before you build an alert on it.

az monitor activity-log list -g $RG --offset 7d \
  --query "[?contains(operationName.value, 'EvictSpotVM')].{time:eventTimestamp, vm:resourceId, status:status.value}" -o table

List Spot VMs that are sitting deallocated:

Resources
| where type =~ 'microsoft.compute/virtualmachines'
| where properties.priority =~ 'Spot'
| extend power = tostring(properties.extended.instanceView.powerState.code)
| where power =~ 'PowerState/deallocated'
| project name, resourceGroup, location, size = tostring(properties.hardwareProfile.vmSize),
          evictionPolicy = tostring(properties.evictionPolicy)

#Test an eviction on purpose

Don't wait for Azure to test your handler. Trigger an eviction:

# Single VM
az vm simulate-eviction -g $RG -n vm-spot-worker01

# One scale set instance (Uniform: instance ID; check the instance list for Flexible)
az vmss list-instances -g $RG -n vmss-spot-workers -o table
az vmss simulate-eviction -g $RG -n vmss-spot-workers --instance-id <instance-id>

The simulated eviction sends the Preempt event and then evicts the VM using its configured policy. Check that:

1. The handler saw the event and drained work within the notice window.
2. Jobs resumed from the last checkpoint, not from the start.
3. Monitoring recorded the eviction and nobody got paged for an expected event.
4. Deallocated VMs were restarted, or deleted instances were replaced by restore or autoscale.

#Common problems

#Prevent it next time

• Use --max-price -1 unless you have a written reason for a ceiling.
• Use Delete for stateless scale set instances and Deallocate only where the disks must survive.
• Check eviction bands in SpotResources before picking sizes, and spread across two or three sizes.
• Run simulate-eviction in every release cycle and after handler changes.
• Run the deallocated Spot VM query weekly and alert on anything deallocated for more than a day.
• Check the current Spot quota and discount for your region rather than assuming last quarter's numbers.