Handling Azure platform maintenance inside the VM with Scheduled Events - drain, approve and recover

#The goal

Platform maintenance, host failures, Spot evictions and scale-in all interrupt VMs. Most of them are announced a few minutes ahead through Scheduled Events in the Instance Metadata Service (IMDS). If your VM reads that feed, it can take itself out of a load balancer, fail over a cluster role or checkpoint a job before the pause or reboot. If it doesn't, the first sign is a 30-second freeze that triggers a cluster failover or a dropped connection storm.

This post builds a small handler, explains the event semantics that cause surprises, and covers the operational settings around it.

#Event types and notice

From Microsoft's Scheduled Events documentation:

Two exceptions matter. On a hardware failure Azure skips the notice and starts recovery immediately. With degraded hardware and predicted failure, NotBefore is typically days in the future.

#The endpoint

curl -s -H Metadata:true \
  "http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01" | jq

Invoke-RestMethod -Headers @{Metadata='true'} -Method GET `
  -Uri 'http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01'

A response with an event:

{
  "DocumentIncarnation": 4,
  "Events": [
    {
      "EventId": "602d9444-d2cd-49c7-8624-8643e7171297",
      "EventType": "Freeze",
      "ResourceType": "VirtualMachine",
      "Resources": ["app-vm-01", "app-vm-03"],
      "EventStatus": "Scheduled",
      "NotBefore": "Mon, 09 Nov 2026 14:20:00 GMT",
      "Description": "Host server is undergoing maintenance.",
      "EventSource": "Platform",
      "DurationInSeconds": 9
    }
  ]
}

Warning

The service is enabled by the first request. That first call can take up to two minutes, events start arriving within about five minutes, and the service is disabled again if nothing calls it for 24 hours. A handler that only runs "when needed" will miss events. Run it continuously.

#Semantics that cause bugs

• Resources is the filter, not the event list. Events for every VM in the same availability set or scale set placement group are delivered to all members. A naive handler on app-vm-02 drains itself for an event that only affects app-vm-01 and app-vm-03. Always check your own name is in Resources.
• Your VM name comes from IMDS, not the hostname: compute.name from /metadata/instance. For scale sets, use the instance name that appears in Resources; compare against what IMDS returns on that instance.
• There is no "Completed" status. An event goes from Scheduled to Started, then disappears. Detect completion by its absence.
• Approval is collective. Approving an event lets it proceed for all VMs in Resources, not just the caller. In a three-node cluster, one node approving early can reboot the other two before they finish draining.
• DocumentIncarnation increases when the event list changes. Use it to avoid reprocessing the same document every second.
• DurationInSeconds is -1 when unknown. Treat unknown as "long" for planning.
• Not every scheduled event appears in the Activity Log or Resource Health, so the in-guest feed is the authoritative early warning.

#A handler you can run

Linux, as a systemd service. It polls once per second (Microsoft's recommendation), acts only on events that include this VM, and approves after the drain completes.

#!/usr/bin/env python3
import json, subprocess, time, urllib.request

IMDS = "http://169.254.169.254/metadata"
SE = f"{IMDS}/scheduledevents?api-version=2020-07-01"
HDR = {"Metadata": "true"}

def get(url):
    req = urllib.request.Request(url, headers=HDR)
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.load(r)

def approve(event_id):
    body = json.dumps({"StartRequests": [{"EventId": event_id}]}).encode()
    req = urllib.request.Request(SE, data=body, headers={**HDR, "Content-Type": "application/json"}, method="POST")
    urllib.request.urlopen(req, timeout=10)

me = get(f"{IMDS}/instance/compute?api-version=2021-02-01")["name"]
seen_incarnation, handled = None, set()

while True:
    try:
        doc = get(SE)
        if doc["DocumentIncarnation"] != seen_incarnation:
            seen_incarnation = doc["DocumentIncarnation"]
            for ev in doc["Events"]:
                if me not in ev["Resources"] or ev["EventId"] in handled:
                    continue
                print(f"{ev['EventType']} {ev['EventId']} notBefore={ev['NotBefore']}", flush=True)
                # Drain hook: must be idempotent and finish well inside the notice window
                rc = subprocess.run(["/usr/local/bin/drain.sh", ev["EventType"]], timeout=600).returncode
                if rc == 0 and ev["EventStatus"] == "Scheduled":
                    approve(ev["EventId"])
                handled.add(ev["EventId"])
    except Exception as e:
        print(f"poll error: {e}", flush=True)
    time.sleep(1)

What drain.sh should do depends on the event:

A simple way to drain behind Azure Load Balancer is to make the probe fail: stop serving the health endpoint, wait for probe interval × unhealthy threshold, then approve.

Caution

In a quorum cluster (SQL Server AG, Pacemaker, etcd), do not auto-approve on every node. Approve only after the cluster reports the affected roles have moved, otherwise an early approval can take down a majority at once.

#Windows

The same logic works in PowerShell as a scheduled task or a service wrapper. The approval call:

$uri  = 'http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01'
$body = @{ StartRequests = @(@{ EventId = $eventId }) } | ConvertTo-Json -Depth 3
Invoke-RestMethod -Headers @{Metadata='true'} -Method POST -Uri $uri `
  -Body $body -ContentType 'application/json'

#Settings around the handler

Enable terminate notifications on scale sets so scale-in and deletes produce a Terminate event:

az vmss update -g rg-app -n vmss-web \
  --enable-terminate-notification true \
  --terminate-notification-time 10

The time is in minutes within the documented 5–15 range.

Guest OS upgrade and reimage events on scale sets are off by default and need the OS image notification profile; check the Scheduled Events page for supported sizes.

Network: the handler must reach 169.254.169.254 directly. Proxies (http_proxy variables, WinHTTP proxy) and host firewalls that block link-local traffic break it silently. Bypass the proxy for that address.

Maintenance configurations control when host updates are applied for dedicated hosts and isolated sizes. They complement, not replace, Scheduled Events.

#Test it

You cannot trigger a platform Freeze on demand, but user-initiated actions produce events with EventSource: User:

az vm restart -g rg-app -n app-vm-01       # produces a Reboot event
az vm redeploy -g rg-app -n app-vm-01      # produces a Redeploy event
az vm simulate-eviction -g rg-app -n spot-vm-01   # Spot: produces Preempt

Watch the handler log and confirm the drain ran, the probe went unhealthy and the approval happened after the drain.

#Prevent it next time

• Bake the handler into the golden image and monitor that the service is running; a stopped handler means 24 hours later the feed is disabled.
• Send handler output to Log Analytics so you have a history of events per VM.
• Alert on VM availability (VmAvailabilityMetric) to catch unannounced hardware failures the feed cannot warn you about.
• Spread cluster nodes across availability zones; events for one zone's hosts then affect fewer members at once.