Home / Azure

AzureTroubleshootingIntermediate

Azure VM disk is slow: finding IOPS and throughput throttling on managed disks

Use the consumed-percentage metrics to tell whether the disk or the VM size is capping I/O, then fix it with a tier change, caching, bursting or a resize.

Deepak Bhardwaj9 min read

Applies toAzure Managed Disks (Standard SSD, Premium SSD, Premium SSD v2, Ultra Disk), Windows and Linux VMs, Azure CLI 2.x

The application team says "the database is slow" and the VM's CPU is fine. High latency with flat throughput usually means the I/O is being capped. On Azure that cap comes from one of two places, and the fix depends on which one:

  • The disk has hit its provisioned IOPS or MB/s.
  • The VM size has hit its own cached or uncached disk limit, no matter how fast the disks are.

Azure publishes a metric for each of these. Check them before you resize anything.

1. Confirm it's storage, from inside the guest

You can do this without RDP or SSH by using Run Command.

Linux (iostat comes with the sysstat package):

az vm run-command invoke -g rg-db -n vm-db-01 \
  --command-id RunShellScript \
  --scripts "iostat -x 5 3"

Look for high await / r_await / w_await (milliseconds per I/O) and %util near 100 on the busy device. Column names vary by sysstat version, so check your build's output.

Windows:

az vm run-command invoke -g rg-db -n vm-db-01 \
  --command-id RunPowerShellScript \
  --scripts "Get-Counter '\PhysicalDisk(*)\Avg. Disk sec/Transfer','\PhysicalDisk(*)\Current Disk Queue Length','\PhysicalDisk(*)\Disk Transfers/sec' -SampleInterval 5 -MaxSamples 3"

If latency on a premium disk is steadily above roughly 10–20 ms while the queue keeps growing, something is throttling it. Before you go further, record which device or drive letter is slow. You'll need its LUN in the next step.

# Map disk names to LUNs
az vm show -g rg-db -n vm-db-01 \
  --query "storageProfile.dataDisks[].{lun:lun, name:name, caching:caching, sizeGb:diskSizeGb}" -o table

2. Disk cap or VM cap? Read the consumed-percentage metrics

The Storage I/O utilization metrics are percentages of the limit, so 100% means you're capped. They're only emitted on VM series that support premium storage.

Metric Hits 100% when
Data Disk IOPS Consumed Percentage A data disk is at its provisioned IOPS
Data Disk Bandwidth Consumed Percentage A data disk is at its provisioned MB/s
OS Disk IOPS Consumed Percentage / OS Disk Bandwidth Consumed Percentage The OS disk is capped
VM Uncached IOPS Consumed Percentage / VM Uncached Bandwidth Consumed Percentage The VM size's limit for disks with caching None (or beyond cache) is reached
VM Cached IOPS Consumed Percentage / VM Cached Bandwidth Consumed Percentage The VM size's limit for cached reads/writes is reached

Query them with the CLI. Split data-disk metrics by LUN, or the average across all disks hides the hot one.

VM_ID=$(az vm show -g rg-db -n vm-db-01 --query id -o tsv)

# VM-level caps
az monitor metrics list --resource $VM_ID \
  --metric "VM Uncached IOPS Consumed Percentage" "VM Uncached Bandwidth Consumed Percentage" \
           "VM Cached IOPS Consumed Percentage" "VM Cached Bandwidth Consumed Percentage" \
  --interval PT1M --aggregation Maximum --offset 2h -o table

# Per-disk caps, split by LUN
az monitor metrics list --resource $VM_ID \
  --metric "Data Disk IOPS Consumed Percentage" "Data Disk Bandwidth Consumed Percentage" \
  --filter "LUN eq '*'" \
  --interval PT1M --aggregation Maximum --offset 2h -o table

Tip

Use Maximum rather than Average. Short saturation spikes during a batch job or backup are averaged away at 5-minute grain, but the users still felt them.

Reading the result

What you see Cause Fix
One LUN at 100% IOPS, VM metrics well below 100% Disk tier too small for the workload Raise the performance tier or move to Premium SSD v2 (section 3)
Disk at 100% bandwidth, IOPS low Large sequential I/O (backups, scans) on a small disk Higher tier, stripe across disks, or move backups off-peak
VM Uncached at 100%, disks below 100% The VM size is the bottleneck Enable read caching where it's safe, or resize the VM (section 4)
VM Cached at 100% Cache limit reached Larger VM size; move write-heavy disks to caching None
Nothing near 100%, latency still high Standard HDD/SSD latency, guest queue depth, or app-level locking Check disk SKU and in-guest queue; look at the database (waits, locks)
Spikes only, then slow Credit-based burst used up Check burst credit metrics (section 5)

3. Fix a disk-level cap

Premium SSD: raise the performance tier without downtime

You can raise a Premium SSD's tier above its size's baseline while the disk stays attached to a running VM, so you get more IOPS without growing the disk.

az disk show -g rg-db -n vm-db-01-data-01 --query "{sku:sku.name, sizeGb:diskSizeGb, tier:tier, iops:diskIopsReadWrite, mbps:diskMBpsReadWrite}" -o table

az disk update -g rg-db -n vm-db-01-data-01 --set tier=P40

az disk show -g rg-db -n vm-db-01-data-01 --query tier -o tsv

Restrictions to know:

  • Only Premium SSD supports tier changes. You can't go below the disk's baseline tier.
  • You can downgrade only once every 12 hours.
  • A shared disk needs all attached VMs stopped before you change it.
  • You pay for the higher tier while it's set. Check current pricing.

Warning

If you manage the disk with Terraform, changing tier there while the VM is running can make Terraform stop and restart the VM. Use the CLI or portal for an emergency tier change, then update your code to match.

Premium SSD v2 and Ultra Disk: change IOPS and MB/s directly

On these disk types, performance is set independently of size:

az disk update -g rg-db -n vm-db-01-data-01 \
  --disk-iops-read-write 12000 --disk-mbps-read-write 400

Check the current limits and how often you can adjust performance in the Premium SSD v2 and Ultra Disk docs, since both have their own rules.

Standard SSD/HDD on a production database

Switching to Premium SSD changes --sku. For a disk that's attached, this usually needs the VM deallocated:

az vm deallocate -g rg-db -n vm-db-01
az disk update -g rg-db -n vm-db-01-data-01 --sku Premium_LRS
az vm start -g rg-db -n vm-db-01

Caution

Deallocating releases a dynamic public IP and wipes the temporary disk (D: on Windows, /mnt or /mnt/resource on Linux). Make sure nothing such as tempdb files or a swap/page file that has to survive lives there in a way that will break startup. Do it in a change window.

4. Fix a VM-level cap

Host caching

Read caching serves reads from the host cache and counts them against the cached limit rather than the uncached one. This can make a large difference for read-heavy data disks.

Disk role Recommended caching
Database data files ReadOnly
Database log files None
Write-heavy or append-only disks None
OS disk ReadWrite (default)
# Set LUN 0 to ReadOnly (index in dataDisks[], check the order from step 1)
az vm update -g rg-db -n vm-db-01 --set "storageProfile.dataDisks[0].caching=ReadOnly"

Warning

Changing caching detaches and reattaches the disk, which causes a brief I/O pause. On the OS disk it restarts the VM. Schedule it, and stop the database service first if the disk holds data or log files.

Resize the VM

If uncached throughput is still capped, the VM size has to grow. Compare the max uncached disk IOPS / MBps columns for the candidate sizes in the VM size docs, then check what the region offers:

az vm list-vm-resize-options -g rg-db -n vm-db-01 -o table
az vm resize -g rg-db -n vm-db-01 --size Standard_E16ds_v5

Resizing restarts the VM. If the new size isn't available on the current hardware cluster, you'll have to deallocate first.

5. Burst credits ran out

Premium SSDs of 512 GiB and smaller (and Standard SSDs) use credit-based bursting. The disk is fast for a while, then drops back to baseline once its credits are spent. Check:

az monitor metrics list --resource $VM_ID \
  --metric "Data Disk Used Burst IO Credits Percentage" "Data Disk Target IOPS" "Data Disk Max Burst IOPS" \
  --filter "LUN eq '*'" --interval PT5M --offset 12h -o table

When used credits reach 100%, the slowdown is expected behavior. You have two options:

  • Raise the tier so the sustained baseline fits the workload.
  • For Premium SSD larger than 512 GiB, enable on-demand bursting. The disk must be detached or the VM stopped. It costs extra, and once enabled it can't be disabled for 12 hours.
az disk update -g rg-db -n vm-db-01-data-01 --enable-bursting true

Prevent it next time

  • Alert on Data Disk IOPS Consumed Percentage and VM Uncached IOPS Consumed Percentage with a Maximum above 95% for 15 minutes. That catches throttling before users report it.
az monitor metrics alert create -g rg-db -n "vm-db-01 uncached IOPS capped" \
  --scopes $VM_ID \
  --condition "max VM Uncached IOPS Consumed Percentage > 95" \
  --window-size 15m --evaluation-frequency 5m --severity 2 \
  --action ag-dba-oncall
  • Size VMs for disk limits as well as vCPU and memory. A VM's uncached limit is often lower than the combined limit of the disks attached to it.
  • Put data and log files on separate disks with the caching settings from section 4, and record them in the build sheet.
  • Run a load test with real I/O patterns (for example fio or DiskSpd) before go-live, and keep the numbers as your baseline.
  • Review burst credit metrics after month-end or batch runs. A disk that runs out of credits every month needs a higher baseline tier.