Azure Resource Manager throttling at scale: 429 TooManyRequests, token buckets and finding the noisy client

#Symptoms

• Deployments, az commands or Terraform runs fail with 429 TooManyRequests, often with the code SubscriptionRequestsThrottled or a resource-provider-specific message, and a Retry-After header.
• Terraform plan takes much longer than it used to, then fails partway through refresh.
• Portal blades show "Too many requests" while a scanner, inventory job or CI run is active.
• One team's automation slows down because another team's automation is running against the same subscription.

The 429 can come from two layers. You need to know which one before you fix anything.

#How ARM throttling works now

Resource Manager uses a token bucket per region. Each request takes a token; the bucket refills at a fixed rate. Microsoft's published defaults (check the "Understand how Azure Resource Manager throttles requests" page for current values):

Two consequences that trip people up:

• A burst of 250 reads goes through instantly, then you are limited to the refill rate. A job that "worked fine in test" with 50 resources fails in production with 3,000, because the burst hides the steady-state limit.
• The bucket is per principal. Ten pipelines sharing one service principal share one bucket. Giving each workload its own identity raises your effective limit and makes the noisy client obvious. There is also an aggregate subscription-wide limit above the per-principal one.

Resource providers apply their own limits on top. Compute, for example, uses short and long windows per API family. You can be well under the ARM limit and still be throttled by Compute because a script calls GET on every VM with instanceView in a loop.

#Step 1: read the headers

The Azure CLI prints response headers with --debug. Make one representative call with the same identity as the failing job:

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/resourceGroups?api-version=2021-04-01" \
  --debug 2>&1 | grep -iE "x-ms-ratelimit|retry-after|x-ms-request-id"

For a provider-level check, call the API the job actually uses:

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.Compute/virtualMachines?api-version=2024-07-01" \
  --debug 2>&1 | grep -iE "x-ms-ratelimit-remaining-resource"

The x-ms-ratelimit-remaining-resource value lists policy names with remaining counts, e.g. Microsoft.Compute/HighCostGet3Min;.... Policy names vary by provider and version; read what your call returns rather than relying on a fixed list.

PowerShell equivalent:

$r = Invoke-AzRestMethod -Method GET `
  -Path "/subscriptions/$sub/resourceGroups?api-version=2021-04-01"
$r.Headers.GetEnumerator() | Where-Object Key -like 'x-ms-ratelimit*'

Tip

Run the header check from inside the failing pipeline with the pipeline's identity. Your own user has its own bucket and will usually show plenty of headroom.

#Step 2: find who is draining the bucket

ARM does not write GET requests to the Activity Log, so the noisiest readers are invisible there. Use these instead.

Writes and deletes are in the Activity Log. With it exported to Log Analytics:

AzureActivity
| where TimeGenerated > ago(1h)
| where CategoryValue == "Administrative"
| summarize Calls = count() by Caller, OperationNameValue
| top 20 by Calls

Throttled writes show up as failures with HTTP 429 in the properties:

AzureActivity
| where TimeGenerated > ago(1d)
| where ActivityStatusValue == "Failure"
| where tostring(Properties_d.statusCode) == "TooManyRequests"
    or tostring(Properties) has "429"
| summarize count() by Caller, bin(TimeGenerated, 5m)
| render timechart

Property layouts in AzureActivity vary; if statusCode is empty, inspect one failed row's Properties column first.

Reads need inference. Line up the time of 429s with what runs then: scheduled pipelines, CSPM/inventory tools, monitoring integrations, Terraform plans, Kubernetes controllers that call ARM (cluster autoscaler, cloud provider, external-dns). If the callers use separate identities, the Caller of their occasional writes tells you which principal is active.

#Step 3: fix the pattern, not the retry count

#Move bulk reads to Azure Resource Graph

Listing every resource, tag or VM state through ARM is the most common cause. Resource Graph is built for this and does not consume ARM read tokens in the same way:

az graph query -q "
Resources
| where type =~ 'microsoft.compute/virtualmachines'
| project name, resourceGroup, location, powerState = tostring(properties.extended.instanceView.powerState.code)
" --first 1000

Resource Graph has its own per-user quota; page results with --skip-token and keep queries to what you need.

#Replace polling with events

Scripts that poll for "did the deployment finish" or "did a resource change" can subscribe to Event Grid system topics for the subscription (Microsoft.Resources.Subscriptions) and react to Microsoft.Resources.ResourceWriteSuccess events instead.

#Tame Terraform

Large states refresh every resource on each plan, and each refresh is at least one ARM read, often more.

terraform plan -parallelism=5          # default is 10; lower it in shared subscriptions
terraform plan -refresh=false          # for fast PR checks; run a full refresh on a schedule

Split monolithic states by lifecycle (network, platform, apps). That bounds the reads per run and limits the blast radius of a slow plan.

#Back off correctly

Azure SDKs and the CLI already retry on 429 using Retry-After. Problems appear when custom code wraps them in its own aggressive retry loop, which multiplies the request rate exactly when the bucket is empty. Honour Retry-After, add jitter, and cap concurrency.

import random, time, requests

def arm_get(url, token, max_tries=6):
    for attempt in range(max_tries):
        r = requests.get(url, headers={"Authorization": f"Bearer {token}"}, timeout=30)
        if r.status_code != 429:
            return r
        wait = int(r.headers.get("Retry-After", 2 ** attempt))
        time.sleep(wait + random.uniform(0, 1))
    r.raise_for_status()

#Spread the load

• One identity per workload, not one shared "automation" service principal.
• Avoid running every scheduled job at minute zero; stagger cron schedules.
• Very large estates benefit from more subscriptions. Limits are per subscription, so a landing zone that puts 5,000 VMs in one subscription will hit provider limits sooner.

Warning

Do not raise CI concurrency to "get through faster" during throttling. Parallel jobs on the same principal drain the bucket faster and extend the outage for everyone in the subscription.

#Prevent it next time

• Log x-ms-ratelimit-remaining-* in custom automation and alert when it drops below about 20% of the bucket.
• Make Resource Graph the default for inventory, reporting and compliance scripts.
• Give each pipeline and tool its own managed identity or service principal so throttling is attributable.
• Review new third-party integrations (CSPM, CMDB sync, FinOps tools) for their ARM polling interval before granting Reader across all subscriptions.
• Load-test automation against a subscription sized like production, not a sandbox with 20 resources.