Home / Azure

AzureTroubleshootingAdvanced

Azure Key Vault throttling (429 Too Many Requests) at scale: find the noisy caller and stop reading secrets on every request

Apps fail or slow down with HTTP 429 from Key Vault during peaks or deployments. Identify who is calling, compare with the service limits, then fix with caching, backoff and vault design.

Deepak Bhardwaj4 min read

Applies toAzure Key Vault (Standard and Premium), Key Vault references in App Service/Functions, Secrets Store CSI driver on AKS, Azure SDKs

Symptoms

  • Requests fail with 429 Too Many Requests from *.vault.azure.net, often with a Retry-After header.
  • Startup storms: scaling out from 10 to 100 instances, or rolling a large AKS deployment, makes every pod fail to start.
  • Cryptographic operations (sign, wrap, decrypt) slow down during batch jobs.

Key Vault limits requests per vault, per region, over a 10-second window, with separate limits per operation type (secrets, keys by type, management). The exact numbers are in the "Azure Key Vault service limits" page; check them for your key types rather than assuming.

Step 1: confirm the throttling

Metrics (enable nothing extra):

KV_ID=$(az keyvault show -n kv-app-prod --query id -o tsv)

az monitor metrics list --resource $KV_ID --metric ServiceApiResult \
  --filter "StatusCode eq '429'" --interval PT1M --aggregation Count \
  --start-time 2026-10-28T08:00:00Z --end-time 2026-10-28T10:00:00Z -o table

az monitor metrics list --resource $KV_ID --metric ServiceApiHit \
  --filter "ActivityName eq '*'" --interval PT1M --aggregation Count -o table

Spikes in ServiceApiHit that line up with 429s tell you when and which operations.

Step 2: find who is calling

With diagnostic settings sending AuditEvent logs to Log Analytics:

// Resource-specific table
AZKVAuditLogs
| where TimeGenerated > ago(4h)
| summarize calls = count(), throttled = countif(HttpStatusCode == 429)
    by CallerIpAddress, Identity = tostring(Identity), OperationName, bin(TimeGenerated, 1m)
| order by calls desc

If your workspace uses the legacy AzureDiagnostics table:

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT" and TimeGenerated > ago(4h)
| summarize calls = count(), throttled = countif(httpStatusCode_d == 429)
    by CallerIPAddress, OperationName, bin(TimeGenerated, 1m)
| order by calls desc

Column names differ between the two tables; check one row first. Typically you'll find one or two identities producing most calls, usually SecretGet in a tight loop.

Step 3: fix the callers

Cache secrets in the application

The most common cause is code that reads a secret per HTTP request or per message. Load once, cache in memory, refresh on a timer or on authentication failure.

// .NET: load secrets into configuration once, refresh periodically
builder.Configuration.AddAzureKeyVault(
    new Uri("https://kv-app-prod.vault.azure.net/"),
    new DefaultAzureCredential(),
    new AzureKeyVaultConfigurationOptions { ReloadInterval = TimeSpan.FromMinutes(30) });

# Python: simple TTL cache around the SDK
import time
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

client = SecretClient("https://kv-app-prod.vault.azure.net/", DefaultAzureCredential())
_cache = {}
def get_secret(name, ttl=1800):
    v = _cache.get(name)
    if v and time.time() - v[1] < ttl:
        return v[0]
    value = client.get_secret(name).value
    _cache[name] = (value, time.time())
    return value

Respect Retry-After

The Azure SDKs retry 429s with exponential backoff by default. Hand-written REST clients often retry immediately, which makes throttling worse. Honour Retry-After and add jitter.

Platform features already cache for you

Feature Behaviour Tip
App Service / Functions Key Vault references Fetched at startup and refreshed periodically, not per request Don't also read the same secrets in code
AKS Secrets Store CSI driver Fetches at pod mount; optional rotation polling Keep rotation-poll-interval in minutes, not seconds; large deployments mount many pods at once
Azure SDK CryptographyClient Can cache public key and do encrypt/verify locally Only private-key operations need the vault

Spread startup storms

  • Use maxSurge and maxUnavailable in AKS rollouts so 500 pods don't mount secrets at the same second.
  • Scale out in steps (autoscale rules with cooldown) for App Service.

Step 4: fix the design if one vault is shared by everyone

Situation Change
Many apps share one vault One vault per application per environment per region
Multi-region app reading a vault in one region A vault per region (limits are per region)
Heavy encrypt/decrypt in batch jobs Envelope encryption: get a data key once, encrypt locally, only wrap/unwrap the data key in Key Vault
Very high crypto throughput or dedicated HSM needs Evaluate Managed HSM, which has its own, separate limits

Tip

Splitting vaults is cheap: Key Vault is billed per operation, not per vault. It also limits the blast radius of an access-policy or RBAC mistake.

Step 5: alert before users notice

az monitor metrics alert create -g rg-monitor -n kv-app-prod-429 \
  --scopes $KV_ID --condition "total ServiceApiResult > 50 where StatusCode includes 429" \
  --window-size 5m --evaluation-frequency 1m --severity 2

Also alert on the SaturationShoebox (vault saturation percentage) metric where available; it shows how close you are to the limit before you hit it.

Prevent it next time

  • Code review rule: no Key Vault calls on the request path.
  • Load test with the real number of instances and watch ServiceApiHit.
  • Keep diagnostic logs on every vault so you can always answer "who is calling".
  • Design one vault per app, per environment, per region from the start.

Related fixes