Home / Azure

AzureTroubleshootingAdvanced

Azure Storage 503 ServerBusy and timeouts under load: account limits, hot partitions and naming that scales

High-volume jobs get 503 ServerBusy, 500 OperationTimedOut or slow blobs from Azure Storage. Tell account-level throttling from hot partitions, then fix naming, spreading, client retries and tier.

Deepak Bhardwaj5 min read

Applies toAzure Storage general-purpose v2 and premium block blob accounts; Blob, Table and Queue services; Azure SDKs and AzCopy

Symptoms

  • Bulk uploads, analytics jobs or many app instances get 503 Server Busy ("The server is busy") or 500 Operation could not be completed within the specified time.
  • Errors come in waves; retries usually succeed.
  • Latency rises long before errors appear (SuccessServerLatency up).

Azure Storage protects itself in two places:

  1. Account level: total request rate and ingress/egress bandwidth per account.
  2. Partition level: each partition (a range of keys) has its own lower target. Many requests to one partition, or to a narrow key range, throttle even if the account total is low.

The current numbers are in the "Scalability and performance targets for standard storage accounts" and Blob/Table/Queue target pages; check them for your account type and region.

Step 1: which kind of throttling is it?

Metrics, split by ResponseType:

SA_ID=$(az storage account show -n stdataprod01 -g rg-data --query id -o tsv)

az monitor metrics list --resource "$SA_ID/blobServices/default" --metric Transactions \
  --filter "ResponseType eq '*'" --interval PT5M --aggregation Total -o table
ResponseType Meaning
ServerBusyError Throttled by the service (503)
ServerTimeoutError Request timed out on the service side (500); often throttling of a partition
ClientThrottlingError Throttled because the client exceeded a target
NetworkError, ClientTimeoutError Client or network side

With resource logs sent to Log Analytics (StorageBlobLogs), find the hot keys:

StorageBlobLogs
| where TimeGenerated > ago(2h)
| where StatusText in ("ServerBusy", "ServerTimeout", "OperationTimedOut") or StatusCode in (500, 503)
| extend container = tostring(split(Uri, "/")[3])
| summarize errors = count() by container, OperationName, bin(TimeGenerated, 5m)
| order by errors desc

// Request rate per prefix: are calls concentrated on a narrow key range?
StorageBlobLogs
| where TimeGenerated > ago(1h)
| extend path = tostring(parse_url(Uri).Path)
| extend prefix = substring(path, 0, 40)
| summarize requests = count() by prefix
| top 20 by requests
  • Errors across the whole account while total transactions are near the published account target: account-level.
  • Errors concentrated on one container, one blob, one table partition key or one queue: partition-level.

Step 2: fix partition hot spots

Blob naming

Blob partitions are ranges of account + container + blob name. Names that increase in order (timestamps, sequential IDs) send all new writes to the end of one range:

logs/2026-10-28T09:00:00.001.json
logs/2026-10-28T09:00:00.002.json     <- all writes land in one hot range

Add a short hash prefix so writes spread:

import hashlib
def blob_name(event_id: str, ts: str) -> str:
    h = hashlib.md5(event_id.encode()).hexdigest()[:3]
    return f"{h}/{ts}/{event_id}.json"

Readers that list by date can use a separate index, or list all prefixes in parallel.

One hot blob

A single blob has its own request target. A "status.json" read by every instance every second, or a lease blob used as a lock, hits it. Cache it, or put it behind a CDN for reads.

Tables and queues

Service Hot spot Fix
Table One PartitionKey for all rows (e.g. today's date) Composite keys such as date_bucket##hash
Queue One queue for all producers/consumers at high rate Several queues, sharded by key

Step 3: fix account-level throttling

  • Spread across accounts: shard by tenant, region or data set. This is the main way to scale beyond one account's targets.
  • Premium block blob accounts give much higher transaction rates and lower latency for small-object, high-TPS workloads.
  • Fewer, bigger operations: batch small writes into larger blobs; use Put Block + Put Block List with larger block sizes for big files.
  • Ask for higher limits: some account limits (for example ingress for very large accounts) can be raised through a support request.

Step 4: fix client behaviour

Bad pattern Better
Immediate retry loops SDK default exponential retry; honour backoff
Thousands of threads per VM Bounded concurrency; tune AzCopy with AZCOPY_CONCURRENCY_VALUE
New client per request One client per process; connection reuse
ListBlobs on a huge container every minute Event Grid notifications or change feed
Many small reads of the same object In-memory or CDN cache

AzCopy example with controlled concurrency and logging:

export AZCOPY_CONCURRENCY_VALUE=64
azcopy copy "/data/export/*" "https://stdataprod01.blob.core.windows.net/raw/?<SAS>" \
  --recursive --log-level=WARNING --block-size-mb=16

Tip

If errors are ClientTimeoutError and server latency is low, the client VM is the bottleneck (CPU, NIC bandwidth, too many threads). Fix that first; Azure Storage isn't throttling you.

Step 5: alerting

az monitor metrics alert create -g rg-monitor -n st-serverbusy \
  --scopes "$SA_ID/blobServices/default" \
  --condition "total Transactions > 100 where ResponseType includes ServerBusyError" \
  --window-size 5m --evaluation-frequency 1m --severity 2

Also alert on SuccessServerLatency p95 rising: it's the early warning.

Prevent it next time

  • Design blob and table keys for spread from day one; never use pure timestamps as prefixes.
  • Plan one storage account per workload or tenant for high-volume systems.
  • Load test with production-like naming and concurrency.
  • Keep resource logs on for critical accounts so hot keys can be found quickly.

Related fixes