Home / Azure

AzureRunbookAdvanced

Sev1: Cosmos DB returns 429 Request rate is large. Throttling storm triage, hot partitions and RU fixes

Requests fail or slow down with 429 (TooManyRequests) and retry storms make it worse. Tell a real RU shortage from a hot partition, add headroom fast, and fix the partition key, indexing or query that caused it.

Deepak Bhardwaj4 min read

Applies toAzure Cosmos DB for NoSQL (provisioned manual and autoscale throughput, serverless); concepts apply to the MongoDB API with different metric names

Symptoms

  • SDK exceptions with status 429 and substatus such as 3200 (RU budget exceeded), with x-ms-retry-after-ms headers.
  • Latency climbs because SDKs retry automatically; eventually requests fail with timeouts.
  • The Cosmos DB account looks "nowhere near" its total RU/s.

Step 1: is it total RU or one partition? (5 minutes)

Portal: Insights > Throughput and Normalized RU Consumption by PartitionKeyRangeId.

ACC=$(az cosmosdb show -g rg-data -n cosmos-prod --query id -o tsv)
az monitor metrics list --resource $ACC --metric NormalizedRUConsumption \
  --dimension CollectionName PartitionKeyRangeId --interval PT1M --aggregation Maximum --offset 1h -o table
az monitor metrics list --resource $ACC --metric TotalRequests \
  --filter "StatusCode eq '429'" --interval PT1M --aggregation Count --offset 1h -o table
Normalized RU pattern Diagnosis
All partition ranges near 100% Not enough RU/s overall
One or two ranges at 100%, others low Hot partition: a key gets most of the traffic
Spikes to 100% only during one job A bulk import, a cross-partition query or a change-feed rebuild

A few percent of 429s with normalized RU near 100% is normal behaviour for a well-used autoscale container. A sustained, high 429 rate with user-facing failures is the incident.

Step 2: add headroom (minutes)

Manual throughput:

az cosmosdb sql container throughput update -g rg-data -a cosmos-prod -d appdb -n orders --throughput 20000

Autoscale (raise the max):

az cosmosdb sql container throughput update -g rg-data -a cosmos-prod -d appdb -n orders --max-throughput 40000

Warning

Increasing RU/s spreads the new capacity evenly across physical partitions. If one partition is hot, doubling RU/s only doubles that partition's share. You may need far more total RU/s than expected, so fix the key soon.

Raising RU/s beyond what current partitions can hold can trigger a partition split that takes time; the change is accepted but completes asynchronously.

Pause or slow the offending batch job if one is running. Most bulk tools accept a max-RU or degree-of-parallelism setting.

Step 3: find the expensive operations

Enable diagnostic settings (resource-specific tables) if they are not on. Then:

CDBDataPlaneRequests
| where TimeGenerated > ago(1h)
| summarize requests = count(), throttled = countif(StatusCode == 429), ru = sum(RequestCharge)
    by OperationName, CollectionName
| order by ru desc

CDBPartitionKeyRUConsumption
| where TimeGenerated > ago(1h)
| summarize ru = sum(RequestCharge) by PartitionKey, CollectionName
| top 10 by ru

Query text with RU charge (needs query text logging enabled; be careful with sensitive data):

CDBQueryRuntimeStatistics
| where TimeGenerated > ago(1h)
| project TimeGenerated, CollectionName, QueryText, ActivityId
| take 50

Step 4: fix the cause

Cause Fix
Partition key with low cardinality or a "celebrity" value (tenant, date) Synthetic key (tenantId_bucket), hierarchical partition keys, or a new container and data migration
Cross-partition queries in a hot path Include the partition key in the WHERE clause; read by id + partition key (point read ~1 RU per KB)
Default "index everything" policy on write-heavy containers Exclude paths you never filter on ("excludedPaths": [{"path": "/payload/*"}])
Large documents updated often Patch API or split hot fields into a smaller document
Retry storms SDK defaults are fine; do not wrap them in another retry loop without backoff

Partition keys cannot be changed in place. Use the container copy feature or a change-feed based migration to a new container, then switch the app.

Step 5: confirm

  • 429 rate back to a low, steady level; no user-facing failures.
  • Normalized RU per partition evenly distributed after any key change.
  • p99 latency back to baseline in the SDK diagnostics or Application Insights dependency telemetry.

Prevent it next time

  • Alert on 429 rate (not just count) and on normalized RU above 90% for 15 minutes.
  • Choose partition keys from the access pattern before go-live, and test with production-like key distribution.
  • Use autoscale for spiky workloads, with a max you have reviewed for cost.
  • Run bulk loads with the bulk executor and a throughput cap, outside business peaks.

Related fixes