Home / AWS

AWSTroubleshootingAdvanced

Sev1: every Lambda function throttling — account concurrency exhausted by one runaway function

Restore service when unrelated Lambda functions throttle because one function consumed the account's concurrency. Find the culprit, cap it with reserved concurrency, protect critical functions and fix the trigger that caused the spike.

Deepak Bhardwaj2 min read

Applies toAWS Lambda (all runtimes), SQS, Kinesis, S3 and API Gateway event sources, AWS CLI v2

Symptoms

  • Many functions return Rate Exceeded / TooManyRequestsException, API Gateway returns 429/502.
  • The account-level ConcurrentExecutions metric is flat at the limit.
  • One function's invocations exploded (a recursive S3 trigger, a burst of SQS messages, a retry storm).

1. Confirm and find the culprit

aws lambda get-account-settings --query "{limit:AccountLimit.ConcurrentExecutions, unreserved:AccountLimit.UnreservedConcurrentExecutions}"

In CloudWatch, chart ConcurrentExecutions per function (metric math or the Lambda console's per-function view) for the last hour. The function at the top is your culprit.

aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name ConcurrentExecutions \
  --dimensions Name=FunctionName,Value=image-resizer --start-time $(date -u -d '-1 hour' +%FT%TZ) \
  --end-time $(date -u +%FT%TZ) --period 60 --statistics Maximum --output table

2. Cap the culprit

aws lambda put-function-concurrency --function-name image-resizer --reserved-concurrent-executions 50

Reserved concurrency is both a guarantee and a ceiling. Setting it to 0 stops the function completely — use that to break a recursive loop.

3. Protect critical functions

aws lambda put-function-concurrency --function-name checkout-api --reserved-concurrent-executions 200

Reserving for critical functions means a runaway elsewhere can't starve them. Don't reserve so much that the unreserved pool is too small for everything else.

4. Fix the trigger

Trigger Typical cause Fix
S3 event writes back to the same bucket/prefix Recursive loop Write to a different prefix or bucket; filter events by prefix/suffix
SQS Backlog drains at full speed Set maximum concurrency on the event source mapping
Kinesis / DynamoDB streams Parallelization factor high, retries Tune parallelization, bisect on error, max retry attempts
API Gateway Real traffic spike or client retry storm Throttling at the stage/usage plan, client backoff 
aws lambda update-event-source-mapping --uuid <mapping-uuid> --scaling-config MaximumConcurrency=20

Lambda has recursive loop detection for some event sources (for example SQS, SNS and S3) that stops loops after a number of invocations; check what it covers today and don't rely on it alone.

5. Recover backlog safely

When you raise concurrency again, the backlog (SQS, stream) drains quickly — check downstream systems (databases, third-party APIs) can take the burst. Raise in steps.

6. Ask for more

If normal peak traffic is close to the account limit, request a quota increase in Service Quotas. Split noisy workloads into separate accounts.

Prevent it next time

  • Alarms on account ConcurrentExecutions above 80% of the limit and on Throttles per critical function.
  • Reserved concurrency on critical and on risky (batch, fan-out) functions.
  • Maximum concurrency on SQS event source mappings.

Related fixes