AWS API throttling at scale: find the noisy caller, tune SDK retries and stop Rate exceeded errors

Deployments and controllers start failing with ThrottlingException, Client.RequestLimitExceeded, TooManyRequestsException or plain Rate exceeded. Usually nothing in your code changed. Control-plane API rate limits are mostly shared per account and Region, and something else is spending the budget. That might be a Kubernetes controller in a hot loop, a monitoring tool polling Describe* every few seconds, or 400 Lambda invocations each calling DescribeInstances. This runbook finds the caller, buys relief, and fixes the pattern.

#Symptoms and what they tell you

Control-plane throttling is distinct from data-plane limits such as DynamoDB throughput or Lambda concurrency. If the failing calls are Describe*, List*, Get*, Create* and Update* against resources, you're in the right runbook.

#Step 1: confirm the scope

Throttling on one tool only, while others work, suggests that tool's own pattern. Throttling across unrelated tools on the same API points to an account-wide budget being drained.

Many services publish call counts to the AWS/Usage namespace (Type=API, Resource=<API name>, Service, Class=None). Graph CallCount for the throttled API:

aws cloudwatch get-metric-statistics --namespace AWS/Usage --metric-name CallCount \
  --dimensions Name=Type,Value=API Name=Resource,Value=DescribeInstances Name=Service,Value=EC2 Name=Class,Value=None \
  --start-time "$(date -u -d '-6 hours' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 60 --statistics Sum --output table

For EC2, the opt-in AWS/EC2/API namespace adds SuccessfulCalls, RequestLimitExceeded, ClientErrors and ServerErrors per API. It's enabled through AWS Support, so turn it on before you need it. AwsTriggeredRequestLimitExceeded in that namespace separates throttling AWS applied during an operational issue from throttling you caused.

#Step 2: find the noisy caller with CloudTrail

Management events record both successful and throttled calls. Throttled ones carry the error code. With CloudTrail logs in Athena (table cloudtrail_logs, partitioned by date):

-- Who calls the most, per API, in the last 2 hours
SELECT useridentity.arn            AS principal,
       eventsource, eventname,
       count(*)                    AS calls,
       sum(CASE WHEN errorcode IN ('Client.RequestLimitExceeded','RequestLimitExceeded',
                                    'ThrottlingException','Throttling','TooManyRequestsException')
                THEN 1 ELSE 0 END) AS throttled,
       approx_distinct(sourceipaddress) AS source_ips,
       arbitrary(useragent)        AS sample_agent
FROM cloudtrail_logs
WHERE eventtime > to_iso8601(current_timestamp - interval '2' hour)
  AND readonly = 'true'
GROUP BY 1, 2, 3
ORDER BY calls DESC
LIMIT 25;

Then break the top principal down per minute, to tell a steady poll from a burst:

SELECT date_trunc('minute', from_iso8601_timestamp(eventtime)) AS minute,
       eventname, count(*) AS calls
FROM cloudtrail_logs
WHERE useridentity.arn LIKE '%role/karpenter%'
  AND eventtime > to_iso8601(current_timestamp - interval '1' hour)
GROUP BY 1, 2
ORDER BY 1, 3 DESC;

useragent usually names the tool (aws-sdk-go-v2/... karpenter, Boto3/... Botocore/..., terraform-provider-aws/..., or a vendor agent). For role sessions, useridentity.arn includes the session name, which often identifies the pod, pipeline or Lambda.

Tip

Without Athena, use CloudTrail Lake, or aws cloudtrail lookup-events with an EventName filter for a quick sample. lookup-events is itself rate-limited to a low request rate per account, so don't loop it from scripts during an incident.

#Step 3: immediate relief

Do these in order. Stop at the first one that brings throttling down.

1. Pause or slow the noisy caller. Scale its Deployment to zero, disable the EventBridge schedule, or set Lambda reserved concurrency to a small number:

   kubectl -n tools scale deploy/inventory-exporter --replicas=0
   aws events disable-rule --name poll-every-minute
   aws lambda put-function-concurrency --function-name tag-sync --reserved-concurrent-executions 2
   

2. Isolate it with a deny. If you can't change the caller quickly, an inline deny on the throttled action for its role stops it from using the budget. It will start failing, but everything else recovers:

   aws iam put-role-policy --role-name inventory-exporter --policy-name temp-throttle-relief \
     --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":"ec2:DescribeInstances","Resource":"*"}]}'
   

3. Ask for a higher rate. Some API rates appear in Service Quotas as adjustable. Many control-plane rates don't, and AWS Support decides those case by case with the CloudTrail evidence. Don't assume an increase is available; check the current quotas for the service.

Warning

Remove temporary deny policies once the caller is fixed, and track them in the incident record. A forgotten deny on a controller role becomes the next outage.

#Step 4: fix retries in clients

Bad retry behaviour turns throttling into a storm: thousands of clients retry at once, empty the token bucket again, and never recover. Every AWS SDK and the CLI v2 support the standard and adaptive retry modes.

• standard: exponential backoff with jitter, a retry quota that stops retrying when most calls fail, default max attempts 3.
• adaptive: standard plus a client-side rate limiter that slows the client when it sees throttling. Best for a single busy client such as a batch job or controller. Less useful for many tiny short-lived clients.

# CLI v2 and most SDKs honour these
export AWS_RETRY_MODE=adaptive
export AWS_MAX_ATTEMPTS=10

import boto3
from botocore.config import Config

ec2 = boto3.client("ec2", config=Config(retries={"mode": "adaptive", "max_attempts": 10}))

# Paginate with server-side filters instead of pulling everything
paginator = ec2.get_paginator("describe_instances")
for page in paginator.paginate(
    Filters=[{"Name": "tag:team", "Values": ["payments"]},
             {"Name": "instance-state-name", "Values": ["running"]}],
    PaginationConfig={"PageSize": 500},
):
    ...

Remove application-level retry loops that wrap SDK calls, which already retry. Three layers of three retries make 27 calls per failure.

#Step 5: fix the call pattern

Account structure matters too. API budgets are per account and Region, so a shared "tools" account that runs inventory, security scanners and CI for the whole organisation concentrates all that load. Spreading noisy automation across accounts is a legitimate scaling lever.

#Prevent it next time

• Alarm on AWS/Usage CallCount for your most critical APIs, and on throttling error counts from CloudTrail via a metric filter or scheduled Athena query.
• Require standard or adaptive retry mode and server-side filtering in internal SDK wrappers and code review.
• Inventory third-party agents (security, cost, CMDB) with their polling intervals. Each one has a share of your budget.
• Before an incident, enable EC2 API metrics in your busiest accounts and save the Athena queries above.
• Load-test automation that fans out, such as per-account Lambdas or StackSets with high concurrency, against a sandbox account's API budget before you run it org-wide.