S3 503 SlowDown at high request rates: prefix scaling, KMS request quotas and clients that make it worse

#Symptoms

• Spark, EMR, Glue or a fleet of workers fail with 503 Slow Down / SlowDown: Please reduce your request rate.
• Uploads to an SSE-KMS bucket fail with KMS.ThrottlingException or You have exceeded the rate at which you may call KMS.
• The bucket was fine for months; a new job or a new naming scheme started the errors.

#How S3 scales

S3 supports at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix. There's no limit on the number of prefixes. When load on a prefix grows, S3 splits partitions, but scaling takes time. During a sudden jump you can get 503s until it adapts.

So 503s usually mean: a sudden burst, concentrated on one prefix range.

#Step 1: confirm and locate

Turn on request metrics (they're not on by default, and are billed as CloudWatch metrics):

aws s3api put-bucket-metrics-configuration --bucket data-lake-raw \
  --id EntireBucket --metrics-configuration Id=EntireBucket

Then watch 5xxErrors, AllRequests, GetRequests and PutRequests for the bucket.

To find which prefixes are hot, use server access logs or CloudTrail data events in Athena:

-- S3 server access logs table
SELECT regexp_extract(key, '^([^/]+/[^/]+/)') AS prefix,
       operation, count(*) AS requests,
       sum(CASE WHEN httpstatus = '503' THEN 1 ELSE 0 END) AS slowdowns
FROM s3_access_logs_db.data_lake_raw_logs
WHERE parse_datetime(requestdatetime, 'dd/MMM/yyyy:HH:mm:ss Z') > now() - interval '2' hour
GROUP BY 1, 2
ORDER BY slowdowns DESC
LIMIT 25;

#Step 2: fix the access pattern

Tip

For analytics, a table format (Apache Iceberg, Delta Lake, Hudi) removes most LIST calls and lets you compact small files without changing readers.

#Step 3: tune clients

SDKs retry 503s with backoff, but defaults may give up too early for bulk jobs, and too many concurrent workers amplify the problem.

AWS SDK / CLI retry mode:

# ~/.aws/config
[default]
retry_mode = adaptive
max_attempts = 10

adaptive adds client-side rate limiting when throttled. Use it for batch jobs, not latency-sensitive APIs.

Hadoop S3A (Spark/EMR): tune fs.s3a.attempts.maximum, fs.s3a.retry.throttle.limit, fs.s3a.retry.throttle.interval, and limit fs.s3a.threads.max / fs.s3a.connection.maximum. Names vary by Hadoop version; check your distribution's docs.

Ramp up gradually: start a big backfill with fewer workers and increase over 10–20 minutes, giving S3 time to scale partitions.

#Step 4: the KMS angle

With SSE-KMS, each PUT calls GenerateDataKey and each GET calls Decrypt, which count against the KMS request quota for the Region (shared by all KMS calls of that type in the account). Heavy S3 traffic can exhaust it and also break other services that use KMS.

Check:

aws service-quotas list-service-quotas --service-code kms \
  --query "Quotas[?contains(QuotaName, 'Cryptographic operations')].{name:QuotaName,value:Value}" --output table

Fix: S3 Bucket Keys. S3 creates a bucket-level key and calls KMS far less often:

aws s3api put-bucket-encryption --bucket data-lake-raw --server-side-encryption-configuration '{
  "Rules": [{
    "ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "aws:kms", "KMSMasterKeyID": "arn:aws:kms:eu-west-1:111122223333:key/abcd-..."},
    "BucketKeyEnabled": true
  }]
}'

Bucket Keys apply to new objects. Existing objects keep per-object keys until rewritten (copy in place, or S3 Batch Operations). The encryption context changes to the bucket ARN, so check key policies that use kms:EncryptionContext conditions.

If you still need more, request a KMS quota increase through Service Quotas.

#Step 5: verify

• 5xxErrors in S3 request metrics back to near zero.
• KMS ThrottlingException gone from CloudTrail (eventSource = kms.amazonaws.com).
• Job runtime stable when scaled back up.

#Prevent it next time

• Design key layouts with a high-cardinality component early in the key for write-heavy data.
• Enable S3 Bucket Keys on every SSE-KMS bucket by default.
• Compact small files as part of the pipeline, not as an afterthought.
• Load test new pipelines at full scale and ramp, not by starting 2,000 workers at once.