Amazon Bedrock ThrottlingException below your TPM quota: max_tokens reservations, burndown rates and bursts

#Symptom

• Calls fail with HTTP 429 ThrottlingException. The message is usually along the lines of "Too many tokens" or "Too many requests", but the wording varies.
• InputTokenCount plus OutputTokenCount in CloudWatch looks far below your tokens-per-minute (TPM) quota.
• Failures come in bursts, often when a batch job starts or concurrency goes up.

#Why it happens below the quota

Bedrock doesn't only count the tokens you end up using. Quota is handled in stages:

Two things follow from this:

• A high max_tokens reserves quota you never use. As an illustration, 20 concurrent requests with 2,000 input tokens and max_tokens of 8,000 reserve 200,000 tokens up front, even if each answer is only 300 tokens.
• Output tokens can count several times over. AWS documents a 5x output burndown for Anthropic models up to version 4.7, higher rates for some newer Claude models, and 1:1 for most other models. Check the current burndown list for your model.

Cache reads (CacheReadInputTokenCount) don't count toward the quota. You're billed on actual tokens, so quota use and your bill won't match.

#Step 1: confirm it's a quota error

Log the error code, message and retry count. SDK retries can hide how often throttling happens.

import boto3
from botocore.exceptions import ClientError

brt = boto3.client("bedrock-runtime", region_name="us-east-1")

try:
    resp = brt.converse(
        modelId=MODEL_ID,
        messages=[{"role": "user", "content": [{"text": "ping"}]}],
        inferenceConfig={"maxTokens": 50},
    )
    print(resp["usage"], resp["stopReason"])
except ClientError as e:
    meta = e.response["ResponseMetadata"]
    print(e.response["Error"]["Code"], e.response["Error"]["Message"],
          meta.get("RequestId"), meta.get("RetryAttempts"))

A quota increase won't fix 503 or 529 errors.

#Step 2: see the throttling per minute

List the model IDs reporting throttles. Use these values rather than assuming a name, because the ID you need depends on how the app calls the model.

aws cloudwatch list-metrics --namespace AWS/Bedrock --metric-name InvocationThrottles \
  --query 'Metrics[].Dimensions[?Name==`ModelId`].Value[]' --output text

Pull the related metrics at one-minute resolution (GNU date):

MODEL_ID="<value from above>"
for METRIC in InvocationThrottles Invocations InputTokenCount OutputTokenCount \
              CacheWriteInputTokenCount EstimatedTPMQuotaUsage; do
  echo "== $METRIC"
  aws cloudwatch get-metric-statistics --namespace AWS/Bedrock \
    --metric-name "$METRIC" \
    --dimensions Name=ModelId,Value="$MODEL_ID" \
    --start-time "$(date -u -d '-1 hour' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
    --period 60 --statistics Sum \
    --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text
done

How to read the results:

• Throttled requests aren't included in Invocations, InvocationClientErrors or InvocationServerErrors, so an error-rate alarm built on those won't see them.
• The InvocationThrottles count depends on your SDK retry settings.
• EstimatedTPMQuotaUsage is an approximation and doesn't include the reservation that drives throttling. It can sit below your quota while you're still being throttled.

#Step 3: read the quota that applies

aws service-quotas list-service-quotas --service-code bedrock --region us-east-1 \
  --query "Quotas[?contains(QuotaName, 'Claude')].[QuotaName,Value,Adjustable,QuotaCode]" \
  --output table

Change Claude to your model family. Quota names vary as models are added, so match by name rather than hard-coding codes.

These are separate quotas, so a raised on-demand quota doesn't help cross-Region calls. Also check for a requests-per-minute quota on your model. If the message mentions tokens per day, look at the account's daily token quota as well.

#Step 4: find who is using the quota

If model invocation logging goes to CloudWatch Logs, group token use by caller. Successful calls appear with token counts, so this shows who is filling the quota rather than who is being rejected.

stats sum(input.inputTokenCount) as in_tokens,
      sum(output.outputTokenCount) as out_tokens,
      count(*) as calls
  by identity.arn, modelId, bin(1m)
| sort in_tokens desc
| limit 20

Then size max_tokens from what each model actually returns:

stats pct(output.outputTokenCount, 99) as p99_out,
      max(output.outputTokenCount) as max_out,
      count(*) as calls
  by modelId

#Step 5: apply the fix

#Lower max_tokens safely

resp = brt.converse(
    modelId=MODEL_ID,
    messages=messages,
    inferenceConfig={"maxTokens": 1200},  # just above measured p99
)
if resp["stopReason"] == "max_tokens":
    log.warning("answer truncated: %s", resp["usage"])

Warning

Setting max_tokens too low cuts answers off mid-sentence and can break JSON output. Alert on stopReason == "max_tokens" after the change and set limits per use case, not one value for everything.

#Limit bursts by tokens, not requests

A requests-per-minute limiter doesn't protect you when request size changes. This limiter reserves the same way Bedrock does. Unused tokens aren't credited back, so it errs on the safe side.

import threading, time

class TokenBudget:
    def __init__(self, tokens_per_minute):
        self.capacity = tokens_per_minute
        self.available = tokens_per_minute
        self.rate = tokens_per_minute / 60
        self.updated = time.monotonic()
        self.lock = threading.Lock()

    def acquire(self, tokens):
        tokens = min(tokens, self.capacity)
        while True:
            with self.lock:
                now = time.monotonic()
                self.available = min(self.capacity,
                                     self.available + (now - self.updated) * self.rate)
                self.updated = now
                if self.available >= tokens:
                    self.available -= tokens
                    return
                wait = (tokens - self.available) / self.rate
            time.sleep(wait)

# Your applied quota x a safety margin, divided by the number of processes
budget = TokenBudget(tokens_per_minute=QUOTA_TPM * 0.8 / PROCESSES)
budget.acquire(estimated_input_tokens + max_tokens)

#Bound your retries

from botocore.config import Config

brt = boto3.client(
    "bedrock-runtime",
    config=Config(
        retries={"total_max_attempts": 6, "mode": "standard"},
        connect_timeout=5,
        read_timeout=120,
    ),
)

Caution

Unbounded retries from many workers keep a throttled model throttled and delay live traffic. Limit attempts, add jitter, cap the queue, and defer low-priority work while throttling lasts.

#Request an increase

aws service-quotas request-service-quota-increase --service-code bedrock \
  --quota-code <QuotaCode from step 3> --desired-value <new value> --region us-east-1

If Adjustable is False, go through AWS Support or your account team. Include your peak tokens per minute, request IDs and UTC timestamps.

#Prevent it next time

Alarm directly on throttles for each model:

aws cloudwatch put-metric-alarm --alarm-name bedrock-throttles-chat \
  --namespace AWS/Bedrock --metric-name InvocationThrottles \
  --dimensions Name=ModelId,Value="$MODEL_ID" \
  --statistic Sum --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching --alarm-actions "$SNS_TOPIC_ARN"

• Review max_tokens against p99 output whenever a prompt or model changes.
• Re-check burndown rates and quotas before a model upgrade. A newer model can use more quota for the same traffic.
• Load test at peak concurrency with production max_tokens values, not just average token volumes.
• Keep bulk jobs and live traffic apart, using batch inference or a scheduled window.