Amazon Bedrock token usage spike: find which app is burning tokens with invocation logs

#The symptom

The Bedrock line on the bill jumped, or the InputTokenCount graph went vertical overnight. Several apps share the account, and the question is "who did this?". CloudWatch metrics tell you which model. Only model invocation logs tell you which caller. Invocation logging is off by default, so turn it on before you need it.

#1. Confirm the spike with metrics

Bedrock publishes per-model metrics in the AWS/Bedrock namespace with the ModelId dimension. Start here, because metrics need no setup.

aws cloudwatch get-metric-statistics --region us-east-1 \
  --namespace AWS/Bedrock --metric-name InputTokenCount \
  --dimensions Name=ModelId,Value=<model-or-inference-profile-id> \
  --start-time 2026-10-21T00:00:00Z --end-time 2026-10-23T00:00:00Z \
  --period 3600 --statistics Sum --output table

Repeat with OutputTokenCount and Invocations. The pattern tells you a lot:

If you don't know the exact ModelId values in use, list them:

aws cloudwatch list-metrics --region us-east-1 --namespace AWS/Bedrock \
  --metric-name Invocations --query "Metrics[].Dimensions[].Value" --output text

#2. Turn on model invocation logging

Logging is configured once per account per region. Destinations must be in the same account and region.

#Create the log group and role

aws logs create-log-group --log-group-name /bedrock/invocations --region us-east-1
aws logs put-retention-policy --log-group-name /bedrock/invocations \
  --retention-in-days 30 --region us-east-1

The role trusts bedrock.amazonaws.com and can write to the log group:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "bedrock.amazonaws.com" },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": { "aws:SourceAccount": "111122223333" },
      "ArnLike": { "aws:SourceArn": "arn:aws:bedrock:us-east-1:111122223333:*" }
    }
  }]
}

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
    "Resource": "arn:aws:logs:us-east-1:111122223333:log-group:/bedrock/invocations:log-stream:aws/bedrock/modelinvocations"
  }]
}

aws iam create-role --role-name BedrockInvocationLogging \
  --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name BedrockInvocationLogging \
  --policy-name write-logs --policy-document file://logs.json

#Enable logging

aws bedrock put-model-invocation-logging-configuration --region us-east-1 \
  --logging-config '{
    "cloudWatchConfig": {
      "logGroupName": "/bedrock/invocations",
      "roleArn": "arn:aws:iam::111122223333:role/BedrockInvocationLogging"
    },
    "textDataDeliveryEnabled": true,
    "imageDataDeliveryEnabled": false,
    "embeddingDataDeliveryEnabled": false
  }'

aws bedrock get-model-invocation-logging-configuration --region us-east-1

Caution

With textDataDeliveryEnabled on, the logs hold full prompts and responses, which can include customer data, PII and secrets pasted by users. Restrict who can read the log group, encrypt it with KMS, keep retention short, and check this with your data protection owner before you enable it in production.

Bodies over 100 KB and binary data only go to S3. If you have large prompts, add an s3Config (or largeDataDeliveryS3Config inside cloudWatchConfig) pointing at a bucket with the Bedrock bucket policy from the AWS documentation.

Note

Invocation logging only captures calls on the bedrock-runtime endpoint (InvokeModel, InvokeModelWithResponseStream, Converse, ConverseStream). Check the current documentation for which other endpoints and features are covered.

#3. Find the caller

Each log record has identity.arn, modelId, operation, input.inputTokenCount and output.outputTokenCount. Open CloudWatch → Logs Insights, pick /bedrock/invocations and run:

fields identity.arn as principal, modelId,
       input.inputTokenCount as inTokens,
       output.outputTokenCount as outTokens
| stats sum(inTokens) as totalIn, sum(outTokens) as totalOut, count() as calls
  by principal, modelId
| sort totalIn desc
| limit 20

identity.arn includes the role and session name (for example assumed-role/orders-api-task/abc123), so ECS tasks, Lambda functions and SSO users show up separately.

#Average prompt size per caller

High tokens per call points at prompt bloat rather than traffic:

fields identity.arn as principal, input.inputTokenCount as inTokens
| stats avg(inTokens) as avgIn, max(inTokens) as maxIn, count() as calls by principal
| sort avgIn desc

#When did it start

fields input.inputTokenCount as inTokens
| filter identity.arn like /orders-api/
| stats sum(inTokens) as tokens, count() as calls by bin(15m)

Line the start time up with deployments, feature flags and scheduler changes.

#Look at the biggest prompts

fields @timestamp, requestId, identity.arn, input.inputTokenCount
| sort input.inputTokenCount desc
| limit 10

Open a few of these records and read input.inputBodyJson. Common finds: the whole chat history resent every turn, twenty retrieved chunks instead of five, or a document pasted in full.

#Run it from the CLI

QID=$(aws logs start-query --region us-east-1 \
  --log-group-name /bedrock/invocations \
  --start-time $(date -d '-24 hours' +%s) --end-time $(date +%s) \
  --query-string 'stats sum(input.inputTokenCount) as tokens by identity.arn | sort tokens desc | limit 10' \
  --query queryId --output text)
sleep 10
aws logs get-query-results --query-id "$QID" --region us-east-1

#4. Attribute shared roles with request metadata

If several apps share one role, identity.arn isn't enough. Callers can tag each request with up to 16 key-value pairs. On Converse it's the requestMetadata field; on InvokeModel it's the X-Amzn-Bedrock-Request-Metadata header.

import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")
response = client.converse(
    modelId="<model-or-inference-profile-id>",
    messages=[{"role": "user", "content": [{"text": "Summarize this ticket."}]}],
    requestMetadata={"team": "support", "feature": "ticket-summary", "environment": "prod"},
)

fields requestMetadata.team as team, requestMetadata.feature as feature,
       input.inputTokenCount as inTokens, output.outputTokenCount as outTokens
| stats sum(inTokens) as totalIn, sum(outTokens) as totalOut by team, feature
| sort totalIn desc

Request metadata is optional per call and Bedrock doesn't enforce it, so set it in a shared client library or gateway. Older SDK versions may not expose the parameter; update boto3 if you get a parameter validation error. Don't put PII in metadata values.

#5. Stop the bleeding

#Prevent it next time

• Keep invocation logging on in every region you use (text delivery optional if policy forbids it).
• Alarm on token volume per model, so the bill isn't your first signal:

aws cloudwatch put-metric-alarm --region us-east-1 \
  --alarm-name bedrock-input-tokens-high \
  --namespace AWS/Bedrock --metric-name InputTokenCount \
  --dimensions Name=ModelId,Value=<model-or-inference-profile-id> \
  --statistic Sum --period 3600 --evaluation-periods 1 \
  --threshold 5000000 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:ops-alerts

• Set the threshold from your own baseline, roughly two to three times a normal hour.
• Give each app its own IAM role, and require requestMetadata in the shared client.
• For billing-grade attribution in Cost Explorer, look at application inference profiles with cost allocation tags. Logs give per-request detail; the bill gives the authoritative total.