Home / AWS

AWSRunbookIntermediate

Sev1: AWS bill spiking right now — find the runaway resource and stop it

Treat a sudden AWS cost spike as an incident. Find the service, account and usage type with Cost Explorer from the CLI, check the usual suspects (NAT data processing, logs, GPU instances, loops), stop the spend and add anomaly detection and budget actions.

Deepak Bhardwaj3 min read

Applies toAWS Cost Explorer, AWS Budgets, Cost Anomaly Detection, AWS Organizations, AWS CLI v2

Signals

  • A Cost Anomaly Detection alert or budget alert.
  • Finance asks why yesterday cost five times more than last week.
  • Unknown instances in a region you don't use (treat as a possible compromise — see the compromised keys runbook).

1. Which service and account?

aws ce get-cost-and-usage --time-period Start=2026-10-17,End=2026-10-25 --granularity DAILY \
  --metrics UnblendedCost --group-by Type=DIMENSION,Key=SERVICE \
  --query "ResultsByTime[].{day:TimePeriod.Start, top:Groups[?to_number(Metrics.UnblendedCost.Amount) > \`100\`].[Keys[0], Metrics.UnblendedCost.Amount]}" --output json

Then narrow by account and usage type:

aws ce get-cost-and-usage --time-period Start=2026-10-22,End=2026-10-25 --granularity DAILY \
  --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Elastic Compute Cloud - Compute"]}}' \
  --group-by Type=DIMENSION,Key=USAGE_TYPE Type=DIMENSION,Key=LINKED_ACCOUNT --output json

Cost Explorer data lags by several hours. For "right now", look at the resources directly.

2. The usual suspects

Usage type contains Typical cause Look at
NatGateway-Bytes Large data through NAT (S3, ECR pulls, cross-AZ) VPC Flow Logs; add S3/ECR gateway/interface endpoints
DataTransfer-Out-Bytes / -Regional-Bytes Egress to internet or cross-AZ chatter Flow Logs by destination
BoxUsage:p4d, g5, … GPU instances left running or launched by a loop aws ec2 describe-instances in all regions
DataProcessing-Bytes (CloudWatch Logs) Debug logging flood Log groups' IncomingBytes metric
Lambda-GB-Second, Request Recursive invocation loop Function Invocations, event sources
TimedStorage-ByteHrs (S3) Versioning without expiry, runaway writes Storage Lens, bucket metrics
Requests-Tier1 (S3) Millions of PUT/LIST from a job CloudTrail data events, access logs

Find running instances everywhere:

for r in $(aws ec2 describe-regions --query "Regions[].RegionName" --output text); do
  aws ec2 describe-instances --region $r --filters Name=instance-state-name,Values=running \
    --query "Reservations[].Instances[].[InstanceId,InstanceType,LaunchTime]" --output text | sed "s/^/$r /"
done

3. Stop the spend

  • Stop or scale down the resource (don't delete data in a hurry): aws ec2 stop-instances, set ASG desired to 0, set Lambda reserved concurrency to 0 to halt a loop:
aws lambda put-function-concurrency --function-name image-resizer --reserved-concurrent-executions 0
  • Lower log levels; set retention on log groups (aws logs put-retention-policy).
  • Add VPC endpoints for S3/DynamoDB (gateway, free) to take traffic off NAT.

Warning

Setting Lambda reserved concurrency to 0 stops all invocations of that function. Use it to break a loop, then fix the trigger and restore concurrency.

4. Unknown resources

If you find resources nobody launched, switch to the security runbook: check CloudTrail for RunInstances callers and contain the credentials.

Prevent it next time

  • Cost Anomaly Detection monitors per service and per account, alerts to on-call chat.
  • AWS Budgets with actions (apply an SCP or stop instances in non-prod) at thresholds.
  • SCPs that deny unused regions and expensive instance families outside approved accounts.
  • Tagging policy so every cost has an owner.

Related fixes