Symptom
"CloudWatch" grows into a top-five line on the bill. Cost Explorer shows most of it under usage types like DataProcessing-Bytes (ingestion) and TimedStorage-ByteHrs (storage).
1. Find the big log groups
Ingestion per log group (last 7 days) from the AWS/Logs namespace:
for lg in $(aws logs describe-log-groups --query 'logGroups[].logGroupName' --output text); do
bytes=$(aws cloudwatch get-metric-statistics --namespace AWS/Logs --metric-name IncomingBytes \
--dimensions Name=LogGroupName,Value="$lg" --start-time "$(date -u -d '-7 days' +%FT%TZ)" \
--end-time "$(date -u +%FT%TZ)" --period 604800 --statistics Sum --query 'Datapoints[0].Sum' --output text)
echo -e "$bytes\t$lg"
done | sort -rn | head -20
Stored size and retention:
aws logs describe-log-groups --query 'logGroups[].[storedBytes,retentionInDays,logGroupName]' --output text | sort -rn | head -20
None in the retention column means never expire.
2. Set retention everywhere
aws logs describe-log-groups --query 'logGroups[?!retentionInDays].logGroupName' --output text \
| tr '\t' '\n' | while read lg; do aws logs put-retention-policy --log-group-name "$lg" --retention-in-days 30; done
Agree retention per type (e.g. app logs 30 days, audit logs longer, or exported to S3). Enforce it for new groups with an AWS Config rule (cw-loggroup-retention-period-check) plus auto-remediation, or set retention in IaC for every log group.
3. Reduce volume at the source
| Source | Fix |
|---|---|
| Lambda functions logging every event at INFO/DEBUG | Use advanced logging controls: JSON format with application log level WARN in prod |
| Containers logging health checks and request bodies | Filter in the app or Fluent Bit before shipping |
| API Gateway full request/response logging | Access logs only; execution logging ERROR |
| VPC Flow Logs to CloudWatch | Send to S3 (cheaper vended-log delivery), query with Athena |
| Debug left on after an incident | Alarm on sudden IncomingBytes increases per group |
aws lambda update-function-configuration --function-name orders-api \
--logging-config LogFormat=JSON,ApplicationLogLevel=WARN,SystemLogLevel=WARN
4. Infrequent Access log class
For logs you keep for investigations but rarely query, the Infrequent Access class has a lower ingestion price. Trade-offs: it supports Logs Insights queries but not features like metric filters, subscription filters or Live Tail (check the current feature list). The class is chosen at log group creation and can't be changed, so create a new group and point the source at it:
aws logs create-log-group --log-group-name /app/orders/audit-ia --log-group-class INFREQUENT_ACCESS
aws logs put-retention-policy --log-group-name /app/orders/audit-ia --retention-in-days 180
5. Long-term retention in S3
For compliance retention, export or stream to S3 (Firehose via subscription filter, or scheduled export) and keep CloudWatch retention short. S3 lifecycle can move them to Glacier classes.
6. Watch Logs Insights query cost
Insights charges per GB scanned. Narrow the time range and log groups, and avoid dashboards that refresh wide queries every minute.
7. Keep it from growing back
- Budget alert on CloudWatch service cost.
- Weekly top-20 log groups by IncomingBytes report.
- Retention set by IaC modules, never left default.
Tip
Most savings come from three or four log groups. Fix those first; setting retention on 900 tiny groups is good hygiene but rarely moves the bill.
Checklist
- Top log groups by ingestion and storage identified
- Retention on every log group, enforced for new ones
- Log levels reduced at the source
- Flow logs and bulk vended logs to S3
- Infrequent Access class for rarely queried logs
- Budget and growth alarms