CloudWatch Logs Insights cheat sheet: queries for errors, Lambda, VPC Flow Logs and CloudTrail

#When to use this

Something is failing and the answer is in CloudWatch Logs, spread across thousands of streams. Logs Insights lets you search, parse and aggregate those logs in one query. You pay for the data scanned, so narrow the time range and log groups before you run anything wide.

#Query shape

Commands are chained with | and run top to bottom:

fields @timestamp, @message, @logStream
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

Built-in fields start with @: @timestamp, @message, @logStream, @log. JSON logs get their fields discovered automatically, so you can filter on level or requestId directly. Nested keys use dots: userIdentity.arn.

#Errors and spikes

Latest errors, ignoring a noisy known message:

fields @timestamp, @logStream, @message
| filter @message like /(?i)(error|exception|fatal)/
| filter @message not like /HealthCheck/
| sort @timestamp desc
| limit 100

Error count over time (switch to the Visualization tab to see the spike):

filter @message like /(?i)error/
| stats count(*) as errors by bin(5m)

Which stream or host is generating the errors:

filter @message like /(?i)error/
| stats count(*) as errors by @logStream
| sort errors desc
| limit 20

Group similar messages so one bad dependency stands out among thousands of lines:

filter @message like /(?i)error/
| pattern @message

Tip

Start with pattern when you don't know what you're looking for. It turns 50,000 error lines into a short list of templates with counts.

#Parsing unstructured logs

Glob parse on an access log line like GET /api/orders 500 1234ms:

parse @message "* * * *ms" as method, path, status, latency
| filter status >= 500
| stats count(*) as hits, avg(latency) as avg_ms by path
| sort hits desc

Regex parse with named groups:

parse @message /user=(?<user>\S+) action=(?<action>\S+)/
| stats count(*) by user, action

JSON application logs need no parsing:

filter level = "ERROR" and service = "checkout"
| stats count(*) by errorCode

#Lambda

Lambda writes a REPORT line per invocation, and Logs Insights discovers its fields (@duration, @billedDuration, @maxMemoryUsed, @memorySize, @initDuration).

Slowest invocations:

filter @type = "REPORT"
| fields @requestId, @duration, @maxMemoryUsed / 1000 / 1000 as mem_mb
| sort @duration desc
| limit 20

Latency percentiles and cold starts over time:

filter @type = "REPORT"
| stats pct(@duration, 50) as p50, pct(@duration, 99) as p99,
        count(@initDuration) as cold_starts, count(*) as invocations by bin(15m)

Memory headroom (are you close to the configured size?):

filter @type = "REPORT"
| stats max(@maxMemoryUsed / 1000 / 1000) as max_used_mb,
        max(@memorySize / 1000 / 1000) as configured_mb

Timeouts:

filter @message like /Task timed out/
| stats count(*) by bin(5m)

#VPC Flow Logs

For flow logs in the default format, delivered to CloudWatch Logs, fields such as srcAddr, dstAddr, dstPort, action, protocol and bytes are discovered automatically. Custom formats may name fields differently. Check a sample record first.

Rejected traffic to a specific instance IP:

filter action = "REJECT" and dstAddr = "10.20.1.15"
| stats count(*) as rejects by srcAddr, dstPort
| sort rejects desc

Top talkers by bytes:

stats sum(bytes) as total_bytes by srcAddr, dstAddr
| sort total_bytes desc
| limit 20

Is anything reaching the database port at all?

filter dstPort = 5432
| stats count(*) by action, srcAddr

#CloudTrail (trail delivered to CloudWatch Logs)

AccessDenied by caller and API:

filter errorCode like /AccessDenied|UnauthorizedOperation/
| stats count(*) as denials by userIdentity.arn, eventSource, eventName
| sort denials desc

Who deleted or stopped something:

filter eventName in ["TerminateInstances", "StopInstances", "DeleteDBInstance", "DeleteBucket"]
| fields @timestamp, eventName, userIdentity.arn, sourceIPAddress, requestParameters
| sort @timestamp desc

Console sign-ins without MFA:

filter eventName = "ConsoleLogin" and additionalEventData.MFAUsed = "No"
| fields @timestamp, userIdentity.arn, sourceIPAddress, responseElements.ConsoleLogin

#Running queries from the CLI

Useful in runbooks and scripts. Times are epoch seconds.

QID=$(aws logs start-query \
  --log-group-names /aws/lambda/orders-api /aws/lambda/payments-api \
  --start-time $(date -u -d '-1 hour' +%s) --end-time $(date -u +%s) \
  --query-string 'filter @type = "REPORT" | stats pct(@duration, 99) as p99 by @log' \
  --query queryId --output text)

# Poll until status is Complete
aws logs get-query-results --query-id "$QID" --query "{status:status,results:results}"

PowerShell equivalent for the time window:

$end   = [DateTimeOffset]::UtcNow.ToUnixTimeSeconds()
$start = $end - 3600
aws logs start-query --log-group-names /aws/lambda/orders-api `
  --start-time $start --end-time $end `
  --query-string 'filter @message like /ERROR/ | stats count(*) by bin(5m)'

Stop a runaway query with aws logs stop-query --query-id $QID. List recent ones with aws logs describe-queries --status Running.

#Gotchas

Warning

Logs Insights charges per GB scanned. A wide query across all log groups over 30 days can cost far more than you expect. Check the current pricing, and save useful queries with Save so the team reuses tested, scoped versions.

#Prevent it next time

• Log in JSON with consistent field names (level, service, requestId). Every query in this sheet gets simpler.
• Save the queries above as saved queries in a shared folder so on-call engineers don't start from scratch.
• Turn the error-count query into a metric filter and alarm, so you hear about the spike before a customer does.
• Set retention on every log group. Never-expire groups cost money and slow down wide queries.