API Gateway returning 403, 429, 502 or 504 - finding the cause with access logs and execution logs

#First: which layer failed?

Turn on access logging with fields that separate gateway errors from integration errors. For a REST API stage:

aws apigateway update-stage --rest-api-id a1b2c3 --stage-name prod --patch-operations \
  op=replace,path=/accessLogSettings/destinationArn,value=arn:aws:logs:eu-west-1:111122223333:log-group:apigw-access-prod \
  'op=replace,path=/accessLogSettings/format,value={"requestId":"$context.requestId","ip":"$context.identity.sourceIp","method":"$context.httpMethod","path":"$context.resourcePath","status":"$context.status","error":"$context.error.message","errorType":"$context.error.responseType","integrationStatus":"$context.integration.status","integrationError":"$context.integration.error","latency":"$context.responseLatency","integrationLatency":"$context.integration.latency"}'

The account needs a CloudWatch Logs role set in the API Gateway account settings before REST API logging works.

For HTTP APIs, set AccessLogSettings on the stage with aws apigatewayv2 update-stage, using the same $context variables.

Query the logs with Logs Insights:

fields @timestamp, status, errorType, error, integrationStatus, integrationError, path
| filter status >= 400
| stats count(*) as n by status, errorType, integrationStatus
| sort n desc

#Status → cause → fix

#403: missing authentication token

This message is misleading: it usually means no route matched, not an authentication problem.

aws apigateway get-resources --rest-api-id a1b2c3 --query "items[].{path:path,methods:keys(resourceMethods||`{}`)}" --output table
aws apigateway get-deployments --rest-api-id a1b2c3 --query "items[0].createdDate"

If you changed resources and didn't create a new deployment, the stage still serves the old version:

aws apigateway create-deployment --rest-api-id a1b2c3 --stage-name prod --description "deploy route fix"

#429: throttling

Limits apply at several levels: the account-level per-region limit, stage and method throttling, and usage plans for API keys.

aws apigateway get-stage --rest-api-id a1b2c3 --stage-name prod --query "methodSettings"
aws apigateway get-usage-plans --query "items[].{name:name,throttle:throttle,quota:quota}"

Check the Count metric against your configured rate to see whether this is a real traffic spike or a misconfigured low limit. Check Service Quotas for the current account limit. Don't assume a number.

Clients should retry 429s with exponential backoff and jitter.

#502: bad Lambda proxy response

For Lambda proxy integration, the function must return this shape:

def handler(event, context):
    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps({"ok": True})   # must be a string
    }

Common mistakes are returning a dict as body, returning nothing, or an unhandled exception. Enable execution logging at ERROR level for the stage to see "Malformed Lambda proxy response":

aws apigateway update-stage --rest-api-id a1b2c3 --stage-name prod \
  --patch-operations op=replace,path=/*/*/logging/loglevel,value=ERROR

Execution logs go to API-Gateway-Execution-Logs_<api-id>/<stage>. Leave the level at ERROR in production. Full INFO logging with data tracing can record request bodies.

Caution

Don't enable data tracing (full request/response logging) on production APIs that carry personal data or secrets.

#504: integration timeout

Compare integrationLatency with the timeout:

fields @timestamp, path, integrationLatency, status
| filter status = 504
| sort integrationLatency desc
| limit 20

• If the Lambda itself times out, its own logs show Task timed out.
• If the backend is genuinely slow, use an asynchronous pattern: accept the request, put it on SQS or Step Functions, and return 202 with a status URL.
• Check whether the integration timeout limit can be raised for your API type in the current documentation, rather than relying on an older figure.

#Lambda-side checks

aws logs tail /aws/lambda/orders-api --since 30m --filter-pattern "?ERROR ?Task ?timed"
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Throttles \
  --dimensions Name=FunctionName,Value=orders-api --start-time $(date -u -d '-1 hour' +%FT%TZ) \
  --end-time $(date -u +%FT%TZ) --period 300 --statistics Sum

Lambda throttling caused by a concurrency limit shows up as 500 or 502 at the gateway, not as 429.

#Prevent it next time

• Access logs with error fields on every stage from day one.
• Alarms on 5XXError (REST) or 5xx (HTTP API), and on Latency p99.
• Contract tests for the Lambda proxy response shape.
• Deploy through CI so that a "forgot to deploy the stage" mistake can't happen.

#Checklist

• Access logs include error type and integration status
• Status mapped to gateway or integration cause
• Stage redeployed after route changes
• Throttles checked at account, stage and usage plan level
• Lambda proxy response shape validated
• 5xx and latency alarms in place