Home / AWS

AWSTroubleshootingAdvanced

Debugging AWS IAM "AccessDenied" errors step by step

Find out who is calling, which policy layer denied the call and why, using STS, CloudTrail, the policy simulator and encoded-message decoding.

Deepak Bhardwaj3 min read

Applies toAWS CLI v2, AWS Organizations

AWS combines several policy layers when it evaluates a request. An explicit Deny in any layer wins, and if no layer allows the call, it's denied. Go through the layers in this order.

1. Who is actually calling?

aws sts get-caller-identity

Many "but I have admin" tickets are the wrong profile, an expired SSO session, or a role assumed through a chain the user forgot about. In pods and Lambda functions, check the identity of the runtime (IRSA/Pod Identity role, execution role), not your own.

2. Read the full error

Newer error messages name the policy type that denied the call:

User: arn:aws:sts::111122223333:assumed-role/app-role/i-0abc is not authorized to perform:
s3:PutObject on resource: arn:aws:s3:::prod-bucket/x because no identity-based policy allows the s3:PutObject action
Phrase in error Layer to look at
no identity-based policy allows Policies on the user or role
explicit deny in a service control policy Organizations SCP
explicit deny in a resource-based policy Bucket, KMS key, queue or other resource policy
permissions boundary The boundary attached to the role
session policy Policy passed during AssumeRole

EC2 and some other services return an encoded message instead. Decode it:

aws sts decode-authorization-message --encoded-message "<blob>" \
  --query DecodedMessage --output text | python3 -m json.tool

The decoded message shows the exact action, resource and context keys, and whether the deny was explicit.

3. Find the call in CloudTrail

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutObject \
  --start-time "$(date -u -d '-30 min' +%FT%TZ)" --max-results 20 \
  --query "Events[].CloudTrailEvent" --output text | jq -r 'select(.errorCode!=null) | [.eventTime,.userIdentity.arn,.errorCode,.errorMessage] | @tsv'

Note

lookup-events only returns management events. S3 object-level and Lambda invoke calls need data events, which you query in CloudTrail Lake or Athena.

4. Simulate the policy

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::111122223333:role/app-role \
  --action-names s3:PutObject kms:GenerateDataKey \
  --resource-arns arn:aws:s3:::prod-bucket/x \
  --query "EvaluationResults[].{action:EvalActionName,decision:EvalDecision,matched:MatchedStatements[0].SourcePolicyId}" --output table

The simulator only checks identity policies and boundaries. It doesn't evaluate SCPs or most resource policies, so if it says allowed but the real call fails, look at those next.

5. Usual causes

  • KMS: the S3 call is allowed, but the role can't use the bucket's KMS key (kms:GenerateDataKey, kms:Decrypt). Check the key policy.
  • Conditions: aws:SourceVpce, aws:PrincipalOrgID, aws:RequestedRegion or MFA conditions that fail silently.
  • Cross-account access: both sides must allow it. That means the identity policy in account A and the resource policy in account B.
  • Wildcard ARNs: arn:aws:s3:::bucket matches the bucket only. Object actions need arn:aws:s3:::bucket/*.

Fix it without over-granting

Use IAM Access Analyzer's policy generation from CloudTrail activity to build a least-privilege policy, instead of attaching *:* to get past the error.

Related fixes