Symptoms
- A role with
AdministratorAccessgetsAccessDenied/UnauthorizedOperation. - The same action works in one account or Region and fails in another.
- Terraform or a pipeline fails after an organisation-wide change nobody announced.
Read the error message first
Many AWS services now say which policy type caused the denial:
| Message contains | Look at |
|---|---|
with an explicit deny in a service control policy | SCPs on the account or its OUs |
with an explicit deny in a resource control policy | RCPs on the account/OU (resource side) |
with an explicit deny in a permissions boundary / because no permissions boundary allows | The role's permissions boundary |
with an explicit deny in a session policy / because no session policy allows | Policies passed at AssumeRole/federation time (e.g. Identity Center, --policy) |
with an explicit deny in a resource-based policy / because no resource-based policy allows | Bucket, key, queue, topic, secret policy |
with an explicit deny in an identity-based policy / because no identity-based policy allows | The role's own policies |
VPC endpoint policy | The endpoint policy on the interface/gateway endpoint used |
Older services or SDKs may only say AccessDenied. Then use CloudTrail.
The evaluation order in one picture
Explicit DENY anywhere? ─ yes ─► DENY
│ no
SCPs allow it (every level from root to account)? ─ no ─► DENY
RCPs allow it (for supported services)? ─ no ─► DENY
Resource policy grants it (and cross-account rules)? ─ yes, same account ─► ALLOW*
Identity policy allows it? ─ no ─► DENY
Permissions boundary allows it? ─ no ─► DENY
Session policy allows it? ─ no ─► DENY
▼
ALLOW
*Simplified: for same-account access a resource policy can grant on its own; for cross-account access both the resource policy and the caller's identity policy must allow.
The key point: SCPs, boundaries and session policies never grant, they only limit. An admin policy can't overcome any of them.
Step 1: find the failing call in CloudTrail
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=Username,AttributeValue=deploy-role-session \
--start-time 2026-10-28T08:00:00Z --max-results 50 \
--query "Events[].CloudTrailEvent" --output text \
| jq -r 'select(.errorCode != null) | [.eventTime, .eventSource, .eventName, .errorCode, .errorMessage] | @tsv'
At scale, query the trail in Athena:
SELECT eventtime, useridentity.arn, eventsource, eventname, errorcode, errormessage, recipientaccountid
FROM cloudtrail_logs
WHERE errorcode IN ('AccessDenied', 'Client.UnauthorizedOperation', 'AccessDeniedException')
AND eventtime > '2026-10-28T08:00:00Z'
ORDER BY eventtime DESC
LIMIT 100;
Step 2: decode encoded messages (EC2 and others)
aws sts decode-authorization-message --encoded-message "<long string from the error>" \
--query DecodedMessage --output text | jq .
The caller needs sts:DecodeAuthorizationMessage. The result lists the action, resource, conditions and matched statements.
Step 3: test with the policy simulator
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam::111122223333:role/deploy-role \
--action-names ec2:RunInstances s3:PutObject \
--resource-arns "*" \
--query "EvaluationResults[].{action:EvalActionName,decision:EvalDecision,org:OrganizationsDecisionDetail.AllowedByOrganizations,boundary:PermissionsBoundaryDecisionDetail.AllowedByPermissionsBoundary}" \
--output table
The simulator shows identity policies, boundaries and (in member accounts) whether Organizations policies allow it. It doesn't model every condition key or resource policy perfectly; confirm with a real call.
Step 4: check each layer
# SCPs that apply to the account (run in the management or delegated admin account)
aws organizations list-policies-for-target --target-id 444455556666 --filter SERVICE_CONTROL_POLICY
aws organizations list-parents --child-id 444455556666 # walk up the OU tree and repeat
# Permissions boundary on the role
aws iam get-role --role-name deploy-role --query "Role.PermissionsBoundary"
# Resource policy example
aws s3api get-bucket-policy --bucket app-artifacts --query Policy --output text | jq .
Typical SCP patterns that surprise people:
| SCP pattern | Effect |
|---|---|
Deny unless aws:RequestedRegion in an allowed list | Global services called through a non-allowed Region endpoint fail; check the SCP exempts IAM, STS, Route 53, CloudFront etc. as intended |
Deny ec2:RunInstances unless tagged | Launches without required tags fail with UnauthorizedOperation |
Deny unless aws:PrincipalOrgID | Calls from outside the org (or via service principals) fail |
| Deny root user actions | Expected; use roles |
Step 5: session policies from Identity Center and federation
Permission sets are roles in each account; their inline policy and the permission set's permissions boundary apply. aws sts assume-role --policy or --policy-arns adds a session policy that limits the session further. Check what your tooling passes; CI tools sometimes add session policies automatically.
Prevent it next time
- Test SCP changes in a sandbox OU first and announce them with the exact error text people will see.
- Keep an "SCP exceptions" role pattern (via condition on a break-glass role ARN) documented and audited.
- Give platform engineers
sts:DecodeAuthorizationMessageand an Athena table over CloudTrail. - Add an IAM Access Analyzer policy check to pipelines that change IAM or resource policies.