Sev1: a bucket policy locked everyone out of S3, including admins. How to regain access

#Symptoms

• Every S3 call on the bucket returns AccessDenied, including GetBucketPolicy and PutBucketPolicy for administrators.
• It started right after a bucket policy change, often one that added "Effect": "Deny" with "Principal": "*" and a condition like aws:SourceVpce, aws:SourceIp or aws:PrincipalArn that no admin session matches.
• Applications that read or write the bucket are failing.

#Why admins are locked out

An explicit Deny in a bucket policy overrides any IAM Allow, including AdministratorAccess. Only the account root user is exempt from bucket policy denies for the bucket policy management actions (GetBucketPolicy, PutBucketPolicy, DeleteBucketPolicy).

#Step 1: confirm it is the bucket policy (2 minutes)

aws s3api get-bucket-policy --bucket acme-orders-data
# An error occurred (AccessDenied) when calling the GetBucketPolicy operation

CloudTrail shows who changed it and the policy text in the request parameters:

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=PutBucketPolicy \
  --start-time $(date -u -d '-1 day' +%FT%TZ) \
  --query "Events[].[EventTime,Username,CloudTrailEvent]" --output text | head -5

Copy the policy from the event. You will need it to rebuild a correct version.

#Step 2a: delete the policy as root (standalone account)

Sign in to the console as the root user of the account that owns the bucket (email address, password, MFA), open the bucket, Permissions > Bucket policy > Delete.

Caution

Root credentials are break-glass. Follow your organisation's procedure for retrieving them, log the use, and sign out immediately afterwards. Do not create root access keys to do this.

#Step 2b: in an AWS Organization with centralized root access

If root credentials were removed from member accounts, use the Organizations privileged task from the management account (or delegated admin for IAM):

aws sts assume-root \
  --target-principal 111111111111 \
  --task-policy-arn arn=arn:aws:iam::aws:policy/root-task/S3UnlockBucketPolicy \
  --duration-seconds 900

Export the returned temporary credentials, then:

aws s3api delete-bucket-policy --bucket acme-orders-data

Centralized root access must already be enabled in IAM for the organisation; check the current IAM documentation for prerequisites and the list of root task policies.

#Step 3: restore service

Deleting the policy removes all bucket-level grants too. If other accounts or services (CloudFront OAC, cross-account roles, log delivery) relied on it, they now fail. Put back a minimal correct policy immediately, then refine.

Write the new policy with an explicit admin exception:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyOutsideVpceExceptAdmins",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": ["arn:aws:s3:::acme-orders-data", "arn:aws:s3:::acme-orders-data/*"],
      "Condition": {
        "StringNotEquals": { "aws:SourceVpce": "vpce-0abc1234" },
        "ArnNotLike": { "aws:PrincipalArn": [
          "arn:aws:iam::111111111111:role/aws-reserved/sso.amazonaws.com/*AWSAdministratorAccess*",
          "arn:aws:iam::111111111111:role/platform-breakglass"
        ] }
      }
    }
  ]
}

Conditions in one statement are ANDed: the deny applies only when the request is outside the endpoint and not from an admin role.

aws s3api put-bucket-policy --bucket acme-orders-data --policy file://policy.json

#Step 4: validate before it is too late

IAM Access Analyzer validates policy syntax and flags risky patterns:

aws accessanalyzer validate-policy --policy-type RESOURCE_POLICY \
  --validate-policy-resource-type AWS::S3::Bucket --policy-document file://policy.json \
  --query "findings[].[findingType,issueCode,findingDetails]" --output table

Then test from each access path: the app (through the VPC endpoint), an admin console session, CloudFront, cross-account roles.

#Step 5: confirm

• App reads and writes succeed; S3 4xxErrors metric back to baseline (request metrics must be enabled).
• get-bucket-policy works for admins.
• Root session ended and its use recorded.

#Prevent it next time

• Bucket policies change only through infrastructure as code with review and validate-policy in CI.
• Every Deny with Principal: * includes an aws:PrincipalArn exception for a break-glass role.
• Test new deny conditions first with a copy bucket.
• Alert on PutBucketPolicy and DeleteBucketPolicy for critical buckets via EventBridge.