S3 AccessDenied through a VPC gateway endpoint: endpoint policies, aws:SourceVpce and the buckets you forgot

#Symptom

• aws s3 ls s3://data-prod-01 works from your laptop but returns An error occurred (AccessDenied) from an EC2 instance or Lambda in the VPC.
• Or dnf update / yum on Amazon Linux suddenly fails, or ECS/EKS can't pull images from ECR (CannotPullContainerError ... 403), right after a security team "tightened" the S3 endpoint.

The IAM role looks correct, and the IAM policy simulator says allowed. That's the clue: something outside IAM identity policies is denying.

#How a request is evaluated through an endpoint

A request from a VPC with an S3 gateway endpoint must be allowed by all of:

1. The caller's IAM identity policy.
2. The VPC endpoint policy.
3. The bucket policy (if it has any Deny, or for cross-account access, an Allow).
4. SCPs/RCPs in the organization, and KMS key policy if objects use SSE-KMS.

An explicit Deny anywhere wins.

#Step 1: confirm the traffic really uses the endpoint

# Which route tables have the S3 prefix list?
aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=vpc-0abc Name=service-name,Values=com.amazonaws.eu-west-1.s3 \
  --query "VpcEndpoints[].{id:VpcEndpointId,type:VpcEndpointType,rts:RouteTableIds,policy:PolicyDocument}"

If the instance's subnet route table is in RouteTableIds, S3 traffic in that region goes through the endpoint. Requests to buckets in another region do not; gateway endpoints are regional.

#Step 2: read the endpoint policy

The default policy allows everything. A restrictive one often looks like this:

{
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": ["arn:aws:s3:::data-prod-*", "arn:aws:s3:::data-prod-*/*"]
  }]
}

Anything not listed is now implicitly denied from this VPC, including AWS-owned buckets your instances depend on:

Names can change; check the current AWS documentation for each service before writing the allow list.

Fix by adding the needed AWS buckets (read-only is enough):

{
  "Effect": "Allow",
  "Principal": "*",
  "Action": ["s3:GetObject"],
  "Resource": [
    "arn:aws:s3:::al2023-repos-eu-west-1-*/*",
    "arn:aws:s3:::prod-eu-west-1-starport-layer-bucket/*"
  ]
}

aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-0123 --policy-document file://endpoint-policy.json

#Step 3: read the bucket policy

aws s3api get-bucket-policy --bucket data-prod-01 --query Policy --output text | jq .

Three patterns cause most incidents:

A. Deny unless from the endpoint. This locks out everyone else, including the console, CI runners and AWS services:

{ "Effect": "Deny", "Principal": "*", "Action": "s3:*",
  "Resource": ["arn:aws:s3:::data-prod-01", "arn:aws:s3:::data-prod-01/*"],
  "Condition": { "StringNotEquals": { "aws:SourceVpce": "vpce-0123" } } }

Add an exception for admin roles with aws:PrincipalArn, or use aws:SourceVpc if several endpoints exist.

B. aws:SourceIp conditions. Through a VPC endpoint the request carries the private IP, and aws:SourceIp doesn't match it. Use aws:VpcSourceIp or aws:SourceVpce instead.

C. Wrong endpoint ID after the endpoint was recreated by IaC. The policy still names the old vpce-.

Caution

A Deny without a principal exception can lock you out of the bucket. If that happens, only the account root user can delete the bucket policy. See recovering from an S3 bucket policy lockout.

#Step 4: find the exact denial in CloudTrail

S3 data events (if enabled) show the endpoint and the error:

-- Athena over CloudTrail logs
SELECT eventtime, useridentity.arn, eventname, requestparameters, vpcendpointid, errorcode, errormessage
FROM cloudtrail_logs
WHERE eventsource = 's3.amazonaws.com'
  AND errorcode = 'AccessDenied'
  AND eventtime > '2026-11-03T00:00:00Z'
ORDER BY eventtime DESC
LIMIT 50;

S3 error messages for newer requests often say which policy type denied the call (for example "explicit deny in a VPC endpoint policy" or "in a resource-based policy"). Read the full message.

#Troubleshooting table

#Prevent it next time

• Manage endpoint and bucket policies in the same IaC repo, with a test that runs aws s3 ls, dnf makecache and an ECR pull from a VPC instance after changes.
• Always include a break-glass aws:PrincipalArn exception in Deny statements.
• Prefer aws:SourceVpc over aws:SourceVpce so recreating an endpoint doesn't break access.