Interface VPC endpoint not working: private DNS, security groups and endpoint policies

#Symptoms

The endpoint's state is available, but one of these still happens:

#1. Split the problem from the client

Run these from the affected host. Replace the Region and service with yours.

SVC=ssm.eu-west-1.amazonaws.com
dig +short $SVC
nc -vz -w 5 $SVC 443
env | grep -i _proxy

On Windows, use Resolve-DnsName and Test-NetConnection -Port 443.

1. If DNS returns public IPs, or anything outside your VPC's ranges, go to step 2.
2. If DNS returns private IPs but TCP 443 fails, go to step 4.
3. If TCP connects and the API returns an authorization error, go to step 5.

Then get the endpoint's configuration:

aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=vpc-0abc1234 Name=vpc-endpoint-type,Values=Interface \
  --query 'VpcEndpoints[].{Id:VpcEndpointId,Service:ServiceName,State:State,PrivateDns:PrivateDnsEnabled,Subnets:SubnetIds,SGs:Groups[].GroupId,ENIs:NetworkInterfaceIds}'

To test the network path without relying on private DNS, call the endpoint's own DNS name:

aws ec2 describe-vpc-endpoints --vpc-endpoint-ids vpce-0abc1234 \
  --query 'VpcEndpoints[].DnsEntries[].DnsName'

aws ssm describe-instance-information --region eu-west-1 --max-items 1 \
  --endpoint-url https://vpce-0abc1234-xyz.ssm.eu-west-1.vpce.amazonaws.com

If that works and the normal call doesn't, your problem is DNS.

#2. Fix private DNS inside the VPC

Private DNS needs three things: the option on the endpoint, both DNS attributes on the VPC, and clients that use the VPC resolver.

VPC=vpc-0abc1234
aws ec2 describe-vpc-attribute --vpc-id $VPC --attribute enableDnsSupport
aws ec2 describe-vpc-attribute --vpc-id $VPC --attribute enableDnsHostnames

aws ec2 modify-vpc-attribute --vpc-id $VPC --enable-dns-support
aws ec2 modify-vpc-attribute --vpc-id $VPC --enable-dns-hostnames
aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-0abc1234 --private-dns-enabled

Warning

Enabling private DNS moves traffic for that service name in the whole VPC onto the endpoint. If the security group or endpoint policy is wrong, working clients break. Check step 4 and step 5 first in production.

Next, check which DNS servers clients use:

DHCP=$(aws ec2 describe-vpcs --vpc-ids $VPC --query 'Vpcs[0].DhcpOptionsId' --output text)
aws ec2 describe-dhcp-options --dhcp-options-ids $DHCP \
  --query 'DhcpOptions[].DhcpConfigurations'

If this shows domain controllers or other custom servers rather than AmazonProvidedDNS, they must forward amazonaws.com queries, or at least the service name, to the VPC resolver.

#Known exceptions

• Conflicting DNS domain. Enabling private DNS fails when another endpoint or private hosted zone already covers that name. Remove the conflict, or leave private DNS off and use the vpce- DNS name.
• S3 public IPs may be correct. With both gateway and interface endpoints for S3, the PrivateDnsOnlyForInboundResolverEndpoint option sends VPC traffic to the gateway endpoint and on-premises traffic through a Resolver inbound endpoint to the interface endpoint. Inside the VPC, you'll see public S3 addresses, and traffic still stays on the gateway endpoint.
• Another Region. A normal endpoint serves its own Region only. For supported services, create a cross-region endpoint with --service-region, which needs the vpce:AllowMultiRegion permission.

#3. Fix resolution from peered VPCs and on-premises

The private DNS zone is only visible in the endpoint's own VPC. A client elsewhere can route to the endpoint's IPs, but it still resolves the service name to public addresses.

The endpoint's DnsEntries field has the alias target name and hosted zone ID. Confirm routing, security groups and NACLs allow the remote client's CIDR on TCP 443 too.

#4. Fix timeouts: the network path

Most timeouts come from the endpoint's security group. It needs inbound TCP 443 from the client's CIDR or security group.

aws ec2 describe-security-group-rules \
  --filters Name=group-id,Values=sg-0endpoint \
  --query 'SecurityGroupRules[?!IsEgress].{Proto:IpProtocol,From:FromPort,To:ToPort,Cidr:CidrIpv4,SG:ReferencedGroupInfo.GroupId}' \
  --output table

aws ec2 authorize-security-group-ingress --group-id sg-0endpoint \
  --protocol tcp --port 443 --source-group sg-0client

Also check:

• Client egress: The client's security group must allow outbound TCP 443 to the endpoint.
• NACLs: They're stateless. On the client and endpoint subnets, allow TCP 443 one way and ephemeral ports back.
• AZ coverage: Each endpoint subnet adds an interface in that AZ. Clients in other AZs can still reach it, but add a subnet in each AZ you use so one AZ failure doesn't take out the endpoint.

aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-0abc1234 \
  --add-subnet-ids subnet-0abc1234 --add-security-group-ids sg-0endpoint

If an AZ is refused, check where the service is offered:

aws ec2 describe-vpc-endpoint-services --service-names com.amazonaws.eu-west-1.ssm \
  --query 'ServiceDetails[].{AZs:AvailabilityZones,PolicySupported:VpcEndpointPolicySupported,PrivateDns:PrivateDnsName}'

If the cause still isn't clear, trace the path from the client to the endpoint with Reachability Analyzer.

#5. Fix AccessDenied: policies

Read the full error message. It often names the policy type:

Check the policy:

aws ec2 describe-vpc-endpoints --vpc-endpoint-ids vpce-0abc1234 \
  --query 'VpcEndpoints[0].PolicyDocument' --output text

If the endpoint belongs to a different account than the caller, the denial may not appear in the caller's CloudTrail.

To test whether the policy is the cause, temporarily restore the default full-access policy:

aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-0abc1234 --reset-policy

Caution

--reset-policy removes a control that may be there for data-exfiltration protection. Save the current policy first, get approval from its owner and restore a corrected version afterwards.

#AccessDenied that starts after adding an endpoint

Through an endpoint, requests no longer carry a public source IP. aws:SourceIp isn't available in the request context, so policies that rely on it stop matching. A Deny with NotIpAddress on aws:SourceIp will then block these requests.

In those policies, use keys that apply to endpoint traffic: aws:SourceVpce, aws:SourceVpc or aws:VpcSourceIp.

Warning

A bad Deny statement in a bucket policy can lock out administrators and applications. Test the change on a non-production bucket first.

#6. Fix partial failures

#Missing companion endpoints

Many tools need several endpoints. Check each service's documentation for the full list.

#Global hostnames

Older SDK configurations may call sts.amazonaws.com rather than a regional name. Run dig on the name the client really uses. If it resolves publicly, make the client regional:

export AWS_STS_REGIONAL_ENDPOINTS=regional

#Proxies

If HTTPS_PROXY is set, calls go to the proxy and bypass the endpoint. Add the private DNS names to NO_PROXY, for example .eu-west-1.amazonaws.com.

#Prevent it next time

• Define a shared endpoint security group in code, allowing TCP 443 from clients.
• Search identity and resource policies for aws:SourceIp before adding endpoints.
• Keep a list of required endpoints per workload, and add subnets when you add AZs.