Symptoms
nslookup db.internal.example.comworks from an EC2 instance in VPC A, but returnsNXDOMAINfrom VPC B.- On-premises servers get
NXDOMAINor the public IP for a name you defined privately. - After you created a private hosted zone (PHZ) for an AWS service domain (for example an interface endpoint zone), some names in that domain stopped resolving.
How resolution works (one paragraph)
Instances use the VPC resolver at the VPC CIDR base + 2 (for example 10.0.0.2), also reachable at 169.254.169.253. That resolver answers from PHZs associated with that VPC, then Resolver forwarding rules, then public DNS. On-prem servers can't use it directly; they need a Route 53 Resolver inbound endpoint.
Step 1: Is the zone associated with the VPC?
aws route53 get-hosted-zone --id <zone-id> --query 'VPCs' --output table
If VPC B is missing, associate it:
aws route53 associate-vpc-with-hosted-zone --hosted-zone-id <zone-id> \
--vpc VPCRegion=ap-south-1,VPCId=vpc-0bbbbbbbbbbbbbbbb
Cross-account (zone in account A, VPC in account B) takes two steps:
# In account A (zone owner)
aws route53 create-vpc-association-authorization --hosted-zone-id <zone-id> \
--vpc VPCRegion=ap-south-1,VPCId=vpc-0bbbbbbbbbbbbbbbb
# In account B (VPC owner)
aws route53 associate-vpc-with-hosted-zone --hosted-zone-id <zone-id> \
--vpc VPCRegion=ap-south-1,VPCId=vpc-0bbbbbbbbbbbbbbbb
# Back in account A, tidy up
aws route53 delete-vpc-association-authorization --hosted-zone-id <zone-id> \
--vpc VPCRegion=ap-south-1,VPCId=vpc-0bbbbbbbbbbbbbbbb
Tip
VPC peering and Transit Gateway don't share DNS. A PHZ must be associated with each VPC that queries it, or those VPCs must forward to a central Resolver.
Step 2: VPC DNS attributes
Both must be true for PHZs to work:
aws ec2 describe-vpc-attribute --vpc-id vpc-0bbbbbbbbbbbbbbbb --attribute enableDnsSupport
aws ec2 describe-vpc-attribute --vpc-id vpc-0bbbbbbbbbbbbbbbb --attribute enableDnsHostnames
aws ec2 modify-vpc-attribute --vpc-id vpc-0bbbbbbbbbbbbbbbb --enable-dns-support
aws ec2 modify-vpc-attribute --vpc-id vpc-0bbbbbbbbbbbbbbbb --enable-dns-hostnames
Also check the DHCP options set. If domain-name-servers points to your own DNS servers instead of AmazonProvidedDNS, instances never ask the VPC resolver, and PHZs are invisible unless your DNS servers forward to it.
aws ec2 describe-dhcp-options --dhcp-options-ids $(aws ec2 describe-vpcs --vpc-ids vpc-0bbbbbbbbbbbbbbbb --query 'Vpcs[0].DhcpOptionsId' --output text)
Step 3: Overlapping zones and "shadowing"
A PHZ is authoritative for its whole domain in the associated VPC. If you create a PHZ for example.com and only add db.example.com, then www.example.com returns NXDOMAIN in that VPC; it no longer falls back to public DNS.
Typical traps:
| You created a PHZ for | What breaks |
|---|---|
Your public domain (example.com) | Every public name not copied into the PHZ |
amazonaws.com or a service domain | Other AWS endpoints in that domain |
| A parent and a child zone both | The most specific zone wins; records in the parent for child names are ignored |
Fix: create PHZs for the most specific name you need (db.example.com, or internal.example.com), not the whole public domain.
Step 4: Resolver rules that hijack the name
A forwarding rule associated with the VPC can send queries for a name to on-prem DNS instead of answering from the PHZ. When both a PHZ and a Resolver rule match a name, the most specific domain wins: a forward rule for internal.example.com beats a PHZ for example.com. When they match the same domain name exactly, the forwarding rule takes precedence.
aws route53resolver list-resolver-rule-associations \
--query "ResolverRuleAssociations[?VPCId=='vpc-0bbbbbbbbbbbbbbbb']"
aws route53resolver list-resolver-rules --query 'ResolverRules[].[Name,DomainName,RuleType,Id]' --output table
If a broad forward rule (example.com → on-prem) shadows names you want from the PHZ, add a SYSTEM rule for the specific subdomain:
aws route53resolver create-resolver-rule --creator-request-id sys-db-1 \
--rule-type SYSTEM --domain-name db.example.com --name use-phz-db
aws route53resolver associate-resolver-rule --resolver-rule-id <rule-id> --vpc-id vpc-0bbbbbbbbbbbbbbbb
Step 5: On-premises clients
On-prem DNS must forward the private domain to a Resolver inbound endpoint in a VPC that is associated with the PHZ.
aws route53resolver list-resolver-endpoints \
--query "ResolverEndpoints[?Direction=='INBOUND'].[Name,Id,VPCId,Status]" --output table
aws route53resolver list-resolver-endpoint-ip-addresses --resolver-endpoint-id <id> \
--query 'IpAddresses[].Ip'
Checklist:
- The inbound endpoint's VPC is associated with the PHZ.
- Its security group allows TCP and UDP 53 from on-prem DNS servers.
- VPN/Direct Connect routes the on-prem DNS servers' subnet to the endpoint subnets.
- On-prem has a conditional forwarder for the domain pointing at both endpoint IPs.
Test from on-prem straight at the endpoint to separate AWS from on-prem forwarding:
dig @10.0.10.10 db.internal.example.com +short
Works → fix the on-prem conditional forwarder. Fails → endpoint, SG, routing or association.
Step 6: Query logs settle arguments
Turn on Resolver query logging for the VPC to see exactly what was asked and answered:
aws route53resolver create-resolver-query-log-config --name vpc-b-dns \
--destination-arn arn:aws:logs:ap-south-1:111122223333:log-group:/dns/vpc-b \
--creator-request-id qlog-vpc-b
aws route53resolver associate-resolver-query-log-config \
--resolver-query-log-config-id <config-id> --resource-id vpc-0bbbbbbbbbbbbbbbb
Prevent it next time
- Centralise DNS: one shared-services VPC with inbound/outbound endpoints, Resolver rules shared through AWS RAM, PHZs associated by automation when a VPC is created.
- Never create a PHZ for your whole public apex unless you mean to run split-horizon and mirror every record.
- Keep
AmazonProvidedDNSin DHCP options unless you have a reason not to. - Add a synthetic DNS check from each VPC and from on-prem for one record in every PHZ.