Sev1: DNS fails inside an EKS cluster. CoreDNS down, throttled or unreachable and every service breaks

#Symptoms

• Application logs: getaddrinfo EAI_AGAIN, no such host, i/o timeout on port 53, Temporary failure in name resolution.
• Every service fails at once, including calls to AWS APIs from pods.
• Sometimes intermittent: one in N lookups times out after 5 seconds.

#Step 1: is CoreDNS running? (2 minutes)

kubectl -n kube-system get deploy coredns
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50
kubectl -n kube-system get endpoints kube-dns

Test from a pod:

kubectl run dnstest --rm -it --image=busybox:1.36 --restart=Never -- sh -c \
  "nslookup kubernetes.default; nslookup sts.eu-west-1.amazonaws.com"

#Step 2a: CoreDNS cannot schedule

Common after node group changes: the only nodes are tainted, are Fargate-only without a profile for kube-system, or are full.

kubectl -n kube-system describe pod -l k8s-app=kube-dns | sed -n '/Events/,$p'

• Fargate-only cluster: create a Fargate profile for kube-system with label k8s-app=kube-dns, then restart the deployment.
• Tainted nodes: add a toleration through the add-on configuration values, or add an untainted node group.

aws eks describe-addon-configuration --addon-name coredns --addon-version $(aws eks describe-addon --cluster-name prod --addon-name coredns --query addon.addonVersion --output text) --query configurationSchema --output text | head -50

#Step 2b: CoreDNS crashes

• Loop detected: the node's /etc/resolv.conf points back at CoreDNS. Fix the node DNS (it should use the VPC resolver) or the forward line in the Corefile.
• A recent manual Corefile edit with a syntax error: compare with the add-on default and roll back.

kubectl -n kube-system get configmap coredns -o yaml
kubectl -n kube-system rollout restart deploy coredns

Warning

If CoreDNS is managed as an EKS add-on, manual ConfigMap edits can be overwritten on the next add-on update. Use the add-on's configuration values instead, with --resolve-conflicts PRESERVE when updating.

#Step 3: the network path

With the VPC CNI and security groups for pods or custom node security groups, DNS traffic (UDP and TCP 53) between pods and CoreDNS pods must be allowed.

aws eks describe-cluster --name prod --query "cluster.resourcesVpcConfig.clusterSecurityGroupId"
aws ec2 describe-security-groups --group-ids <node-sg> --query "SecurityGroups[].IpPermissions[?FromPort==\`53\`]"

Network policies can also block it: a default-deny egress policy without an allow rule for kube-dns breaks DNS for that namespace.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: shop }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
      podSelector: { matchLabels: { k8s-app: kube-dns } }
    ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]

#Step 4: throttling at the VPC resolver

The Route 53 Resolver limits packets per second per network interface (check the current quota). All CoreDNS forwarding from one node shares that limit. Large clusters with a couple of CoreDNS replicas hit it; symptoms are intermittent timeouts under load.

Check the ENA counter on the node hosting CoreDNS:

ethtool -S eth0 | grep linklocal_allowance_exceeded

A rising value means packets to the resolver are being dropped. Fixes:

• More CoreDNS replicas spread over more nodes, so the load spreads over more ENIs.
• Cache longer in CoreDNS (cache 30 or more).
• NodeLocal DNSCache on every node, which caches and reduces packets to CoreDNS and the resolver.

#Step 5: reduce the query volume (ndots)

With the default ndots:5, a lookup of api.example.com first tries several search-domain suffixes, multiplying queries. For pods that call many external names:

spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"

Or use fully qualified names with a trailing dot (api.example.com.).

#Step 6: scale DNS properly

aws eks update-addon --cluster-name prod --addon-name coredns \
  --configuration-values '{"replicaCount":4,"autoScaling":{"enabled":true,"minReplicas":4,"maxReplicas":12}}' \
  --resolve-conflicts PRESERVE

Configuration keys depend on the add-on version; check the schema from step 2a. Add a PodDisruptionBudget and topology spread so replicas land in different AZs.

#Step 7: confirm

• The dnstest pod resolves internal and external names repeatedly without delay.
• CoreDNS metrics (coredns_dns_requests_total, coredns_forward_responses_total, SERVFAIL rate) stable.
• linklocal_allowance_exceeded no longer increasing.

#Prevent it next time

• At least 2-4 CoreDNS replicas across AZs, autoscaling on, PDB in place.
• NodeLocal DNSCache for large or chatty clusters.
• Alerts on CoreDNS SERVFAIL rate, latency, and pod restarts.
• Network policy templates always include the DNS allow rule.