AKS intermittent DNS timeouts: 5-second delays, CoreDNS overload, conntrack races and the ndots trap

#Symptoms

• Outbound calls are usually fast but some take almost exactly 5 seconds (or 10, 15).
• Logs show Temporary failure in name resolution, getaddrinfo EAI_AGAIN or i/o timeout on DNS lookups, mostly under load.
• The problem disappears when you call an IP instead of a name.

Five seconds is the default resolver timeout in /etc/resolv.conf. A delay of exactly 5 s means a DNS packet was lost and retried.

#Step 1: measure it from inside a pod

kubectl run dnsprobe --image=nicolaka/netshoot --restart=Never -- sleep 3600
kubectl exec dnsprobe -- cat /etc/resolv.conf
kubectl exec dnsprobe -- bash -c '
for i in $(seq 1 300); do
  s=$(date +%s%N); getent hosts api.contoso.com >/dev/null || echo FAIL
  e=$(( ($(date +%s%N)-s)/1000000 )); [ $e -gt 1000 ] && echo "slow: ${e} ms"
done'

A few slow: 50xx ms lines confirm packet loss somewhere in the DNS path.

#Step 2: check the ndots multiplier

Kubernetes sets options ndots:5. Any name with fewer than five dots is tried with every search domain first:

api.contoso.com.default.svc.cluster.local   -> NXDOMAIN
api.contoso.com.svc.cluster.local           -> NXDOMAIN
api.contoso.com.cluster.local               -> NXDOMAIN
api.contoso.com.<vnet>.internal.cloudapp.net -> NXDOMAIN
api.contoso.com                             -> answer

Each lookup can become 10 queries (A and AAAA for each). That multiplies CoreDNS load and the chance of a dropped packet.

Fixes, per workload:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
      - name: single-request-reopen   # glibc only, see step 3

Or use fully qualified names with a trailing dot (api.contoso.com.) in configuration.

#Step 3: conntrack races on UDP

glibc sends the A and AAAA queries in parallel from the same socket. Under load, both packets can race through the node's netfilter conntrack table and one is dropped (insert_failed). The client then waits 5 seconds.

Check on a node (with a debug pod):

kubectl debug node/aks-nodepool1-12345678-vmss000000 -it --image=nicolaka/netshoot
chroot /host
conntrack -S | grep -o 'insert_failed=[0-9]*'      # rising numbers = races
cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max

#Step 4: is CoreDNS overloaded?

kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl -n kube-system top pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 | grep -iE "timeout|error|i/o"

If you scrape CoreDNS metrics, look at:

sum(rate(coredns_dns_requests_total[5m]))
sum by (rcode) (rate(coredns_dns_responses_total[5m]))
histogram_quantile(0.99, sum by (le) (rate(coredns_dns_request_duration_seconds_bucket[5m])))
sum(rate(coredns_forward_requests_total[5m]))

AKS scales CoreDNS with an autoscaler based on cluster size. On clusters with few nodes but very chatty pods it can be too few replicas. Tune the coredns-autoscaler ConfigMap in kube-system (the linear parameters) rather than editing the CoreDNS deployment, which AKS reconciles.

Extra CoreDNS settings, such as a longer cache or conditional forwarders, go in the coredns-custom ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  contoso.server: |
    contoso.local:53 {
      errors
      cache 60
      forward . 10.0.0.4 10.0.0.5
    }

kubectl -n kube-system rollout restart deployment coredns

#Step 5: is the upstream the bottleneck?

CoreDNS forwards external names to the VNet's DNS: Azure-provided DNS (168.63.129.16) or your custom DNS servers.

• Azure-provided DNS has a per-VM query rate limit (documented in the Azure DNS limits). All pods on a node share that node's limit, so a busy node can hit it. Caching in CoreDNS (and node-local caches) reduces upstream queries.
• With custom DNS servers (domain controllers, DNS forwarders, Private DNS Resolver), check their CPU, packet drops and query logs during the slow periods.

# From a node: how fast is upstream?
kubectl exec dnsprobe -- dig @168.63.129.16 api.contoso.com +stats | grep "Query time"

#Decision table

#Prevent it next time

• Scrape CoreDNS metrics and alert on SERVFAIL rate and p99 latency.
• Set ndots: 2 in your standard pod templates and Helm charts.
• Run a node-local DNS cache on busy clusters.
• Make HTTP clients reuse connections, so services don't resolve names on every request.