OKE pods failing DNS lookups: CoreDNS scaling, ndots, custom forwarding and pod network security rules

#Symptom

Pods log getaddrinfo EAI_AGAIN, no such host, i/o timeout against 10.96.0.10:53, or Java UnknownHostException. The failures come in one of three patterns:

• Intermittent, under load: most lookups work, but there are 5-second stalls and timeouts at peak traffic.
• Specific names only: cluster services resolve, but on-prem names or OCI private zone names don't.
• Everything, from some nodes: pods on one node pool or subnet can't resolve anything.

Each pattern has a different cause. Work out which one you have before you change anything.

#Checks

#1. Find where it fails

Run a debug pod on the affected node and query each hop directly:

kubectl run dnsdbg --rm -it --restart=Never --image=busybox:1.36 \
  --overrides='{"spec":{"nodeName":"<node-name>"}}' -- sh

# inside the pod
cat /etc/resolv.conf
nslookup kubernetes.default.svc.cluster.local      # through the kube-dns ClusterIP
nslookup objectstorage.<region>.oraclecloud.com    # external name through CoreDNS
nslookup db01.corp.example.com                     # private or on-prem name

Then query a CoreDNS pod IP directly, which skips the Service and kube-proxy:

kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
# inside the debug pod
nslookup kubernetes.default.svc.cluster.local <coredns-pod-ip>

If the pod IP works and the ClusterIP doesn't, look at kube-proxy, conntrack or the Service. If neither works from this node but both work from another node, look at the network path (security rules). If everything resolves except private names, look at forwarding.

#2. Check CoreDNS health and capacity

kubectl -n kube-system get deploy coredns
kubectl -n kube-system top pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 | grep -Ei "timeout|refused|SERVFAIL|i/o"
kubectl -n kube-system get pods -l k8s-app=kube-dns \
  -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.status.containerStatuses[0].restartCount}{"\n"}{end}'

Restarts with OOMKilled, CPU sitting at its limit, or only one or two replicas for a cluster with hundreds of nodes all mean CoreDNS can't keep up. If all the replicas are on the same node or fault domain, one node failure takes out cluster DNS.

#3. Measure the ndots multiplier

With the Kubernetes default of ndots:5, a lookup for api.partner.com (two dots) is tried against every search domain before the absolute name. Each attempt is usually an A query plus an AAAA query, so one application lookup can turn into eight or more queries, most of which return NXDOMAIN.

kubectl -n kube-system logs -l k8s-app=kube-dns --tail=2000 | grep -c NXDOMAIN

The query log only exists if the log plugin is enabled in the Corefile. If NXDOMAIN answers for *.svc.cluster.local suffixes make up most of the traffic, the volume problem comes from your workloads, not from CoreDNS.

#4. Check the security rules on the pod subnet (VCN-native)

With VCN-native pod networking, CoreDNS pods take their IPs from the pod subnet, and DNS between pods is ordinary VCN traffic that security lists and NSGs filter. If a new node pool uses a different pod subnet, or someone tightened an NSG, UDP and TCP 53 to the CoreDNS pods can be dropped while everything else works.

oci network nsg rules list --nsg-id <pod-nsg-ocid> --all \
  --query 'data[?protocol==`"17"` || protocol==`"6"`].{dir:direction,src:source,dst:destination,ports:"udp-options"}' \
  --output table

Both directions need to allow UDP 53 and TCP 53 between all pod subnets. With flannel, the equivalent rule is between the worker node subnets.

#Cause to fix

#Fixes

#Scale CoreDNS through the add-on

On enhanced clusters, CoreDNS is a managed add-on. Change the replica settings through the add-on configuration rather than by editing the Deployment, because the add-on manager reverts manual edits.

oci ce cluster get-addon --cluster-id <cluster-ocid> --addon-name CoreDNS

oci ce cluster update-addon --cluster-id <cluster-ocid> --addon-name CoreDNS \
  --configurations '[
    {"key":"minReplica","value":"3"},
    {"key":"nodesPerReplica","value":"8"},
    {"key":"coreDnsContainerResources","value":"{\"requests\":{\"cpu\":\"200m\",\"memory\":\"128Mi\"},\"limits\":{\"memory\":\"256Mi\"}}"}
  ]'

The keys come from the OKE CoreDNS add-on configuration reference. Run oci ce cluster update-addon --help to check the flag names on your CLI version. Leaving out the CPU limit avoids CFS throttling on a component that only needs CPU in short bursts. Add topologySpreadConstraints through the add-on configuration so the replicas spread across fault domains.

#Reduce query amplification per workload

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
      - name: single-request-reopen

ndots: 2 keeps the short-name lookups for in-namespace services, such as orders and orders.prod, and sends external names straight to the absolute query. musl-based images such as Alpine ignore single-request-reopen, so for those, test with TCP or use a glibc base image.

#Forward private domains with coredns-custom

Don't edit the coredns ConfigMap. OKE reads a coredns-custom ConfigMap in kube-system, and every key in it must end in .server:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  corp.server: |
    corp.example.com:53 {
        errors
        cache 30
        forward . 10.20.0.10 10.20.0.11
    }

kubectl apply -f coredns-custom.yaml
kubectl delete pod -n kube-system -l k8s-app=kube-dns   # force a reload

On enhanced clusters, the CoreDNS add-on also has a customizeCoreDNSConfigMap setting. Check that it allows your customization. The forward targets should be the VCN resolver listener endpoints or your on-prem DNS servers. If you forward to on-prem servers, the CoreDNS pods need a route to them through the DRG, and the security rules must allow it.

Warning

Restarting every CoreDNS pod at once causes a short cluster-wide DNS outage. Use kubectl -n kube-system rollout restart deploy/coredns during business hours, and only after you confirm that more than one replica is running.

Caution

On basic clusters, OKE can delete CoreDNS customizations during internal cluster updates. If production depends on private-domain forwarding, run it on an enhanced cluster, or have a pipeline check for and reapply coredns-custom.

#Fix the pod subnet path

Add stateful ingress and egress rules for UDP and TCP 53 between all pod subnets, or from all pod NSGs to the NSG that the CoreDNS pods use. Then repeat check 1 from a node in the failing pool.

#Prevent it next time

• Alert on CoreDNS restarts and on the coredns_dns_responses_total{rcode="SERVFAIL"} rate. On Kubernetes versions where the add-on enables coreDnsEnhancedMetrics, scrape those metrics too.
• Make the DNS check (check 1) a synthetic job that runs on every node pool after each upgrade or new subnet.
• Set dnsConfig in your base Helm chart so every workload starts with sensible ndots.
• Keep coredns-custom in Git and apply it with the rest of the cluster add-on configuration.
• When you add a pod subnet, update the DNS security rules in the same change.