Home / AWS

AWSTroubleshootingAdvanced

EKS worker nodes not joining the cluster

Fix nodes that launch but never appear in kubectl get nodes, by checking the node role mapping or access entries, endpoint reachability, bootstrap and the VPC CNI.

Deepak Bhardwaj3 min read

Applies toAmazon EKS 1.29+, managed node groups and self-managed nodes

Symptoms

  • EC2 instances are running, but kubectl get nodes doesn't list them.
  • The managed node group is stuck in CREATING, or reports NodeCreationFailure: Instances failed to join the kubernetes cluster.

1. Run the AWS troubleshooting automation

aws ssm start-automation-execution --document-name AWSSupport-TroubleshootEKSWorkerNode \
  --parameters "ClusterName=prod-eks,WorkerID=i-0abc123"

It checks IAM, security groups, endpoint access, DNS and the kubelet in one go.

2. Is the node's IAM role authorized?

Clusters use access entries (newer) and/or the aws-auth ConfigMap (legacy).

aws eks describe-cluster --name prod-eks --query "cluster.accessConfig.authenticationMode"
aws eks list-access-entries --cluster-name prod-eks
kubectl -n kube-system get configmap aws-auth -o yaml

The node instance role ARN must be mapped as system:nodes, or have an access entry of type EC2_LINUX/EC2_WINDOWS. A common mistake is using the instance profile ARN instead of the role ARN, or including a path in the role ARN.

The role also needs AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, and the CNI policy (unless you use IRSA or Pod Identity for the CNI).

3. Can the node reach the API server?

  • Private endpoint only: nodes must be in the cluster VPC (or routed to it), and the cluster security group must allow 443 from the nodes.
  • Public endpoint only: nodes in private subnets need a NAT gateway to reach it.
  • No internet access at all: you need VPC endpoints for ec2, ecr.api, ecr.dkr, s3 (gateway), sts, and logs if used.
aws eks describe-cluster --name prod-eks --query "cluster.resourcesVpcConfig.{public:endpointPublicAccess,private:endpointPrivateAccess,cidrs:publicAccessCidrs}"

4. Look at the node

aws ssm start-session --target i-0abc123
sudo journalctl -u kubelet --no-pager | tail -50
sudo cat /var/log/cloud-init-output.log | tail -50       # bootstrap / nodeadm errors
Log shows Cause
Unauthorized from the API server IAM mapping (step 2)
dial tcp ...:443: i/o timeout Endpoint reachability (step 3)
failed to get cluster CA / bad endpoint Wrong bootstrap user data, or a custom AMI not built for this version
cni plugin not initialized The VPC CNI (aws-node) isn't running, or the subnet ran out of IPs

5. Subnet IP exhaustion

The VPC CNI gives each pod a VPC IP. Small subnets run out quickly.

aws ec2 describe-subnets --subnet-ids subnet-0aaa subnet-0bbb --query "Subnets[].{id:SubnetId,free:AvailableIpAddressCount}" --output table

Fixes: larger subnets, prefix delegation (ENABLE_PREFIX_DELEGATION=true on aws-node), or secondary CIDRs with custom networking.

6. Version skew

Nodes can't be newer than the control plane, and shouldn't be more than the supported number of minor versions older. Make sure the AMI release matches the cluster version.

Tip

Managed node groups with the EKS-optimized AMI avoid most bootstrap problems. Custom AMIs should be built from the official EKS AMI recipes.

Related fixes