Karpenter not launching nodes for pending pods: NodePool requirements, EC2NodeClass status, limits and capacity errors

Pods are Pending, the cluster has room in the budget, and Karpenter does nothing, or it creates NodeClaims that disappear or never register. Karpenter fails quietly: it doesn't post an event on the pod for every reason it rejected a launch. The cause is in one of four places, and you can tell which by how far the request gets.

#Find the stage first

# Namespace depends on how you installed Karpenter (kube-system or karpenter)
KNS=kube-system

kubectl get nodepools
kubectl get ec2nodeclasses
kubectl get nodeclaims -o wide
kubectl logs -n $KNS -l app.kubernetes.io/name=karpenter -c controller --since=30m \
  | grep -Ei 'error|could not|incompatible|launch|insufficient' | tail -50

If kubectl get nodeclaims is empty while pods are pending, the problem is scheduling simulation or limits. If NodeClaims appear and vanish, it's launch. Watch them live with kubectl get nodeclaims -w and run kubectl describe nodeclaim <name>. The Conditions block (Launched, Registered, Initialized, Ready) tells you where it stopped.

#Stage 1: no NodeClaim at all

#Pod constraints that no NodePool can satisfy

Karpenter works out whether any instance type allowed by any NodePool could host the pod. Its log line usually reads like incompatible with nodepool "default", daemonset overhead=..., no instance type satisfied resources ... and requirements .... Common mismatches:

• A nodeSelector or required node affinity on a label the NodePool doesn't define or restrict, for example node.kubernetes.io/instance-type: m5.large while the NodePool only allows karpenter.k8s.aws/instance-category In [c, r].
• A toleration missing for a taint set in spec.template.spec.taints.
• Requests larger than any allowed type once DaemonSet overhead is counted. A pod requesting 15 CPU can't fit a 16 vCPU node after kube-reserved and DaemonSets.
• Topology spread or zone affinity pointing at a zone the EC2NodeClass subnets don't cover.
• A PVC bound to an EBS volume in us-east-1a while the NodePool's subnets only exist in 1b and 1c.

Dump what the NodePool actually allows and compare it with the pod spec:

kubectl get nodepool default -o jsonpath='{.spec.template.spec.requirements}' | jq .
kubectl get pod <pod> -o json | jq '{ns:.spec.nodeSelector, aff:.spec.affinity, tol:.spec.tolerations, req:[.spec.containers[].resources.requests]}'

#NodePool limits reached

spec.limits caps the total resources a NodePool may provision. When it's reached, Karpenter stops launching from that pool without warning on the pod.

kubectl get nodepool default -o jsonpath='{.spec.limits}{"\n"}{.status.resources}{"\n"}'

If status.resources.cpu is at or near spec.limits.cpu, raise the limit or add a second NodePool with a lower weight as overflow. Nodes stuck in a bad state still count against the limit, so clear orphaned NodeClaims before raising it.

#NodePool or EC2NodeClass not Ready

A NodePool whose nodeClassRef points at a missing or unready EC2NodeClass is skipped.

kubectl get nodepool default -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
kubectl get ec2nodeclass default -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'

The EC2NodeClass reports whether its subnet, security group, AMI and instance profile selectors resolved (condition names vary by version; check your build's output). An empty status.subnets or status.amis means a selector matched nothing. A tag such as karpenter.sh/discovery: <cluster> might be missing from the subnets, or an amiSelectorTerms alias might name an AMI family that doesn't exist for your Kubernetes version.

kubectl get ec2nodeclass default -o jsonpath='{.status.subnets}{"\n"}{.status.securityGroups}{"\n"}{.status.amis}{"\n"}' | jq .

#Stage 2: NodeClaim created but never launched

kubectl describe nodeclaim shows Launched as Unknown or False with a reason such as LaunchFailed and a message carrying the EC2 error.

When EC2 returns an insufficient-capacity error, Karpenter deletes the NodeClaim, marks that instance type/zone/capacity type as temporarily unavailable, and retries with the next option. With narrow requirements there may be no next option. The NodeClaim churn you see is that loop. The fix is diversity, not retries:

spec:
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]

Only include arm64 if your images are multi-arch.

To see what EC2 actually rejected, look for CreateFleet errors in CloudTrail. Fleet calls can succeed at the API level while individual launch specs fail, so check responseElements.errorSet too:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateFleet \
  --max-results 20 --query 'Events[].CloudTrailEvent' --output text \
  | jq -r 'fromjson | [.eventTime, .errorCode // "ok", (.responseElements.CreateFleetResponse.errorSet.item // "" | tostring)] | @tsv' | head

#Stage 3: instance running, node never registers

The EC2 instance exists (tag karpenter.sh/nodeclaim) but there's no Node. Karpenter eventually times out the registration and terminates the instance, so look quickly.

1. Access entry for the node role. With API authentication mode, the node IAM role needs an access entry of type EC2_LINUX (or EC2_WINDOWS):

   aws eks list-access-entries --cluster-name prod
   aws eks create-access-entry --cluster-name prod \
     --principal-arn arn:aws:iam::111122223333:role/KarpenterNodeRole-prod --type EC2_LINUX
   

   With the aws-auth ConfigMap, check that the role is mapped with groups system:bootstrappers and system:nodes.

2. Bootstrap failure. Custom userData on the EC2NodeClass is merged differently per AMI family. AL2023 uses nodeadm NodeConfig MIME parts, and Bottlerocket uses TOML. A malformed part stops kubelet starting. Read the console output:

   aws ec2 get-console-output --instance-id i-0abc123 --latest --output text | tail -80
   

3. No route to the API server. Private-endpoint clusters need the node security group to reach the cluster security group on 443. Subnets without NAT also need VPC endpoints for ECR, STS and S3 so images pull.

#Stage 4: registered but not initialized

The node shows up but stays NotReady, or keeps karpenter.sh/unregistered or your startupTaints. Typical causes: the VPC CNI can't attach ENIs (subnet out of IPs), a startup taint that the agent meant to remove it never runs, or a GPU NodePool where the device plugin isn't scheduled, so nvidia.com/gpu is never advertised and Karpenter doesn't consider the node initialized.

kubectl describe node <node> | sed -n '/Taints/,/Unschedulable/p;/Conditions/,/Addresses/p'
kubectl get pods -A -o wide --field-selector spec.nodeName=<node>

Warning

Deleting NodeClaims or NodePools to "reset" Karpenter drains and terminates the nodes behind them. On a pool running production pods, fix the EC2NodeClass or requirements in place and let Karpenter reconcile.

#Prevent it next time

• Keep requirements broad: at least three instance families, several sizes and every AZ you have subnets in. Narrow pools are the main cause of capacity loops.
• Alert on pods Pending for longer than a few minutes, and on NodePool status.resources approaching spec.limits.
• Ship Karpenter controller logs to CloudWatch Logs and keep a saved query for could not launch and incompatible with nodepool.
• Pin AMIs (alias: al2023@v<date>) in production and promote them through environments, rather than using @latest everywhere.
• Reapply the controller IAM policy and CRDs with every Karpenter upgrade. Permission drift between releases often shows up as UnauthorizedOperation.