Optimization: EKS cost with Karpenter consolidation, spot diversification and right-sized requests

#Where EKS cost hides

• Pod requests far above real use, so nodes look full but sit idle.
• Nodes that never get replaced by cheaper/smaller ones.
• Spot pools limited to one instance type, so interruptions and fallbacks to on-demand.
• Many small clusters each paying for system pods and control plane.

#1. Requests vs usage

kubectl top pods -A --sort-by=cpu | head -30
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory'

For cost by namespace, enable split cost allocation data for EKS in Billing (adds pod-level cost to the Cost and Usage Report) or use Kubecost. Then right-size requests using VPA in recommendation mode or Goldilocks.

#2. Karpenter consolidation

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
      requirements:
        - { key: karpenter.sh/capacity-type, operator: In, values: ["spot", "on-demand"] }
        - { key: kubernetes.io/arch, operator: In, values: ["arm64", "amd64"] }
        - { key: karpenter.k8s.aws/instance-category, operator: In, values: ["c", "m", "r"] }
        - { key: karpenter.k8s.aws/instance-generation, operator: Gt, values: ["5"] }
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m
    budgets:
      - nodes: "10%"
      - nodes: "0"
        schedule: "0 8 * * mon-fri"      # no voluntary disruption 08:00-18:00 UTC on weekdays
        duration: 10h
  limits:
    cpu: "400"

• WhenEmptyOrUnderutilized lets Karpenter replace or remove nodes when pods fit elsewhere or on a cheaper node.
• Budgets cap how many nodes can be disrupted at once and block disruption in sensitive windows.
• limits stop runaway scale-out.

#3. Spot done properly

• Allow many instance types (categories, sizes, generations); Karpenter picks from the deepest, cheapest spot pools (price-capacity-optimized).
• Run interruption handling (Karpenter's SQS interruption queue) so nodes drain on the 2-minute notice.
• Keep stateful and single-replica critical workloads on on-demand:

nodeSelector:
  karpenter.sh/capacity-type: on-demand

• Spread replicas across zones and nodes (topology spread constraints) so one interruption doesn't take all replicas.

#4. Graviton

Multi-arch images (docker buildx --platform linux/amd64,linux/arm64) let Karpenter choose arm64 nodes, which are often cheaper for the same performance. Check every image (including sidecars and DaemonSets) has an arm64 variant before allowing arm64 in the NodePool.

#5. Protect what must not move

metadata:
  annotations:
    karpenter.sh/do-not-disrupt: "true"     # on pods running long batch jobs that can't restart

And PodDisruptionBudgets for services, with values that still allow one pod to move (maxUnavailable: 1). A PDB of zero blocks consolidation forever.

#6. Reduce fixed overhead

• Fewer, shared clusters with namespaces and quotas instead of one cluster per small team (where isolation allows).
• DaemonSets: each runs on every node, so trim unnecessary agents. Larger nodes amortise DaemonSet overhead.
• Scale dev/test node pools to zero out of hours (NodePool limits or scheduled scaling of workloads).

#7. Measure

Warning

Aggressive consolidation plus tight requests plus spot can create constant pod churn. Watch restart counts and p99 latency after each change, and use disruption budgets to slow things down.

#Checklist

• Requests right-sized from real usage
• Karpenter consolidation, expiry and budgets configured
• Spot diversified with interruption handling
• Critical workloads pinned to on-demand with sensible PDBs
• Multi-arch images; Graviton allowed where ready
• Cost by namespace tracked