Where OKE cost comes from
| Item | Notes |
|---|---|
| Worker nodes (managed node pools) | Compute + boot volumes for every node, used or not |
| Virtual nodes | Billed by pod resources, no nodes to manage |
| Enhanced cluster fee | Per-cluster charge for enhanced features; basic clusters have no cluster fee (check current pricing) |
| Load balancers, block volumes | Created by Services and PVCs. Easy to forget |
1. Requests vs usage
kubectl top nodes
kubectl top pods -A --sort-by=cpu | head -30
kubectl describe nodes | grep -A8 "Allocated resources"
When requests are far above usage, right-size them (VPA recommendation mode or Goldilocks) before changing nodes. Otherwise the autoscaler keeps adding nodes for reserved but unused capacity.
2. Cluster Autoscaler add-on
Enable the Cluster Autoscaler as a cluster add-on (enhanced clusters) or deploy it yourself, with min/max per node pool:
nodes: "1:10:ocid1.nodepool.oc1..aaaa…" # min:max:node-pool-ocid, one entry per pool
scale-down-unneeded-time: 5m
scale-down-utilization-threshold: 0.6
Things that block scale-down: PDBs with zero allowed disruptions, pods without controllers, pods with local storage, safe-to-evict: "false". The autoscaler logs say why a node wasn't removed.
3. Managed nodes or virtual nodes?
| Managed node pools | Virtual node pools | |
|---|---|---|
| You manage | Node shapes, OS images, upgrades (cycled by OKE) | Nothing at node level |
| Billing | Whole nodes | Pod CPU/memory requested |
| DaemonSets, privileged pods, GPUs | Supported | Limited (check feature support) |
| Best for | General platforms, special needs | Bursty or simple stateless workloads, small teams |
Virtual nodes need enhanced clusters. Mixed setups are common: a managed pool for system components and DaemonSet-heavy workloads, virtual nodes for burst.
4. Shape choice
- Ampere (Arm) node pools for workloads with multi-arch images: often the best price per core. Use node labels/affinity (
kubernetes.io/arch: arm64). - Flexible shapes sized so DaemonSet overhead is a small share of each node (very small nodes waste a large fraction on system pods).
- Preemptible node pools with taints for batch and CI (see the preemptible instances post).
5. Basic vs enhanced clusters
Enhanced clusters add features (virtual nodes, add-on management, workload identity, larger scale, SLA). For small dev/test clusters that don't need them, basic clusters avoid the cluster fee. Keep production on enhanced if you rely on those features.
6. Clean up what Kubernetes created
kubectl get svc -A --field-selector spec.type=LoadBalancer
kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,STATUS:.status.phase,SIZE:.spec.capacity.storage
- Load balancers for test services that nobody uses (use a shared ingress controller instead of one LB per service).
- Released PVs with
Retainpolicy that still hold block volumes. - Deleted clusters can leave LBs and volumes behind. Check the compartment after deleting a cluster.
7. Non-production
Scale dev node pools to zero out of hours (node pool size 0, or the autoscaler min 0 with workloads scaled down), and delete ephemeral test clusters automatically.
Warning
Lowering requests too far and enabling aggressive scale-down at the same time causes pod evictions and CPU contention under load. Change one thing at a time and watch latency and restarts.
Checklist
- Requests right-sized from usage
- Cluster Autoscaler per pool, scale-down blockers fixed
- Virtual nodes for suitable workloads
- Ampere and preemptible pools where images and workloads allow
- Basic clusters for dev/test where features aren't needed
- Orphaned LBs and volumes cleaned up