OKE operations cheat sheet: oci ce commands for clusters, node pools, add-ons, upgrades and work requests

This is a reference for the oci ce commands you use during upgrades, incidents and capacity changes. It assumes you already have a working cluster and a CLI profile. Every command takes --profile and --region. For upgrades, add --wait-for-state SUCCEEDED to wait on the work request rather than polling.

Set these once per shell:

export C=<compartment-ocid>
export CL=<cluster-ocid>
export NP=<node-pool-ocid>

#Inventory

# clusters in a compartment with version and endpoint type
oci ce cluster list -c $C --lifecycle-state ACTIVE --all \
  --query 'data[].{name:name,ver:"kubernetes-version",type:type,id:id}' --output table

# private/public endpoint and whether the public IP is enabled
oci ce cluster get --cluster-id $CL \
  --query 'data.{endpoints:endpoints,cfg:"endpoint-config"}'

# node pools with version, shape and size
oci ce node-pool list -c $C --cluster-id $CL --all \
  --query 'data[].{name:name,ver:"kubernetes-version",shape:"node-shape",size:"node-config-details".size,id:id}' --output table

# node state as OKE sees it (compare with kubectl get nodes)
oci ce node-pool get --node-pool-id $NP \
  --query 'data.nodes[].{name:name,ip:"private-ip",state:"lifecycle-state",fd:"fault-domain",err:"node-error"}' --output table

# virtual node pools
oci ce virtual-node-pool list -c $C --cluster-id $CL --all --output table

If kubectl get nodes shows a node that oci ce node-pool get reports as DELETED, or the other way round, the cause is a failed node replacement. Check the work requests.

#kubeconfig and tokens

# private endpoint kubeconfig (use from bastion/VPN/inside VCN)
oci ce cluster create-kubeconfig --cluster-id $CL --file $HOME/.kube/config \
  --region <region> --token-version 2.0.0 --kube-endpoint PRIVATE_ENDPOINT

# generate a token manually - what kubectl's exec plugin runs
oci ce cluster generate-token --cluster-id $CL --region <region>

Tip

The kubeconfig runs oci ce cluster generate-token as an exec plugin. For CI runners, add --profile <name> to the args in the kubeconfig, or set OCI_CLI_AUTH=instance_principal. A missing profile is the usual cause of Unauthorized in pipelines that work on a laptop.

#Versions and upgrade order

# Kubernetes versions OKE currently offers
oci ce cluster-options get --cluster-option-id all \
  --query 'data."kubernetes-versions"'

# images valid for node pools (filter on your target version)
oci ce node-pool-options get --node-pool-option-id all \
  --query 'data.sources[?contains("source-name",`OKE-1.3`)].{name:"source-name",id:"image-id"}' --output table

Upgrade order: control plane first, then node pools. Upgrade the control plane one minor version at a time. Node pools can lag the control plane, but only within the Kubernetes version skew policy. Check the current OKE skew rules before you skip node pool upgrades across several releases.

# 1. control plane
oci ce cluster update --cluster-id $CL --kubernetes-version <vX.Y.Z>

# 2. node pool: set version + matching image, then cycle nodes
oci ce node-pool update --node-pool-id $NP \
  --kubernetes-version <vX.Y.Z> \
  --node-source-details '{"sourceType":"IMAGE","imageId":"<oke-image-ocid>"}' \
  --node-pool-cycling-details '{"isNodeCyclingEnabled":true,"maximumSurge":"1","maximumUnavailable":"0"}' \
  --force

maximumSurge/maximumUnavailable accept counts or percentages. Surge needs spare capacity and quota for extra nodes. Set maximumUnavailable above 0 only if your PodDisruptionBudgets and spare capacity can cover the missing nodes. Run oci ce node-pool update --generate-param-json-input node-pool-cycling-details to see the fields your CLI version supports.

Warning

Updating a node pool without enabling cycling changes only the template, so existing nodes keep running the old version until you replace them. Check kubectl get nodes -o wide after every update. Don't assume the version changed.

#Drain behaviour and safe node removal

# how long OKE waits for evictions before giving up / forcing
oci ce node-pool update --node-pool-id $NP \
  --node-eviction-node-pool-settings '{"evictionGraceDuration":"PT30M","isForceDeleteAfterGraceDuration":false}' --force

# remove one bad node and shrink the pool by one
oci ce node-pool delete-node --node-pool-id $NP --node-id <node-ocid> \
  --is-decrement-size true --override-eviction-grace-duration PT15M --force

# replace a node without decrementing: delete it, OKE recreates to size
oci ce node-pool delete-node --node-pool-id $NP --node-id <node-ocid> --force

# reboot or rebuild a single node's boot volume (enhanced clusters)
oci ce cluster reboot-cluster-node --cluster-id $CL --node-id <node-ocid>
oci ce cluster replace-boot-volume-cluster-node --cluster-id $CL --node-id <node-ocid>

evictionGraceDuration is ISO 8601 with a maximum of PT60M. If you set isForceDeleteAfterGraceDuration to true, OKE terminates nodes even if PodDisruptionBudgets are still blocking eviction. Use it when a stuck upgrade costs more than a short disruption.

# scale
oci ce node-pool update --node-pool-id $NP --size 6 --force

If the Cluster Autoscaler manages a pool, it owns the pool's size, and manual changes to --size get overridden.

#Add-ons (enhanced clusters)

oci ce cluster list-addons --cluster-id $CL --output table
oci ce addon-option list --kubernetes-version <vX.Y.Z> \
  --query 'data[].{name:name,versions:versions[].{v:"version-number"}}'
oci ce cluster get-addon --cluster-id $CL --addon-name CoreDNS

# pin, configure or upgrade an add-on
oci ce cluster update-addon --cluster-id $CL --addon-name CoreDNS \
  --configurations '[{"key":"minReplica","value":"3"}]'

oci ce cluster install-addon --cluster-id $CL --addon-name CertManager
oci ce cluster disable-addon --cluster-id $CL --addon-name <name> --is-remove-existing-add-on false

Add-on names are case-sensitive. Copy them from list-addons or addon-option list. Manual kubectl edit changes to a managed add-on's resources get reverted, so make configuration changes through update-addon. Run each command with --help to confirm the flag names on your CLI version.

#Work requests: what actually failed

oci ce work-request list -c $C --cluster-id $CL --all \
  --query 'data[0:10].{op:"operation-type",status:status,start:"time-started",id:id}' --output table

oci ce work-request get --work-request-id <wr-ocid>
oci ce work-request-error list -c $C --work-request-id <wr-ocid> --all
oci ce work-request-log-entry list -c $C --work-request-id <wr-ocid> --all \
  --query 'data[].message'

#Credentials and endpoints

# rotate cluster CA/credentials (two phases; plan a node cycle in between)
oci ce cluster start-credential-rotation --cluster-id $CL --auto-completion-delay-duration P5D
oci ce cluster complete-credential-rotation --cluster-id $CL

# move the API endpoint to a different NSG / toggle public IP
oci ce cluster update-endpoint-config --cluster-id $CL \
  --nsg-ids '["<nsg-ocid>"]' --is-public-ip-enabled false

Caution

After complete-credential-rotation, kubeconfigs, service account integrations and nodes that still use the old CA stop working. Regenerate kubeconfigs and cycle the nodes before the auto-completion delay runs out. Check the OKE documentation for the exact sequence your cluster version needs.

#Quick health one-liner

kubectl get nodes -o custom-columns=N:.metadata.name,V:.status.nodeInfo.kubeletVersion,R:.status.conditions[-1].type \
 && oci ce node-pool list -c $C --cluster-id $CL --all --query 'data[].{n:name,v:"kubernetes-version"}' --output table

Run it before and after every upgrade. The two version columns should match.