Home / OCI

OCITroubleshootingAdvanced

Sev1: OKE cluster API unreachable or Unauthorized — endpoint, NSG and kubeconfig token fixes

Restore kubectl access to an Oracle Kubernetes Engine cluster. Check the cluster state, public vs. private endpoint, the endpoint NSG and route, and the OCI CLI token that the kubeconfig uses for authentication.

Deepak Bhardwaj3 min read

Applies toOCI Container Engine for Kubernetes (OKE) basic and enhanced clusters, OCI CLI 3.x, kubectl

Symptoms

  • Unable to connect to the server: dial tcp ...:6443: i/o timeout.
  • error: You must be logged in to the server (Unauthorized).
  • exec plugin: invalid apiVersion or errors from oci ce cluster generate-token.
  • CI/CD deploys to the cluster fail at the same time for everyone.

1. Cluster state and endpoints

oci ce cluster get --cluster-id $CL --query 'data.{state:"lifecycle-state", version:"kubernetes-version", endpoints:endpoints, cfg:"endpoint-config"}'
  • lifecycle-state should be ACTIVE. UPDATING during an upgrade can briefly affect access.
  • endpoints lists public-endpoint and/or private-endpoint. A cluster with only a private endpoint is unreachable from the internet by design.

2. Timeout: network path

Check How
Kubeconfig points at the endpoint you can reach `kubectl config view --minify
Endpoint subnet NSG / security list allows TCP 6443 from your source oci network nsg rules list --nsg-id <endpoint-nsg>
Private endpoint: route from your network (VPN/FastConnect, DRG, bastion) Route tables, DRG attachments
Public endpoint: your egress IP is allowed in the NSG Compare with curl ifconfig.me

Regenerate kubeconfig for the right endpoint type:

oci ce cluster create-kubeconfig --cluster-id $CL --file $HOME/.kube/config --region eu-frankfurt-1 \
  --token-version 2.0.0 --kube-endpoint PRIVATE_ENDPOINT     # or PUBLIC_ENDPOINT

No route to a private endpoint? Use the OCI Bastion service with a port-forwarding session to the endpoint IP on 6443, or OCI Cloud Shell with a private network connection to the VCN.

3. Unauthorized: the token

OKE kubeconfigs call oci ce cluster generate-token on every request. It fails when:

  • Your OCI CLI config profile changed, the API key was deleted or rotated, or the session token expired (oci session refresh / oci session authenticate).
  • The OCI_CLI_PROFILE in the kubeconfig env section points at a profile that no longer exists.

Test it directly:

oci ce cluster generate-token --cluster-id $CL --region eu-frankfurt-1 | head -c 200; echo
oci iam region list --profile DEFAULT >/dev/null && echo "CLI auth OK"

4. Unauthorized: RBAC

The OCI IAM identity authenticates; Kubernetes RBAC authorises. Members of the tenancy Administrators group are cluster-admin; everyone else needs a binding to their OCID or group OCID:

kubectl create clusterrolebinding ops-admins --clusterrole=cluster-admin --group=<group-ocid>

If no one with access is left, a tenancy administrator can regenerate a kubeconfig and recreate the bindings.

5. CI pipelines

Pipelines usually use an API-key user or instance principal. Check:

  • The API key fingerprint in the pipeline secret still exists on the user.
  • The IAM policy allows use clusters in the compartment for that user or dynamic group.
Allow group 'Default'/'ci-deployers' to use clusters in compartment prod
Allow dynamic-group 'Default'/'devops-build' to use clusters in compartment prod

Prevent it next time

  • Document how to reach the private endpoint (Bastion session commands) in the on-call runbook.
  • Track API key age and expiry for CI users; prefer resource/instance principals.
  • Grant cluster-admin to a group, not individuals, and keep at least two admins.

Related fixes