Home / OCI

OCITroubleshootingIntermediate

Sev1: OKE pods can't pull images from OCIR — expired auth token, wrong username format and IAM policies

OKE workloads fail with ImagePullBackOff from OCI Container Registry. Fix the pull secret username and auth token, IAM policies for nodes, and network access to the registry.

Deepak Bhardwaj3 min read

Applies toOracle Container Engine for Kubernetes (OKE), OCI Container Registry (OCIR), kubectl, OCI CLI 3.x

Symptoms

  • New pods stuck in ErrImagePull / ImagePullBackOff.
  • Events show unauthorized, denied: Anonymous users are only allowed read access on public repos or i/o timeout.
  • Often when the auth token of the user in the pull secret was deleted, or the user left the company.

1. Read the event

kubectl get pods -A | grep -E "ImagePull|ErrImage"
kubectl describe pod <pod> -n <ns> | sed -n '/Events/,$p'
Message Cause
unauthorized / denied Bad pull secret credentials or token revoked
not found / manifest unknown Wrong repository path or tag
i/o timeout / no such host Node can't reach <region>.ocir.io

2. Check the image path

OCIR paths look like:

<region-key>.ocir.io/<tenancy-namespace>/<repo>:<tag>
e.g. fra.ocir.io/axyz123abc/orders-api:1.42.0

oci os ns get   # your tenancy namespace
oci artifacts container image list --compartment-id $COMP --repository-name orders-api \
  --query 'data.items[].{v:version, created:"time-created"}' --output table

3. Recreate the pull secret

Username format:

  • Default identity domain / federated users: <tenancy-namespace>/<domain-or-idp>/<username>, for example axyz123abc/oracleidentitycloudservice/ci-bot@example.com.
  • Older IAM users without identity domains: <tenancy-namespace>/<username>.

Check the format your tenancy needs in the OCIR docs; it is the most common mistake.

Create a fresh auth token for a service user (not a person):

oci iam auth-token create --user-id <service-user-ocid> --description "OKE pull INC12345"

Copy the token value; it is shown only once. Then:

kubectl create secret docker-registry ocir-pull -n app \
  --docker-server=fra.ocir.io \
  --docker-username='axyz123abc/oracleidentitycloudservice/ci-bot@example.com' \
  --docker-password='<auth-token>' \
  --dry-run=client -o yaml | kubectl apply -f -

Make sure the deployment references it:

spec:
  template:
    spec:
      imagePullSecrets:
        - name: ocir-pull

Or attach it to the namespace's default service account:

kubectl patch serviceaccount default -n app -p '{"imagePullSecrets":[{"name":"ocir-pull"}]}'

Tip

A user can have only a small number of auth tokens (check current limits). If creation fails, delete an unused one with oci iam auth-token delete.

4. IAM policy for the service user

Allow group ocir-pullers to read repos in compartment apps

The user must be in that group. For pushing from CI, manage repos is needed.

5. Network path from worker nodes

Private worker nodes reach OCIR through a NAT gateway or a service gateway (OCIR is in the Oracle Services Network). Check:

kubectl run nettest --rm -it --image=busybox:1.36 --restart=Never -- nslookup fra.ocir.io

If DNS fails or the route is missing, fix the worker subnet route table (service gateway or NAT) and security list egress on 443.

6. Retry pulls

kubectl delete pod -n app -l app=orders-api --field-selector=status.phase=Pending
kubectl rollout status deployment/orders-api -n app

Prevent it next time

  • Use a dedicated service user for pulls, owned by the platform team, with token rotation documented.
  • Consider image pull without secrets using OKE workload identity / instance principals where supported for your cluster type.
  • Pin images by digest in production.
  • Alert on pods in ImagePullBackOff for more than a few minutes.

Related fixes