Home / OCI

OCITroubleshootingIntermediate

OKE pods in ImagePullBackOff from OCIR - fixing registry authentication, paths and network access

OKE pods fail to pull from OCI Container Registry because of a wrong image path, a missing or wrong pull secret, an expired auth token, policies, or no network route. Read the pod event, then fix the matching layer.

Deepak Bhardwaj4 min read

Applies toOracle Container Engine for Kubernetes (OKE) managed and virtual nodes; OCI Container Registry (OCIR); kubectl

Read the event first

kubectl describe pod <pod> -n <ns> | sed -n '/Events/,$p'
Event text Cause
not found / manifest unknown Wrong repository path or tag
unauthorized / authentication required / denied Pull secret missing or wrong, auth token expired, or private repo without access
i/o timeout / dial tcp ... connection refused No network path from the nodes to OCIR
toomanyrequests Rate limiting on the registry
no match for platform in manifest Arm/x86 mismatch between the image and the node shape

1. Image path

The OCIR image path format is:

<region-key>.ocir.io/<tenancy-namespace>/<repo-name>:<tag>

For example fra.ocir.io/axyz123abcd/orders/api:1.4.2. The namespace is the Object Storage namespace, not the tenancy name. Many regions also accept the full region identifier as the host (for example eu-frankfurt-1.ocir.io).

oci os ns get --query data --raw-output
oci artifacts container image list --compartment-id $COMP --repository-name orders/api \
  --query "data.items[].{ver:version,created:\"time-created\"}" -o table

Check the tag actually exists. A CI build that failed after the tag was referenced is a common cause.

2. Pull secret (private repositories)

Create an auth token for the user or service user that pulls (Identity > Users > Auth tokens). Then:

kubectl create secret docker-registry ocir-pull -n orders \
  --docker-server=fra.ocir.io \
  --docker-username='axyz123abcd/svc-oke-pull' \
  --docker-password='<auth-token>' \
  --docker-email=noreply@example.com
  • The username is <namespace>/<username>.
  • For identity-domain users, it's <namespace>/<identity-domain-name>/<username> (for example axyz123abcd/OracleIdentityCloudService/svc-oke-pull or axyz123abcd/Default/svc-oke-pull). Check the format for your tenancy in the OCIR documentation.
  • The password is the auth token, not the console password.

Reference it in the pod spec or the service account:

spec:
  imagePullSecrets:
    - name: ocir-pull

# or attach to the namespace's default service account
kubectl patch serviceaccount default -n orders -p '{"imagePullSecrets":[{"name":"ocir-pull"}]}'

Secrets are namespaced. A secret in default doesn't help pods in orders.

Test the same credentials from a workstation:

docker login fra.ocir.io -u 'axyz123abcd/svc-oke-pull'
docker pull fra.ocir.io/axyz123abcd/orders/api:1.4.2

3. Policies

The user behind the auth token must be allowed to read the repository:

Allow group oke-pullers to read repos in compartment apps

Without a pull secret, OKE nodes can pull from OCIR using the node's instance principal if a policy allows it:

Allow dynamic-group oke-nodes to read repos in compartment apps

Check that the dynamic group's matching rule covers the worker node instances (for example ALL {instance.compartment.id = '<workers-compartment-ocid>'}). Instance principal pulls depend on your OKE version and node type, so check the current OKE documentation before you rely on them.

4. Auth token expired or deleted

Auth tokens don't expire by default, but they get deleted when people leave or when a user hits the per-user token limit and someone removes an old one. Use a dedicated service user for pulls, record which token the cluster uses, and rotate it on purpose:

kubectl create secret docker-registry ocir-pull -n orders --docker-server=fra.ocir.io \
  --docker-username='axyz123abcd/svc-oke-pull' --docker-password='<new-token>' \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment -n orders

5. Network path

Nodes in private subnets need either:

  • a service gateway with the "All Services in Oracle Services Network" CIDR route, or
  • a NAT gateway.

Check the worker subnet's route table and the security list or NSG egress rule for 443. Test from a debug pod:

kubectl run nettest -n orders --rm -it --image=busybox:1.36 --restart=Never -- \
  sh -c "nslookup fra.ocir.io && wget -S -O- https://fra.ocir.io/v2/ 2>&1 | head -5"

A 401 from /v2/ means the network is fine: the registry is reachable and asking for credentials.

6. Architecture mismatch

Ampere A1 nodes need arm64 images. Build multi-arch images:

docker buildx build --platform linux/amd64,linux/arm64 -t fra.ocir.io/axyz123abcd/orders/api:1.4.2 --push .

Checklist

  • Image path uses region key, namespace and an existing tag
  • Pull secret in the pod's namespace with namespace/username and auth token
  • Policy lets the user or dynamic group read repos
  • Service account token owner documented, rotation planned
  • Nodes reach OCIR via service gateway or NAT
  • Multi-arch images for mixed node pools

Related fixes