OKE Workload Identity: fix NotAuthorizedOrNotFound by checking policy, cluster type and signer

#Symptom

A pod that should call OCI services, such as Object Storage or Vault, fails with:

ServiceError: {'status': 404, 'code': 'NotAuthorizedOrNotFound',
'message': 'Authorization failed or requested resource not found.'}

Or it gets a 401 NotAuthenticated, or the SDK reports that it can't get a resource principal token. The same call works from your laptop.

OCI returns 404 instead of 403 when it won't confirm a resource exists, so that message can mean "no permission" and "wrong name" alike.

#How Workload Identity works

OKE issues a token for the pod's Kubernetes service account. The SDK exchanges that token for an OCI resource principal session token. IAM then evaluates policies written for that workload, using the namespace, service account name and cluster OCID. There's no API key on the pod, and no dynamic group of nodes.

#Step 1: confirm the cluster type

oci ce cluster get --cluster-id "$CLUSTER_ID" \
  --query 'data.{type:type,name:name,k8s:"kubernetes-version"}'

ENHANCED_CLUSTER is required. A BASIC_CLUSTER can be upgraded to enhanced:

oci ce cluster update --cluster-id "$CLUSTER_ID" --type ENHANCED_CLUSTER

Warning

You can't change an enhanced cluster back to basic, and enhanced clusters may carry a cluster fee. Check the current pricing and confirm before you upgrade.

#Step 2: see which identity the pod is using

NS=payments; POD=api-7d9c

kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.serviceAccountName}{"\n"}'
kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.automountServiceAccountToken}{"\n"}'

If serviceAccountName says default, your policy for api won't match. Set it in the deployment:

spec:
  template:
    spec:
      serviceAccountName: api
      automountServiceAccountToken: true

#Step 3: use the right signer

The most frequent application-side error is using another signer. Instance principal signers use the node's identity, and resource principal signers for Functions won't work on OKE either.

Python:

import oci

signer = oci.auth.signers.get_oke_workload_identity_resource_principal_signer()
client = oci.object_storage.ObjectStorageClient(config={"region": "ap-mumbai-1"}, signer=signer)

namespace = client.get_namespace().data
print(namespace)
print(client.list_objects(namespace, "app-data").data.objects[:5])

Go:

provider, err := auth.OkeWorkloadIdentityConfigurationProvider()
if err != nil { log.Fatal(err) }
client, err := objectstorage.NewObjectStorageClientWithConfigurationProvider(provider)

Requirements to check:

• A recent enough SDK. Older versions don't include the OKE signer, so check the release notes for your language.
• Set the region if the SDK can't detect it. An environment variable is commonly used:

env:
  - name: OCI_RESOURCE_PRINCIPAL_VERSION
    value: "2.2"
  - name: OCI_RESOURCE_PRINCIPAL_REGION
    value: ap-mumbai-1

The OCI CLI inside a pod takes its own option; check whether your CLI build supports it with oci --help | grep auth:

oci os ns get --auth oke_workload_identity

#Step 4: write the policy correctly

Allow any-user to read objects in compartment app-data where all {
  request.principal.type = 'workload',
  request.principal.namespace = 'payments',
  request.principal.service_account = 'api',
  request.principal.cluster_id = 'ocid1.cluster.oc1.ap-mumbai-1.aaaaexample'
}

All these must be exact:

any-user looks alarming, but the conditions restrict it to one service account in one cluster.

Caution

Leaving a condition out widens access a lot. Dropping service_account allows every pod in that namespace. Dropping all of them could allow any principal in the tenancy. Always review the full set.

Policies apply through the compartment hierarchy. List what exists:

oci iam policy list --compartment-id "$COMPARTMENT_ID" --all \
  --query 'data[].{name:name,statements:statements}'

#Step 5: match the verb to the API call

A policy that allows read objects won't allow every related call:

Encrypted resources need more. If the bucket uses a customer-managed key, the service itself must be allowed to use that key, which is a separate policy.

#Step 6: rule out the target

print(client.get_bucket(namespace, "app-data").data.compartment_id)

Check that the compartment in your policy is the one holding the resource, and that the region in your client is the bucket's region. A wrong region is reported as the same 404.

Audit logs show which principal and which decision was recorded. In Logging > Search:

search "<tenancy-ocid>/_Audit"
| where data.response.status = '404'
| sort by datetime desc

Field names can vary; open one event to see its structure. See OCI Audit logs introduction.

Policy changes can take a short time to apply, so wait a minute before you test again.

#Fast decision table

#Prevent it next time

• Create one service account per application, not one per namespace.
• Keep policies in Terraform, with the cluster OCID coming from an output instead of being typed by hand.
• Run a small startup test in the pod, and fail fast with a clear message if the call doesn't work.
• Related: instance and resource principals and the IAM policy cheat sheet.