Home / OCI

OCITroubleshootingIntermediate

Sev1: apps on OCI suddenly get NotAuthorizedOrNotFound — instance principal and dynamic group fixes

Fix workloads that use instance or resource principals and start failing with 404 NotAuthorizedOrNotFound. Check the dynamic group matching rule, the policy statement and its domain prefix, compartment moves, and the token on the instance.

Deepak Bhardwaj2 min read

Applies toOCI IAM identity domains, dynamic groups, instance principals, resource principals (Functions, OKE), OCI SDKs and CLI

Symptoms

  • ServiceError: 404 NotAuthorizedOrNotFound from SDK calls on an instance that worked before.
  • Started after an instance was recreated, moved to another compartment, or after IAM was reorganised into identity domains.

1. Test from the instance

oci os ns get --auth instance_principal
oci secrets secret-bundle get --secret-id <secret-ocid> --auth instance_principal

If os ns get fails, the instance can't get a principal token at all (network to the metadata service or IAM endpoints). If it works but the specific call fails, it's a policy or matching problem.

2. Does the dynamic group still match the instance?

curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/instance/ | jq '{id, compartmentId, definedTags}'

Compare with the dynamic group rule (identity domain → Dynamic groups):

ANY {instance.compartment.id = 'ocid1.compartment.oc1..aaaa'}
ALL {instance.compartment.id = 'ocid1.compartment.oc1..aaaa', tag.app.role.value = 'web'}
Change Breaks rules that use
Instance recreated instance.id = '...'
Instance or compartment moved instance.compartment.id = '...'
Tag removed or renamed tag.<ns>.<key>.value

Prefer compartment or tag-based rules over instance OCIDs.

3. Does a policy grant the dynamic group?

Allow dynamic-group 'Default'/'app-instances' to read secret-family in compartment prod-security
Allow dynamic-group 'Default'/'app-instances' to manage objects in compartment prod-data where target.bucket.name = 'app-exports'

Check:

  • The domain prefix: a dynamic group in a non-default domain must be referenced as '<domain>'/'<name>'.
  • The policy is attached at the right compartment (the target resource's compartment or an ancestor).
  • Conditions (where target.bucket.name = ...) still match renamed resources.

4. Resource principals (Functions, OKE)

Functions use resource.type = 'fnfunc' in dynamic group rules:

ALL {resource.type = 'fnfunc', resource.compartment.id = 'ocid1.compartment.oc1..aaaa'}

For OKE pods, use OKE Workload Identity policies with request.principal.type = 'workload' and namespace/service account conditions; check the current documentation for the exact attribute names.

5. Token and network on the instance

  • The SDK fetches a token from the instance metadata service (169.254.169.254) and federation endpoint (auth.<region>.oraclecloud.com). Egress must allow HTTPS to OCI services (NAT gateway or Service Gateway with All Services).
  • Instance clock must be correct (NTP); large skew breaks request signing.
  • Long-running processes cache tokens; they refresh automatically in current SDKs — update old SDK versions.

6. Wait for propagation

Dynamic group membership and policy changes usually apply quickly but can take a few minutes. Retest before changing more.

Prevent it next time

  • Tag- or compartment-based matching rules; avoid instance OCIDs.
  • Policies and dynamic groups in Terraform alongside the workload.
  • A health check in the app that calls a cheap authorised API at start-up and fails loudly.

Related fixes