Home / OCI

OCIRunbookAdvanced

Sev1: OCI IAM policies deleted or overwritten — teams and workloads lose access

Restore access after IAM policies were deleted or a policy update replaced its statements. Confirm what's missing with Audit, rebuild statements from IaC or evidence, handle identity domain admin lockout, and keep policies in version control.

Deepak Bhardwaj2 min read

Applies toOCI IAM with identity domains, IAM policies, dynamic groups, OCI Audit, OCI CLI 3.x

Symptoms

  • NotAuthorizedOrNotFound (HTTP 404) for operations that worked yesterday — OCI returns 404 rather than 403 when you lack permission.
  • Instance-principal workloads fail to read secrets or write to buckets.
  • Users see empty compartments in the console.

1. Confirm the change

oci audit event list --compartment-id $TENANCY --start-time $(date -u -d '-24 hours' +%FT%TZ) --end-time $(date -u +%FT%TZ) --all \
  --query 'data[?contains("data"."event-name", `Policy`)].{t:"event-time", e:"data"."event-name", who:"data"."identity"."principal-name", res:"data"."resource-name"}' --output table

Policies live in the compartment where they're attached; check the tenancy root and parent compartments too. UpdatePolicy replaces the full statement list.

2. Who still has admin?

Members of the tenancy Administrators group keep full access through the built-in tenant admin policy, which can't be deleted. Use one of them to repair.

oci iam policy list --compartment-id $TENANCY --all --query 'data[].{name:name, statements:statements}' --output json | head -50

3. Rebuild the statements

Best source, in order:

  1. IaC (Terraform oci_identity_policy) — re-apply it.
  2. An export or Git copy of policies.
  3. Audit event request details and your documentation.
  4. Error messages from failing apps, to build minimal statements.
oci iam policy create --compartment-id <comp-ocid> --name app-team-policy \
  --description "Restored 2026-10-24" \
  --statements '[
    "Allow group '"'"'Default'"'"'/'"'"'app-team'"'"' to manage instance-family in compartment prod-app",
    "Allow group '"'"'Default'"'"'/'"'"'app-team'"'"' to use virtual-network-family in compartment prod-network",
    "Allow dynamic-group '"'"'Default'"'"'/'"'"'app-instances'"'"' to read secret-family in compartment prod-security"
  ]'

Quoting gets ugly in shells — put statements in a JSON file and use --statements file://statements.json.

With identity domains, refer to groups and dynamic groups as '<domain-name>'/'<group-name>'. Policies written without the domain name apply to the Default domain.

4. Identity domain admin lockout

If the problem is the identity domain itself (admins removed, MFA lockout):

  • Another user with the Identity Domain Administrator role, or a tenancy administrator, can restore roles.
  • Keep a break-glass local user in the Default domain with MFA, exempt from federation, stored credentials in a safe.
  • If no one can sign in, open an Oracle Support SR — this takes time, so plan the break-glass path in advance.

5. Test

oci os object list --bucket-name app-data --auth instance_principal   # from an app instance

Test as a member of each affected group (or with their API key) — policy changes apply within seconds, but cached sessions may need a refresh.

Prevent it next time

  • All policies in Terraform; manage policies restricted to a small IAM admin group.
  • Events rule on DeletePolicy / UpdatePolicy → Notifications.
  • Prefer several small policies per team/compartment over one large policy that a single bad update can wipe.

Related fixes