OCI Resource Manager apply job failed or stuck - runbook

#Symptom

A Resource Manager apply or destroy job ends in Failed, or sits in In progress far longer than usual and blocks every other job on the stack. Part of the change may already be live. The next plan shows resources to create that already exist, or the plan won't start at all.

Resource Manager locks the stack state while a job runs, so only one job runs per stack at a time. A hung job blocks the whole stack until it finishes or you cancel it.

Set a few variables first:

export STACK=<stack-ocid>
export C=<compartment-ocid>

#1. Find the failing job and read the real error

List the latest jobs on the stack, newest first:

oci resource-manager job list --stack-id $STACK --sort-by TIMECREATED --sort-order DESC --limit 5 --output table
export JOB=<job-ocid>
oci resource-manager job get --job-id $JOB

The summary rarely contains the cause. Pull the Terraform log at ERROR level and above:

oci resource-manager job get-job-logs --job-id $JOB --all \
  --level-greater-than-or-equal-to ERROR --query 'data[].message' --output json

For the full raw console output (what you'd see running terraform apply locally):

oci resource-manager job get-job-logs-content --job-id $JOB

Look for the first Error: line. The resource address under it (with oci_core_instance.app[1]) tells you what failed. Everything after the first error is usually a consequence.

#2. Map the error to a fix

Tip

Re-run a plan job before every retry. If you go straight to an apply that's auto-approved, it applies whatever is in the config now, not what you reviewed before the failure.

#3. A job is stuck in progress

First confirm it is actually stuck. Large Exadata, database or OKE resources can legitimately take an hour or more. Check the log timestamps for recent activity:

oci resource-manager job get-job-logs --job-id $JOB --sort-order DESC --limit 20 --query 'data[].[timestamp,message]'

If the provider is still printing Still creating... [45m0s elapsed] for a resource that is clearly progressing in the console, wait.

If nothing moves, cancel gracefully first:

oci resource-manager job cancel --job-id $JOB --wait-for-state CANCELED

A graceful cancel signals Terraform to stop. The job can still finish part of its work before it ends as Canceled.

Caution

--is-forced true stops the job immediately. Oracle warns that a forced cancel can leave the state file out of step with the real resources. Use it only when a graceful cancel doesn't complete, and expect to do the state checks in section 4 afterwards.

oci resource-manager job cancel --job-id $JOB --is-forced true

#4. After a partial apply: compare state with reality

Download the current state before changing anything. This is your restore point:

oci resource-manager stack get-stack-tf-state --stack-id $STACK --file stack-before-fix.tfstate

Then run drift detection to see which resources no longer match:

oci resource-manager stack detect-drift --stack-id $STACK
oci resource-manager stack list-resource-drift-details --stack-id $STACK --output table

Then run a new plan job and read the plan, not just its status:

oci resource-manager job create-plan-job --stack-id $STACK --wait-for-state SUCCEEDED --wait-for-state FAILED
oci resource-manager job get-job-logs-content --job-id <plan-job-ocid>

Three outcomes:

• Plan shows only the remaining changes. State is consistent. Apply from that plan.
• Plan wants to create something that already exists. The apply created it but didn't record it in state. Import it (section 5).
• Plan wants to destroy and recreate something that is fine. Usually a changed attribute or an ignored default. Fix the config, or add a lifecycle { ignore_changes = [...] } block for that attribute.

Apply only from the reviewed plan:

oci resource-manager job create-apply-job --stack-id $STACK \
  --execution-plan-strategy FROM_PLAN_JOB_ID --execution-plan-job-id <plan-job-ocid>

#5. Bring an orphaned resource back into state

Resource Manager has no import button for single resources. Two working options:

Option A: import block in the config (Terraform 1.5 and later). Add it to the stack's configuration, upload it, then plan and apply:

import {
  to = oci_core_instance.app[1]
  id = "ocid1.instance.oc1.eu-frankfurt-1.<unique_id>"
}

oci resource-manager stack update --stack-id $STACK --config-source ./stack.zip --force

Remove the import block once the apply succeeds.

Option B: fix the state locally and push it back. Use this when you need terraform state rm or state mv as well:

oci resource-manager stack get-stack-tf-state --stack-id $STACK --file terraform.tfstate
terraform init
terraform import 'oci_core_instance.app[1]' ocid1.instance.oc1.eu-frankfurt-1.<unique_id>
terraform plan            # must show no unexpected changes
oci resource-manager job create-import-tf-state-job --stack-id $STACK --tf-state-file terraform.tfstate

Warning

The import-state job replaces the stack's state. Make sure no one runs a job on the stack while you work, and keep stack-before-fix.tfstate until the next clean apply.

#6. Roll back or pin the Terraform version

To go back to the resources as they were after an earlier successful apply, Resource Manager offers rollback jobs. Plan the rollback first and review it:

oci resource-manager job create-plan-rollback-job --help    # check the parameters on your CLI version
oci resource-manager job create-apply-rollback-job --stack-id $STACK \
  --execution-plan-rollback-strategy AUTO_APPROVED --target-rollback-job-id <good-apply-job-ocid>

Rollback changes infrastructure. It doesn't restore data in deleted volumes or databases.

If errors started after you uploaded new configuration, check which Terraform version the stack uses and update it to one your code supports. Supported versions vary; check the Resource Manager docs for the current list.

oci resource-manager stack get --stack-id $STACK --query 'data."terraform-version"'
oci resource-manager stack update --stack-id $STACK --terraform-version 1.5.x --force

#Prevent it next time

• Always run plan, then apply with FROM_PLAN_JOB_ID. Avoid AUTO_APPROVED on production stacks.
• Keep stacks small (network, data, app tier) so a failure or long job doesn't block everything.
• Grant manage orm-jobs only to the pipeline or the people who own the stack. Give everyone else read orm-family.
• Download the state (get-stack-tf-state) as a pipeline artifact before every production apply.
• Schedule a drift detection run and alert on drift, so console changes don't surface in the middle of an incident.
• Don't change Terraform-managed resources in the console. If you must during an incident, write the change down and bring the config in line afterwards.