OCI API calls failing with 429 TooManyRequests: a runbook for throttled automation, Terraform and scanners

Terraform applies fail halfway with 429-TooManyRequests. Scripts print ServiceError: {'status': 429, 'code': 'TooManyRequests', ...}. Teams that changed nothing see their pipelines go red too. OCI throttles API calls per tenancy and per region. One busy caller, often an inventory scanner or a polling loop, can use up the budget for everyone. The fix is to find that caller, bring the call rate down, and make every client back off the same way.

#What 429 tells you

Oracle doesn't publish a per-operation rate for most APIs, and limits differ by service. Design for "back off and spread out". Don't try to aim just under a number.

#Step 1: stop the bleeding

1. Pause the obvious bulk callers: CSPM or inventory scanners, ad-hoc export scripts, and Terraform plans running in a loop.

2. Lower Terraform parallelism for any apply that must finish now:

   terraform apply -parallelism=3
   

3. Let the SDK and CLI retry instead of wrapping them in your own tight retry loop. Remove --no-retry from CLI scripts and don't set the retry count to zero in emergency scripts.

Caution

Killing a Terraform apply mid-run because of 429s can leave resources created but not recorded in state. Let it fail on its own, then run terraform plan and import anything orphaned before you re-apply.

#Step 2: find the caller

Audit records API calls with the principal and user agent. Use Logging Search to rank callers over the throttled window:

search "<tenancy-ocid>/_Audit"
| summarize count() as calls by data.identity.principalName, data.identity.userAgent
| sort by calls desc

Then narrow to one principal and see which operations it calls:

search "<tenancy-ocid>/_Audit"
| where data.identity.principalName = 'svc-inventory'
| summarize count() as calls by type
| sort by calls desc

What to look for:

• A user agent string like Oracle-PythonSDK/..., Oracle-JavaSDK/..., terraform-provider-oci/... or a vendor scanner name. This tells you the tool even when every tool shares a service user.
• Instance principal callers (the principal is an instance OCID). Map it to the host with oci compute instance get.
• Steady high volume of Get or List events from one principal. This is almost always polling.

Audit coverage and field names differ by service and event version. If a call type you expect isn't there, it may not be audited. Add client-side logging to confirm. Check the Audit event schema for your services.

Tip

Give every automation its own user, dynamic group or at least its own user agent suffix. Ranking callers takes seconds when each automation has its own identity. When everything runs as terraform-admin, it takes hours.

#Step 3: fix the call pattern

Almost every throttling incident comes from one of these patterns.

Search replaces the compartment loop:

oci search resource structured-search \
  --query-text "query instance resources where lifeCycleState = 'RUNNING'" \
  --limit 1000

Waiters replace hand-written polling:

oci compute instance action --instance-id ocid1.instance.oc1... --action STOP \
  --wait-for-state STOPPED --wait-interval-seconds 30 --max-wait-seconds 1200

Events replace polling entirely for "tell me when X happens". A rule on com.oraclecloud.computeapi.terminateinstance.end that posts to Notifications or a Function makes no calls at all while it waits.

#Step 4: make every client back off the same way

#OCI CLI

The CLI retries throttled calls by default. Control it per call:

oci os object list -bn big-bucket --all --max-retries 8

In shell loops, sleep between iterations and add jitter. sleep $((RANDOM % 5 + 1)) is crude but better than nothing.

#Python SDK

Recent SDK versions apply a default retry strategy that covers 429, timeouts and most 5xx errors. Check your SDK version's documentation for the current default and the environment variable that turns it off. For bulk jobs, set the strategy explicitly so behavior doesn't change when the SDK is upgraded:

import oci

retry = oci.retry.RetryStrategyBuilder(
    max_attempts_check=True, max_attempts=10,
    total_elapsed_time_check=True, total_elapsed_time_seconds=900,
    retry_max_wait_between_calls_seconds=60,
    retry_base_sleep_time_seconds=2,
    service_error_check=True,
    service_error_retry_on_any_5xx=True,
    service_error_retry_config={429: []},
    backoff_type=oci.retry.BACKOFF_FULL_JITTER_EQUAL_ON_THROTTLE_VALUE,
).get_retry_strategy()

signer = oci.auth.signers.InstancePrincipalsSecurityTokenSigner()
compute = oci.core.ComputeClient(config={}, signer=signer, retry_strategy=retry)

Retries only absorb bursts. A job that sends ten parallel workers into a throttled API still fails, more slowly. Add a client-side rate limiter in front of the SDK when you fan out:

import threading, time

class TokenBucket:
    def __init__(self, rate_per_sec, burst):
        self.rate, self.capacity = rate_per_sec, burst
        self.tokens, self.ts = burst, time.monotonic()
        self.lock = threading.Lock()

    def take(self):
        while True:
            with self.lock:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.ts) * self.rate)
                self.ts = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
            time.sleep(1 / self.rate)

bucket = TokenBucket(rate_per_sec=5, burst=10)
# bucket.take() before every API call in every worker thread

Start low (a few calls per second per job) and raise it only while 429s stay at zero.

#Terraform

The OCI provider retries 429 and 500 responses automatically, with backoff and jitter. Two provider settings control it:

provider "oci" {
  region                 = "eu-frankfurt-1"
  retry_duration_seconds = 300   # minimum time to keep retrying 429/500
  # disable_auto_retries = true  # don't, unless you have a reason
}

retry_duration_seconds has no effect on 400, 401, 403, 404 or 409 errors, and it's ignored when disable_auto_retries is true. Newer provider versions also support a retries configuration file for per-status-code control. Check your provider version's documentation. In CI, combine retries with -parallelism of 3 to 5 for large states. Don't run plans for every workspace at the same moment.

#Step 5: verify recovery

• Rerun the Logging Search ranking. The top caller's volume should be down by an order of magnitude, not by a few percent.
• Run a known-good Terraform plan and a CLI smoke test from a different team's pipeline. Both should pass on the first attempt.
• Watch for new 429s in client logs for a full day of scheduled runs, including the on-the-hour peaks.

#Prevent it next time

• One identity and one user agent suffix per automation, so Audit can rank callers right away.
• Write down call-rate budgets for scanners and inventory jobs, and run them off-peak with jittered schedules.
• Use Search and Events instead of list loops and polling in new tooling. Reject them in code review.
• Split large Terraform states and set CI defaults for -parallelism.
• Before adding a third-party tool that calls OCI APIs, find out how it paces calls and whether you can cap its rate.